I would guess it’s a combination of your speed and accuracy…
we need to have the air, tongue release, and fingering happen all at the proper time…if one of these is off it won’t work. It’s not so bad on most notes but you have found a problem interval for you.
I would practice slowly from several notes going down to D: G D, G# D, A D, Bb D, B D
go slow enough to make them all work, I bet by the time you get to B it will be better because you’re putting this specialized focus on it.
If it doesn’t work smoothly do it slower and more accurately until it does. also try tonguing everything and then no tonguing at all.