Keyboard regularly wants to disconnect from Mac when using Mac Virtual Display

My primary use for the AVP is using it as a virtual display. I can connect to my Mac Studio M2 Max, no problem. However, my keyboard is regularly switching away from my Mac and wants to be used for a different app in Vision OS.

I'm using an Apple Magic Keyboard with Touch ID, plugged in. However, I've also used my wired SteelSeries gaming keyboard to test as well. Both behave the very same way - they work, but SO often, they stop giving input on my Mac. For days I didn't know what was happening and I'd have to disconnect the Mac Virtual Display. Then today I figured out that the keyboard is being "taken" by a different app in Vision OS.

So say for instance, I'm working away on my Mac, but then I look over at the YouTube video I'm watching in Safari (the Vision OS app of Safari, not Safari on my Mac). Well then what happens is my keyboard now "belongs" to that Safari app and won't work with my Mac. I have to "pinch" my Mac Virtual Display to get the keyboard back over here, and very often that doesn't work. I tap tap tap, it won't switch over. Sometimes I can drag my Virtual Display around to a new location and then it will capture my keyboard again, but it seems super buggy.

It makes working on my Mac with the AVP as a display not reliable or practical.

It seems the whole point of being able to work with the AVP as your Mac Virtual Display is that you can have al these other windows around you in your virtual space - iTunes, Safari, Messages, etc..

BUT - as soon as I look at those other windows, my keyboard then "moves" to those apps and I can't get it back to my Mac easily.

Wish there was a way to keep the Mac keyboard paired to the Mac to avoid this problem.