Airplay 2 to Apple TV - volume greyed out on iPhone

I have 2 x Apple TV’s 4th gen and an AirPlay express and have been using them with AirPlay 2 and my iPhone XR for multi room audio.

However, on one of the Apple TV’s, if I am on the same wi fi network as the Apple TV, when I stream music to it, the volume slider on my iPhone is greyed our and there is no way to control the volume. If I change network (to 4g, so not the same network as the Apple TV) the volume slider comes back onto my iPhone and I can control the volume. Obviously the downside of being in 4g is that I can no longer do multi room. It only happens on 1 Apple TV. The other Apple TV and the airport express both have the volume control slider available on my iPhone when on the same network as my iPhone. I’m going to try resetting the Apple TV later to see if that helps, but any guidance would be appreciated.