My iPhone speakers switch to phone volume without me adjusting anything.

when I’m listening to music (on Apple Music), it will randomly adjust the volume. The volume switches from speaker volume to phone call volume, I know it’s not being turned down because the volume dial stays where it is (even at full volume). I could not find anything in the setting for Apple or Apple Music to fix this. I have to swipe out of Apple Music to make it go back to normal, but half the time it just happens again. Any ideas?