Siri completely ignores HomeKit requests except for when it is activated using the side button

So i have a ceiling light and a lamp and they are compatible with homekit

They are able to be controlled with siri, however they only work when i activate siri from the side button.

if i just start saying ‘siri, turn off the lights’ it just ignores my request and leaves

is there a hidden toggle i have to switch?

i have multiple iphones, however i put only one iphone near me at all times and the others are far away.