iPad as monitor for mac mini - Magic Keyboard does not work

I have a mac mini (M2) and an ipad pro (newest), and I have added the ipad as a display for the mac mini hoping to be able to work from my bed on my mac mini using the ipad as control surface. Problem is that the trackpad and mouse of the magic keyboard seem to have no effect on anything I do in the continuity platform, it only works for the iPad OS parts. (Funny thing is that the Apple Pencil does seem to work a bit). Now it seems I will need to take my keyboard and mouse to my bed office which is not smooth enough. Is there any way I can get the magic keyboard to work instead? That is, have the magic keyboard control my mac mini through continuity mode? btw same issue whether I use mirror / extend / main display solution - and I have all universal control and sidecar features switched on, so I cannot really identify the issue, and hope for some help.