My brother can use my phone with face id

It is much more unusual than you would think. Usually these posts are a result of someone playing a joke on someone else and having the person who owns the iPhone stand behind them to their left or right and then holds the phone up and shows them they can open the iPhone. Actually the True Depth camera can "see" the owner as it is wide angle. I've told this before but the actress Torrey DiVitto had a twitter post showing that her younger sister could unlock her iPhone. She was sitting behind her sister in a restaurant booth. When she turned towards the wall in the booth her sister could no longer unlock the iPhone.

If you resemble each other enough, yes, that can happen. Disable FaceID and use a passcode that you don't give him.

Setup Face ID again in a room without your brother in it.

Don't share your passcode, either.

Disable Face ID and use a passcode only. It is not unusual for siblings to be able to unlock a phone.