martijn278, how many GB of storage are your photos taking up in iCloud?
I assume, you have already enabled the option "Optimize iPhone Storage" in the Settings > Apple Account > iCloud > Photos?
It may take a while until the photos and videos will get optimized. But even optimized the photos will need some local storage on your iPhone. Your device will store smaller, optimized versions for browsing, also albums, faces, other metadata for searching. And it will try to keep the most recently used photos downloaded in the full resolution, so you do not need to download them over and over again and waste bandwidth or or have to wait for the download. On my iPhone the optimized versions are taking up roughly 10% to 20% of the size of the Photos Library in iCloud. I would not keep more photos in iCloud than you can afford to keep optimized on your devices.
For example, ich you are willing to use 10GB of storage locally on your iPhone, do not keep more than 100GB of photos and videos in iCloud.
How much storage are your photos taking up on your iPhone compared to the storage in iCloud? Are they already fully optimized?