Cannot remove iCloud Drive folders with Optimize Mac Storage

Hello there, for at least months now I am having this problem where I can't remove a downloaded folder from iCloud Drive while "Optimize Mac Storage" is activated. I can still remove independent files but not a whole folder wich is a big pain obviously since I'd have to go through hundreds of subfolders and select thousands of files sometimes.

So the thing is that when right-clicking on a folder the option is displayed in the context menu and not grayed out:

"Download entfernen" is german for "Remove Download".

But when clicking on it I would only get this error wich translates to "The folder "Adobe" can not be ejected" wich is a very weird wording in the first place. And no, it's not just the Adobe folder, that's just an example.

The "Remove Download" works when "Optimize Mac Storage" is activated though but then I don't have manual control over what's offline accessible and whats not wich is not an option.

Basically what that means is that my whole iCloud is downloaded on my Mac at all times and that obviously takes up a whole lot of space wich does not need to be blocked.

Does anyone else experience the same issues? Are there any solutions to the problem?

Best regards from Germany,

Felix