Deleting 'referenced files' from Photos andf Finder

It is the expected behavior that the referenced files remain in the Finder folders, because they might be referenced from other Photos Libraries as well.

Before you delete a photo in Photos, you may want to use the command "File > Show Referenced File in Finder" and then delete the referenced file yourself, after deleting the reference in Photos.

I would not use a referenced library in Photos, because the support for referenced files is very limited. It is a lot of work to fix broken references or to restore a referenced library from a backup, if the drive needs replacing.

Photos will not touch any file it does not manage, and as you told it to not manage your image files, then it doesn't. To amplify what Léonie said, running a referenced library in Photos is a storing up issues for the future. The app simply does not have the tools for the task, importation and deletion become multi-stage tasks and should you need to move the library, move the files, or if the path to the files changes for any reason it's a work of pain to reconnect the links. Frankly, I've no idea why Apple even leave that possibility with the app. Put bluntly: if you want to use Photos, don't use a referenced library. If you want to use a referenced library, then don't use Photos - pretty much very other app in this space only run referenced libraries and have the capability to do so.

If you "don't want to use any form of referenced system," then CHECK the “Copy items to the Photos library” option in Photos > Settings > General and use File > Consolidate on any photos and videos that were imported with the Copy option UNchecked..