How can I stop Photos from trying to ID something that is not a person (or is a total stranger)?

I've scanned in hundreds of family pics dating back to the 1920s.

In order to help ID people I have a Smart Album where

Person/Pet is [blank]

to show people who haven't been face identified. Problem is, the album is crowded with photos where a child is holding a doll (and Photos is trying to ID the doll), or there's a statue in the background, has non-family members who died before World War 2, or total strangers. Is there any way to tell Photos to ignore faces for these?