Using my whole photo library on the Apple TV screensaver

… why you cannot choose …

Unfortunately, any possible explanation wouldn’t overcome your issue. It is programmed the way that it is.

1. The screen saver loads images locally (copied to the Apple TV box disk), to a limited set (by total size and/or total number or other non-public algorithm), so that no network access is needed for the screen saver.
2. The slide show feature (instead of screen saver) can request images in real time, and is not limited by size or number, as the Apple TV disk isn’t used.
3. You still need to select an album, not the whole library, but you can create an album that is all, or almost all, of the library.

You may give feedback to Apple, if you want: Feedback - Apple TV - Apple