HomePod mini flakiness such as you're describing usually points to one or more WiFi issues (see: If HomePod mini or HomePod isn't responding - Apple Support). HomePods, particularly stereo pairs, require simple unfettered peer-to-peer local area network access in addition to decent Internet connectivity.
Modern WiFi routers often implement local security measures to police peer-to-peer communications (one infected PC or smartphone can spread malware, etc.), so you want such features off or configured to make an exception for your stereo pair. Likewise, WiFi routers often try to optimize (potentially crowded) local WiFi spectrum usage by auto-switching devices between 2.4GHz and 5.0GHz or between channels within these frequencies. These "features" can make a pair of HomePods lose track of each other as their network connections are changed and it can confuse various HomeKit things.
Next time your HomePods get flakey, go reset your WiFi router (then restart HomePods if necessary). You might also want to Google your WiFi router model and HomePod (both in quotes) to see if you can spot any configuration suggestions for keeping the HomePods happy…
