Ok I figured out the problem. It has something to do with a process called TrustedPeersHelper which I think manages the various devices on your account. I noticed this was running a lot of CPU time on my mac so I was researching if others had the problem and found this link (https://www.reddit.com/r/MacOS/comments/dmbuj5/macos_catalina_trustedpeershelper_always_running/)
I'll summarize since it's not totally easy to follow but basically there are three database files located in "~/Library/Keychains" (you can go to the "Go" menu and select "Go to Folder" and paste that in it will take you there. The files are called "com.apple.security.keychain-defaultContext.TrustedPeersHelper" with the extension .db .db-shm and .db-wal .. it seems for some reason these got corrupted and became quite large (in my case over 15mb each). If you delete them they will be recreated immediately (I'm assuming from scratch without whatever the error was). I then restarted all my devices and my watch is back to normal battery life!
If you want the shortcut to delete the files you can open the Terminal window and paste in:
rm -f ~/Library/Keychains/*/com.apple.security.keychain-defaultContext.TrustedPeersHelper.{db,db-shm,db-wal}
then hit enter and restart everything and you should be ready. The original article where I found this said they had to log into their apple account again but that wasn't my experience.
I also found that this has improved my iPhone battery performance (though it wasn't terrible) but it does seem better now. Also my MacBook seems to be working with less CPU churn.
Hope this helps others.