You'll want to move the folder that contains the actual music files and move the "Music Library.musiclibrary" file on its own with it. Only you can know what that is. I can't tell you which one, because I don't know where the actual music files are.
You don't need to move the top level Music Folder in your user profile though.
If you moved from iTunes to Apple Music then the folder with the actual media files should in the iTunes Media folder.
Otherwise it would be in the Music folder's Media folder.
Basically you need to verify where your tracks are pointing to. You can do that in the Music app by clicking on a track and then tapping Command+I to get the track info, and then going to the File tab, to see the actual location of the file.
On my library for instance, since I also upgraded from iTunes to Apple Music the tracks are actually located in a folder path like this: iTunes / iTunes Media / Music / Artist Name / Album Name / Track Name on the external drive.
While the Music Library file is at the same level as the iTunes folder.