Unfortunately the Mac mini Server Mid 2010 with 10.14 is no longer supported.
But it's solved by now.
After many detours I got to the right person in the Enterprise department at Apple Support.
If anyone else is looking for this solution, this is the way to go:
- existing server app, for me it was version 5.3, and in the Macintosh HD/Library delete the folder "Server
- Empty trash
- Restart system
- make sure that the supposedly deleted items (point 1.) are actually deleted.
- Open the App Store app and go to the server app under "Purchases". It should show the button "INSTALL" on the right.
- Install "macOS Server". The correct version (5.6.3) will be downloaded and installed.
- Reboot the system
So it worked well for me.
It would be nice if Apple would create a support article for it.
These are things that are not simply self-explanatory.