Thanks mmatt555, that makes it much clearer. Here's what I think you need to do. It doesn't involve using the Touch Bar itself.
In any application that supports it, bring up the Apple Colour Picker. It doesn't have to be Numbers: Pages or Keynote, or any third-party app that uses the Colour Picker will work:
By default you'll see just two rows of swatches at the bottom, but you can drag the top and bottom of the swatches panel up and down respectively to expose more.
I don't think it matters which of the options you choose in the top panel - above I've selected Colour Palettes, below I've selected Crayons. Now select an object within the application - a shape or a cell, and then click on one of the colour names.
You should now find that you can select individual colours in the lower swatches panel - if successful, it will be surrounded by a green border:
I have to say that it isn't very easy to activate that selection border. Dragging a new colour into the swatches panel seems to work, but you may need to persevere.
Once you have a colour selected, press the Delete key on your keyboard. This colour will be removed from the swatches panel in the Colour Picker - and from your Touch Bar:
Before:
After:
(I deleted the ochre colour to the left of the blue.)
Click on another colour in the swatches panel (it should now be selected immediately) and again press delete. Rinse and repeat for every colour you want to remove.
As I say, it's not an entirely straightforward procedure but it does seem to work. Hope it helps you.
Cheers,
H