And takes too long or is too annoying to drag them out of the sets—delete the sets? Mind, if you crash or otherwise destroy prefs, you have to save them as a set, which I believe means dragging them all back out again.
Personally, I'd be fine with the option to have NO sets for those who have no need to separate their colors for any reason, but I don't know if that would work with the panel as designed, plus I don't know if it would be their priority over everything else since there is a workaround.
What I'm saying is you probably should go ahead and make it a feature request/idea (it's not a bug/problem because it's as designed), but use the workaround or you'll drive yourself crazy waiting for it to be changed.
I was trouble shooting with Photoshop, they led me to this area. It will be redesigned if enough people enjoy the idea of the feature of image icons only. Thanks for your idea of finding feature request idea.