Photoshop 2019: Refine edge brush tool in Select and Mask not working

Could you be a bit more specific? It works for me...

Even after decontaminating it to 100 it still erases some part of the mask

So brush back that part. What you are saying is that the brush does work, but not always 100% correctly. That is to be expected, and that is why you have a normal brush in this tool as well.

How do I do that just on the masked area? While I'm in Select and mask and have my mask selected the back ground too gets selected by the brush...
E.g. in the image below