I earlier reported what I thought might be a bug in Photoshop CC 2018. I have discovered that while it is not a bug, I had stumbled upon an interaction between a Vibrance/Saturation layer and the RGB Working Space that I was not aware of. The purpose of this post is to describe this interaction so that others aren’t surprised by it, or, if they are, I hope that this will help them understand what is going on.
In short, if the working space (in Edit/Color Settings...) has a smaller gamut than the document’s gamut, then colors in the document will be clipped to (approximately) the working space, if there is a Vibrance/Saturation layer in the document. If you apply the Vibrance layer (for instance, by flattening the document), the colors in the flattened document will also be clipped.
I found this out when I installed PS CC 2018 and requested the default preferences. I had always worked with my RGB Working Space = Pro Photo RGB, but a default installation sets the working space to sRGB. I noticed that most of the colors in my images were much less saturated, and the hues had also shifted (a small amount). After a lot of testing (and a very worried post in this forum), I finally figured out that the cause was the value of RGB Working Space that was set when Photoshop was launched.
I was able to confirm that this behavior has always been present in Photoshop, for all the versions I have at hand — from PS CS 6 to PS CC 2018. And, as far as I can tell, there has been no change in the behavior over these various versions.
I tested a number of adjustment layer types, and, as far as I can tell, it is only Vibrance/Saturation layers that are affected by the RGB working space. All the other adjustment layer types that I tested give the same colors regardless of the RGB working space. This effect is only seen when a document has colors that are outside of the working space (either before the Vibrance/Saturation layer is applied or afterwards).
I haven’t found any documents that describe this behavior, but if you look in the Photoshop help document (for PS CS 6 through CC June 2016) you see this comment: “About color working spaces: A working space is an intermediate color space used to define and edit color in Adobe applications.” This suggests that when Photoshop has a Vibrance/Saturation adjustment, it transforms the colors into the working space, applies the adjustment, and then transforms the colors back to the document space.
There is nothing I can find that tells the user that Vibrance/Saturation layers interact specially with the working color space, or that it is the value of the color space when Photoshop is launched, that is the determining value.
Of course, none of this should be an issue for you unless you try to edit documents with an embedded large-gamut space while Photoshop has a working space with a smaller gamut.