Layer-Masks in RGB Mode and RGB Channels take on the Grey Mode Dot-Gain Color Settings Errors

  • 1
  • Problem
  • Updated 10 months ago
This is a long-standing problem with how Photoshop applies and calculates "color settings" incorrectly across the board with CMYK and Greyscale (Duotone as well) and Multichannel color modes.  I thought that RGB Mode was the one "safe space" I could confidently work within without the completely miscalculated Gamma / Dot-Gain problems of the other modes.  Unfortunately I have just discovered that this also affects all Layer-Masks in RGB Mode, and also if making a "Control-Select" of the RGB Channel (or of course one of the R, G, or B channels).  First issue is that not only does the "Grey" setting in Color Settings affect the "preview" of the individual channels... but it also affects what gets selected when doing a Control-Select of that channel in order to make a "greyscale-to-transparency" selection.  The second part of this issue is that Layer-Masks has become a standard way of applying transpareny to a layer, and what I just discovered is that because Layer-Masks use a "Channel" to achieve the masking,  this channel will take on the Grey Mode dot-gain or gamma settings.  I usually have an sGrey profile (similar to a 2.2 Gamma profile) set for the Grey Mode Color Settings, to correct the issue of greyscale previews (20% dot-gain default setting is totally wrong, dot-gain is applied incorrectly across the board in all of these modes).  The problem is that even within RGB Mode, any layer mask will then actually affect the layer based on the Grey Mode Color Setting.  Some people use specific curves in that setting, so I can't just write into my actions that I develop a switch to sGrey in the color settings, and I can't write an action that forces the correct masking method without resorting to clipping-masks,  and I can't create the correct transparency for a clipping mask without control-selecting the RGB channel, unless I create an action that literally goes through each grey level of the layer as solid-pixel selections and then fill them in with opacity... which leads to the other long-standing problem of being limited to 100% levels of Opacity settings instead of 256 levels for true 8-bit opacity.   This is a huge problem compounding upon all the other color-mode issues mentioned above, but it leaves me frustrated that even RGB mode is getting affected by these incorrectly calculated color settings.   

If I can create a true 256-level opacity layer out of the greyscale of a layer, then I can use it as a clipping-mask to create the true 8-bit opacity masking without the Grey Mode color setting affecting the mask, therefore having a masking method that is not going to be affected by the user's "dot-gain" settings in Grey Mode, and not have to change their grey-mode setting or suggest them to change it to sGrey or a Gamma 2.2.  The problem then is that opacity settings cannot be set to more than 0 to 100% levels... which is something like 6.5 bits (6-bit being 64 levels, and 7-bit being 128 levels).   

I'm sure this will never be corrected by the Photoshop programmers, there are now way too many people relying on the incorrectly calculated color-mode settings in their work... but I was surprised to see that the Grey-Mode settings are affecting all layer-masks and control-selections of RGB Channels even in RGB Mode.  Looking at testing with Adjustment Layers, I also just realized there is no workaround for adjustment layers to be set to a Clipping-Mask Layer in order to mask the adjustments... the clipping layer must be visible, so trying to use a layer with transparency built-in to mask an adjustment-layer will not work, it only works on a color-fill layer or another layer of pixels above.  I call the Greyscale, CMYK, Duotone, and Multichannel modes in Photoshop "Danger-Zones" for all of these dot-gain color-setting issues, but now I have to warn users of all these issues even within RGB mode??  I'm truly shocked at how poorly developed this program really is, even with all the supposed resources Adobe should have to be able to think ahead like when creating the entire technique of layer-masking using alpha channels... I understand not wanting to correct the long-standing color-setting calculation issues because of how many users probably rely on the incorrect errors in their work and would be shocked by it being changed, but I wish there was at least an option to turn those things off and look at the true values as they really are, and also not have layer-masks in RGB affected by those other mode color settings.  Using Photoshop for 20 years and disappointed constantly at how many errors are present throughout the program in the most fundamental ways dealing with color settings for editing, appearance and printing, etc.  
Photo of Max Chroma

Max Chroma

  • 2 Posts
  • 0 Reply Likes
  • Frustrated, Disappointed

Posted 10 months ago

  • 1
Photo of eartho

eartho, Champion

  • 1479 Posts
  • 497 Reply Likes
Ugh. Never noticed that before and it's really weird behavior. Why would a mask be dependent on dot gain settings instead of working in absolute mode? Especially in RGB. I'm sure when that was originally coded in the early 90's, there might have been a good reason, but why is it still that way, i wonder?
Photo of Max Chroma

Max Chroma

  • 2 Posts
  • 0 Reply Likes
It can be a complex situation where I suppose considering alpha channels "viewed by themselves" are "greyscale"... then at least it needs an sGrey similar to using sRGB to make the right gamma correction (which is a whole other topic)... but the original problem with "dot-gain" was that it was not applied to a sGrey already...  dots on paper respond more to a thing called the Contrast-Sensitivity Function in human vision.... basically to get a 50% grey we really want to just print a 50% dot as close as possible, the eyes will blur it into a proper 50% grey because of that contrast-sensitivity function (from the right distance, it blurs to a grey)... it doesn't need "gamma correction" because it is not the same as luminance or light intensity.   

So the dot-gain thing has been broken forever because of that... it creeps into greyscale and duo-tone and multichannel and even CMYK mode...   

But let's consider all of that a different complaint or "problem"...  the real issue is that suppose alpha needs to be computed without the sGrey and it gets corrected because everything is viewed in RGB from a sRGB or similar "gamma-correction" curve...  even if all these are functioning as they are supposed to..  then the real "bug" here is that it does not update consistently.

So for example, I suppose I can suggest people to use sGrey in their grey curve to make sure things like generating luminosity masks are going to have the same curve as the actual values of the greyscale from the luminosity in RGB...   the problem is that if you change the curve in grey settings,  then you click on one of the alpha-mask channels, it only "updates" that one channel to the new grey curve setting, and therefore the layer-mask, and how it applies alpha masking to an adjustment layer... but then all other layer-mask channels are still using the previous greyscale setting.   

So taking all the other stuff as supposedly "how it needs to function"..( like using an sGrey curve in greyscale mode in order to actually view the greyscale levels the same as they appear in sRGB)..  the main bug here is that it doesn't update all the layer-masks to whatever setting you change the grey curve to until you click on them, one at a time.   Personally I feel it shouldn't be responding to the greyscale settings at all, it should use the RGB color setting, and also the R, G, B, and extra channels in RGB mode should not respond to the grey setting either.   So it is just not consistent except they assume looking at a single channel (even a R channel when you don't have "view channels in color" set) to be looking at a greyscale channel.    

Since most people leave the greyscale set to "20% Dot Gain" which is the default, (and which is totally wrong, it is actually lighter appearing than if you printed it perfectly, and way lighter if you were really getting 20% dot-gain... because it is not applying the curve on top of an already sGrey corrected view... but again that is all kind of a separate issue).. so with the default then any actions I write which don't change a user's grey profile to sGrey, (if they even have an sGrey curve to choose), the actions will most likely use the 20% dot-gain grey setting and produce luminosity masks or other control-select alpha transparencies with the wrong curve applied.   

I hope they could at least fix the part where it is not applying to all the masks when updating the grey curve... apparently it is non-destructive (you can change the curve and it changes the mask -when you click on it- and can be changed back)... but any adjustments made or color fills etc that use that alpha mask on the layer or group will be processing using that grey curve.     The problem is also in the original selections made from a channel to get an alpha transparency selection,  even selecting the RGB all as one (it is basically selecting Luminosity there,) but since it "sees" it as a greyscale channel then it applies the greyscale curve setting as well.   

I agree with you that working in RGB mode should use the RGB Color setting even when looking at single channels or mask channels... since I can look at a completely neutral layer and it applies the sRGB (or whatever curve I have set), why doesn't it consider the alpha channels and therefore masks on layers as just neutral grey in RGB and let the RGB Color setting do the "gamma correction".    Very frustrating, as I said I like to choose how to simulate the other modes properly all within RGB so as to avoid all these issues with the dot-gain or curves applied in Duotone, Grey, Multichannel, CMYK....  but now I must worry that even in RGB there is this bleed-through of the greyscale curve settings.   SMH
(Edited)