Photoshop: Layer Masks Overly-sensitive/Ghost Images

  • 1
  • Problem
  • Updated 2 days ago
  • (Edited)
I am shooting a small object twice - once lit with softboxes from the top, and then again only backlit, and I use the backlit image to create a perfect layer mask that punches out the background from my small object.

My issue is how sensitive the layer mask is - I can edit and edit to achieve a perfect black background, but once I paste my image into a layer mask, there's all sorts of new areas of white to edit out. Shown below, I took screencaps of the image as originally shot, as edited to be black for masking, as pasted into the layer mask with before-unseen detail popping up all over the image, and my biggest issue -

I painted out the toothpick with a %100 opaque, %100 flow paintbrush tool, but you can still see a ghost image of the toothpick coming out of the front of my object? How is this possible? If the layer masks weren't so sensitive, this project would have maybe 50% less editing to do.

This is all very subtle, but the detailed applications of the photos require me to be this picky. As far as I'm concerned this is a problem with the software, I should be able to generate images from one file and paste them into a mask without having the software reinterpret them.


1- Mask photo as shot -


2 - Mask Photo with lighter elements PAINTED OUT - 


3 - Mask Photo after being pasted into Layer Mask, new details become visible -


4 - Final Image, with details visible that had previously been 100% painted out -
Photo of Nicholas Riggs

Nicholas Riggs

  • 4 Posts
  • 0 Reply Likes

Posted 1 week ago

  • 1
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
As far as I'm concerned this is a problem with the software, 
Too be blunt: Your claim does not seem to begin to make sense. 

I recommend you add something to your workflow: 
• Apply the Layer Style »Stroke« set to »Position> Outside« and a glaringly obvious color to the Layer with the Layer Mask 
• Use the Burn Tool set to »Range> Shadows« to address the areas you had missed
• Invert the Layer Mask and address the »insides«, too 
• After resolving the problem remove the Stroke 

 But keep in mind that you should do this at View > 100% (or larger) to make sure you don’t miss some stray pixels. 
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
I painted out the toothpick with a %100 opaque, %100 flow paintbrush tool, but you can still see a ghost image of the toothpick coming out of the front of my object? How is this possible? 
This would be possible if you did not paint out the region in question accurately. 
Add a very steep Curves Layer to the edited image that’s intended to be used as the Layer Mask and it becomes obvious where you actually missed non-black pixels. 
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
I should be able to generate images from one file and paste them into a mask without having the software reinterpret them.
Your screenshot is clipped to closely to be sure whether you edited the image (that is intended to be the Mask) in RGB or grayscale. 
If you should have edited it as an RGB image then a reinperpretation would naturally be unavoidable when converting it to a grayscale image. 

Could you provide the images for testing? 
What are Photoshop’s  Color Settings and the images’ Color Space? 
Photo of Nicholas Riggs

Nicholas Riggs

  • 4 Posts
  • 0 Reply Likes
My original image is opened and edited in a greyscale colorspace before being pasted into the layer mask.

I understand that there are remaining non-black pixels in my mask image, my issue is that those details are not visible in the source-image itself, but only appear once it is pasted into the mask layer. It's not possible to paint or burn those details out unless I can see them, and my preferred workflow is to do all that editing before I paste it into the mask layer.

Take a look at the files - https://www.dropbox.com/sh/n3zqvmbm767mspp/AAC5aVVUXsZuqQymR9vLa7zCa?dl=0

The test_mask shows how the file looks before it is pasted in, and yes, I know that there are non-black pixels present in the image, but unless you fiddle with the levels or curves adjustment, they're not visible at that point.

The test_issue has the masked image duplicated several times on top of itself to show the non-black pixel areas more clearly - there are areas in the mask that are 100% black pixels that are still allowing detail from the underlying photo to show through. The toothpick that extends out from the object to the right, and the ghosted images all around the frame is 100% painted out black in the mask, but is still visible in the final composite.

I'm not trying to blame sloppy work on Photoshop, I just can't figure out what is allowing certain details to show through masks even when they are 100% black pixels in the masking layer.
(Edited)
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
 there are areas in the mask that are 100% black pixels that are still allowing detail from the underlying photo to show through. 
You seem to be mistaken – it’s only 99% in many parts.   


I understand that there are remaining non-black pixels in my mask image, my issue is that those details are not visible in the source-image itself, but only appear once it is pasted into the mask layer. 
Maybe you should not use »Grey Gamma 2.2« as the grayscale space. 

but unless you fiddle with the levels or curves adjustment, they're not visible at that point.
It may be necessary to incorporate temporary Adjustment Layers into your workflow if you do not want to edit the actual Layer Masks. 
Photo of Nicholas Riggs

Nicholas Riggs

  • 4 Posts
  • 0 Reply Likes
So I'm basically seeing these effects as a result of the brush tool not painting at 100% even though I am selecting the lowest value available in the program... Interesting.

I had been assuming that I was somehow seeing detail come through areas of the mask I had been totally painted black with the brush tool, but apparently those areas only have a 99% k value.
Photo of Nicholas Riggs

Nicholas Riggs

  • 4 Posts
  • 0 Reply Likes
Hot damn, I just checked the color picker and realized that selecting the black off the lower right hand side (full saturation) provides a 100% k value and selecting the black from the left hand side (zero saturation) only provides a 99% k value.
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
Just use the single key shortcut D to reset fore- and background color (and X to switch them when necessary) every now and then if you want to paint true black and white. 

So I'm basically seeing these effects as a result of the brush tool not painting at 100% even though I am selecting the lowest value available in the program
Not unless you have some unnecessary Pressure Sensitivity, Fading or similar settings. 
Photo of christoph pfaffenbichler

christoph pfaffenbichler, Champion

  • 1251 Posts
  • 185 Reply Likes
I think that aside from your workflow choices another thing may have helped confuse the issue: 
Layer Masks’ appearance on screen does not have a »perceptive« relation to their effect but a numerical one. 

The screenshots each show two windows of the same image, left the composite image (a black Solid Color Layer with a Layer Mask atop a white background), right the Layer Mask itself. 
Once the Grayscale Working Space is set to »Dot Gain 15%« and the other time to »Grey Gamma 2.2«. 
So while the Mask looks different (maybe especially noticeably in the 50% area) the values are the same and therefore the effect in the RGB image is the same.