Camera Raw/Lightroom: Enhanced Profile Issue: CSV looktable files with large dimensions are not able to be embedded in presets

  • 4
  • Problem
  • Updated 11 months ago
  • (Edited)
First, I love having the ability to embed custom HueSatMap style looktables inside the enhanced presets in ACR 10.3. I think this will be a gamechanger and some of the results I'm getting in testing are jaw-dropping.

However, I'm finding that when I try to create a new enhanced profile in ACR, it will not embed looktables which have large dimensions (for example, 90, 30, 30). I have no trouble with smaller tables (for instance, 36, 16, 16). 

It's really important to be able to have a table this size because the changes are mapping to linear Prophoto RGB space. So the coordinate system for value is very dense close to white, and then becomes considerably less dense. So in a small table size, there is very little control over the outer limits.

(And Adobe themselves uses tables with 90, 30, 30 dimension in many of their own DCP files emulating camera styles). 

Is there a way that this can be fixed, so that we can embed larger table sizes inside ACR? 

OR - if you could you give us the tool to encode our looktables directly rather than having to do it through ACR? It looks pretty close to Ascii85? 
Photo of Nathan Johnson

Nathan Johnson

  • 26 Posts
  • 22 Reply Likes

Posted 2 years ago

  • 4
Photo of Simon Chen

Simon Chen, Principal Computer Scientist

  • 1674 Posts
  • 576 Reply Likes
It is intentional to restrict the size of the LUT (I think it is restricted to 32x32x32) to reduce the rendering performance impact and the memory and disk space required to store/load/sync settings with such large LUT.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4709 Posts
  • 1276 Reply Likes
Why do DCPs sometimes have larger look tables (hue-saturation maps), if they cause rendering performance issues?
Photo of Simon Chen

Simon Chen, Principal Computer Scientist

  • 1674 Posts
  • 576 Reply Likes
The DCPs are accessed by reference and the bulk of the data does not get carried around in the XMP sidecar or the settings tables stored in the catalog. 
Photo of John R. Ellis

John R. Ellis, Champion

  • 4709 Posts
  • 1276 Reply Likes
The enhanced color profile applied to a photo is also stored by reference in the .xmp sidecar:

   crs:CameraProfile="Adobe Standard"
   crs:CameraProfileDigest="AC58BA900C3A001F052B43DA5615508D"
Photo of John R. Ellis

John R. Ellis, Champion

  • 4709 Posts
  • 1276 Reply Likes
Oops, Adobe Standard is a DCP.  But Adobe Color is an enhanced profile, and it's look table is stored by reference in the .xmp sidecar:

   <crs:Look>
    <rdf:Description
     crs:Name="Adobe Color"
     crs:Amount="1.000000"
     crs:UUID="B952C231111CD8E0ECCF14B86BAA7077"
     crs:SupportsAmount="false"
     crs:SupportsMonochrome="false"
     crs:SupportsOutputReferred="false">
    <crs:Group>
     <rdf:Alt>
      <rdf:li xml:lang="x-default">Profiles</rdf:li>
     </rdf:Alt>
    </crs:Group>
    <crs:Parameters>
     <rdf:Description
      crs:Version="10.3"
      crs:ProcessVersion="10.0"
      crs:ConvertToGrayscale="False"
      crs:CameraProfile="Adobe Standard"
      crs:LookTable="E1095149FDB39D7A057BAB208837E2E1">
Photo of Simon Chen

Simon Chen, Principal Computer Scientist

  • 1674 Posts
  • 576 Reply Likes
That only applies to the default LUTs built by Adobe. Those default (or well known) LUTs are embedded in Adobe's code and shipped with every copy of Lightroom. In the case that you want to build and use your own custom LUTs in your custom profile, Lightroom does't know them and the full custom LUT table would need to be embedded in the XMP. This is the same reason that Lightroom does not need to embedded the Adobe Standard.dcp, because it is installed with every copy of Lightroom. That is not the case with custom dcp or custom LUT.
(Edited)
Photo of John R. Ellis

John R. Ellis, Champion

  • 4709 Posts
  • 1276 Reply Likes
Make sense, thanks much.
Photo of Nathan Johnson

Nathan Johnson

  • 26 Posts
  • 22 Reply Likes
Simon - the documentation mentions that the max RGB LUT is 32x32x32... but does not mention the max dimensions for dng_hue_sat_map"  look tables. 

Whatever the max dimension is, it seems to be smaller than the RGB LUT... 

I just tried a dng_hue_sat_map csv at 32x32x32 and it would NOT parse it.  :(

The documentation does mention that "the Adobe Raw profiles use 36 hues, 16 saturations, and 16 values." 

In my testing, this is fine for make broad changes to an image. 

But to me, the beauty of the hue_sat_map is making very precise, finely tuned adjustments that wouldn't be possible any other way (and in dimensions that I get to chose)... and 36x16x16 gives me LESS precision than an RGB LUT...


Photo of Max Wendt

Max Wendt, Employee

  • 16 Posts
  • 22 Reply Likes
You have a bit more flexibility with look tables than you do RGB tables. The maximum size cannot exceed 36 * 32 * 16, but you can have more that 36 hue divs by reducing the number of sat or val divs.

The maximum values for each are:
hues: 360
sats: 256
vals: 256

You can arrange them however you like, as long as the product of the three of them doesn't exceed (36 * 32 * 16).
(Edited)
Photo of John R. Ellis

John R. Ellis, Champion

  • 4713 Posts
  • 1277 Reply Likes
"could you give us the tool to encode our looktables directly rather than having to do it through ACR? It looks pretty close to Ascii85?"

See this related feature request for the RGB LUTs in enhanced profiles: https://feedback.photoshop.com/photoshop_family/topics/camera-raw-and-lightroom-document-the-format-...

The ASCII85 encoding is easy to figure out. The stumbling block seems to be the compression algorithm -- I tried all the common ones without success.
Photo of Max Wendt

Max Wendt, Employee

  • 16 Posts
  • 22 Reply Likes
Note that you don't need to use linear ProPhoto - you can specify the encoding when you make your looktable. Using sRGB still provides you with plenty of power; I made all of the initial profiles using sRGB with divs of 36, 16, and 8. 
Photo of Nathan Johnson

Nathan Johnson

  • 26 Posts
  • 22 Reply Likes
Max - can you speak to the advantages / disadvantages of sRGB vs linear ProPhoto for encoding in the dng_hue_sat_map? As far as I can tell, tables with the sRGB encoding would still it's entries somewhat clustered towards white, no?
Photo of Max Wendt

Max Wendt, Employee

  • 16 Posts
  • 22 Reply Likes
No - the sRGB encoding is non-linear, and the distributions of the values are more even. Making a few quick "toy" tables will help you get a feel for how the divisions map; you can target the highlights, shadows, and midtones in a few different table to get a sense of what values the divisions affect.
Photo of Nathan Johnson

Nathan Johnson

  • 26 Posts
  • 22 Reply Likes
Max - thanks so much for your response on this. Yes, I've made a number of "toy" tables. 

As a developer though, I want to get more than just a "sense" of what values each division affects (as well as how much each values is affected by the scale factor). I'm interested in getting mathematical precision.

Just as an illustration, I have this toy table:  https://www.dropbox.com/s/g507vbaeqbxfsww/playground.png?dl=0  that is equally divided between 36 hues, 10 sat division, and 10 val divisions. In theory, I should be able to precisely target any pixel on this table.  



Let say for illustration that I wanted to completely desaturate the entire 9th "column" of saturation levels. So I build this 36x10x10 table (sRGB encoded), and set the sat scale to 0.0 for all entries where sat is 9/10 (link here: https://www.dropbox.com/s/3vkiypp8rikahkw/srgb-sat-9-10.csv?dl=0 ) Here's what results:



Not exactly what I expected...

Now if I make sure my playground image is in the ProPhoto color space https://www.dropbox.com/s/u86s9w6mjfet1uz/prophoto-playground.tif?dl=0 , things get way closer to what I would expect, but is still off in that it seems to be effecting brighter vals differently when it should only be effecting saturation (also, the experiment doesn't give me a lot of confidence that I can expect the nearly the same results from the looktable on a JPEG  vs linear RAW file..)




I'm also having a hard time understanding the math behind the scale factor... for instance, if I set a scale factor on all saturation to 0.5 ( https://www.dropbox.com/s/l05owgkf35jlw0u/srgb-sat-halved.csv?dl=0  ), I would expect HSB measurements to show half the saturation they were before. But this is not the case. For instance, saturation that was previously at 100% is now at 11-40% saturation (weirdly depending on brightness - with the brightest values being severely desaturated). 



Very strange.

Any input you could provide on how to understand mathematically where the divisions are being made, and how the scale factors are being applied would be greatly appreciated! 

-Nathan
Photo of Todd Shaner

Todd Shaner, Champion

  • 1601 Posts
  • 540 Reply Likes
There appears to be a similar issue with using even the Adobe enhanced camera profiles in a Develop preset as described here:

https://feedback.photoshop.com/photoshop_family/topics/color-profile-not-applied-with-preset

I'm not seeing the issue the OP mentions, but do see the issue of mentioned by Anthony Blackett in his reply. Using any of the 'Adobe' named profiles other than 'Standard' produces different results when applied using a preset than when applied directly in the Basic panel.

Photo of Cameron Rad

Cameron Rad

  • 161 Posts
  • 48 Reply Likes
This isn't the same issue.
Photo of Todd Shaner

Todd Shaner, Champion

  • 1601 Posts
  • 540 Reply Likes
Cameron, I didn't say it was the same issue, but as Nathan Johnson mentions:
"And Adobe themselves uses tables with 90, 30, 30 dimension in many of their own DCP files emulating camera styles."
This appears to be the cause of the issue at the new post I mentioned when using the Adobe Color, Landscape, Portrait, and Vivid enhanced profiles in a preset. I'm hoping someone here can investigate further what's happening and determine a solution.
So any ideas guys on the cause?
Photo of Cameron Rad

Cameron Rad

  • 161 Posts
  • 48 Reply Likes
That isn't the cause. The issue you're talking about is regarding presets in Lightroom not loading/displaying profiles properly.

It isn't an issue with the HSV Look Tables or HSV Look Table based Profiles as the profiles themselves can load just fine, when selected on their own. Also new profiles can be made in Adobe Camera RAW using the HSV Look Tables from those Adobe profiles and they work just fine.

What that thread is talking about is presets in Lightroom Classic CC made with Adobe Enhanced Camera Profiles (with HSV Look Tables), aren't loading the profiles correctly. 

If you follow the Enhanced Camera Profile SDK, you can see how to build profiles with HSV Look Tables copied from Adobe Color, Vivid, etc. What you will notice when you select those Adobe profiles as a base, is that it isn't just an HSV Look Table. There's also a tone curve. This may be a contributing factor in the differences seen in lightroom. One may need to select "Treatment & Profile" as well as the point curves.
(Edited)
Photo of Todd Shaner

Todd Shaner, Champion

  • 1601 Posts
  • 540 Reply Likes
Thanks Cameron.

"What that thread is talking about is presets in Lightroom Classic CC made with Adobe Enhanced Camera Profiles (with HSV Look Tables), aren't loading the profiles correctly."

I understand the difference between DCP and XMP profiles. The issue at the other post is that applying the Adobe supplied Enhanced profiles using a custom Develop preset produces different rendering results than direct application using the Basic panel 'Profile' selector.

"There's also a tone curve. This may be a contributing factor in the differences seen in lightroom. One may need to select "Treatment & Profile" as well as the point curves."

I still get the same difference in rendering when checking everything in the Develop preset settings as shown below. Again, this is happening when using the Adobe supplied Enhanced camera profiles: Adobe Color, Adobe Landscape, Adobe Portrait, and Adobe Vivid. Not user created enhanced XMP profiles. That's why I said it appears to be a "similar issue!"
Clearly there is an issue with the Adobe supplied Enhanced Camera Profiles when used in a Develop Preset.

Please run your own check let me know if you see the same issue. Thank you kindly!
(Edited)
Photo of Cameron Rad

Cameron Rad

  • 161 Posts
  • 48 Reply Likes
Please run your own check let me know if you see the same issue. Thank you kindly!

Ran my own check using Develop Presets with Adobe Enhanced Camera Profiles and don't see what you're talking about. 





(Edited)
Photo of Todd Shaner

Todd Shaner, Champion

  • 1601 Posts
  • 540 Reply Likes
Sorry for the confusion. To demonstrate the issue:

1) Create two virtual copies of the colorchecker.
2) Reset both copies to remove any settings.
3) Apply the Adobe Color profile to the first copy using the Basic panel 'Profile' selector.
4) Create a new Develop preset using ALL of the settings in the first copy.
5) Apply the new Develop preset to the second copy.6) Observe differences in color rendering (especially in the Blue patches).
(Edited)
Photo of Cameron Rad

Cameron Rad

  • 161 Posts
  • 48 Reply Likes


No issue happened for me.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4711 Posts
  • 1277 Reply Likes
Following Todd's recipe on my LR 8.1 / Mac OS 10.13.6, I observe differences in rendering:

https://www.dropbox.com/s/za4h06zl8n0ipvx/profile-preset-problem.2019.01.02.mov?dl=0
Photo of Todd Shaner

Todd Shaner, Champion

  • 1601 Posts
  • 540 Reply Likes
Thanks John for confirming.
So it's happening on both my Windows 10 Build 1803 system and John's Mac OS X 10.13.6 system using LR 8.1. I also checked a Windows 7 SP1 system using LR 8.1 and see the same issue.
Cameron what version of OS and LR are you using?
(Edited)