Camera Raw & DNG: Lens Profile Creator can't handle 36 MP DNGs or JPGs from Pentax K-1

  • 1
  • Problem
  • Updated 3 years ago
  • (Edited)
First the software used and the environment:

Adobe Lens Profile Creator, version 1.0.4. for Windows
on a 64-bit Windows 10 computer with 16 GB RAM, 128 GB system SSD.

The problem: I tried to calibrate a lens for my Pentax K-1 and its 36 MP sensor - and found that the Adobe Lens Profile Creator (ver. 1.0.4.) was not willing to process the full resolution DNGs with the required test chart shots. It gave a warning that - looking for the corners of the test chart - the memory requirements were exceeded and that I should try with a lesser number of images. I reduced the nine provided shots to three, but got the same warning - starting with the first pic analyzed.

I then converted the DNGs to full resolution JPGs using LR CC and tried again - same result.

For a third try I limited the longest edge of newly converted JPGs to 3000 px - and the lens profile creation process with the same pics (now downsized) went through. But I found that the created profiles weren't available for my DNGs - and that's what I wanted them for. 

I found a workaround for that - simply switched the RAW tag in the created .LCP file to "true" - but of course am not satisfied with this solution. I would really like to be able to process the full rez DNGs without any workarounds. Are the 36MP DNGs/JPGs simply too big for processing?

Phil

Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes

Posted 3 years ago

  • 1
Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Hi Phil,

The Lens Profile Creator has not been updated in several years, so there may be a technical limitation with the larger K1 images. 

Would you be willing to upload the images to DropBox (or similar) so that I may do some troubleshooting?

If so, please send me a link to ccastleb@adobe.com.

Also, please let me know which lens(es) you are attempting to profile, if it isn't obvious in the image metadata. I ask because Pentax typically sends Adobe pre-built profiles to be included in ACR/LR. It's possible we may already have a profile on the schedule for a future release.

Regards,

- Chris
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
Hi, Chris!

Thanks for the quick reply. 

The lens I profiled on the Pentax K-1 is a Sigma 24mm AF Super Wide II for K-Mount. It's an older lens that doesn't have specific lens info in the metadata - LR just identifies is as "Sigma" (like another Sigma I have, an older lens too - the Sigma 50-500 f4.0-6.3 APO DG for K-Mount which also identifies as "Sigma" without any further info).

The next lens I plan to profile on the K-1 is a Pentax FA J f4.0-5.6 18-35 - AL - and boy, this lens does seriously need profiling ... :-)

I'm uploading a 374 MB ZIP file right now - it contains the nine f11 DNGs I made profiling the Sigma 24mm lens mentioned above - via WeTransfer. You'll get some download info from WeTransfer hopefully in the next 30 minutes or so.  I have further profile shot groups available if you need them - from f2.8 up to f22 of the same lens.

Thanks a lot for your help!

Regards,

Phil

Edit: The WeTransfer upload has been completed. You should get a corresponding message soon.

Phil
(Edited)
Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Hi Phil,

I replied to the file transfer message and then I read this follow up. :)

I was able to build the profile for you and attached it to my reply. Not sure yet why the ALPC is not working on your system.

Feel free to forward the other calibration shots to me and I will generate the profiles for you until we can figure out why ALPC isn't working for you.

Regards,

- Chris
(Edited)
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
Thanks very much for your help, Chris! I received the .lcp file.

But see attached a screen shot with the warning message popup - I was just trying again to create the profile with exactly the same DNGs I sent you.  

Thanks for your offer for further profiling. Since a lot of data is involved - I plan to do  180 single shots for the FA J 18-35 profile (4 apertures, 5 focal lengths, 9 shots each) giving around 7 GB - I'm a bit hesitant to bother you with it. :-) We'll see. Maybe you'll find a solution before I find the time to take all those shots. ;-)

One extra thing:  I tried varying the  size of the Calibration Cache in the preferences dialog, but this didn't help either. Default was 1 GB, but even with a setting of 20 GB the behavior was the same.

Phil

(Edited)
Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Have you tried adjusting/increasing your virtual memory/paging size in the Windows 10 Performance options?
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
Hi, Chris - should've posted my new infos here as a comment, but put it as a reply to the original post in this thread instead. Thought I'd let you know ...

Phil
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
OK, Chris - I have done some further testing on my computer and two additional machines - all with the same files I sent you.

First I followed your suggestion and increased virtual memory in Win10 on my computer - it was on automatic handling (with a maximum short of 3.000 MB) - I switched it to manual, minimum 5.000 MB, maximum 10.000 MB, re-booted Windows so that the new settings could take effect, and tried again - same error message appeared in ALPC, again while processing the first pic.

Hadn't any more time this morning, but later had access to two other machines - first a fairly old Mac (but running with the current Mac OS El Capitan), second a Win10/64bit machine again with 16 GB of RAM, but using a different processor - an i7; my own computer uses an AMD A10.

First tried everything on the Mac: No problems at all, everything went smoothly, profile was created fast.

Then tried the Win10/i7 machine: processing went further than on my Win10 computer - but didn't finish either: it stopped at the 6th pic with the same error message as posted before,

Finally I increased virtual memory on this computer too - switching again from automatic handling to manual, but this time trying settings of 20.000 MB minimum and maximum. After a re-boot ALPC was started again - and this time created the profile completely and without problems. Wow.

Still have to check if the 20.000 MB min/max settings for Win10 virtual memory also works on my home machine. Since there is a difference in handling the files if Win10 automatically allocates virtual memory - my computer stops processing at the first file, the other one with the i7 at the sixth - I sincerely hope that I can get my machine to work through all files too. Will have access again to it in about 10 hours.

Meanwhile any recommendations on your side how to adjust min and max setting for virtual memory?

Phil
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
Just tested: 20.000 MB min/max setting for virtual memory works on my home machine too. Profile creation went through. Interesting though that 16 Gigs of RAM are not sufficient to run the calibration ... OK, but problem solved. Thanks a heap, Chris!
Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Happy to hear that it helped!

Again, the ALPC hasn't been updated in a long time. It is also a 32-bit application. I'm pretty sure that limits its use of system ram, thus the reliance on virtual memory. There are no current plans to release a new version of ALPC, but should that change, updating to 64-bit would be high on my list of improvements.
Photo of KrenFor78

KrenFor78

  • 14 Posts
  • 0 Reply Likes
Hi again, Chris!

Unfortunately the problem was only partly solved. Memory management of ALPC is so weird that I can - with my new virtual memory setting - profile exactly one set of nine images. If I try to profile a second one in the same session - and for my wide angle zoom I had to profile 19 sets of nine images yesterday - I get the same memory error message as before. I had to close ALPC after profiling one set, restart it, reload the current project, select the next set and profile that.

A very slow and cumbersome process which I gave up on after the first five sets or so - another reason being that processing of the large DNGs took a long time on my AMD A10 machine, and due to heavy vignetting of my lens wide open a lot of those shots with the calibration target placed in the corners weren't detected by ALPC.

I instead turned to my "solution" described in the first post in this thread - converting the DNGs to 3.000 px wide JPGs, processing those and enabling the "RAW" flag in the resulting profile. This was done much faster, and I was able to process all 19 sets in one go - without any memory errors. Also with the JPGs APLC could detect almost all charts (I upped exposure half an EV before exporting, that helped with the strong light falloff in the corners).

I have to test the resulting .lcp file a bit, but it sure looks good so far - applied to JPG and to DNGs alike.

So if you ever update APLC, it really would be a good idea to make it 64 bit compatible - as you suggested yourself. And hopefully the memory problems can be solved with that.

Phil
(Edited)
Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Official Response
Thanks for the follow up, Phil. 

Given the ALPC is 32-bit, I am not surprised it struggles with the larger images and multiple profiling within a single session. Also, there may be other implementation issues with how it handles memory independent of the ability to address larger amounts of system RAM. I'll bring this up with the team as something to consider addressing in the not-too-distant future.

- Chris
Photo of steve flynn

steve flynn

  • 4 Posts
  • 0 Reply Likes

How do lens manufacturers produce profiles for use with larger megapixel sensors if ALPC struggles with memory issues? Do they use different software?

Photo of Chris Castleberry

Chris Castleberry, Camera Raw Engineer

  • 376 Posts
  • 114 Reply Likes
Hi Steve, 

A lens profile can be made independently of the sensor megapixel count. What is important in making an accurate profile is the size (e.g. 35mm, APS-C, Micro Four Thirds, etc.) of the sensor relative to the maximum image circle supported by the lens. For example, as long as the profile for a 35mm full-frame lens is generated for the entire maximum image area (i.e. on a full-frame body), it can be used on both a 35mm and APS-C body. 

Regards,

- Chris
Photo of Simon Chen

Simon Chen, Principal Computer Scientist

  • 1552 Posts
  • 503 Reply Likes
@steve, 

If your camera has a setting to reduce the image resolution when capturing the raws for creating the lens profiles, try that as a workaround. The resulting lens profile generated from these lower resolution image capture would be just as effective for the lens geometric and vignette corrections, regardless of the image resolutions. For correcting the chromatic aberrations, Lr/ACR currently relies on an automatic algorithm to estimate and remove them. So it does not matter. either.
(Edited)