Lightroom: Very large increase of 32bit .tif files while working in LR

  • 1
  • Problem
  • Updated 5 years ago
  • (Edited)
Hi,

issue: large 32bit .tif file size increase after importing a 32bit .tif file into LR and heavily working on this 32bit .tif file in LR.

For shooting interiors of churches, I very often use AEB with 5 exposures and merge these 5 files to a 32bit .tif file. Because merging the 5 exposures in PS HDR Pro often shows some colors casts, I now use the LR plugin "Photomatix Merge to 32Bit" (latest version).

The plugin exports and merges e.g. five Canon 5D Mk III files (each around 22MP / 25 MB) and imports a 32bit .tif file with around 130MB. Now when I start to work for hours on this 32bit .tif file in LR, the file size increases heavily, often to 300-500MB, sometimes even to 1GB++.

Please note that I do NOT leave LR with this 32bit file. Usually I do firstly a lot of global adjustments (temp, tone, sharpening, noise, grain, etc) and then a lot of local adjustments (2-5 graduated filters, 5-15 adjustment brushs with many, many brush strokes ) IN LR.

And the longer I work on such a 32 bit .tif file the slower LR becomes. Switching to 100% view takes often up to 10-20 seconds. Rendering the preview in library often takes 30-60 seconds! Switching from library to develop (and rendering the develop view) takes often 10-20 seconds).

Maybe the file size (hundreds of MBs, sometimes 1GB++) is the cause of the bad performance. But in each case I think that the file size should not increase so heavily.

I would expect hat LR just writes some develop metadata regarding my adjustments (like a part of the the content of a sidecar file) into some metadata space in the 32bit .tif file. And these information should not consume hundreds of MB...

So please fix this increase in file size and waste of storage space.

Thanks.

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes

Posted 6 years ago

  • 1
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
If LR is increasing the size of your 32-bit file because of continuous metadata updates as you make changes, then turn off the Automatically Write XMP Metadata to Files in LR Preferences and see if things work differently. With that turned off, if there is any increase in the filesize, then LR is doing something else to the file.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
Also, if Photomatix has the capability to produce a file in another 32-bit format other than TIFF that LR can still read, does LR behave similarly, or does it create a sidecar file, then?
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve,

thanks for the hints. I will watch the behaviour with "Automatically Write XMP" set to OFF. Then I will post the result here.

I guess that LR is doing something else. Metadata (and even many brushstrokes) take very few storage space (look at a sidecar file of a RAW file). Here file size should be in the range of KB, not hundred of MB.

But let see what happens with the OFF settings.

Manfred
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
What happens if you also make the TIF readonly before you start? You may get an error at the point you do whatever it is that increases the size.

The large increase in size suggests copies of the image data embedded within the TIF at various points in the processing.

Do you record Snapshots of the image as you process?

Does your LR database also increase by a similar size after you process the image? For the best test, import a unadjusted 32-bit image into a *new* LR catalog. If you use an existing catalog, then the extra storage might just be put into previously deleted database records, rather than actually making the filesize larger so the measurement would be impossible.
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
I wonder if LR is saving the file with different compression options.

Can you create small examples (maybe through cropping) and post them so we can examine the file structure?
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
That is another good idea.

Maybe the file does not increase incrementally, but when LR saves the TIF out as it writes the metadata changes, perhaps it is saving it as uncompressed or much less compressed than Photomatix did, so the size you're seeing is the uncompressed TIF size had it been saved in Photoshop. The slowness in LR may be due to LR caching things in memory and eventually using up all the memory, rather than the size of the TIF increasing.

Can you create a 32-bit HDR image with Photomatix using a small section of each LR image--by defining an identical crop in LR, so that the file isn't 100s of MB but a few 10s, instead. ZIP up a copy of that smaller-dimension HDR fresh out of Photomatix before it is touched by LR and upload it to www.dropbox.com then post a public download link to it here.

If you also want to process that smaller dimension file the way you normally would in LR and then zip it up and also upload that and post the public download link, here.
Photo of Allan Olesen

Allan Olesen

  • 64 Posts
  • 6 Reply Likes
Compression would only explain some of it. Uncompressed 32 bit per color channel, 3 color channels per pixels, and 22 megapixels should result in roughly 252 MB of image data.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi,

sorry, I forgot to mention that I enabled the option "Use Half Floating Point Format" in the LR plugin "Photomatix Merge to 32Bit".

Disabling this option results in around 260 MB .tif files after merging. Enabling this option results in around 130MB .tif files.

Manfred
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 812 Reply Likes
Yes, that cuts the uncompressed data size in half. But then the data is not directly editable, and other applications may change the 16 bit floats to 32 bit floats.

Again, to understand what you're seeing, we need example files.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Chris,

thanks for the clarification. My understanding as well as my experience is that LR does NOT change 16 bit floats to 32 bit floats. Maybe other applications do this...

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi,

in the meantime I have worked on a sample file, see

http://dl.dropbox.com/u/23843309/32bi...

The first file "test1--no-half-floating-point-format--auto-xmp-enabled--immediately-after-merge-by-photomatix.tif" was created by merging 5 raw files and was exported as original immediately after the merge. File size is about 4MB

The second file is "test1--no-half-floating-point-format--auto-xmp-enabled--after-editing-in-LR.tif" which I exported as original after working for about 5-10 minutes on 32bit tif file in LR. I added some gradients and several brushs with many brush strokes (as well as eraser brush strokes). File size more than 9MB!

So within a few minutes I could easily double the file size...

Thanks.

Manfred
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 812 Reply Likes
The difference is almost entirely XML metadata.

And it appears to contain hundreds of small paint/mask operations (each one is a separate bit of metadata, and really adds up). The basic color adjustment data doesn't take up much - but all those paint/mask operations take up a ton of space (around half the file).

Hmm, there appear to be multiple XMP packets in the large file. Each one getting larger than the previous. Looks like 15 packets total (from counting the end tags).

At first glance it looks like Lightroom has a bug in their XMP code that appends XMP data instead of replacing it.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Chris,

thanks for investigating this issue quickly. Hopefully the LR team can fix it soon.

If you need more information please let me know.

PS: Beside the issue, the ability of LR to import and edit 32 bit file is GREAT! It allows to easily achieve "natural looking" HDR images. My wish would be that the LR team integrates the merge to 32 bit in LR (without the color casts of PS HDR PRO) and enhance the 32bit editing facilities...

Thanks.

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Chris & Steve,

I just did another test, see

https://dl.dropbox.com/u/23843309/tes...

I now merged again the same 5 exposures (now with half floating point format enabled) but disabled XMP auto writing in the catalog settings.

The first file "test2--half-floating-point-format--auto-xmp-disabled--immediately-after-merge-by-photomatix.tif" ist the merge image immediately after the Photomatix merge. File size is about 2 MB.

The second file "test2--half-floating-point-format--auto-xmp-disabled--after-editing-in-LR-for-some-minutes.tif" was exported after working some minutes in LR (with Auto XMP disabled). File size is after some minutes about 2.5MB.

Hmmm, I do not understand why the file size changed. I did NOT save metadata to the file manually and AUTO XMP was disabled. My understanding is that in this case LR should NOT write anything to the file, all infomation should kept in the catalog.

What do you think? Maybe Chris can inspect the .tif file with some utilities to see the content and the changes to the file.

Thanks.

Manfred
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 812 Reply Likes
Both files contain XMP metadata. The larger one contains a single XMP packet with a lot of tone curves and painting/masking data (local corrections).

That's better than the multiple packets, but LR did still write all the metadata into the file instead of writing a sidecar file.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Chris,

my understanding is: if Auto XMP is enabled then LR should update the catalog AND the tif file (some metadata space within the tif file).

If Auto XMP is disabled then LR should ONLY update the catalog and should NOT write anything to the tif file or a sidecar file (i.e. there will not be a sidecar file)

Manfred
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2612 Posts
  • 333 Reply Likes
That is my understanding as well.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
Manfred, you can check for metadata by opening each small TIF in Windows Wordpad and using Ctrl-End to find the bottom and then page up a little to see the extent of the human-readable text.

Did you restart LR after setting the Auto-Write-XMP off? I would also flag the file as readonly and see if that prevents metadata from being written.

Another question, when you say Exported, are you creating a new copy of the file with the LR Export process?

What LR shouldn't be doing is writing XMP metadata to the original TIF as you're editing it, but if you're actually making a new copy of the file with LR then it easily could be embedding what edits you've made as it writes this new copy, itself.

The process for creating the post-LR-edit copy of the file you upload to dropbox should be to do edits, then exit LR and make a copy of the original file using the operating system Explorer or Finder and rename that, not make a new copy with LR using the Export process if that's what you're doing.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve,

thanks for the hint about inspecting the metadata in tif files.

What I have seen now: in the first file (exported as original immediately after merge by Photomatix) there is mainly a "rdf:Description" (XML entry), nothing more. In the second file (exported as original after working some minutes) there are 10,000+ lines with "crs:GradientBasedCorrections".

Regarding restart LR after setting the Auto-Write-XMP off: if am unsure about this i.e. I cannot remember for sure.

Regarding creating a new copy of the file with the LR Export process. I exported all files AS ORIGINAL. What LR does (according to my understanding) is: write all updates to the file (if any) and then do a plain copy of the file. Therefore I think it is identical to: closing LR and copying the file with OS mechanism.

I just did another test, see https://dl.dropbox.com/u/23843309/tes...

Now I disabled Auto XMP and restarted LR. Then I merged again the 5 exposures, exported as original immediately after the merge. This is file "test3--half-floating-point-format--auto-xmp-disabled--immediately-after-merge-by-photomatix.tif". Then I worked some minutes on the file (gradients, brush strokes, global changes) and exported again as original. This is file "test3--half-floating-point-format--auto-xmp-disabled--after-editing-in-LR-for-some-minutes.tif".

In Wordpad I can now see that the first file just contains minimal XML stuff (creator, GPS etc). But the second file contains a lot of adjustment XML entries! According to my understanding this should NOT happen. Auto XMP was disabled and I did NOT save metadata manually.

So it seems that (1) according to the last test LR writes XML entries to the file where it should not and (2) according to the first test LR appends XML metadata instead of replacing/overwriting it :-(

If something is missing please let me know.

Manfred
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
Can you please post a copy of the original file after adjusting in LR, not an Export As Original copy of it?

Export As Original should keep the image-pixel data unchanged which it seems to be doing, but this discussion is about metadata being added not image-pixels being different.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve,

I just did another test, see https://dl.dropbox.com/u/23843309/tes...

I again merged 5 exposures. Closed LR, copied the file. Then I worked few minutes on this file (gradients, brushs, global adjustments). Closed LR and copied file. Opened LR again and copied as original.

The two first files copied by OS means (after closing LR) are identical (verified via diff tool) and seem to contain no larger XML stuff at the end of the file.

The file exported as original is larger and contains XML stuff at the end. This is new to me. My understanding was that export as original and copying via OS means should result in the same file.

OK, something new learned. But ... I strongly think that the naming "exporting as original" is really more than confusing. Then you can start naming apples now bananas... But that is another thread :-;

Manfred
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
Export normally changes the image-pixel data. So the Original in Export as Original means Original-Pixel-Data, not Original-File-Before-Import. LR doesn't keep a copy of the file before it was imported, and metadata and previews data may have been written to it depending on other settings, so LR would have no way to revert to a before-Import state, and so apparently writing all the metadata to the file as if you'd done Save Metadata to File is what it does even when the Original-Pixel-Data is still there.

The main question is with your original post, did the reported size increase happen due to your editing in LR or due to your using Export As Original?

Because there were multiple XMP envelopes in the initial post-edit copy you uploaded, It seems that it probably happened due to your editing with Auto-Write-XMP turned on and there is a bug, but a confirmation would be nice to see, since the Export-As-Original wasn't doing what you thought.

Can you repeat the process used with the very first pair of files, except using the OS file-copy instead of Export As Original to create the sample file, but with Auto-Write-XMP turned on so we can clarify if there is an bug, still?
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve,

>... LR doesn't keep a copy of the file before it was imported...

For sure, I am aware of this. But in the case above, when Auto XMP is off and I did not manually save metadata, then when exporting as original I would expect a COPY of the file, not a "modified copy". For me there is no reason why LR should write metadata to the file during export.

>...did the reported size increase happen due to your editing in LR or due to your using Export As Original?

I am very sure that it happened due to my editing. While I was editing in LR I watched the file size in Windows Explorer. And what I saw was a steadily growing file size - without any export.

>...Can you repeat the process used with the very first pair of files,

Done (with another image), see https://dl.dropbox.com/u/23843309/tes...

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve and Chris,

during the last 10 days or so my catalog has grown from 4.5GB to now 7GB! During this time I have imported only a few images and worked on only few 32bit .tif images.

I guess this huge increase in catalog size has to do with the many, many adjustments I have done to only a few .tif images (as described in the postings above).

I suspect that when Auto XMP setting was enabled and LR appended the XMP data to the .tif file again and again (instead of overwriting it), LR has also added too much data to the catalog.

Three days ago I have run sqlite3_analyzer.exe against my LR catalog and I have seen that around 80-90% of all db pages have been history data.

So please advice how I can

- shrink my catalog to a "usual" size (I always optimize the catalog when doing my daily LR backup)

- prevent LR to write so much data to the catalog (around 2GB+ within 10 days when working only with few images)

Thanks.

Manfred
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2613 Posts
  • 333 Reply Likes
This is a longshot but might work:

Export your 32-bit TIFs' adjustments as a new catalog, remove these exact same images from your main catalog, optimize your catalog and hopefully it's much smaller, then re-import the 32-bit image adjustments from the newly-created catalog.

It is possible that the exponentially-growing history isn't accessible to the Export-to-catalog process, just like it isn't accessible to the history-display (right?) in LR, so perhaps it'd disappear in an export-import round-trip.

Obviously you want to backup your main catalog before removing anything from it, and if the images are in any sort of collection the export-import may remove that relationship, although I'm not sure because I don't use collections.

The only likely fix is to have Adobe figure out what they did wrong and make it so it doesn't continue to happen in a future version of LR. As far as Adobe shrinking any current exponentially-growing history, I doubt they'd make a special utility to help the handful of 32-bit people out, but you can always hope.

If the dead-weight get's lost in the export-import round-trip then you can always export your entire catalog to a new catalog, and if it's not gone, yet, then import that catalog into a new catalog and see if the final result is smaller.
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve,

thanks for the hints and advice.

What I have done in the meantime: I manually deleted the dev history of 2 images and optimized the catalog. Now it is 1.5GB smaller!

Maybe I will try to export some or all images and reimport them. But I am unsure what this will do with my many other images and how I can verify that all went fine.

Sure, in theory all should work. But LR is not theory, it is a piece of software with some or even many bugs...

Nevertheless I hope the LR team tries to find the cause of this excessive catalog size growth.

Thanks.

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve and Chris,

one addition to my postings. The 32bit development facility of LR is great. Now I have 16 to 20 EV values in a single image. I can increase exposure or shadows globally or locally without seeing any noise. For me this is an unbeatable LR feature!

But for really using it LR should be designed and implemented to use many, many local adjustments. 32bit development without many local adjustment makes no sense to me.

What is also missing for 32bit processing: selection/masking features similar to PS. In PS I can select/mask a part of the image and then apply some adjustments. In LR it is painful to exactly paint a mask (imagine e.g. painting/masking the windows of the interior of a cathedral).

Unfortunately PS is not designed to fully work with 32bit images. Some functions work but many are disabled. So LR is the only practical alternative.

So please forward these remarks to the LR team. I hope that they can and will make 32bit processing right and a joy to work with.

Thanks.

Manfred
Photo of Manfred Schneider

Manfred Schneider

  • 35 Posts
  • 0 Reply Likes
Hi Steve & Chris,

another "glitch". After working some hours on a 32bit .tif file when opening this in PS CS6 for further refinement I have to wait around 10 minutes (!) after pressing CTRL+E until PS CS6 shows the image. This is on a Win7/Intel i7-3770/32GB machine.

Then I exported the 5 RAW files as well as the 32bit .tif file as catalog. Now the catalog has a size of around 50MB. Then I started LR with this new catalog and did a CTRL+E for invoking PS CS6. I had to wait more than 15 minutes until the image was rendered.

It seems that ACR have to read a huge set of metadata. This is hardly acceptable.

Please ask the LR Team to work on this issue too.

Thanks.

Manfred