Lightroom: 3.4 is corrupting my source JPEG's (no, really!)

  • 2
  • Problem
  • Updated 5 years ago
  • Solved
  • (Edited)
Now I know what you're thinking, but hear me out ... I've been working on this problem for a few days now, and I've googled plenty of threads where people have thought that LR was corrupting their images, and friendly people gently pointed out that LR doesn't modify your source images, and that the problem was probably a bad hard drive, memory card, or otherwise - but this is different.

The specs:

Lightroom 3.4
OS X 10.6.7 - all latest updates
Model Identifier: MacPro5,1
Processor Name: Quad-Core Intel Xeon
Processor Speed: 2.8 GHz
Number Of Processors: 1
Total Number Of Cores: 4
L2 Cache (per core): 256 KB
L3 Cache: 8 MB
Memory: 8 GB

The situation:

A few days ago, I started noticing corrupted images in my LR catalog. Opening the images in Finder / Preview / Photoshop showed the same corruption, so it wasn't an issue with LR previews or thumbnails - the files were really toast.

I used ImageVerifier to search out all possible bad files, and was able to restore good versions of the (JPEG) images from backups - about 50 total, so far so good.

I verified that the files looked good in Finder / Preview, then re-opened Lightroom and navigated to the restored images. The images I had just restored started updating their thumbnails in LR, and right before my eyes several of them came up corrupted. I looked back in Finder, and sure enough, the files were bad again - with a current timestamp on them. It appears that LR had updated the files with a new thumbnail or metadata when reading the images, and corrupted them in the process.

I've repeated this process twice, with the same results. Again, I was able to restore the corrupted files from backups, but I'm now afraid to open up my master catalog and browse around for fear of destroying more images.

Some things I've tried:

I took a set of the restored images (now known to be good, but the same files had been corrupted by LR previously), and copied them to a new folder for testing. I created a new blank catalog, and Added those images to it. I added keywords, adjusted development settings, etc. then "Saved" the settings to the XMP data in the JPEG images. Timestamps all updated on the files, so I know data was saved into them - but nothing got corrupt. I tried copying the original known good images over these newly updated ones - LR didn't complain, but rebuilt it's thumbnails silently, and no corruption. I tried copying corrupt files over them - sure enough, new thumbnails rebuild showing them corrupted in LR. I then copied the good files back again, and LR happily rebuilt the previews/thumbs again with no issues. Basically, I could not reproduce the problem with a fresh catalog.

So this makes me think that something is wrong with my master catalog file - but I have 100,000 images in it, with tons of keywords and developer settings. How can I repair or migrate this catalog without loosing my settings, or corrupting more files?

The facts:
  • This is hard drive data only - no camera cards are involved here. Some of these images were copied to the hard drive years ago, and according to my backup archives were in good condition as recently as 1 month ago.
  • The hard drive appears to be good - Disk Utility reports no errors, and data only gets corrupted when I view the files in LR.
  • The appearance of the corrupted files seems to correlate with the time I upgraded from LR 3.3 -> LR 3.4, though that may be coincidence.
  • I've attached an example of a file corrupted by LR.

    Next, I will try downgrading to LR 3.3, but it's late, I'm tired, and I don't want to be risking more files when I can't think straight and don't trust myself to restore them properly.

    If anyone has any other suggestions I would be most appreciative - but for the moment, this puts a bit of a kink in my photography work-flow :)

    Thanks for any help you can offer!

    Here's an example corrupted file:



    And here's the original known good image:

  • Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    • frustrated

    Posted 8 years ago

    • 2
    Photo of Lee Jay

    Lee Jay

    • 990 Posts
    • 136 Reply Likes
    Do you have autowrite xmp set to on?
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    Yes, I did. But I have now turned it off. I can confirm that corruption occurs when I hit CMD-S to save the meta-data to the file.

    I had previously worked for years without that setting enabled, and I probably did enable it around the time that I upgraded to LR 3.4. I had turned it on after reading a number of recommendations to use it - but at the time, I did not realize it would save data directly into the JPEG as opposed to an XMP sidecar file.

    The corruption issue only seems to affect JPEG images - which is the majority of my database. I do have some CR2 raw files, and TIFs from other sources, but these have been unaffected.
    Photo of Dan Tull

    Dan Tull, Employee

    • 172 Posts
    • 38 Reply Likes
    Does the Cmd+S (manual) corruption still occur only in the original catalog and not in a fresh catalog? I'd be very interested in getting some sample files to at least see the corruption results and (better yet) to directly reproduce the issue myself, too.
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    No, the CMD-S function does not corrupt the sample images in a fresh catalog, only in the original catalog.

    The two image attached to this thread show a before/after corruption. If you mean you would like a copy of the Catalog, I'm open to that, but it'd have to be some sort of physical delivery (the catalog file alone is 1.4gb). Let me know if you want to explore that.
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    I'm even open to doing a remote session if you want to screen share to my system - I can demonstrate the issue, and then you can let me know what files would be helpful to you to debug it.
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    Ok - I did a little more work on this, and I've got it boiled down to an example catalog, and a set of images which reproduce this problem in about 75mb.

    I copied my master catalog, then moved aside my image library folder so that it wouldn't find it. Then I took one set of images known to be a problem, and made a copy of it aside the new demo catalog. I re-pointed that folder in the demo catalog to the new location, and was able to reproduce the problem using the new demo catalog and set of images.

    I then restored them, and removed all other file folders and collections from the demo catalog and optimized it. It's now down to about 2mb in size for the catalog itself, which has just 16 images in it. I can package up the catalog, a folder of images, and a second copy of the images as "known good" and upload it somewhere if you'd like.

    Would you like me to first demonstrate the problem (i.e. corrupt the files with LR) before uploading? That way you'll have an example of bad files, along with a known good copy, and you can attempt to restore/reproduce the problem yourself?
    Photo of Dan Tull

    Dan Tull, Employee

    • 172 Posts
    • 38 Reply Likes
    That sounds perfect. Having the original cat + files (to try to repro internally) as well as an example of a corrupted JPEG to examine should give me a good starting point to investigate. Based on how much you've reduced the file set size, a zip file with all of it should still come in well under 100 MB limit for free transfer services like transferbigfiles.com, yousendit.com, or Adobe's own SendNow service. My adobe.com e-mail handle is dtull, if you want to send them my way. Thanks for taking the time to narrow down a small set, that's a huge help when investigating issues like this!
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    If I lock or otherwise write-protect all of my source images, would that prevent LR from writing data to them, even if a user presses CMD-S (possibly accidentally?)
    Photo of Beat Gossweiler

    Beat Gossweiler

    • 238 Posts
    • 38 Reply Likes
    On Windows, marking a file Read-Only will prevent LR from writing into it (without error message). On Mac, you'll have to try (possibly with removing access rights "write" from your images for all three authorities).

    Beat
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    I'll try that - nothing that a 'chmod -R ugo-w *' in a Terminal window cant fix :)
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    I've tried this now, and it seems to work pretty well for the time being. It is a little disconcerting that LR doesn't generate any error message when it's unable to write the XMP data - it just goes about as if everything is working normally, but no files get updated if they're write-protected.

    I'll have to figure out a workflow for new incoming files - convert to DNG maybe? (Yeah, I admit it - I've been in LR for years, and I still haven't standardize on DNG yet ...)
    Photo of Lee Jay

    Lee Jay

    • 990 Posts
    • 136 Reply Likes
    If you have a catalog and some files that can create this issue in a repeatable manner, I would guess that Adobe would want those. Do you?
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    I've managed to create a reproducible example now (see above).
    Photo of Lee Jay

    Lee Jay

    • 990 Posts
    • 136 Reply Likes
    Doug's effort here is an example of a really tricky bug to nail down, yet he did an amazing job of nailing it down. And here's the result:

    http://blogs.adobe.com/lightroomjourn...
    Photo of Jeffrey Tranberry

    Jeffrey Tranberry, Sr. Product Manager, Digital Imaging

    • 15208 Posts
    • 2184 Reply Likes
    Yup. Kudos to Doug. This is great help to the product team and the Lightroom community.
    Photo of Dan Tull

    Dan Tull, Employee

    • 172 Posts
    • 38 Reply Likes
    Definitely. Doug handed this one to us on a silver platter, which made it a piece of cake to file and route to be fixed.
    Photo of Tom Hogarty

    Tom Hogarty, Lightroom Product Manager

    • 32 Posts
    • 37 Reply Likes
    Doug,
    As I mentioned in the Lightroom Journal blog post, the team really appreciates your help in tracking down this bug. (http://blogs.adobe.com/lightroomjourn...)

    Regards,
    Tom Hogarty
    Lightroom, Camera Raw and DNG Product Manager
    Photo of Doug Cooper

    Doug Cooper

    • 10 Posts
    • 0 Reply Likes
    Wow - getting kudos from the LR team was totally unexpected, but very nice.

    I have to say, I'm very impressed with the rapid and professional response to my posting from the LR Development team. There's a lot of commercial software out there that has terrible support (even if you can get someone to acknowledge that you might have discovered a bug, good luck getting it actually fixed).

    You guys rock, and have really set the bar for quality customer interaction and support.
    Photo of Jeffrey Tranberry

    Jeffrey Tranberry, Sr. Product Manager, Digital Imaging

    • 15208 Posts
    • 2184 Reply Likes
    Thanks. I have to say I appreciate how well written your problem was and the constructive tone you used - despite encountering a frustrating issue. It's a great example of how this site can be leveraged to achieve successful results.
    Photo of Lee Jay

    Lee Jay

    • 990 Posts
    • 136 Reply Likes
    Hear! hear, Jeffrey!
    Photo of Victoria Bampton - Lightroom Queen

    Victoria Bampton - Lightroom Queen, Champion

    • 4724 Posts
    • 1819 Reply Likes
    Agreed, an excellent example! Shame we can't sticky this thread really!
    Photo of Jeffrey Tranberry

    Jeffrey Tranberry, Sr. Product Manager, Digital Imaging

    • 15211 Posts
    • 2185 Reply Likes
    Official Response
    This issue is addressed in the Lightroom 3.4.1 update and Adobe Camera 6.4.1 update.

    Choose Help>Updates... to install.

    Keeping Photoshop Up-To-Date: http://adobe.ly/PS-Up-To-Date
    Photo of Sarah Valentine

    Sarah Valentine

    • 16 Posts
    • 0 Reply Likes
    Can someone walk me through this??? PLEASE PLEASE...my brain is swimming due to this issue and I have crazy deadlines with holiday portraits. I am right brain now and my left brain is hiding...
    Photo of Victoria Bampton - Lightroom Queen

    Victoria Bampton - Lightroom Queen, Champion

    • 4731 Posts
    • 1823 Reply Likes
    Sarah, this thread is 3 years old, about an old version of Lightroom. Perhaps you'd like to tell us what's going on?
    Photo of Sarah Valentine

    Sarah Valentine

    • 16 Posts
    • 0 Reply Likes
    Thanks for responding Victoria. I was just saying a little prayer. I work daily in Lightroom probably not to the best of my ability yet. I am a full time photog. I have quite a few photos. My main concern is to get back what I had recently. I may by accident moved files around that I should not have. Basically what happened is that I opened my lightroom this room and it asked me questions that I was not awake for and the next thing I know everything is gone!
    Photo of Victoria Bampton - Lightroom Queen

    Victoria Bampton - Lightroom Queen, Champion

    • 4731 Posts
    • 1823 Reply Likes
    I've just found your other thread Sarah. I'll go have a read.
    Photo of Sarah Valentine

    Sarah Valentine

    • 16 Posts
    • 0 Reply Likes
    This morning