Lightroom: Photo integrity check

  • 19
  • Idea
  • Updated 2 years ago
  • (Edited)
Lightroom currently has the possibility to search for missing photos, which is a very important function. Unfortunately this does not warn about the case where some photos are still present, but corrupted. Given the large amount of data stored on most computers, and the long storage time, this corruption is bound to happen sooner or later (such examples show up regularly on forums). The user therefore needs to be warned as early as possible so that corrective action can be taken (e.g. retrieve from backup) before it’s too late.

Proposition: implement an automatic photo integrity check.

One way of implementing this would be to calculate the checksum of photos on import for all file types (I understand this is already done for DNG s). The verification against checksum could be done either during the regular catalogue optimisation, or at the user’s demand in a way similar to the command ‘Library’>’Find Missing Photos’. This could be then saved into a temporary collection for review by the user.

Lightroom, make me feel confident that my photo collection is sound and safe.
Photo of quid quomodo

quid quomodo

  • 8 Posts
  • 2 Reply Likes
  • hoping to know that my photos are in safe hands

Posted 6 years ago

  • 19
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
You pretty much have this if you force recreation of all your previews.
Photo of quid quomodo

quid quomodo

  • 8 Posts
  • 2 Reply Likes
Could you please explain how this works? If one forces recreation of all the previews, does Lightroom – at the end of the process – provide a list of photos which are corrupted?

In this case it may be doable for a one-off check. But as Rob says, it is not practical to rebuild previews weekly or even monthly for a 100,000 photo collection.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
It would be good not to have to re-render all photos from scratch, just for a quick integrity check.
Photo of quid quomodo

quid quomodo

  • 8 Posts
  • 2 Reply Likes
Hal P Anderson has recently highlighted the following link:
http://dpbestflow.org/data-validation...
It is possible to run the free DNG converter on folders containing raw files to check whether the conversion (and therefore the raw files health) is OK.

Pros: it's currently the only way to check a large number of raw files.
Cons: It can take a long time, requires disk space for the second copies, and is limited to raw files (no jpg or video).
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
What are the advantages of this over recreation of previews?
Photo of quid quomodo

quid quomodo

  • 8 Posts
  • 2 Reply Likes
If the recreation of previews provides a list of corrupted files, then the advantage of this method is: none. But I don't know whether the preview regeneration will provide such a list.

If the recreation of previews doesn't provide a list of corrupted files, and instead requires manual visual inspection of each photo, then the advantage of this method is a lot of time saved.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
As it stands, *if* Lightroom has to render a photo from raw data, it will pin a corruption badge to it, but some (most/all) of us would probably prefer to know about this immediately, before happening upon the image down the road... I would think that comprehensive detection (e.g. when backing up catalog) would be a good idea, as well as putting corrupt files detected in a collection and notifying user of the problem when encountered during normal use.

This reply was created from a merged topic originally titled
Lightroom: better detection and handling of corrupt photo files.
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
Really needs this
Backing up the database isn't just enough, we need to trust our photos backup as well.
First we need to compute a rawimagedigest for all the photo types
It would be nice to store it into the metadata of all the photos. But storing it in the database only would already be a great step forward.
Then we need to be able (as Rob said) to run a check and create a collection of all the corrupted images.
That is the minimum.
The bonus would be to provide a ui to recover these images from a back up.
-Select image to recover
-Point to the backup folder
-Lightroom scan the folder and finds the backed up images
-Lightroom check the rawimagedigest of these back ups
-If rawimagedigest is Ok lightroom copy the back ups, to the original image location, and replace the original images. (LR would handle image name modification thanks to its database)
-Lightroom update the copied image with the metadata stored in its database.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Y'all may be interested in the 'Raw File Integrity Checker' plugin (free, written by me):

http://www.robcole.com/Rob/ProductsAn...
Photo of Benjamin Espe

Benjamin Espe

  • 2 Posts
  • 0 Reply Likes
Does anyone know if Adobe is working on this? You guys have already mentioned some good points why and how to care about image integrity and if the filesystem of my operating system doesn't care of it, I wish Lightroom does. Lightroom is perfect to take care of the integrity of my pictures and I'd be glad to here that they are working on it.

Any insider knowledge?
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Adobe sometimes labels stuff "under consideration" or "in progress", but only sometimes.. (they could surprise us by implementing something not so labeled).

Non-Adobe insiders are sworn to secrecy, so unless Adobe chooses to enlighten, we stay in the dark..

My guess: Adobe won't implement this feature. If you want integrity checking, convert to DNG! (in other words, it's on a relatively short list of advantages of DNG over other formats, so maybe best for DNG not to eliminate this perk). It would be really easy to implement, yet they haven't, so.. Note: in the beginning, non-raw DNGs did not enjoy integrity checking, now (@Lr5 I think): all DNGs do - when that change was made would have been a perfect opportunity to include other file-types, but they didn't...

Of course there is the "Raw (and non-raw) File Integrity Checker" plugin, but there is a draw-back in having such info handled in proprietary fashion - if the plugin (or equivalent) doesn't work 10 years from now, then the integrity check info has become useless. And anyway it's limited - check codes are over whole file, not just static data (like image data), so saving xmp internally kills the integrity checking.

Rob
Photo of Benjamin Espe

Benjamin Espe

  • 2 Posts
  • 0 Reply Likes
Ohh man.. this topic is really annoying. Of course I can change everything to DNG, would be far more convenient if my camera supported it initially..

Probably Adobe is just not interested in this topic. They want to push their DNG format. It would be great if Adobe would support correcting errors on DNGs as well.
Photo of Jacques-Jean Tiziou

Jacques-Jean Tiziou

  • 29 Posts
  • 8 Reply Likes
Yep, this sucks.

There should be a built-in way to easily:

– verify selected sets of folders of images (without having to export copies of them)

– verify the entire archive at a given time

– enable rolling background verification so that the entire archive is verified periodically

– access an archive verification report, that would tell us something like "All images are verified as completely readable as of X date"

– get warnings about corrupted files, and see them in a separate collection with a nice interface for retrieving batches of images from backups. (Aka if twenty files are corrupted, and I point LR to a source to retrieve a backup, the whole batch of images are matched and copied in place of the corrupted files. And of course verified again.)

Right now it sucks to run into this kind of thing, because it invariably happens when you're actually looking to access the files:

Photo of Bryn Forbes

Bryn Forbes

  • 157 Posts
  • 21 Reply Likes
Having imported a batch of files with intermittent corruptions, only some of the corrupt ones were "unable to build preview" so maintaining a checksum to see if the raw files has changed at all is necessary to find corrupt files, as well as providing a way to help find missing files which is the number one problem in lightroom students I have taught. In terms of dollars I know students who get frustrated with missing files and stop using lightroom / cancel subscription (often just because drive letters have changed, or they accidentally reimported the folders and they got moved). Having the checksum stored would allow a possible future feature of "looking for my missing files anywhere in this folder tree"
Photo of Linwood Ferguson

Linwood Ferguson

  • 23 Posts
  • 17 Reply Likes
Someone pointed me to this suggestion while I was ranting in another forum about the requirement to use DNG in order to get integrity checking in lightroom (and even then it's only half-way checking since it checks only image not metadata portions of the file).

With photographers with home-quality computers accumulating multiple terabytes of data, bit rot is a real and growing concern. 

The idea of trying to render an image to see if it is corrupt is flawed -- major corruption can occur in an image and it will still render successfully (meaning no error message).  It may be totally hosed visually of course, but no human can routinely look at 100,000 images to see if any are corrupt.

The answer really lies in integrity checking file systems like zfs, btrfs or refs if Microsoft finishes it, but that is a long way from most photographers' systems. 

Adobe could VERY simply handle this - pull a checksum of raw images on import, stash it with the image data in the catalog, not the image.  Periodically let a run check that the checksum remains unchanged, and collect the images failing for review.

It's pretty fast, easy, and solves the problem without requiring people convert to DNG.  For those not writing metadata to DNG, it also provides a more complete check of DNG files than the DNG checksum does.

This is not rocket science.  It's pretty easy.  I see one posting above hinting at a solution, but the link is broken.  I wrote my own if anyone is interested (https://lrvalidate.codeplex.com/ ), free, all source code provided.  But people will be reluctant (rightly so) to use anything that writes to the catalog not from Adobe.

Adobe could do this in a few man days' effort, and get in front of the encroaching epidemic of people discovering "I had not looked at my files in 5 years and now some of them are unusuable what happened?!?!" complaints.

Bit rot.  If you don't know the name now, you will eventually.  :(