Lightroom 3 RAW files appear corrupted, but not when opened in Picasa

  • 1
  • Problem
  • Updated 7 years ago
  • Solved
when importing RAW files from Canon 7D I get very strange "psychedelic bar code" over much of my photo. The file looks fine on camera, and when viewed via Picasa. What is Lightroom doing to my files? I'm sorry I can't describe this better.
Photo of Barrie Raik

Barrie Raik

  • 6 Posts
  • 1 Reply Like

Posted 7 years ago

  • 1
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
Have you installed all the updates (and bug fixes) for Lightroom?

Yes, that looks like the result of a corrupt file, so unless you're seeing an old, unfixed bug - I'm not sure how Picasa could open the file without corruption.
Photo of Barrie Raik

Barrie Raik

  • 6 Posts
  • 1 Reply Like
Lightroom says it is up to date version 3.4.1
Of 63 photos I took all but about 10 have this problem, always on the right side.
Where do you think the corruption is taking place? in the camera? on the CF card? or in Lightroom?
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
Most likely on the CF card, or when reading from the CF card into the computer (I've had plenty of bad cards and bad readers).
Photo of Barrie Raik

Barrie Raik

  • 6 Posts
  • 1 Reply Like
thank you Chris. I just used a free CR2 viewer, and the files are not corrupted when read from the card, but are corrupted on my HD. Therefore, the culprit is the card reader. I hope this is true.
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 135 Reply Likes
Doesn't Picasa only show the embedded JPEG, and not the raw data?
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
Well, if that's the case - yeah, that would explain it, too.
Photo of Andrew Rodney

Andrew Rodney

  • 509 Posts
  • 90 Reply Likes
They (Picasa) indicate they have raw support for some cameras. That it looks fine on the camera would further indicate the raw is toasted and the embedded JPEG or secondary JPEG is OK.

Don’t format that card, it may be possible to copy the file off directly, then open and get a non corrupted file. Or maybe not.
Photo of Barrie Raik

Barrie Raik

  • 6 Posts
  • 1 Reply Like
I bought a new card reader.
The CF card is able to be read and copied to my hard drive with the new reader.
The RAW files look fine in Lightroom now.
I think the problem was that the files got corrupted when being read and copied by my old reader. However, just in case, I got a new CF card too.
Thanks everyone for your help.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I think it would be good for Lightroom to be able to do validity checking upon import, and afterward too. IMNSHO, this should have been detectable and detected by software, not the user. I don't know how much Lightroom could do if it's just bogus raw data being read the same way each time, but whatever they can do, they should do in this realm... (are there no data check codes in these raw files?, is the faulty data read the same way each time?...)

To quote Chris Cox: "I've had plenty of bad cards and bad readers"

Summary: I don't consider this problem "Solved" until it won't happen again in the future. Put another way: Barrie Raik's new reader doth not a Lightroom solution make.

-R
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
There are no data check codes (per se) in these files, no.

There is only so much an application can do if the underlying device and file APIs don't report a problem.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Adobe should be able to apply similar validity checking whether converting or not.

If there are too many advantages reserved for the DNG format, some people will switch raw converters, instead of file formats.
Photo of Andrew Rodney

Andrew Rodney

  • 509 Posts
  • 90 Reply Likes
>I don't consider this problem "Solved" until it won't happen again in the future.

Its the reader. The problem is solved and yet, can never be solved in all cases into the future. Data can get corrupted or lost at any time.

I had a similar issue using a PCMCIA slot to copy files into LR on a laptop. I was on location, also copied them to a Sanho HyperDrive. The files on the Hypderdrive looked fine, the files on the Macbook looked like the one above. That’s how I knew the PCMCIA slot was funky. NEVER format your camera card until you verify the data (converting to DNG is a help there, having LR build high quality previews is another). Then back that up.

Expecting LR to continuously check the file integrity is off the charts. If you have ONE copy of a file, that file can get corrupted at any time due to any number of drive or other issues. Ensure the data is sound, make a couple back ups.

You can also get a product like ImageVerifier. Once you run that on a pile of images, you’ll see how incredible time intensive the process is. Its something you can’t have on going in LR, the product would screech to a halt. Verify on import, examine files, backup. Simple.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I'm not suggesting Adobe prevent corruption, I'm suggesting they take steps to detect it when it happens, which may not be as onerous as you think. It would make sense to do it upon import, before offering the opportunity to delete from card. Then all the information would be computed and stored for subsequent checking, maybe as an option for catalog backup / optimization / integrity check.
Photo of Andrew Rodney

Andrew Rodney

  • 509 Posts
  • 90 Reply Likes
They can’t prevent it, they can detect it and can be a darn slow process (play with ImageVerifier on a few hundred raws and don’t hold your breath). If you convert to DNG, you’ll get some degree of detection although not to the level of ImageVerifier (Hash Checking, hence its speed). Same in terms of having LR build 1:1 previews (its got to get that data from the raw). Again, that’s a far faster and useful process but a speed hit. So if you ask for either DNG conversion or larger previews, if the raw is really in bad shape, you’ll get an error long before you format the card.

The problem is thus solved.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I think what Adobe meant by "Solved" is the problem was not due to a bug in Lightroom, but due to a faulty card reader. From that point of view, I agree - its solved. My reason for saying "perhaps one shouldn't be so hasty to dismiss" is that there is still a missing piece in this puzzle - its still possible for detectable corruption to go undetected, and even get backed up. That's the hole I'd like to see plugged. That goes for both the catalog, and the images in it.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Image validation using ImageVerifier, when check-codes are available is .2 seconds per image. So, it would take less than an hour to validate every image in my catalog. I wouldn't necessarily want to do this every time I back-up/optimize my catalog, but sometimes - yes. (additional structure checking adds about a second per image, so would require a few hours more to complete).

Summary: I think to whatever extent possible Lightroom should detect and report bad or suspicious image data for all file formats. Then when data deemed valid (perhaps conditioned by user approval), check-codes should be stored for future validation, to be done along with other integrity checks, e.g. as option when backing up / optimizing / catalog-integrity-checking / other problem checking...

Adobe - do you still read these threads after marked "Solved"?
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
Yes, we keep reading as long as you keep updating (partly to filter out the SPAM :-).
Photo of Andrew Rodney

Andrew Rodney

  • 566 Posts
  • 93 Reply Likes
>I think to whatever extent possible Lightroom should detect and report bad or suspicious image data for all file formats.

It already does.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Lightroom does some incidental checking, but it could have more robust data validation mechanisms.
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
How much slower do you want every image to process in order to handle that validation?
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
It wouldn't matter to me, as long as the initial image was not being withheld until completion. Obviously, 99.9X% of the time the images are fine. So, having it done in a lazy fashion after import would be fine. I'd just leave my card in the slot until Lightroom said all images have passed scrutiny, at which point, if they all looked good to me too, I'd approve they be deleted from card, and check-codes could be stored for future validation...
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 813 Reply Likes
The validation has to be done as the file is read in, otherwise you have to re-read it to validate (totally not worth the hassle).
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I can imagine two kinds of problems:
1. Reader not consistent.
2. Reader consistent.

I agree with you regarding type 1 - you need to re-read to see if same the second time. But, I assume there are also type 2 problems (correct me if I'm wrong). In which case, one must resort to sherlock holmes-ing...

Anyway, I'd be happy to take responsibility for image certification - that's why I suggested the option to save check-codes after locking/finalizing, since that's when the user has for sure given his or her stamp of approval. Don't get me wong, I could also continue to argue for import validation ala software - to the extent that its possible, it would make a very nice complement to post-import checking, its just that there are some things humans are better at than computers - this may be one of them. So, to me, its the ongoing pre-backup validation that is the piece I miss the most.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
If one experiments for a while one will see that much corrupted image data escapes Lightroom scrutiny that could be caught with a little more attention given to it.
Photo of Barrie Raik

Barrie Raik

  • 6 Posts
  • 1 Reply Like
I am amazed that my problem has generated all this discussion. I just wanted to add that for someone with limited knowledge about how LR works it really would have been helpful for an error message to appear. I didn't even know enough to know how to investigate the problem - the term "corrupted" didn't occur to me until this week.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I'm hearin' ya. Although its harder to tell whether a raw image is corrupt straight from the card, than it is to tell if its been corrupted later, it is "overly generous" to claim Lightroom is already doing all it can in this regard, in my opinion.
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
This is the real issue. In order for an error message to be shown, an application has to know that there is an error condition. In this case the entire OS assured the application that all was well. It was only when Lightroom actually created its own preview that the damage was made obvious.

There was no indication that anything was wrong, and no problems were raised by the parts of the system that are /supposed/ to be looking out for this stuff.

As others have pointed out, there are some ways an application can determine if some sorts of problems are likely. But there is no way to determine many common problems during import; at least, not without the additional cost of more time per photo when importing.

At most, there are a set of problems that can be looked for, there are a set of problems that have a very high cost to look for, and another set of problems that are, for all intents and purposes, completely impossible to determine. At least, for a user application on the two platforms we are interested in.

Personal computers and their peripheral systems are just not designed with this sort of thing in mind beyond the basics. There is a tremendous cost associated with serious data integrity that few are willing to pay for.

I mean, has anyone every looked at the USB spec, and the liberties manufacturers take with it? I'm amazed any of it works.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
After further consideration, the ability to assess corrupt images upon import is hardly necessary - the naked eye does a fine job of identifying them. I mean, it didn't take long to identify Barrie's reader problem - the corrupted images nearly always have a "signature" look to them. Barrie (and many others like him) found out promptly by consulting the forum... What would be more important to me is the ability to save check codes for future checking, as part of the backup process. They could be saved even as an option when finalizing/locking, for example.

Summary: The bigger problem, for me, than detection of corrupt images upon import, is not detecting corrupt images ever. I hate it when 10 years down the road I go to fetch an image and only the top half comes in... And of course, at that point, all of the backups have the same problem. (Yes, I know that a backup strategy that doesn't always overwrite can also remedy this, but I don't do this for photos, nor do a lot of other people, and even if one does do this, it can be a "challenge" to go back and find the most recent non-corrupt copy).

I acknowledge that this would not prevent corruption from happening, but it would prevent corrupted images from being backed up, and allow one to recover immediately.

This kind of corruption detection is super-easy to implement, and is something that should just be done, in my opinion - hardly worth talking about. I don't know enough about raw image data to know how hard it would be to toss in a little "suspicious data pattern" checking upon import, but I consider that to be of lesser importance.

For those who don't know: DNG files do have these check codes embedded in the file, however they are not checked by Lightroom except when the DNG file is revisited. So this proposal is to save the same check codes for non-DNG files too, in the catalog, and check them as part of the robustened catalog / photo integrity checking invokable on demand and/or as part of the backup options.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
See related idea about robustened photo problem detection: http://feedback.photoshop.com/photosh...

And another idea for robustened catalog integrity checking:
http://feedback.photoshop.com/photosh...
Photo of Andrew Rodney

Andrew Rodney

  • 566 Posts
  • 93 Reply Likes
>After further consideration, the ability to assess corrupt images upon import is hardly necessary - the naked eye does a fine job of identifying them.

Sure it is. That’s the time to fix the issue if possible and the time to avoid formatting the card to continue shooting. Its after you know the DNG is solid, you’ve built high quality thumbs, that you back up like a madman. THEN format the card.

>This kind of corruption detection is super-easy to implement.

It would be quite easy to import a fixed way and never know the image was corrupted. Then, ignorance being bliss, format the card only to find later the data you imported without proper checks (convert to DNG, build high quality thumbs, neither of which are a requirement) that your data is hosed.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
OK - let's have integrity checking upon import as well - sounds good to me...
Photo of Andrew Rodney

Andrew Rodney

  • 566 Posts
  • 93 Reply Likes
>OK - let's have integrity checking upon import as well - sounds good to me...

We already do!
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Adobe stopped a little short, in my opinion.