LightRoom 4.4 Moving files to NAS using Library module permanently lost ~9% of files

  • 1
  • Problem
  • Updated 4 years ago
  • (Edited)
Decided to move a large collection of RAWs to my NAS (synology diskstation 1513+ on shared windows drive) for better redundancy and backups using the Library module to drag to the other disk (and save time relinking files in the catalog, etc!)

After letting LR chug along and finish w/o any error notification, I start noticing that a number of photos in my collections have started complaining that it can't find random files. Some photos were ones I recently touched and they shouldn't be missing, so searched and was horrified when it failed to find the file by name on the entire computer, every drive, every partition.

Ran the "Find missing photos" feature, and it lists 1212 files that are missing. I moved moved roughly 13k photos, so there's almost a 9% drop rate. Went to the file system, the original source folders are gone as expected. Went to the target drive, found MOST of the photos, in exactly the original file structure, just like you would expect.

But upon closer inspection I'm horrified to see random NEF files are missing. _DSC_1234.NEF is missing, but _DSC_1234.XMP exists! LR managed to move over the sidecar file, but not the critical RAW. Often surrounding files like, _DSC_1233.NEF/XMP and others are unaffected. They seem to cluster somewhat around certain folders, which leads me to suspect it's choking on an access lock mid-copy, but I have no proof.

I don't know why this has happened, I was using LR during the move to work, but not going anywhere near many of the files that were lost (moved 2013 files, was working on 2014 files, briefly touching some smart collections that might have included some 2013 files).

Right now, I've used R-studio to do hardcore file recovery on the drive and am slowly recovering the fragments of deleted NEFs still left on the disk. I've managed to recover a few already, and have 16k left to sift through. But already I've verified that within the hundreds of GBs of recovered files, some RAWs of my deceased father are lost forever. A majority thankfully survived the move, but some were irreplaceable, and I'm pretty bummed about it.

I doubt there's anything anyone can do to help me find files already wiped from disk. I get you guys probably have to build to multiple platforms and have to rely on very OS-specific quirks that make things hard. Software is hard.

Maybe LR on the back end relies on a file system OS move command that errors for whatever reason (random file lock due to transient access by LR or something else, network hiccup to the NAS-on-shared-drive, I don't know) but it chugs happily along. I really wish it would do a much more atomic and safe, copy/verify with hash (or even simple file existence)/then delete instead of whatever it does now.

But I really hope that the devs can look back into this issue and implement a more robust copy/move algo. This loss hurts quite a bit, but other users probably stand to lose more than I did. If there's any more info I can provide to help, please contact me.

Lightroom version: 4.4 [891433]
Operating system: Windows Vista Service Pack 2 (Build 6002)
Version: 6.0 [6002]
Application architecture: x64
System architecture: x64
Photo of Randy Au

Randy Au

  • 2 Posts
  • 0 Reply Likes
  • sad

Posted 4 years ago

  • 1
Photo of Benjamin Warde

Benjamin Warde, Employee

  • 483 Posts
  • 74 Reply Likes
Hi Randy,

Yikes, that sounds nasty (and you sound remarkably calm and patient, given the seriousness of the situation). I have not heard of this issue previously, if anyone else has experienced this problem, please add your info to this thread.

The Lightroom team has been made aware of your post, and may be able to determine if a Lightroom bug is responsible. In the meantime, based on your description, it sounds like your photos were not backed up anywhere? Assuming that's the case, I have a method which might be able to (sort of) recover some of the photos anyway. It's somewhat involved, so I'll email you directly.

Thanks,
Ben
Photo of Benjamin Warde

Benjamin Warde, Employee

  • 483 Posts
  • 74 Reply Likes
Hi Randy,

Actually, I just realized that the lengthy instructions I was going to email you are already posted on our website:

http://helpx.adobe.com/lightroom/kb/e...

This will not recover missing raw files. But it may recover JPEG versions of many of your photos, in some cases possibly even full resolution JPEGs. So, it definitely falls into the "better than nothing" category. Let me know if this helps at all, or if you have any questions about using the script.

Thanks,
Ben
Photo of Randy Au

Randy Au

  • 2 Posts
  • 0 Reply Likes
Thanks for the info, I didn't know that there was a tool to recover JPEG previews, and it's certainly better than nothing when I exhaust my other options with the data recovery. Went through more recovered files today and found a few more precious keepers hiding in the pile, will probably be a few more days of patient work.

My situation sorta sucks, and I can't say I'm exactly happy about it. But I know this is the sort of bug that no one intentionally ignores, and throwing a fit isn't going to bring back overwritten bits on disk.

Googling around, variants of the bug have been reported years ago but it couldn't be reproduced to be fixed. Originally I was just going to recover my data and move on with life, but figured since it happened to me, it will happen to other people and you guys should hear about it from someone who's calmed down =).

I doubt I can reproduce the bug on demand myself but the other relevant details I can provide is that I have a slowish HDD (7400rpm 2TB seagate I think, but it could even be 5400), and I moved a ton of files, then touched collections +smart collections that reference files being moved as well as non-moving ones. My desktop system is also pretty old, 4-core q6600 w/ 8gb of ram and a very weak video card. The HDDs are fairly new, within about 2 years and generally healthy. The NAS is less than a year old and works fine over gig-e.

Another symptom I notice that might be related is when I work on a batch of photos, mark a bunch to be rejected. then hit ctrl+backspace and hit "delete files on disk." occasionally the first photo in the rejected set (when it switches over) will flash a message like "could not locate file, or is offline", and fail to delete the first 1 or 2 photos while the rest delete. It almost is like it was busy reading the NEF to generate the preview and locked itself out from deleting the photo for a moment. I certainly press the delete option faster than it can load everything, so maybe there's some kind of race condition happening.

For now, I'll go back to moving my files around manually via Explorer where I've got Teracopy installed and is pretty good about verifying files w/ CRC before a move is fully completed, all while being pretty speedy. Hopefully you guys can do something similar to what they do in future versions.
Photo of Benjamin Warde

Benjamin Warde, Employee

  • 483 Posts
  • 74 Reply Likes
Hi Randy,

Thanks for the additional info. One note: best to run the preview extraction script as soon as possible. The scrip will extract the largest preview that exists for each photo. Lightroom keeps smaller previews around indefinitely, but by default 1:1 (i.e., full resolution) previews are deleted after 30 days. So, you are potentially losing some full res previews each day that you wait to run the script.

Thanks,
Ben
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4427 Posts
  • 1638 Reply Likes
For an easier option than the script, you might also like to take a look at this plugin. http://regex.info/blog/lightroom-good.... There's also a link on that page to another which offers slightly different functionality, so there's plenty of choice (although we'd much rather you didn't have to use them!)
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 384 Reply Likes
*** Adobe's script does not assign proper icc-profile, nor transfer any metadata, so your recovered previews will be off color-wise, and be completely devoid of metadata. For that reason (and others), consider using one of the other more robust plugins instead.