Lightroom: When moving many files to a NAS, some files get permanently deleted/lost.

  • 1
  • Problem
  • Updated 6 years ago
  • (Edited)
Recently, I bought a ReadyNAS and a few harddrives to go with it, and decided to move all my 35594 pictures to it.
As Lightroom has some silly features like not being able to move multiple folders unless they have a root folder, I moved most of these files via Explorer and then had to sync them one by one for hand (which is annoying, as there's one folder for every day – however, that's another question). For the rest of the 20 000 photos, I decided to move them with Lightroom.

Big mistake.
Results:
0 missing files from the files I moved with Explorer and resynced.
3049 missing files from the files I moved with Lightroom.

The files simply don't exist anymore, on either of the disks. I've searched every single drive for them completely by file name, date, everything. They're gone.
This has nothing to do with disk failure, or user error. Only the files I moved with Lightroom are affected.

This was running Lightroom 3.5 (updating to 3.6 as we speak), and Windows 7 x64. The external drive that I moved files from is an iomega 1TB one, and the NAS is a ReadyNAS Ultra 4 running X-RAID (RAID-5 derivative) with 4 Ecogreen F4's.
Photo of Stephen Kirk

Stephen Kirk

  • 6 Posts
  • 0 Reply Likes
  • outraged

Posted 6 years ago

  • 1
Photo of Stephen Kirk

Stephen Kirk

  • 6 Posts
  • 0 Reply Likes
What makes me annoyed is that I can't blame the NAS, but only Lightroom – because I've transferred files using FTP, plain Windows Explorer, and AFP (from my MacBook Air) without any problems at all. It's
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
This sounds like a serious problem that might be difficult to duplicate in house to fix.

Is there a pattern as to which files are missing, like a few from each folder that might indicate some mis-reading of a file-operation-completion-status, or many at a time and then everything ok for several folders and then back to a bunch at a time that migth suggestion a transient condition either on your network or in your hardware or a race-condition not being handled correctly by code, that cause the problem, or perhaps files without XMP sidecars or only files with XMP sidecars, or some other condition with the individual files, themselves.

As an aside, is there a reason you didn't use Add Parent Folder in LR to temporarily provide a root folder to move things more in groups?
Photo of Stephen Kirk

Stephen Kirk

  • 6 Posts
  • 0 Reply Likes
It's more of a case of "a few from each folder" – say, every fifth or sixth picture. Alas, the former.

The reason I didn't use "Add Parent Folder" is that I didn't know that feature existed – despite googling for roughly an hour so I didn't have to right click and find all six hundred folders one by one. Alas, I'm glad I didn't find that – as that would've lead to even more lost files, as all files moved with Windows Explorer transferred successfully.

What saddens me is that this was the step before the backup – I just needed to sort out the files first. I wouldn't had mind if this occurred some other day, as I'd have backups in that case.

All files that are lost are virtually identical in terms of metadata et cetera.

The ReadyNAS is plugged directly into the computer with a gigabit network cable. My best guess would be that it means that Lightroom cannot handle transfers over network decently. I've done internal transfers of lots of pictures without any of these losses, but I've seen a thread on the Adobe forums of someone having a similar problem.
Photo of Stephen Kirk

Stephen Kirk

  • 6 Posts
  • 0 Reply Likes
An example from an album: From the first 50 pictures, these seem to have been lost:
2, 6, 10, 15, 17, 22, 26, 30, 34, 39, 43, 47.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
A few from each folder seems like something easy to repeat.

I usually would use another network device plugged into a switch or router of some sort, not directly wired between, but maybe that's how your ReadyNAS is designed to work, directly cabled. Do the missing ones continue through the entire set of folders?

I wonder if there is a connection limit on the ReadyNAS and LR isn't checking an error status, somewhere, and was assuming the copy continued when the socket was closed due to some being overwhelmed with too many simultaneous copies occuring at the same time, or did you wait for one folder to finish before you started another? If you did not, perhaps using the Add Parent Folder method would have worked since LR would only be moving one file at a time, insetad of many simultaneous copies.
Photo of Stephen Kirk

Stephen Kirk

  • 6 Posts
  • 0 Reply Likes
Usually the ReadyNAS is supposed to be plugged into a router or switch, but I'm forced to connect it straight to my PC for now due to facility reasons. The missing pictures are in all folders transferred using Lightroom, but not those I've transferred with plain Explorer.

Some of the folders were in a root folder (a "2011" folder), and these are the ones that I moved using Lightroom. Therefore, it should've only moved one file at a time as usual (which it appeared to do, as I saw the file counter increment and decrement by one, not several).

My best guess would be that the "write success" check is broken when transferring over network.
http://forums.adobe.com/thread/871689...

From now on, I'll make sure to transfer using the OS instead... :(
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
It would probably also be helpful, assuming the problem is unknown to Adobe and otherwise not fixed in 3.6, if you could reproduce it, somehow.

If you can reproduce it, then varying one aspect and trying again, will help narrow down some of the necessary conditions. For example, if you move from a folder on your computer's internal HD to the NAS does it still happen, which takes the external drive out of the picture, and a variation would be to move from the external drive to the internal drive, to remove the NAS from the picture to see if it still happens. Moving the files in the opposite direction would also help isolate if it is a reading problem or a writing problem on one of the devices.

The more that you can keep the computer the same the better if there is any interest in helping Adobe determine what is happening as well as your feeling comfortable that you may have fixed something so it won't happen, again.