Lightroom: More than 12 Hours to Backup catalog with 58K images

  • 1
  • Problem
  • Updated 6 years ago
  • Solved
  • (Edited)
In W7 64bit with 32GB & i-7 2700K, LR4.1, LR catalog of 58K images stored on separate internal SATA3 2TB drive with about 380GB free, I started weekly catalog backup up 7/29 @ 9:10pm with no other tasks running except usual background stuff. 12 hours later job was only approximately 2/3's done (comparing current backup size vs. last week's size). Checking Process Explorer I find LR is only using a constant 12.5% of the available power! Shouldn't things go faster? Shouldn't LR be smart enough to grab more resources? What might cause this kind of unacceptable slowness?

Just copying the complete catalog manually is so much faster that I wonder if that isn't the better option? For reconstruction purposes, what extras value does using the LR backup get me that I don't get via the manual route?

TIA
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes

Posted 6 years ago

  • 1
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 134 Reply Likes
A normal backup of my 130K catalog takes just a couple of minutes, so something is wrong. Did you have it do a check first?
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
Yeah, something is not right in terms of hardware. You have a failing disk or disk adapter, or one of the cables is bad or getting a lot of interference.
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes
Failing hardware is very non-specific and improbable any way -- whole rig is newish (3 months) and there are no problems in other apps. For example, simple copy uses same hardware so you would think any problem with hardware would show up there as well as in LR, yet it doesn't.

However, the newest thing is an Intel 60GB SSD rigged for disk caching since my boot drive occupies 180GB. I wonder if the SSD could be interfering? I'll disable it and give things a try. If that doesn't help, then the culprit smells like the software.

BTW, what about the question of simple copy vs. LR backup? Is there any benefit to using LR backup vs. a simple backup copy?
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes
PS: My initial post is easy to misread -- my catalog isn't 58K big, it has 58000 images. The catalog itself is 28.4GB comprising 64K+ files. Hmm... That's BIG. But last weeks catalog was about the same size, maybe several thousand smaller, and yet it only took about 8 hours to complete.

The 2700K has 4 cores and each is hyperthreaded (I think), so that's 8 virtual cores. That means LR is only using 1 virtual core (1/8 = 12.5%). But maybe that's all it takes to completely flood the SATA channel, that is, more processor power can't push any more data down the SATA channel if its already full. Still the question is why did 100% of the job get done last week in 8 hours, while this week after 12 hours only about 70% of the job was done?

I've canned nearly 4K images so far, and I'll try again tonight with the SSD disabled to see how it goes.
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 134 Reply Likes
My catalogs contain over 100,000 images and occupy over 50GB including previews and take just a couple of minutes to backup. You are talking about the Lightroom catalog backup function, correct?
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes
"You are talking about the Lightroom catalog backup function, correct?"

Yah, that's it. Last night after deleting about 4K more images and correcting a minor mismatch between my catalog structure and actual file layout, I completed an LR catalog backup in less than 2 hours. I don't know how much of the difference in completion time is due to the file anomaly vs. the amount of free space on my hard-drive, although I suspect the latter. In any case, I'm back to my normal backup times. Note that the SSD had no apparent role in the problem.

Still, only a "couple of minutes to backup" 100K images is extraordinary by my standards. Are you adding many new images between each backup? Or is your catalog relatively static? If you're not adding much, maybe the backup function is using essentially incremental backup. In my case, I'm adding at least a thousand images each week, including both new captures and PS renditions of existing captures, plus I'm deleting old images to keep my total space down, which means basically LR backup has to redo the whole thing each time I backup the catalog.

Also, how big is your drive space? Are you RAIDing? I'm going to move up to a 3TB drive -- 4TB is not cost-effective yet, and RAID is too gimmicky for me.

Basically I haven't found much info on how LR catalog backup works or what factors affect its operational speed, and apparently no one else has a handle on it either. The alternative, of course, is regular incremental backup from the system level. My images and catalog are all and exclusively on a separate drive, which makes this feasible.
Photo of John Spacey

John Spacey

  • 101 Posts
  • 20 Reply Likes
Makes sure you have plenty of space on your C:\ OS drive - Lightroom uses this for temp files while backing up.
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes
John,

At last! Someone who actually knows what's going on. Thank you. That's exactly what I suspected when I got to deleting all those old images. I moved over to a 3TB drive last night and I'm hoping that will solve the problem for at least a half-year :-) Since I shoot Canon rather than Nikon there's really no imminent likelihood my RAW files will double in size, and there's no great new space grabber, like shooting EDOF or HDR (which I've already compensated for).

Bruce
Photo of Dan Santillo

Dan Santillo

  • 8 Posts
  • 0 Reply Likes
The suggestion to check your cables isn't so bad. I'm forever experiencing slow down (not just with LR, with Windows generally) and then my working drive will become unusable. I solve it every time by powering down, fiddling with SATA cables and powering back up - it isn't just me who has this problem with SATA cables. I've even tried the cables that clip into place, but no luck. Really must put a dab of superglue on them as last time LR couldn't restart until I deleted the preview cache...

My catalogue with 70,000 photos backs up in around a minute.
Photo of Bruce Cyr

Bruce Cyr

  • 22 Posts
  • 0 Reply Likes
" fiddling with SATA cables and powering back up"

Dan,

Thanks, but I've never had a problem with SATA cables. OTOH, USB ports in Windows are a constant source of frustration -- they're here, they're gone, I never know when one is going to go into hiding. This has been constant over 3 generations of Windows, half a dozen mobos, and diligent updating of drivers.

Bruce