Lightroom: Library Mode Performance - Loading Images

  • 2
  • Problem
  • Updated 4 years ago
  • (Edited)
I apologize if this is covered by someone else but I haven't found a similar issue with a specific resolution.

My primary issue is that, seemingly in the past few months maybe longer, I'm unable to quickly review images in Library mode. I knew that LR 3.4 had performance issues, some related to this area, so I've been patient. I just tried the 3.5 RC and the situation is the same. So now I'm at wits end.

My understanding is that when I'm in Library mode I'm dealing with the Previews. My Previews are set for 2048 pixels and my review window size is around 1700x1200 pixels, so I'm under that size even with this 30" monitor. And yet, when I walk through images to review them they will work fine for a bit and then come up completely pixelated and the "Loading..." badge pops up for 2 to 5(!) seconds while it does whatever the heck it is doing.

These are RENDERED PREVIEWS coming off my 7200RPM hard drive and they take about as much time as the Develop module takes sometimes. I can't figure this out for the life of me. Until, perhaps 3.4, this was always a benchmark of Lightroom for me - I could easily review and rate photographs and then go back and further tweak the ones I liked best. Now it's nearly useless as I hit a brick wall reviewing images.

I'm reviewing Canon raw images, typically from 5DMk2 and 7D, but I'm seeing similar issues with older cameras and, quite frankly, that shouldn't matter because I'm looking at rendered previews off the internal drive, right?

With LR3.5 apparently not resolving this issue I'm stuck. I've read that deleting and rebuilding the catalog's previews might be useful. Any evidence this is a real fix -- I'll burn a few days worth of time doing this if it helps but I'm not interested in wasting my time on wild guesses (although since I can't really get any real work done anyways, I may be left with no other alternatives).

Are there any diagnostics, logging modes or ANYTHING I can do that could help pin this down?

Many thanks,

- dave

A few specifics:

MacPro, 4 core (2x2.66GHz processors)
9GB of RAM (LR taking less that 2 GB of it at the moment)
7200RPM drive where the catalog/previews live (internal system drive)
7200RPM drive where the 50GB raw cache lives (internal drive, not system)
5000RPM drive where the photo images live (external drive, firewire 800)
Catalog has 100K+ images.
Been using Lightroom since V2. This catalog was rebuilt when V3 came out (normal upgrade process if I recall).
Catalog optimized regularly.
Photo of David Griffin

David Griffin

  • 6 Posts
  • 0 Reply Likes
  • frustrated

Posted 7 years ago

  • 2
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4427 Posts
  • 1637 Reply Likes
When you upgraded from v2, did you rebuild the previews file manually? If not, I'd give that a shot.

Find your catalog and rename the Previews.lrdata file to something else (renaming rather than deleting so you can put it back if this idea doesn't work!). Then open LR, which will be lacking previews, select photos and go to Library menu > Previews > Render Standard Previews.

See if it's any further once it's done. If it's not, just delete the new previews and rename the old one back again, but if it is faster, then I'd set it re-rendering all of the previews over the next few nights. The preview mechanism was completely rebuilt in 3.x and in the early days we saw a lot of speed issues that were solved by rebuilding the previews.
Photo of David Griffin

David Griffin

  • 6 Posts
  • 0 Reply Likes
Thank you Victoria. I don't recall rebuild the previews myself -- just whatever the V2 to V3 process did and following whatever instructions were hanging around (I know the upgrade process made a new catalog database).

I will try the "test" of rebuilding previews to see if that helps. I would like to mention that I spent many happy months with Lightroom 3 before this started happening -- it definitely raised its head in the past few point releases (although I admit there are plenty of other variables at work...)
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4427 Posts
  • 1637 Reply Likes
The upgrade process 'borrows' the old previews file, and it seems there were some leftover gremlins in the previews upgrade.

You're probably right, there may be something else going on if you were ok until recently, but it's a good one to rule out.

If that doesn't do the trick, next in line would be to try a clean test catalog and just import some of the problem photos and see if you have the same problems in that clean catalog. Not that I'm suggesting you start again, don't worry, but it can help provide clues.
Photo of David Griffin

David Griffin

  • 6 Posts
  • 0 Reply Likes
Well so far this has worked well. I renamed the previews file and started rebuilding the previews with the current session and there were no delays. I've rebuilt previews for 2011 (not many, just a bit under 5000 at this point) and random tests of photos from this year show no loading delay.

All indications are we are on the right track. Tonight I'll have it crank away on the rest of the library and we will see if this trend continues.

Speaking of clues (and variables) one of the things I've done over the past year that might "wake the gremlins" you spoke of is that I converted a significant portion of my library to DNG files (anything over a couple of years old). I don't know if this has any impact on the previews but I thought it might be worth mentioning.

Thank you again Victoria for your suggestion of the partial rebuild proof of concept. I will update this topic tomorrow, hopefully with good news and an opportunity to close it out and provide someone else with a datapoint in case they run into the same situation.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4427 Posts
  • 1637 Reply Likes
That's great news David, hopefully that'll do the trick.
Photo of David Griffin

David Griffin

  • 6 Posts
  • 0 Reply Likes
I continue to test this. About a 1/3 of my catalog's previews have been rebuilt - I did the past couple of years plus 2005 which are DNG files (so that's about 30K previews rebuilt). BTW I'm using the 3.5RC for these tests.

Right now it is much better than it was, but I'm still seeing that dreaded 3-5 second "Loading..." badge pop up in the Library Loupe view -- maybe once every 10-20 images when viewing a random folder and walking through the images (at a steady but brisk rate of one per second or so). Before it was nearly every image so its tolerable although still confusing as to why it happens so much.

I hope to have the rest of the catalog's previews done by tomorrow so I can gauge if there is a performance degradation as the preview database gets larger or holding steady.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 384 Reply Likes
I think Lightroom has some bugs in the change-detect department. I've also noticed Lightroom re-rendering previews when it seems like it shouldn't have to. I haven't scrutinized enough to provide a report, but I've noticed it. Also, I've noticed it *not* re-rendering when it should (in the case of file changed externally, sometimes it takes a really long time and/or maybe a bit of intervention before the it will be updated - sorry I don't have it down to the gnat's nose before mentioning).
Photo of David Griffin

David Griffin

  • 6 Posts
  • 0 Reply Likes
At the 2/3 mark of rebuilding I think I can confidently say this is degrading back to what I saw before. Folders that I have rendered in the past now are increasingly hard to walk through in Loupe mode without the Loading... badge popping up. Sometimes it is a short 1 second wait, but it often reverts back to that 4-5 second Loading... (when the short delay happens there never seems to be any change in the Loupe image, the 4-5 second delay always has a pixelated image). The more previews I built the more often this condition would appear.

I'm inclined to agree with Rob that there seems to be either a consistent bug in the change-detect department or, if I could speculate further, it almost feels like the UI asynchronously asks the database for a preview and if it doesn't get an answer in a few milliseconds it decides to rebuild the preview instead. As the preview database increases the lag time to retrieve a particular preview also increases and crosses this threshold.

In summary, rebuilding the preview database has helped somewhat but as it grew the performance has gone back down again - even with the 3.5RC release.

I don't want to go to multiple catalogs, but without any other avenues to explore it appears that this particular experiment points to a catalog size penalty that ultimately makes LR unusable for fast image-to-image reviews.

Thanks Victoria -- it was exciting for a while to think their was a "old database gremlin" in there but I think this is something else (and can't even speculate what it might be.) If there's instrumentation I can turn on, I'd be happy to do so.

- dave

Data points: Old previews 75GB, rebuilt standard previews 55GB (for 78K images).
Photo of Jim Burgess

Jim Burgess

  • 87 Posts
  • 11 Reply Likes
I have experienced the same thing regarding the "loading" message and slowdown of image display. Here's what I've observed:

Sysytem Configuration: Mac Pro 3,1 with 12GB RAM,
ATI 5770 driving dual Apple monitors (23" and 27").
LR catalog and previews are on a 1GB system drive with 50% free.
LR images are distributed on 3 internal hard drives.

When quickly browsing through photos in Loupe view making selections, about every 10-20 images the display pauses with a "Loading" message for several seconds (4-10). The image is highly pixelated, and not viewable.

It occurs if either standard previews or full-size previews have been generated.

Looking at Activity Monitor, here's what is happening during the delay:
- CPU time for the LR process shoots up to 200% or more.
- System CPU time is about twice the process CPU time.
- There isn't any noticeable disk activity.

Note that it doesn't make any difference...
- If the LR catalog has just been optimized.
- If the LR catalog and previews are relocated to a drive other than the system drive.
- Which of the 3 internal drives containing images is being accessed.
- If the ACR Cache is relocated. Of course the cache doesn't come into play for JPGs, but for raw files it's location doesn't matter. Size of the cache is 50GB, and it's been tested on drives that have more than 25% free space.

The only way I have been able to "cure" the problem is if I delete the previews folder, and start over by building new previews. The last time I did this the previews folder was up to 30GB. Note that I don't build previews for everything in my catalog which is well over 100,000 images. Rather I build them as as needed or when I do an import of new images.

The previews database is now up to about 44,000 items and I'm starting to see the "loading" slowdown again.

Also, I have my preferences set to delete 1:1 previews after 1 week, but honestly I don't think they are being deleted or otherwise managed properly because the number of previews never decreases, and it seems to rise faster than the number of images I add to the catalog. Based on what I and other users have seen, there appears to be some serious issues regarding previews in LR when there are a large number of images in the catalog. A time permits I am going to try to quantify this more.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 384 Reply Likes
Jim - one thing:

If your standard preview size is set too high, 1:1 previews will never be discarded.

I totally understood this behavior not too long ago but at the moment I'm drawing a blank.

Anyway, it's sensible and consistent with how preview subsystem is designed (even if totally non-intuitive) - try dropping standard preview size a notch and you should be able to manually discard 1:1 previews (you will need to rebuild standard previews first). Also, presumably, auto-discard would start working too.

R