Lightroom: Suspected memory leak: 3.4.1

  • 9
  • Problem
  • Updated 7 years ago
  • (Edited)
I've been moving images from one machine to another all day today. Simple process. Export as catalog from laptop. Import from catalog on workstation. From the workstation, rename all images and move to the final folder. After a couple of hours and maybe 5 export/import/rename/move iterations, LR was running at a crawl. Activity Monitory showed 77 threads and 1.5GB of virtual memory in use by LR. This compares to 23 threads and 190MB of virtual after a reboot and restart of LR.

sorry i don't have any more details.
Photo of Frederick Moor

Frederick Moor

  • 15 Posts
  • 0 Reply Likes

Posted 7 years ago

  • 9
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 376 Reply Likes
I don't know if your particular problem started with 3.4.1, but there are still leaks...
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 132 Reply Likes
I'm not sure. I really thought I had one. I had been running multiple imports and memory was climbing like crazy during preview generation. Eventually, it had absorbed all of my 14GB and slowed to a crawl but it seemed to be almost done so I just let it go. Memory leak, right? Well, it eventually did finish, all those preview generation threads stopped and the memory usage fell right back to normal. So it wasn't really a leak, just an over-allocation like it was allocating just a little faster than it was completing and releasing.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 376 Reply Likes
Good point Lee Jay - not all apparent leaks are... - its so much better than it was that I pay a lot less attention these days (although I do restart Lightroom occasionally to "replenish" memory). Sorta begs the question: "would Frederick Moors system have returned to normal if he let it go long enough?
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
Hmm. Batch rename and catalog import definitely had significant leaks (or at least accumulations while they were underway) at one point, but I thought they'd been fixed. I'll have to look and see if I can confirm.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 373 Reply Likes
I've not done batch rename in a long while, and I haven't seen leaks doing imports recently, but then I rarely import more than a hundred or two photos at a time. I'll keep my eye out for any leaks and what the leaky conditions are if it occurs.
Photo of Calvin Hilton

Calvin Hilton

  • 6 Posts
  • 0 Reply Likes
It's not unusual for me to see 1/2 GB memory used by LR 3 but today I'm seeing over 1 GB memory used (according to Windows Task Manager) after not a lot of activity in a single catalog since starting LR. I've just been reviewing about 200 photos and doing a little tweaking.

One thing that I just did that I don't do very often is create a simple Develop preset and applied it to about 15 photos. Preset is only Camera Standard and Daylight WB.

I'm running Win 7 32-bit.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
The various caches scale based on your system RAM (but only up to a point). I don't have the code handy to do the math for limits. In my testing, I have ways to force the caches to clear to look for leak patterns. The other way you can look is to repeat the operation that seems to cause excessive usage. If it plateaus, it's probably just a result of caching. If it climbs indefinitely, it's a leak. Note the initial climb when the caches fill will likely happen quickly. The leak (if any, I'm sure there are at least a few) is probably smaller and slower to accumulate.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 373 Reply Likes
So, its possible the numbers I quoted are low, compared to a system with 12-32GB (I have 8). And, its possible they are high compared to a system with 2-6... - YMMV...
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
Something like that.

To complicate matters a bit more, ACR has its own memory cap and a scratch file, so some leaks result in that file getting big instead of memory usage continuing to climb. We've knocked off a couple of those and there's one other one that we're going to try to backport into the next 3.x dot.

As for 32-bit Windows, the cache sizes there are a bit on the conservative side to try to cut down on weird out of memory errors due to fragmentation. For Windows XP, I see LR a bit over 1 GB as normal for some tasks, but if it starts getting up in the 1.3-1.4 GB+ range things might start to wig out a bit.

There's a memory panic mechanism that'll kick in and try to let go of cache data to get more space. A real leak will probably eventually make the app crash, but it'll usually get really slow for a while before it does.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 373 Reply Likes
Dan - thank you for taking the time to provide this more specific and useful information :-)
Photo of didi

didi

  • 54 Posts
  • 9 Reply Likes
Interesting! Thank you!
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 376 Reply Likes
Just had Lightroom take about 2.9G, Lr + Computer started to slow... - presumably leak. Was doing a lot of spot removal earlier, dunno if coincidence.
Photo of didi

didi

  • 54 Posts
  • 9 Reply Likes
Spot removal definitely has a memory leak.
Photo of Frederick Moor

Frederick Moor

  • 15 Posts
  • 0 Reply Likes
;-)
Photo of Mitch Miller

Mitch Miller

  • 3 Posts
  • 0 Reply Likes
Lightroom 3 *appears* to be leaking memory when rolling through images and making adjustments in the Develop module. Memory just keeps climbing until LR became too slow for my sanity (i.e. the sliders becoming active would take longer and longer; starts out less than 250ms, but eventually grows to over 1500ms).

Using Lightroom 3.4.1 x64 on Windows 7 x64. Intel i7 (quad core) with 6Gb RAM, 500Gb hybrid drive with 128Gb free.

Using Windows TaskManager to monitor memory utilization -- it starts about the 2Gb mark and slowly grows to over the 4Gb mark, when the response times begin to slow significantly (most notably the time to switch between images, and the availability of the sliders).

This reply was created from a merged topic originally titled
Lightroom 3.4.1 memory leak in Develop module.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
Yes, there is a fix for an accumulation matching that symptom in an internal build. We're planning to include it in the next dot/camera updates release, barring some problem with backporting it to the 3.x branch.

That's separate from the issue that started this thread, I think.

Also, to reference the earlier post about spot healing, from some long running stress tests I've been running internally, spot healing causes the memory footprint of a single open image to exceed the accounting estimate used for cache sizing. A couple of such mis-estimates would cause elevated memory, but combined with the bug above, which is actually caused by too many images to be open at once when walking through Develop loupe view can result in really high memory usage.

It's not a leak in the typical sense of memory that is allocated and not freed because the pointers are left dangling, but it has the same practical effect unless you have access to the API for forcing the caches to clear.

There's also some general responsiveness issues with dust spotting that are proving harder to fix (especially if you get lens corrections in the mix, too), so even with memory staying in line, it'll tend to feel sluggish, but the high memory use sure doesn't help matters there.
Photo of Mitch Miller

Mitch Miller

  • 3 Posts
  • 0 Reply Likes
Thanks Dan.

Is there anything that we can do until the next release to avoid the problem? Ex: does switching between Library and Develop modules before advancing to the next image avoid the memory accumulation?
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
I did some investigation and did find something you can do to keep memory lower, but it's a somewhat destructive workaround. If you clear the history periodically as you go, it'll significantly the reduce memory usage and it may improve the responsiveness of the local correction tools (spot and brushing, especially) as well. Like I said, it's a bit destructive and definitely a workaround and not a real solution.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
I did find another script that I can use to drive memory too high, but the amount of strokes I was laying down was pretty high. The entire image in my case was completely covered with hundreds of strokes. It's possible it'd ramp higher if I applied more different adjustment channels to the brushes. If there's an easily back portable change to whatever fixes this issue, I'll push for it to get included in the next dot release as well.
Photo of Steve Holcroft

Steve Holcroft

  • 5 Posts
  • 0 Reply Likes
Any progress on this? I won't be using LR next summer if this doesn't get sorted soon.
As far as I'm concerned LR3 isn't fit for professional use with this bug.
Photo of Charles Juszczak

Charles Juszczak

  • 2 Posts
  • 0 Reply Likes
I'm experiencing this issue with the Exposure brush and all I have to do is open the brush to run in to this problem. The program becomes so slow it is completely unusable.

I literally went and bought a new computer today suspecting that may be my issue (went from a Core 2 Duo MacBook Pro with 4GB of RAM to an i7 iMac with 4GB now and 16GB tomorrow when the upgrade arrives), of course now I found this thread.

I'm still running the trial of Lightroom (because of the Exposure brush vs. none in Aperture 3) and haven't broken the seal on the copy I picked up the other day because of this problem.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
What do you mean "open the brush"? Just clicking the brush icon to open the settings drawer causes memory to go up? On an image with no adjustments?
Photo of Charles Juszczak

Charles Juszczak

  • 2 Posts
  • 0 Reply Likes
My apologies, yes just clicking the brush icon (on an images with lots of brush strokes already) causes the memory to go up.

I just tried opening the image with the brush strokes (all exposure) in the develop module (actually without clicking the brush icon) and it immediately used 2.87 GB of RAM (according to the OS X activity monitor) and slowed to a crawling pace and then moved to another photo and restarted Lightroom and it only used 483 MB and it remained very usable.

Sorry if that's confusing...
Photo of Steve Holcroft

Steve Holcroft

  • 5 Posts
  • 0 Reply Likes
Dan I don't do a lot of brushing - just a couple of faces maybe. Nevertheless, Lightroom is almost unusable, so it's not just excessive use that's causing it. My PC is also brand new.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
I ran a script that toned down the number of brush strokes and I do still get high (though not leaking -- it plateaus in a way that makes me suspect this is just an over-sized cache) memory usage patterns. The system that it's on has 8 GB of RAM and it uses 5 GB. I'm not seeing a drastic impact on responsiveness, but this is using a version that has another fix that might be helping. I'll need to run this same script with a 3.4.1 build and see if it's worse there to confirm.

It's always so devilishly hard to tell whether I've managed to create the same bug customers are seeing or not. I really need to find a way to plumb an action recording system so I can get a log and see what actions were being performed and combine it with my existing mechanism for logging memory usage, CPU time etc, to see if that provides me better clues as to why people get such different behavior.
Photo of Steve Holcroft

Steve Holcroft

  • 5 Posts
  • 0 Reply Likes
Any updates Dan?
As far as I'm concerned right now LR3.4.1 is unfit for purpose. I don't expect to have to reboot every ten minutes when I'm going through hundreds of images. I'm not even doing a lot of work on any of them but sooner rather than later it will lock up.

Adobe's response to this so far has not been good enough.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
Did you try the 3.5 RC? It has a fix that may help if I'm understanding what you're seeing correctly. There's technically still a way to get high memory and corresponding disk churn, but my experiments so far indicate that it happens primarily when doing lots of local corrections.

Note that there is one known crasher bug for certain adjustments which we have cornered internally.
Photo of Steve Holcroft

Steve Holcroft

  • 5 Posts
  • 0 Reply Likes
Dan I've just tried 3.5 RC and it does seem much better. Thanks for your reply.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
One other remark on this particular issue (specific to walking through and adjusting multiple images in Develop). If folks find that even LR 3.5 is hanging on to too much memory and thrashing a bit as a result, I also requested (and got) a configuration switch for disabling all caching of negatives.

It'll be in the final version of 3.5 so that if there are still people having trouble, we can more easily distinguish between memory usage due to loaded images that have big footprints that are mis-estimated by the cache retention logic (known problem that we're investigating) vs. other kinds of actual leaks that we don't really understand well (yet).
Photo of Steve Holcroft

Steve Holcroft

  • 5 Posts
  • 0 Reply Likes
I had to upgrade to 3.4.1 recently because of a new camera purchase, so I can't go back to 2.7 which worked fine.
Can I download any other version that works with a 60D?
I will have to try Capture One this week as I'm beginning to make clients wait far too long for their images.