Lightroom SDK: why does memory use skyrocket on some systems but not others, when uncommited catalog updates accumulate?

  • 1
  • Question
  • Updated 7 years ago
I have multiple plugins that don't increase memory consumption even 1% on my machine when running a whole catalog update (with-write-access-do, or with-private-write-access-do), yet on some user's machines they were running out of memory before finishing the update (last user reported oom at ~2500 photos updated).

I have solved this problem by breaking the update into 1000 photo chunks (exiting the with-do gate every 1000 photos to flush accumulated update info to catalog, which frees up the ram), but still 'tis a puzzlement: I would expect same behavior on all systems.

In my case its win 7/64(8GB), last user to report problem was running same OS with 4GB. Not sure about Macs.

I suppose this could have been a problem report instead, but since the problem is solved for me, I was thinking of it more as a curiosity... But, it may very well still be a problem for some plugins/authors/users, and may be related to other memory issues...
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes

Posted 7 years ago

  • 1
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
How dirty buffers are committed is indeterminate. But, more to the point, we have very little information about what actually caused the OOM reported to us. This exception bubbles up to the plug-in from a variety of causes, and knowing this real cause is the key.

Because memory allocation and garbage collection is also completely indeterminate.

The right thing to do is to batch up your requests, as you say.

Ideally we would never want a plug-in to take down the app in this manner, but the fact is that plug-ins share the same Lua image with the app. In most cases we are forced to run our code in an LrTask of some sort (provided or not) which forces us to eventually release control to a non-plugin thread (though yielding at critical spots or in tight loops is probably a good idea.)

The problem here is that the access gates are atomic, so I suspect other routines may not get much attention under the right load (and this includes GC.)

What happens if you yield() inside the gate instead of breaking the work up explicitly?

If the access gates eventually represent an open database cursor or log that cannot be flushed until the gate is closed, then there isn't much else you can do (or should do) than batch up your requests.

[Edit, upon rereading the SDK, I seriously doubt this is the case. I suspect judicious use of yield() /should/ allow the app to get to housekeeping.]

As an aside, I'd really like a more expanded coroutine/task/threading explanation and examples in the next release of the SDK.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Thanks John,

Its "funny" - I previously had yields in the update loops but took them out, deeming them unnecessary.

Generally, its obvious when a yield is necessary: the UI is unresponsive, and/or there's a spinning ball or doughnut...

Since neither of these things was happening (UI remained responsive, progress indicator was progressing...), I assumed there was an implied yield somewhere, and the explicit yield was redundant.

After further consideration, I think you may be right - although the explicit yield may not be necessary on all systems, it may be necessary on some systems.

I haven't confirmed this yet, but at the moment it seems a likely culprit...
I'll see if I can find a guinea pig (er, I mean willing participant) to test this for me, and will report back.

PS - Enhanced notes in the guide about tasking would be worthwhile for many of us...

Rob
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
The access gates have to be run in a context where yield() can be invoked, and you can call it from the gate.

This is the best way to pass the baton back to the rest of the app so it can flush buffers, release memory and generally get some stuff done -- not just update the UI.

I've gotten into the habit of yielding whenever I start a new task, or around (or in) longer loops.

Do get back to us on this. I'd like to know if my complete guess is at all accurate.

This is also a place where someone from Adobe should chime in. *hint*
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
John, I did some research - one of the problem plugins has had a yield in the main loop since the beginning; there was a different plugin from which I had removed it. So, I think it has something to do with the catalog update instead.

PS - beware of yield inefficiency (Windows only). I have a yield function that yields every 20th call.