Lightroom: Make use of extra cores to process multiple photos in parallel

  • 36
  • Idea
  • Updated 3 years ago
  • Implemented
  • (Edited)
I was exporting 50 photos on my newly built Core I7-3770 machine, which is a CPU has 4 cores 8 threads. From the task manager, I noticed that only 50~60 percent of CPU was used.

If exporting 1 photo only takes half of the CPU power, why not Lightroom process 2 or more photos at one time for those systems have extra power?
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
  • unsure

Posted 6 years ago

  • 36
Photo of jdv

jdv, Champion

  • 728 Posts
  • 56 Reply Likes
I suspect the export process is memory and (to some extent) i/o bound, not CPU bound. The assumption based on looking at task manager that 100% utilization would mean more ability to export is not based on real observations.

Not only is the task manager a very rough and not very accurate picture of the work your computer is doing, it is not in the least predictive. And, like I said, it depends if the problem at hand (i.e., exporting renditions) is CPU bound or not.
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
Hi John,
Thanks for the reply.

However, I have Lightroom 4.1 running with 16GB ram on a Windows 8 64bit system. So I don't think memory is a bound here for just 50 photos. And my catalog, raw cache files and original raw are stored on SSD drive and I guess there is no i/o bound here as well. But just try to eliminate these cause, I will try to use the Resource Monitor the monitor the CPU/Memory/Disk usage during export next time when I doing the export.

BTW, I have tried to turn off HT for my CPU and I got almost exactly same amount of time used for exporting 50 photos. So Lightroom is actually not using the extra threads provided by the CPU at all.
Photo of Butch_M

Butch_M

  • 290 Posts
  • 112 Reply Likes
Scott Kelby offered a tip in a video some time back that you could save time and invoke more resources by exporting in simultaneous batches ...

Say you need to export 300 images to jpeg to upload to the web ... select 150, then export ... immediately select the remaining images and export ... the long hand method to multitasking export ...
Photo of eric teela

eric teela

  • 5 Posts
  • 4 Reply Likes
BUT, this is what the program should be PROGRAMED to do for us. (sorry for yelling)
Photo of Dan Berdal

Dan Berdal

  • 34 Posts
  • 2 Reply Likes
exactly! You can force lightroom to do this by breaking up your export into several groups and exporting them simultaneously... For the record adobe camera raw when hosted in 64 bit photoshop exports photos in the way we're requesting.
Photo of Lynn Bayer

Lynn Bayer

  • 6 Posts
  • 0 Reply Likes
performance on LR5 is very bad and this doesn't work any more so good as it worked with LR4.
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
Hi Butch_M,

Thanks for the suggestion! Yeah, this method definitely would help. I will try that next time.

But I also would like this could be built into the Lightroom export engine as well, if possbile.
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
Report:

I have just tried the manual parallel processing method provided by Butch_M.

By exporting 56 photos in one go, it took 125 seconds
By manually select 30 photos export first and immediately select the remaining 26 export next, it finished in 100 seconds.

That's about 25% speed increase and that is a lot!

I believe if this can be done within the software it could be even faster.
Photo of Michael Decker

Michael Decker

  • 38 Posts
  • 0 Reply Likes
Yes, especially as they could ensure, that x processes are working in parallel.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4135 Posts
  • 1473 Reply Likes
Yeah, they've left it not maxed out, so that you can continue working while the export runs in the background, but a 'turbo export' checkbox in the Export dialog would certainly get my vote.
Photo of Dan Berdal

Dan Berdal

  • 34 Posts
  • 2 Reply Likes
This would be ideal! leave it up to the user. There are times that you want to be able to use your computer for other tasks, and there are times when you would prefer to max out your system resources to finish your export asap.
Photo of Butch_M

Butch_M

  • 290 Posts
  • 112 Reply Likes
There is a fine line in resource management ... Not all users have the same desires and/or requirements ...

Sometimes I return from an event and need to post 4,000 to 6,000 images to my online shopping cart ASAP as time is money in getting the the pictures in front of potential customers ... other times I only need to export a few dozen images ... while I move on and perform other tasks ...

I agree with Victoria that allowing the option, based upon the current workflow circumstance would be the most appropriate method ... it would be the best of both worlds ... mainly because it would allow the user to be in control, not an arbitrary decision of the underlying code ...
Photo of john beardsworth

john beardsworth

  • 998 Posts
  • 219 Reply Likes
Coincidentally, today I was playing with controlling an export through code . Breaking a 267-file export into 3 roughly-simultaneous exports saved 60% of the start-to-finish export time. I may develop it a little further - for example letting the user choose how many simultaneous batches to run - but the existing code does bypass the standard export dialog so I'm not sure it would be compatible with other plugins.
Photo of Robert Frost

Robert Frost

  • 392 Posts
  • 51 Reply Likes
Hi John, Can you do the same for preview rendering?

Bob frost
Photo of john beardsworth

john beardsworth

  • 998 Posts
  • 219 Reply Likes
I don't believe so, Bob, since there's no SDK access to previews. You can see what I did with exports here.
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
John, this really opened a new idea! Thanks! I will look at the SDK and see what can be done there..
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
I think it's best to use all available horse power, and let process priority dictate how it gets shared among apps. Apps shouldn't have to artificially restrict CPU usage - that's what the OS/priority system is for. On my machine, all cores are sometimes utilized at 100% during export, just not for as much of the time as I would expect (goes up to 100% for a while, which is good, then drops down, for longer than I would expect, which is bad, then back up...). I run Lightroom at below-normal priority - long exports have no noticeable impact on other normal priority apps (I started doing that so I could watch HD video without any stuttering, while exporting...). With Lightroom at normal priority, there is occasional stuttering during HD video playback... Other apps doing CPU-intensive tasks (e.g. video/audio compression, use CPU at 100% almost the whole time, without interfering with other higher priority apps (same priority app performance may be degraded, lower priority app performance will get no cpu while higher priority apps are contending for it).

It's long been a mystery what Lightroom is doing sometimes - CPU not maxed out, but ultra-fast disks don't help much - hmmm.... what's the bottle-neck then?

It could be that Lightroom's got some serious problems in the multi-tasking department which is accounting for some of it's ongoing performance woes - even when functioning normally.

If Lightroom is truly so "memory-bound" (I'm not convinced yet), I'd really like to understand that better...
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
Definitely agree, resource allocation should be a task of OS but not Lightroom itself. It should use all resource available to speed up its processing.
Photo of Bryn Forbes

Bryn Forbes

  • 157 Posts
  • 21 Reply Likes
I suspect it's that it's using a single export process and multithreading the filters it's applying but some are only single threaded. So it's 100% when it's a really efficient algorithm, and low when it's single threaded (zip compression for example)
Photo of sean

sean

  • 261 Posts
  • 43 Reply Likes
Here's what Adobe should do---> Take a look at Final Cut X and how they tackle real-time rendering.

If Adobe says that they bind the CPU load for exports to keep some overhead CPU for running the UI, I say they simply look at what's going on in the interface. If the mouse is idle, boost the CPU usage to execute export (or other job) as fast as possible. But, as soon as I start moving in the module, clicking on photos and doing other tasks within LR, scale the CPU back on that export. Final Cut X has a user-set timeout period before the CPU kicks into rendering mode. Same could apply to LR. Get up from the computer and 5 seconds after no action, LR floors the CPU.

It's patently silly that we have to divide up our exports into 3 groups to get the software to use 100% of my cpu power. As a pro, when I export, 99% of the time I want it now. I'm not moving onto other jobs. This has been an issue since the beginning.

[If I were wanting to move onto other jobs, I'd want to have some sort of job queue to run at midnight, but that's for another thread.]
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
I agree - If Adobe has down-throttled export CPU usage so UI remains responsive, they really need to optimize... - that would *not* be considered "best practice", programming-wise.

On the other hand, you may find Lr4.3RC1 is better in this regard. In previous versions anyway, parallel exports greatly increase export throughput on some systems, but on others: not so much. - They fixed some processor utilization issues in .3 candidate.
Photo of sean

sean

  • 261 Posts
  • 43 Reply Likes
Hmm.. Im using 4.3RC. Don't see improvement there.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Dunno what to say: I think Lr has been "CPU allocation challenged" since day one. I also have apps that can utilize all cores at 100% (for extended periods of time) when doing cpu intensive computation - dunno why Lr can't too, or doesn't. I get better CPU allocation than some, but even on my machine the average CPU consumption during a multi-second export is way under 100% (not sure exact number...). I would expect that to be primarily CPU bound, and hence very close to 100% most of the time. Maybe I don't understand the nuances/details, or maybe Adobe really hasn't programmed Lr "correctly" - just dunno... If down-throttling for UI responsiveness is the rationale, then something is not being done optimally, or so I speculate from the comfort of my padded armchair...
Photo of sean

sean

  • 261 Posts
  • 43 Reply Likes
I have a 6-core system. With one export job, all 6 cores are in use but at 30%. When I stack up 3 export jobs, I can peg the CPU at a full 100%, all 12 threads.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Yep - I think it's high time Adobe took a good hard look at their software from a CPU allocation perspective. - you shouldn't have to stack up export jobs like that...

But since you do, consider using publish services to help facilitate simultaneous exports, or ExportManager if your exports aren't via publish services.

The same thing that keeps Lr from being as fast as possible during exports (only partial CPU utilization) is also keeping it from being as fast as possible when rendering for develop...
Photo of sean

sean

  • 261 Posts
  • 43 Reply Likes
That last bit there... since all of the dev stuff is serial [photo 1, then 2, then 3, then 4] doesn't it make sense to be CPU aware and prerender 2,3 and 4 in the same time it takes to render the current image in the dev panel? In other words, yes, I can confirm that selecting a new image uses 6 cores and 6 threads at about 25%. That means my CPU (1/2 of the threads are idle) and hella powerful GPU are sitting around while I wait as well. Seems like there's a ton of room for optimization here, but I'll admit as you, I'm mostly armchair.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Sean,

You're preaching to the choir: I don't see any reason why the CPU couldn't be far more utilized when rendering a single image for develop. And regardless, pre-caching (aka look-ahead caching) for develop would be a worthwhile optimization, in my (armchair) opinion.

PS - I have no idea whether the GPU could be tapped for rendering tasks - I would guess: not really.

Rob
Photo of eric teela

eric teela

  • 5 Posts
  • 4 Reply Likes
When converting RAW images, process one image per core. Similar to SETI@home. It may not be efficient to split the conversion of an image across multiple cores but running 4-16 conversions at a time would speed up the overall batch. I don't know if this means that ACR would need to spawn as many instances of itself as there are cores or not. Of course this is memory and disk bandwidth limited, but that can be addressed by the user.

This reply was created from a merged topic originally titled
Efficient Multi Core Processing of RAW files.
Photo of eric teela

eric teela

  • 5 Posts
  • 4 Reply Likes
I'm running a quad core i5 with 20GB ram and a 500MB/s SSD(neither is maxe'd or even taxed), the max processor use I ever see is 300% fluctuating greatly. Seti@home runs 4 instances of itself, PEGS the processors at 400% (100 each) for days on end and still manages to gracefully allow other programs (even photoshop) to work normally like a well written background process should. Perhaps Adobe could consider making a hefty donation to Seti@home for them to optimize their photo applications for multi core processing?
Photo of Damon Brodie

Damon Brodie

  • 12 Posts
  • 2 Reply Likes
My typical workflow after importing is to keyword the imported photos. Even with previews pre-rendered, scrolling down in grid view is a bit painful. As I'm working away on my current screen full of thumbnails, LR could be doing whatever it needs to do to make it faster to render the thumbnails further down.

Another way to describe it would be to take better advantage of idle CPU and memory.
Better multi-threadedness would be appreciated too on imports and exports.

This reply was created from a merged topic originally titled
Lightroom: Use idle CPU in grid mode and multi-threaded utilization (performance).
Photo of dmitry.zaslavsky

dmitry.zaslavsky

  • 2 Posts
  • 0 Reply Likes
Lightroom doesn't seems to fully utilizing the CPU/memory. For example on export on my quad-core CPU, utilization often drops to 40-60%. It seems like the writing of the resulting files needs to be done asynchronously with reading/writing files

This reply was created from a merged topic originally titled
Full resource utilization.


Note: This topic was created from a reply on the Lightroom: Use idle CPU in grid mode and multi-threaded utilization (performa... topic.

This reply was created from a merged topic originally titled
Full resource utilization.
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
I am glad that so many people think the same as me. I hope that Adobe could consider any of the idea here
Photo of eric teela

eric teela

  • 5 Posts
  • 4 Reply Likes
Dear Adobe, please respond to this post. I think all we want is an honest answer as to why your program does not work in this fashion. If you are working on it, wonderful. If you don't have enough resources, let us sign up to support this initiative. If there are other considerations we have not brought up, please discuss.
Photo of Alessandro Avenali

Alessandro Avenali

  • 21 Posts
  • 0 Reply Likes
I'd like Lightroom would definitely make something better with my CPU, multithreading and multitasking itself. Speed up the exporting process or let people make a good use of OS multitasking capabilities!
E.g.: I'd like to work on a different catalog (a different wedding for example) while exporting my 1.000/2.000 images set(s) of the first catalog.
Photo of Arnold Bartel

Arnold Bartel

  • 170 Posts
  • 40 Reply Likes
When exporting a webgallery the workaround to do it in several parallel jobs often doesn't work (as it does with the normal picture export).
For me this means that the hint can just be a workaround and Adobe should focus on optimising the use of ressources with multicore computers.
Photo of Michael Decker

Michael Decker

  • 38 Posts
  • 0 Reply Likes
I've similar issue during importing more then 2,000 files:

Windows 7 SP1; 64 Bit
Intel i7 with 4 cores, enabled hyperthreading and 3.6 GHz
-> 8 virtual CPUs
24 GB RAM

Only up to 25 % of my whole CPU resources used by Lightroom, so it took up 20 minutes.

If it would use all cores, it could do it 4 times faster: also only in 5 minutes!

Some for updating the previews after changing pictures. One preview after the other was updated. No parallel processing. Only simple and slow sequencially processing.

I'm really disappointed, as multi core CPU are not rare anylonger since years!
Photo of Michael Decker

Michael Decker

  • 38 Posts
  • 0 Reply Likes
The same issue is there, if you use Bidge and Camera RAW:
http://feedback.photoshop.com/photosh...
Photo of Michael Decker

Michael Decker

  • 38 Posts
  • 0 Reply Likes
So the similar feature request for Bridge and Camera RAW is now:
http://feedback.photoshop.com/photosh...
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Here is a simple script you can use to divide export into multiple simultaneous tasks.

Export in Parallel:


Note: sequence numbers for filenaming purposes will be partitioned appropriately.

Free, written by me:
http://www.robcole.com/Rob/ProductsAn...

Rob
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
You bet Dan, and yeah: if you are memory-challenged in the first place (whether due to amount installed or non-optimal memory-management/bugs..), this script may make things worse.. - in addition to using more CPU, it will also use more memory doing exports in parallel than sequentially. Lr *should* release used memory, but as most of us know - it doesn't always..

FWIW: This script only supports (non-publishing) exports without post-process actions, it does not support publish services, nor web galleries.
Photo of Bryn Forbes

Bryn Forbes

  • 157 Posts
  • 21 Reply Likes
hopefully Adobe will be inspired by the real world results of your script and apply this to standard preview generation (smart previews build so much faster) and web pages and books and so on.
Photo of f_rele

f_rele

  • 7 Posts
  • 1 Reply Like
Link is not working
Photo of John R. Ellis

John R. Ellis, Champion

  • 3539 Posts
  • 907 Reply Likes
Unfortunately, Rob Cole hasn't been heard from since January 2015, and his site went offline soon after. (Hope everything is ok...)

With respect to his script, note that LR CC 2015 / 6 now fully utilizes multiple processors for export, so the script is unnecessary. In fact, some people now complain that LR is too aggressive in using multiple processors for export, to the point that interactive use is impossible while an export is in progress.
Photo of f_rele

f_rele

  • 7 Posts
  • 1 Reply Like
Thanks for the info. I have LR6 but I was just curious how punish my 6 cores to maximum. Glad to hear that LR6 utilizes all cores better and external script is not necessary.
Photo of Issac Issac

Issac Issac

  • 2 Posts
  • 0 Reply Likes
hi, could someone please tell me how to install the export in parallel scripts in lightroom? or even how to get it. i used to have it before but after i cleaned my hard drive and installed windows again i lost it and cant find it anywhere.

thanks
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes
The author's Web site, robcole.com, has been down for many weeks. Not clear if it's coming back, which would be too bad.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 3791 Posts
  • 1252 Reply Likes
Lightroom CC/6 now does this automatically
Photo of Issac Issac

Issac Issac

  • 2 Posts
  • 0 Reply Likes
Could you tell me how its done please? I have 5.7. Many thanks
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 3791 Posts
  • 1252 Reply Likes
It happens automatically in 6 - you don't do anything. You can't do it in 5 without a plug-in or setting it up manually (e.g. select some, set them exporting, select some more, set them exporting at the same time)
Photo of Spring Feng

Spring Feng

  • 8 Posts
  • 1 Reply Like
I am glad that after 3 years this thread is still alive...

Yes, Adobe has finally addressed this issue with latest CC/6 updates... although is poorly implemented in my opinion.

1. The whole UI is not responding when the export is taking progress.. I don't mind lightroom taking all resources when exporting (i.e. other programs could run slowly when exporting is happening), but freeze its own UI doesn't make sense.
2. The overall export speed is NOT increased based on my unscientific testing.. although LR is now using the full CPU threads, but for some funny reason, the per photo exporting speed dropped a lot and seems totally killed the effort of utilizing multiple cores...
Photo of John R. Ellis

John R. Ellis, Champion

  • 3298 Posts
  • 825 Reply Likes
"The whole UI is not responding when the export is taking progress.. "

Please add your vote and opinion about this to this topic: http://feedback.photoshop.com/photosh...
Photo of John R. Ellis

John R. Ellis, Champion

  • 3589 Posts
  • 928 Reply Likes
Spring Feng wrote, "The overall export speed is NOT increased based on my unscientific testing.. although LR is now using the full CPU threads, but for some funny reason, the per photo exporting speed dropped a lot and seems totally killed the effort of utilizing multiple cores..."

My perception was different, so I did a small experiment on my 4-core MacBook Pro, exporting 20 raw files as quality-60 JPEGs. CC 2015.1.1 was 1.41x (41%) faster than LR 5.7.1. Total CPU utilization increased by 1.51x, from 53% to 80%. (LR probably shouldn't use more than 80%, to ensure reasonable interactive response during the export.) This kind of speedup from using multiple processors is pretty good and what I would have expected.

A couple of import things to note:

- Versions of LR prior to CC 2015.1.1 / 6.1.1 had a serious bug in export, often causing it to do significantly more work than necessary. So if you're on such an earlier version, consider upgrading (but to 2015.1.1, not 2015.2.1!).

- The progress bar in the upper-left corner is updated much more frequently in LR 5.7.1 than CC 2015.1.1, giving the impression that LR 5.7.1 is much zippier. LR 5.7.1 updates it after each file is exported, and it shows the filename; whereas CC 2015.1.1 updates it only 8 times (after 12.5% of the total work is done), and it doesn't show filenames.

Details of the experiment:

- MacBook Pro (late 2013), Core i7, 2.6 GHz, 4 cores/8 logical processors, L2 256 KB/core, L3 6 MB, 16 GB memory, 1600 MHz DDR3, SSD.

- The raw files were 5472 x 3648 .arw's, 21 MB, with all the basic settings adjusted, sharpening, and one stroke of the local adjustment brush.

- The JPEGs were exported the original size, quality 60, and were 3.9 MB.