Skip to main content
Adobe Photoshop Family

8 Messages

 • 

484 Points

Thu, Aug 30, 2012 2:01 AM

Implemented

37

Lightroom: Make use of extra cores to process multiple photos in parallel

I was exporting 50 photos on my newly built Core I7-3770 machine, which is a CPU has 4 cores 8 threads. From the task manager, I noticed that only 50~60 percent of CPU was used.

If exporting 1 photo only takes half of the CPU power, why not Lightroom process 2 or more photos at one time for those systems have extra power?

Responses

704 Messages

 • 

8.5K Points

8 years ago

I suspect the export process is memory and (to some extent) i/o bound, not CPU bound. The assumption based on looking at task manager that 100% utilization would mean more ability to export is not based on real observations.

Not only is the task manager a very rough and not very accurate picture of the work your computer is doing, it is not in the least predictive. And, like I said, it depends if the problem at hand (i.e., exporting renditions) is CPU bound or not.

8 Messages

 • 

484 Points

8 years ago

Hi John,
Thanks for the reply.

However, I have Lightroom 4.1 running with 16GB ram on a Windows 8 64bit system. So I don't think memory is a bound here for just 50 photos. And my catalog, raw cache files and original raw are stored on SSD drive and I guess there is no i/o bound here as well. But just try to eliminate these cause, I will try to use the Resource Monitor the monitor the CPU/Memory/Disk usage during export next time when I doing the export.

BTW, I have tried to turn off HT for my CPU and I got almost exactly same amount of time used for exporting 50 photos. So Lightroom is actually not using the extra threads provided by the CPU at all.

322 Messages

 • 

7.5K Points

8 years ago

Scott Kelby offered a tip in a video some time back that you could save time and invoke more resources by exporting in simultaneous batches ...

Say you need to export 300 images to jpeg to upload to the web ... select 150, then export ... immediately select the remaining images and export ... the long hand method to multitasking export ...

5 Messages

 • 

266 Points

BUT, this is what the program should be PROGRAMED to do for us. (sorry for yelling)

35 Messages

 • 

694 Points

exactly! You can force lightroom to do this by breaking up your export into several groups and exporting them simultaneously... For the record adobe camera raw when hosted in 64 bit photoshop exports photos in the way we're requesting.

6 Messages

 • 

178 Points

performance on LR5 is very bad and this doesn't work any more so good as it worked with LR4.

8 Messages

 • 

484 Points

8 years ago

Hi Butch_M,

Thanks for the suggestion! Yeah, this method definitely would help. I will try that next time.

But I also would like this could be built into the Lightroom export engine as well, if possbile.

8 Messages

 • 

484 Points

8 years ago

Report:

I have just tried the manual parallel processing method provided by Butch_M.

By exporting 56 photos in one go, it took 125 seconds
By manually select 30 photos export first and immediately select the remaining 26 export next, it finished in 100 seconds.

That's about 25% speed increase and that is a lot!

I believe if this can be done within the software it could be even faster.

38 Messages

 • 

662 Points

Yes, especially as they could ensure, that x processes are working in parallel.

Champion

 • 

5.8K Messages

 • 

102.6K Points

8 years ago

Yeah, they've left it not maxed out, so that you can continue working while the export runs in the background, but a 'turbo export' checkbox in the Export dialog would certainly get my vote.

35 Messages

 • 

694 Points

This would be ideal! leave it up to the user. There are times that you want to be able to use your computer for other tasks, and there are times when you would prefer to max out your system resources to finish your export asap.

322 Messages

 • 

7.5K Points

8 years ago

There is a fine line in resource management ... Not all users have the same desires and/or requirements ...

Sometimes I return from an event and need to post 4,000 to 6,000 images to my online shopping cart ASAP as time is money in getting the the pictures in front of potential customers ... other times I only need to export a few dozen images ... while I move on and perform other tasks ...

I agree with Victoria that allowing the option, based upon the current workflow circumstance would be the most appropriate method ... it would be the best of both worlds ... mainly because it would allow the user to be in control, not an arbitrary decision of the underlying code ...

1.3K Messages

 • 

22.5K Points

8 years ago

Coincidentally, today I was playing with controlling an export through code . Breaking a 267-file export into 3 roughly-simultaneous exports saved 60% of the start-to-finish export time. I may develop it a little further - for example letting the user choose how many simultaneous batches to run - but the existing code does bypass the standard export dialog so I'm not sure it would be compatible with other plugins.

442 Messages

 • 

6.6K Points

Hi John, Can you do the same for preview rendering?

Bob frost

1.3K Messages

 • 

22.5K Points

I don't believe so, Bob, since there's no SDK access to previews. You can see what I did with exports here.

8 Messages

 • 

484 Points

John, this really opened a new idea! Thanks! I will look at the SDK and see what can be done there..

4.5K Messages

 • 

76.3K Points

8 years ago

I think it's best to use all available horse power, and let process priority dictate how it gets shared among apps. Apps shouldn't have to artificially restrict CPU usage - that's what the OS/priority system is for. On my machine, all cores are sometimes utilized at 100% during export, just not for as much of the time as I would expect (goes up to 100% for a while, which is good, then drops down, for longer than I would expect, which is bad, then back up...). I run Lightroom at below-normal priority - long exports have no noticeable impact on other normal priority apps (I started doing that so I could watch HD video without any stuttering, while exporting...). With Lightroom at normal priority, there is occasional stuttering during HD video playback... Other apps doing CPU-intensive tasks (e.g. video/audio compression, use CPU at 100% almost the whole time, without interfering with other higher priority apps (same priority app performance may be degraded, lower priority app performance will get no cpu while higher priority apps are contending for it).

It's long been a mystery what Lightroom is doing sometimes - CPU not maxed out, but ultra-fast disks don't help much - hmmm.... what's the bottle-neck then?

It could be that Lightroom's got some serious problems in the multi-tasking department which is accounting for some of it's ongoing performance woes - even when functioning normally.

If Lightroom is truly so "memory-bound" (I'm not convinced yet), I'd really like to understand that better...

8 Messages

 • 

484 Points

Definitely agree, resource allocation should be a task of OS but not Lightroom itself. It should use all resource available to speed up its processing.

161 Messages

 • 

3.2K Points

I suspect it's that it's using a single export process and multithreading the filters it's applying but some are only single threaded. So it's 100% when it's a really efficient algorithm, and low when it's single threaded (zip compression for example)

409 Messages

 • 

7.7K Points

8 years ago

Here's what Adobe should do---> Take a look at Final Cut X and how they tackle real-time rendering.

If Adobe says that they bind the CPU load for exports to keep some overhead CPU for running the UI, I say they simply look at what's going on in the interface. If the mouse is idle, boost the CPU usage to execute export (or other job) as fast as possible. But, as soon as I start moving in the module, clicking on photos and doing other tasks within LR, scale the CPU back on that export. Final Cut X has a user-set timeout period before the CPU kicks into rendering mode. Same could apply to LR. Get up from the computer and 5 seconds after no action, LR floors the CPU.

It's patently silly that we have to divide up our exports into 3 groups to get the software to use 100% of my cpu power. As a pro, when I export, 99% of the time I want it now. I'm not moving onto other jobs. This has been an issue since the beginning.

[If I were wanting to move onto other jobs, I'd want to have some sort of job queue to run at midnight, but that's for another thread.]

4.5K Messages

 • 

76.3K Points

I agree - If Adobe has down-throttled export CPU usage so UI remains responsive, they really need to optimize... - that would *not* be considered "best practice", programming-wise.

On the other hand, you may find Lr4.3RC1 is better in this regard. In previous versions anyway, parallel exports greatly increase export throughput on some systems, but on others: not so much. - They fixed some processor utilization issues in .3 candidate.

409 Messages

 • 

7.7K Points

Hmm.. Im using 4.3RC. Don't see improvement there.

4.5K Messages

 • 

76.3K Points

Dunno what to say: I think Lr has been "CPU allocation challenged" since day one. I also have apps that can utilize all cores at 100% (for extended periods of time) when doing cpu intensive computation - dunno why Lr can't too, or doesn't. I get better CPU allocation than some, but even on my machine the average CPU consumption during a multi-second export is way under 100% (not sure exact number...). I would expect that to be primarily CPU bound, and hence very close to 100% most of the time. Maybe I don't understand the nuances/details, or maybe Adobe really hasn't programmed Lr "correctly" - just dunno... If down-throttling for UI responsiveness is the rationale, then something is not being done optimally, or so I speculate from the comfort of my padded armchair...

409 Messages

 • 

7.7K Points

8 years ago

I have a 6-core system. With one export job, all 6 cores are in use but at 30%. When I stack up 3 export jobs, I can peg the CPU at a full 100%, all 12 threads.

4.5K Messages

 • 

76.3K Points

Yep - I think it's high time Adobe took a good hard look at their software from a CPU allocation perspective. - you shouldn't have to stack up export jobs like that...

But since you do, consider using publish services to help facilitate simultaneous exports, or ExportManager if your exports aren't via publish services.

The same thing that keeps Lr from being as fast as possible during exports (only partial CPU utilization) is also keeping it from being as fast as possible when rendering for develop...

409 Messages

 • 

7.7K Points

That last bit there... since all of the dev stuff is serial [photo 1, then 2, then 3, then 4] doesn't it make sense to be CPU aware and prerender 2,3 and 4 in the same time it takes to render the current image in the dev panel? In other words, yes, I can confirm that selecting a new image uses 6 cores and 6 threads at about 25%. That means my CPU (1/2 of the threads are idle) and hella powerful GPU are sitting around while I wait as well. Seems like there's a ton of room for optimization here, but I'll admit as you, I'm mostly armchair.

4.5K Messages

 • 

76.3K Points

Sean,

You're preaching to the choir: I don't see any reason why the CPU couldn't be far more utilized when rendering a single image for develop. And regardless, pre-caching (aka look-ahead caching) for develop would be a worthwhile optimization, in my (armchair) opinion.

PS - I have no idea whether the GPU could be tapped for rendering tasks - I would guess: not really.

Rob

5 Messages

 • 

266 Points

8 years ago

When converting RAW images, process one image per core. Similar to SETI@home. It may not be efficient to split the conversion of an image across multiple cores but running 4-16 conversions at a time would speed up the overall batch. I don't know if this means that ACR would need to spawn as many instances of itself as there are cores or not. Of course this is memory and disk bandwidth limited, but that can be addressed by the user.

This reply was created from a merged topic originally titled
Efficient Multi Core Processing of RAW files.

5 Messages

 • 

266 Points

I'm running a quad core i5 with 20GB ram and a 500MB/s SSD(neither is maxe'd or even taxed), the max processor use I ever see is 300% fluctuating greatly. Seti@home runs 4 instances of itself, PEGS the processors at 400% (100 each) for days on end and still manages to gracefully allow other programs (even photoshop) to work normally like a well written background process should. Perhaps Adobe could consider making a hefty donation to Seti@home for them to optimize their photo applications for multi core processing?

12 Messages

 • 

510 Points

8 years ago

My typical workflow after importing is to keyword the imported photos. Even with previews pre-rendered, scrolling down in grid view is a bit painful. As I'm working away on my current screen full of thumbnails, LR could be doing whatever it needs to do to make it faster to render the thumbnails further down.

Another way to describe it would be to take better advantage of idle CPU and memory.
Better multi-threadedness would be appreciated too on imports and exports.

This reply was created from a merged topic originally titled
Lightroom: Use idle CPU in grid mode and multi-threaded utilization (performance).

2 Messages

 • 

74 Points

8 years ago

Lightroom doesn't seems to fully utilizing the CPU/memory. For example on export on my quad-core CPU, utilization often drops to 40-60%. It seems like the writing of the resulting files needs to be done asynchronously with reading/writing files

This reply was created from a merged topic originally titled
Full resource utilization.


Note: This topic was created from a reply on the Lightroom: Use idle CPU in grid mode and multi-threaded utilization (performa... topic.

This reply was created from a merged topic originally titled
Full resource utilization.

8 Messages

 • 

484 Points

8 years ago

I am glad that so many people think the same as me. I hope that Adobe could consider any of the idea here