Excessive time taken to create Collections

  • 1
  • Problem
  • Updated 1 month ago
Good morning,

I rely heavily on Collections to organize and manage images in the context of how they are used. Putting together a book, for example, that has, say, 100 or 200 images; I create a collection for each image. I can then isolate all the ‘source’ images that may contribute in some way to the final result (the image that appears in the book).


This requires me to create a lot of collections. I don’t know how many collections are now in my Collection library but it’s easily in the 1,000s perhaps by now into the 10,000s.


I run a Mojave Mac using LR Classic. The catalogue contains ~110,000 images. If I start from scratch (i.e., immediately post-startup from Shutdown) creating collections, the 1st new collection appears in a second or less. That’s fine. With every succeeding new collection, the time taken increases. After 5 or 6, the swirling beach ball of death starts to appear. By the time I doing the 10th or 11th it takes 10+ seconds to create a collection. By the time I get to the 15th it’s taking 18 secs. And it just gets worse from there.


Then I start again. This is quite time-consuming and annoying. More importantly, it leads to mistakes.


What’s going on? 


While I’m at it, it would VERY useful to have some way of creating collections in bulk.

Cheers,
R

Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes

Posted 1 month ago

  • 1
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 5029 Posts
  • 1961 Reply Likes
> Putting together a book, for example, that has, say, 100 or 200 images; I create a collection for each image. I can then isolate all the ‘source’ images that may contribute in some way to the final result (the image that appears in the book).

Koro, can you explain a bit more about your workflow? I don't quite understand why you have a collection for each image. How does that help? Why do you need to do that?
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
Hello Victoria,

Thank you for your response.

In short: there are multiple source images for each single final image, i.e., the one that actually appears in the book. In virtually every case the photos are historical photos – what I refer to as ‘analogue’ or ‘chemical’ (as contrasted with digital) photos. For any given photo I might have chemical negatives, paper prints, printed versions, sometime from online sources, even microfilm, and, believe it or not, tintypes-the real thing in the original form. The quality is highly variable – a negative, for example, isn’t always the best option. They often enough turn up with, for example, thumbprints right over the subject’s face. There may multiple versions of identical photos with different resolutions, different print qualities, sometimes they are damaged. 

But there are many digitally-sourced images involved as well.

Out of all of that, I need to select the one that warrants spending more time bringing it up to the best possible output quality. I find it easiest to do this if they are all together in the same place.

Yes, I could devise a way to do this with keywords but my keyword structure already has ~2500 unique entries. But, yes, one of the advantages of doing it with keywords is that the keywords stay with the metadata whereas membership in a collection does not (serious trap for the uninitiated or unwary, BTW).

Essentially, I don't see keywords as an appropriate mechanism for categorising the end-use of an image. In my catalogue, keywords do: Who, What, When, & Where. Things like 'Fred's 3rd birthday' (which might have been at multiple Where) are done with collections. 

In this job what I need is not all the photos of Fred, but all versions of that single photo of Fred, you know, the one where he has chocolate cake in his hair taken at his 3rd birthday. And it may be that it has to be extracted from another photo – and so on.

Assembling the material in this way also allows me to bring together all the various bits that contribute to the final result required. For example, bringing together all the things required to build a genealogical chart (using Photoshop) that will appear as a single image in the final work. Or a time sequence of school photos: all the school photos of Fred from 1953 to 1965... .

This is a typical collection list of images (Figures, in this case) for a chapter in a book. BTW these collections are far from complete – it might indicate 1 or 2 but all of these collections will end up with 5 maybe 10 images sourced form anywhere in the catalogue and perhaps appearing in more than one collection.
 


When I have gone through them all the winner! – 1 in each collection -- get 5 stars. If I filter the parent collection set on star rating 5, all the images that will end up in the book are isolated. I can now put just those into another collection to prepare them for being exported. In that collection they are reordered so they are in the same order that they will be in the final book. Then they are exported (and renamed with a sequence number) directly to the folder that InDesign expects them to be in and they are linked into the final InDesign file.

I hope this answers your question.

Cheers,
Koro


Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
Your organization makes sense, thanks for explaining it.

Taking 10 seconds to create a collection is clearly a performance bug. There are other performance bugs with collections as well, adding and deleting photos: 
https://forums.adobe.com/message/9063934#9063934 .

Some workarounds to consider:

- Keywords. You said they weren't appropriate but I didn't understand the given reason: "Things like 'Fred's 3rd birthday' (which might have been at multiple Where) are done with collections." I use an Events parent keyword for things like birthdays and graduations, e.g. "John's birthday" for all of John's birthdays over the years. If I want the birthday for a particular year, I can easily combine keyword filtering with date filtering.  I also sometimes introduce additional top-level keywords for very specific "dimensions" of my data, e.g. the source of a photo.  If I don't want these keywords exported when I publish the photos on line, I uncheck the attribute Include On Export.

- Stacks. You could put all of the "analogues" in a single stack rather than a collection.

- The Title field (or User Comment, or Headline, or any field you're not currently using).  You can search these fields using the Library Filter bar or smart collections. The built-in features of LR don't let you see the complete list of unique Title values, as you can with collections or keywords, but you can use the Any Filter or Data Explorer plugin to see those values and all the photos with any given value, e.g.




Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
Good morning John,

Thank-you for your comments and suggestions. Keeping in mind that my question was about performance, I’ll swerve away from that to explain a little more about my approach.

I’ve looked at the link you included briefly – not yet sure what to make of it but I will have another look.

Keywords: As I wrote the sentence "Things like 'Fred's 3rd birthday' (which might have been at multiple Where) are done with collections." My mind was busy in the background crafting ways to do it with keywords and it was easy to come up with a sensible set of keywords to do the job: WHO | Fred, WHAT | Event | Birthday Party, WHERE | **Fred’s Place** (as a location), and WHEN | **Fred’s birthday** (i.e. as a disaggregated date).

My history with Lightroom goes back several years (really swerving away here...) to when I needed but could not find an important piece of paper. I scoured the office and the house from top to bottom and could not find it. I started using Lightroom to manage documents not photographs. Not exaggerating: I gathered together every piece of paper in office and house and I photographed them with my El Cheapo, V1 digital camera. I organised all that paper into NUMBERED (not named) file folders. I created a keyword structure to find the photos of documents (say, passports, insurance, etc). Then I constructed a hierarchy of collections which matched physical locations: Office | Grey Filing Cabinet | Drawer 2 | Folder 056. I put the photos of the docs that are held in F56 into that collection. If I want to find the house insurance policy, I use keywords. I can see the photo of the house policy and if I click on the little rectangle badge, it tells me what folder the object represented by the photo is in. Phew...  It works. I have refined Objects_X_Location quite a lot and it is very reliable.

During that time (my early days of LR use) I went back and forth between keywords and collections: which is best for rational categorisation? How can they work together so they don’t both do the same thing? It doesn’t make any sense (to me) to have a keyword WHO | Fred Blogs (which tags any photo that includes Fred) and a collection **All Photos of Fred**.  What does make sense is a collection “Fred’s 3rd birthday party” which is the set of all photos taken at that event some of which do not include Fred. 

To my way of thinking the specifics of this are in the grey area between collections and keywords. I could do keywords WHAT | Event | Birthday which is a sensible keyword. What isn’t a sensible keyword is WHAT | Event | 3rd Birthday and at that point I turn to collections.

Stacks: I DO NOT use stacks. In the early days of my LR use, stacks seemed like a great idea. But I had some very nasty results with them with things getting lost or things being inadvertently included in stacks where they didn't belong. That forced me into a lot of rebuilding (eliminating stacks and reorganising the mess they left behind) and I don’t need to tell you the atmosphere was blue in this office... . Managing them was nightmarish. Maybe they have improved but I am very wary of them.

As it stands most (all?) of the searchable text fields are already used for purposes not related to what I’m currently doing.

I have definitely moved on to using LR for its primary purpose: managing digital images (as contrasted with managing insurance policies...). But I’m still wed to the idea that there is a demarcation between keywords and collections and I try to keep the grey area between them to a minimum. Keywords are for finding things, collections are for managing groups of like.

All of this discussion is great but I’m still puzzled why LR get so quickly and so thoroughly bogged down when creating collections.

Thanks again for your suggestions.

Cheers,
Koro

Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
"I’m still puzzled why LR get so quickly and so thoroughly bogged down when creating collections."

It's clearly a bug, probably the use of an inappropriate algorithm whose performance only gets noticeably bad when there are very large numbers of collections. 
Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
My previous link explains how deletion of photos from a collection uses an inappropriate "n-squared" algorithm. With an n-squared algorithm, the time to perform an operation on a set of n elements is proportional to n^2 (n squared).  For small values of n, n^2 doesn't matter -- can you notice the difference between 0.001 and 0.01 seconds? But as n gets large, n^2 gets huge, and the user starts to notice. Deleting photos from a collection of size 1000 is more than fast enough for most users, while deleting photos from one of size 10,000 is painfully slow -- rather than just 10 times as slow (as you might expect), it's roughly 100 times as slow (10^2).

Most users have dozens of collections at most, usually no more than a couple hundred. You're out of the ordinary, having thousands or tens of thousands, tens or hundreds of times larger than the typical user. So the time needed to create a collection in your case is disproportionately large.

Technically, the bug here might have something other than n-squared behavior, but it's got the same characteristics -- the algorithm is fine with small numbers of collections but breaks down with large numbers.
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
Good morning John,

I've counted the Collection Sets (~1,500), Collections within those is about more-or-less doubles that; so altogether ~4,500 collection entries accumulated over 6-8 years of LR use. Most of these are devoted to the Object_X_Location system. I ask myself: 'Do I really need to have all of those?' The answer, of course, is "no" and maybe it's time to review my use of my own carefully crafted system, re-balance 'necessary' and 'nice to have'.

I could create a keyword solution but it's not clear that I even need that. Beyond that, creating and maintaining a KW hierarchy has its own set of challenges (which I'll explore if you're interested).

The challenge is testing. It would seem that I'm pressing up against the limits of the software; intellectually easy to understand, emotionally irksome.

As always, thank you for your thoughts.
R
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
Hello again John,

After backing everything up, I have deleted (in one hit) about 1,000 collection sets and their contained collections (~4,000 entries). Unquestionably the performance has improved but not as much as one might hope. For now please accept that at face value.

FYI: the delete required 83 seconds.

I now need to retreat and -- in the context of what I'm working on now -- look again at how the role of keywords might be extended further towards what I now expect of collections.

If you are interested in the result (assuming it departs significantly from the current approach), let me know and I'll contact you.

Cheers,
R
Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
Re stacks, LR does have needlessly complicated rules for stacks, especially with regards to collections and folders.  Adobe botched that one. But as long as you use stacking primarily with collections or primarily with folders and don't try to mix the two, large numbers of us find stacking quite useful and straightforward to work with.
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
I will have another look at it. It may have improved since I had trouble with it.
Photo of Todd Shaner

Todd Shaner, Champion

  • 1488 Posts
  • 501 Reply Likes
Have you tried Optimizing the catalog (File> Optimize Catalog)? Also try closing the Metadata panel or set it to 'Default.' That's a stretch as to the cause, but worth trying. Also close all open Collections folders except the one you're working in.
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
I have explicitly optimised (File > Optimize Catalog) the catalogue several times recently. I always make the collections views as compact as possible because LR's behaviour is to jump to the top when the 'add' operation is complete. It displays the same behaviour in the keyword panel. I don't understand the logic behind this behaviour but I take steps to minimise the effect of it -- which means collapsing things as much as possible.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
Re the Collections panel jumping when you create a new collection, you might add details of your issue to this existing bug report: 

https://feedback.photoshop.com/photoshop_family/topics/fix-dropdown-menu-position-when-duplicating-smart-collections-sets?topic-reply-list%5Bsettings%5D%5Bfilter_by%5D=all
Photo of John R. Ellis

John R. Ellis, Champion

  • 4531 Posts
  • 1213 Reply Likes
Re keywords: 

I agree one should have a clean demarcation between uses of collections and keywords -- otherwise lies insanity. 

I think it makes sense to let LR's actual capabilities guide you in the choice of when to use keywords versus collections.  I've closely examined LR's detailed capabilities, and there are three significant differences:  

- collections can impose a specific order on a set of photos;

- keywords can be stored persistently in photo metadata independent of the catalog;

- keywords admit of easier and more powerful searching.

There are a number of other less significant differences -- see here for details:  
https://feedback.photoshop.com/photoshop_family/topics/lightroom-classic-help-with-setting-up-collec...

Given this, many of us use a simple rule: Use collections to make groups of photos where order is important (e.g. for books, renaming with sequence numbers for export, publishing to an online service).  Use keywords for everything else.

The Quick Collection is a specific exception to this rule -- LR's user interface makes it very convenient to create temporary groupings of photos with the Quick Collection, even if the ordering of the photos doesn't matter.

You wrote:

"Keywords are for finding things, collections are for managing groups of like."

The more you think about this, the more that distinction evaporates.  A keyword applied to a photo expresses the relationship that the photo is or has the thing or property named by that keyword. Thus all photos with that keyword are a "group of like".

Conversely, you give the example, "a collection 'Fred’s 3rd birthday party' which is the set of all photos taken at that event some of which do not include Fred."  It's entirely conceivable you'll want to search for those photos at some point in the future.
Photo of Koro

Koro

  • 27 Posts
  • 6 Reply Likes
Good morning again John,

All true, all good points that underscore the recognition that keywords and collections are two facets of the fundamental purpose of LR: rationalise the process of managing large collections of images.

Cheers,
R