Lr Classic CC has no way to be sure if photos are in the catalog. zero.

  • 1
  • Problem
  • Updated 5 months ago
  • (Edited)
This post contains a video, for a full easy explanation of my situation. 
Video
https://www.youtube.com/watch?v=sThVzs0Q5pc&feature=youtu.be

The Issue in a Nutshell

You might say Import does not detect duplicate photos. So you have no way to point at various folders and be sure you imported that data. So I have no easy way to find out if my master catalog contains all my various drives (that I am sorting out) data.

I discovered this after doing a catalog import, I then did a standard import, not clicking import, pointing it at the photos folder of the catalog I just catalog imported, just to check if catalog import had worked. I selecting 'Don't import suspected duplicates', expecting to see zero or only a handful of orphaned files caused by incorrect deletion. It sees everything, every photo in the catalog! 

The main things I tried which I think relevant to say...

  • Since pointing it at the original /photos location on the other catalog drive did not work... I copied it to the same drive = no difference. (Some people talk about external drives causing issue, it ruled that out)
  • Pointed it at its own /photos directory = no new photos detected
  • Pointed it at its own /photos/folders I copied out to the desktop = new photos incorrectly detected.
  • Create a new empty catalog and did a catalog import of the mothership catalog, repeat = same result in all cases of above tests.
  • Try with one single folder of just 78 images = the same
  • Tried on another laptop with the same version of Adobe to rule out virus, bug, software, laptop, etc... = no difference, it's not my kit.
History
I started with a detailed Community Forum Post: https://forums.adobe.com/thread/2443613 and then since I discovered Adobe don't bother checking those, I then moved to a support case. Then I was asked to close my Adobe Case Number ADB-928182-W8X1 and post it here as a bug. After spending about 3 days working on this problem the lovely people at Adobe really touched my heart by giving me a one month credit for the Photography plan. How totally touching, moving, rewarding, understanding of them. No bug (or do they call it an enhancement request these days) provided, I feel like posting it here is a just a place for customers to vent, nothing more.

I think catalog systems are a terrible way to design software. I like how Exposure uses single tiny files in the images folder, as it means multi-users can also work on the same data together. Maybe it's time to move off Lr.

Version
I am using most up to data Lr and stuff, Lr Classic CC 7.1
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
  • angry I can't trust data security of Lr

Posted 5 months ago

  • 1
Photo of Robert Frost

Robert Frost

  • 392 Posts
  • 51 Reply Likes

Have you renamed all your files already on your computer?

Bob Frost

Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Hi Bob, I never rename or touch the RAW files before import from card, merge, or anything, I only deal with RAW. I let Lightroom import and manage them from the card. The only rename that is sometimes done is within Lightroom once a photo is ready to publish, at that point IMG_1234 might become a better clearer name for sharing and posting online. I rename in Lightroom using the Library tab and within the side panel. In terms of directory structure, again I never touch this except on occasion I might rename a few folders inside the Library folder tree within lightroom to have a more meaningful name on the end, e.g. YYYYMMDD turns to YYYYMMDD D.R. Congo. But when I compared data not seen as duplicate it was not a folder that had a meaning full name added, nor was it a folder that had any files renamed inside it, just 20170226.
(Edited)
Photo of Robert Frost

Robert Frost

  • 392 Posts
  • 51 Reply Likes

I seem to remember someone saying that LR compares the filenames of new files with those in its catalog. So if you have changed the name of the file in the catalog, it will think the same file with its original name is new. Should be easy to try. I may be wrong; my memory is not so good these days!


Bob Frost

Photo of Jim Wilde

Jim Wilde, Champion

  • 269 Posts
  • 82 Reply Likes
Yes, I guessed that would be the case. I finally got some memory cells working and recalled the issue that I had seen (and reported to Adobe) several years ago, and tested it again (in much the same way as you did) to verify my memory.

Basically, what happens is that, as you may know, Lightroom's duplication checking is based on three criteria....file-name, capture date, and file length. When you import images these three bits of information are stored in a field (called the Import Hash) within one of the catalog database tables....and it's that hash that's subsequently used to compare against new imports. 

Generally that all works well, except that when you do the "Import from Another Catalog" the contents of the Import Hash for all the images being imported from the other catalog are not brought in, i.e. after the import from the catalog, the Import Hash is blank for all the imported images. Thus when you subsequently attempt to import those same images from another source, the duplicates checking fails because there's no Import Hash to check against. I have a feeling there are some other instances as well where something similar can happen, but will definitely need to trawl through some old emails and/or forum posts to confirm that.

I hope that's at least explained what you have been seeing. I'll contact Adobe again about this.....it may be a bit of an edge case, but it's still something that should be avoidable.  
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Hi Jim, it's just mind blowing to think it's been an issue for years. My heart nearly dropped when you explained it like that. Many people have been told (like me) by Adobe support at times to Create a New Catalog and do a catalog import it, as sometimes they think a catalog file might be broken or some other reasons. So if just about everything has that hash empty, I am many others have got no way to know for sure if I have all the data without that function working, especially since I was just merging 3 catalogs. Which leads to the question, how can I trust Catalog Import? lol

I was considering capture One or Exposure for a long time, because of speed and for the RAW engine. since this is in such a mess and I spent so long testing, trouble shooting, maybe now is the time to jump. :( I have wasted a whole week on this, what with copying NAS testing unwinding things, logically checking things, writing down file sizes, names, data, waiting for copies or import to run. What a nightmare!

Don't Adobe not have a bug forum or public bug listing? I had no way to check if something had been an issue before. Support just wanted to close the ticket as fast as possible.

I worry that the life and energy has gone out of Classic CC.
(Edited)
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Jim, what is feedback.photoshop.com's affiliation with adobe? I had the strange experience of a support guy closing a ticket with this page left open asking me to log a ticket here, but if feels like a independent forum?
Photo of Jim Wilde

Jim Wilde, Champion

  • 269 Posts
  • 82 Reply Likes
This IS the official Adobe site for users to post bug reports/suggestions/feature requests, etc. It is managed and monitored by Adobe, and some team members are supposed to read all threads here. So technically you have formally reported an issue, I've added some details about how I think it occurs, so now it's up to Adobe to investigate and prioritise a fix. 

Regarding your earlier point about your concern that the life and energy has gone out of Classic, I don't really share that concern as yet. If it had I don't think they would be trying as hard (as they are reported to be) to fix those long-standing performance issues which have vexed many users over recent versions. So for me it's too early to say that the commitment to Classic has gone, let's see what happens throughout this year, I think by then we'll have a much better sense of their intentions regarding both Classic and LRCC.
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Thanks for clarifying. I just think it's strange they get the user base such as you and me to do all the hard work. Perhaps I am so frustrated I make the point too strong, but they clearly just divided their team on to two separate versions of Lr, and that's never good in software development. We can expect slower fixes and their attentions and efforts to be split :(  From what I read Lr cloud version has lots of features missing compared to Classic, so I bet that is the focus. I think the speed issue took them years to address, probably because behind the scenes they were working on this cloud version, which is more a consumer focused product aimed at Mac Photos and Google Photo type of users wanting something more in the cloud.

As a side note, Import Catalog also has bugs if you repoint a catalog import at one you already did, to check you go everything, it can't distinguish between the files that were previously imported as virtual copies with changes added, it thinks those are new again. I just can't trust it to manage data, I think I'll have to use Photo Mechanics and change my workflow, which I don't want to do but hey...

Now I am down a rabbit hole with third party plugins like the Teekesselchen open source duplicate locator. Comparing folder by folder for 5 years and 192,000 photos. Let's hope they still bother with Classic CC, I think this should be a priority and not a side bug, as it's about being confident in your data.

Most importantly, It would be nice to have a real bug database online, so users like me can search and find this if its already log. But I guess that would make them vulnerable to people seeing how long this has been an issue, and cause more user discontent. So they create a community forum and let third party people answer it. Sounds like a way for developers who want to avoid official responses to engage disguised as helpful community members, and take the heat off Adobe for such things.

Still, should look on the bright side, they extended my Photography plan by one month for free, worth 10 pounds. x
(Edited)
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Tested again and still present in 7.2
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
How does this forum work, is this site just a pool of discontent and a bucket of 'maybe we will work on this' forum? Do the hours and days of tests by the users get turned into a tracked bug with a reference no? Will we even get notified of a buy fix? I don't see how. Do I have to test with each version that keeps coming out again just hoping that someone resolved it. Do you have any freelance jobs in testing and QA going? ;)

ps. we all know the strategy behind the new non-classic version is to be able to make lots of money by charging reoccurring revenue for file storage, and take over the need for Dropbox, and increase profits. So Adobe's business focus has to be focused on bringing non-classic up to classics functionality, and things like this will be ignored.
(Edited)
Photo of john beardsworth

john beardsworth

  • 998 Posts
  • 219 Reply Likes
I wouldn't bother with conspiracy theories (cock up is always more likely anyway). But like any developer, Adobe evaluate bugs according to a range of factors and then prioritise them accordingly. When they are fixed, they are sometimes mentioned here, but I wouldn't bet on it.

Realistically, I wouldn't expect this bug to be fixed in the near or far future - it is pretty obscure and too rarely encountered. So you might find it more profitable to adapt your practices to ensure it doesn't continue to be a problem for you.
(Edited)
Photo of Andy Beales

Andy Beales

  • 13 Posts
  • 1 Reply Like
Hi john,

I just wish support don't tell people like me to 'just create a new empty catalog and do a catalog import' as this causes the issue, or tell people who use laptops and desktops to use a travel catalog on your laptop and merging it, or anyone merging catalogs from each year into one master, so I think it affects a lot of people but they don't know it, or don't both checking old cards and drives.
The bug totally destroys anyone's ability to check easily if a directory, memory card, or old drive is present in my catalog without manually checking contents and totals of photos. So for me it's about nobody should ever rely on Lr to check if all the data is in the catalog.

I am not a conspiracy theory, that's a bit strong. It's clear business strategy sense that the guys at Adobe would double revenue if they push people towards online storage fees, so hence CC has to be the focus more and more... which is why I guess things like this will not be fixed. Resources are now split on two different products.

I already have changed my workflow. I can no longer have a master 1 catalog to rule them all. Only way is 1 catalog per project. 
(Edited)
Photo of john beardsworth

john beardsworth

  • 998 Posts
  • 219 Reply Likes
My preference for a travel catalogue has changed over the years. I used to rely on the Import/Export Catalog roundtripping, and I am pretty sure I would have remembered if I had hit this bug. But since the introduction of smart previews I simply take my single master catalogue on the road, with as many smart previews as I expect to need. So no roundtripping, no fragmentation of control over my pictures and workflow.

I really don't think the new "Lightroom CC" has any effect on fixing this bug - it looks like it's been hanging around for years.