Lightroom Classic: Merged directories using Finder (macOS Mojave) - how do I update Lightroom database?

  • 1
  • Question
  • Updated 6 months ago
  • (Edited)
Hi everybody, and Hapoy New Year! I have a question. I recently upgraded to an iMac Pro. I started from scratch, versus using OS-X’s migrate mechanism. I decided that this would also be a good time to properly organize my disparate directories (some already within Lightroom, and some not) into a directory structure already employed by Lightroom. Instead of merging within Lightroom, I did so in finder, figuring that Lightroom would be smart enough to update on its own. It did not do so, at least not that I can see. So, how do I instruct Lightroom to rescan everything, convert new photos in a given directory to DNG if required, etc. ? Thank you, in advance, for your guidance. :)
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like

Posted 6 months ago

  • 1
Photo of Johan Elzenga

Johan Elzenga, Champion

  • 2011 Posts
  • 826 Reply Likes
You should have informed yourself rather than 'figuring that Lightroom would be smart enough to update on its own'. Then you would have found that this is not the case and so you should have used Lightroom to do the reorganisation. Do you still have the old Mac with the old directories as they are in the Lightroom catalog? If so, it's probably easier to replace your new directory structure with the old one, and then start again, this time doing it by using Lightroom.

Lightroom can 'reconnect' to missing images and/or missing folders (https://www.lightroomqueen.com/lightroom-photos-missing-fix/), but if you changed the folder hierarchy and moved images around, then that may be a very time consuming task.
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
You are right, I should have informed myself. Thank you.
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
Oh, and yes, I do have my old iMac, and I do even have RAID (mirrored) backups of the images on there. I also have backups of my external drives (which hold a portion of the photos, and all of the combined directories), but the server I back them up on is currently down, so I won't have access to it for a while, until I figure that out. No issue except I cannot connect. I'll solve that pretty soon though. Anyway, so there is nothing lost, just significant inconvenience. Thank you again. :)
(Edited)
Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
Lightroom is ignorant of any file operations performed outside of Lightroom.  This is by design as Lightroom is a database application—albeits, constructed on a poor man's database—and not a file management system or operating system.

This is made pretty clear in virtually any manual or Lightroom book published by third parties.

However, Lightroom is also notorious when it comes to file handling as it does not message the operating system to perform the file handling; it does it itself.  This might sound wonderful as Johan implies, but Lightroom bypasses the protections inherent in the operating system.  For example, Lightroom does not verify the file is in the destination folder before deleting the file in the source folder.  The operating always does this verification.

The problem here is—and many, including myself have experienced this—that if Lightroom hangs, and only briefly, there is a good chance that your images go into the ether never to be seen again (unless you have a backup).

From my experience, Lightroom can hang for any number of reasons; e.g., competing for RAM with other applications, competing for CPU time with other applications, issues with the GPU, dogs barking, and the earth rotating on its axis.

Never trust Lightroom to perform any file operations.  Do it outside of Lightroom under the inherent protections of the operating system and follow Victoria's brillant solutions in the link Johan provided above.  (Lightroom is pretty good at finding images if you can point it to the correct directories.)
(Edited)
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
I have this lack of trust of Lightroom, although this was based more on suspicion and observation than research, as you clearly have done. Thank you.
Photo of Paul Grant

Paul Grant

  • 49 Posts
  • 17 Reply Likes
As stated previously make all changes within Lightroom.   That beng said there are some simple solutions to get all your content back in order.   Not knowing your old and new file strucure its hard to give you specifics.   I would suggest you Remove all the folders from Lightroom.    Just control click on the folder in the libary view.   Then remove the folder and all the contents.    You will get a dialog to delete from Libary or Hard Disk.   Do not delete from Hard Disk.    Once all folders are removed from Lightroom the Import and Add your new Top Level Folder.   As part of the Dialog Box say ADD.   Not Copy or Move.    All your files and folder structures should now be in place.
Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
And what happens to his edits?
Photo of Paul Grant

Paul Grant

  • 49 Posts
  • 17 Reply Likes
That totally depends on what options were set in prefernces.   This is why I always leave "Write Metadata to XMP" on all the time.  Some say shut if off to speed up performance.   I have had too many issues in the past and I have learned my lesson.   The alternativer to my above approach is simple.  Select ALL Photos in the current Libray and do a "Command + S" to "Save all Metadata" to the xmp files. 
Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
Yep.  Probably should have mentioned that in your suggestion as we're dealing with a newbie.  No sense in inadvertently causing him more grief than he already has to deal with.

I too keep 'Write Metadata to XMP' on all the time.  I've never experienced a performance lag because of that, just the slight hassle of wanting to quit LR and finding out it is still writing metadata.  (Given that I have little trust in Lightroom to remeber anything, I usually choose to not shutdown until it is done.  Wish there was an activity indicator for this.)

Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
By default, the "Write Metadata to XMP" is disabled. However, I've just enabled it, thanks to your suggestion, and will keep it enabled moving forward. I'm guessing minimal performance degradation with iMac Pro's SSD, and even if the degradation is noticeable, I'm not a pro, this is all just for fun, so I have no issue waiting those extra few seconds.  For my past error, I'll write existing information to XMP, as Paul has suggested. Thank you, I really appreciate the expert help!
Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
Yes.  Adobe does pick the damnedest things to select (or not) as defaults in its perferences.  Makes one wonder if any of the developers are actually working photographers.

Like I stated above, I've never noticed any performance lag on either my MacBook Pro (2013) with 16GB or my MacPro with 128GB RAM with xmp writing selected.  The 'performance' issue is that Lightroom is not especially preppy on writing xmp files and after a big imort with a lot of keywording—never mind develop work—it is still writing xmp files when you want to quit.

Lightroom claims it will pick up where it left off, but I have seen folders with raw files and missing xmp files AFTER a restart and new session has completed.

I simply do not believe anything Adobe says about Lightroom and data integrity when it is based upon SQLite which is an open source freebie database with poor referential integrity.
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
Wow, it’s surprising that Adobe is not employing a more robust solution.

My computer is an iMac Pro, 10-core, 4TB SSD, 64GB RAM, 16GB VRAM. I’m surprised that Premiere Pro and After Effects only use one core each, and assume that additional cores aren’t in any way leveraged by Lightroom either. I just purchased and started using an Atamos Ninja V 4K monitor, which also does ProRes and ProRes Raw recording. Employment of ProRes makes Premiere Pro scrubbing stutter-free (not having to deal with AVHCD or .264 codecs).
Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
People have been complaining for years about Adobe's inability to take advantage of multiple cores beyond basic baby steps.  Heck, Lightroom cannot even use multiple GPUs!

Now, some will tell you that Lightroom can use multiple cores, and they base this claim on the fact that one can see 100% - 200%plus CPU utilization on some utlities (iStat, for example).  In other words, if Lightroom is using 240% CPU utilization, then by fiat it must be running on three (3) cores.

Uh huh.  I must have been sick that day during my graduate course on Symmetric Mutli-Processing (SMP) at the University of Minnesota.

Regardless, if this inference is true, there is an inherent loss of perfomance in SMP using buses or crossbar switches is the bandwidth and power consumption of the interconnect among the various processors, the memory, and the disk arrays.  Mesh architectures avoid these bottlenecks, but there are serious programming challenges  with this kind of architecture because it requires two distinct modes of programming; one for the CPUs themselves and one for the interconnect between the CPUs.

Given that Adobe cannot even maintain good referential integrity of the database, I highly doubt the ability of their programmers to make effective use of today's processers.  I won't even speak to the lack of financial committment on Adobe's part to effectively implment SMP.
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
Yes, I think you're right about LR not using multiple cores. You'd think that if Premier Pro and After Effects are unable to leverage multiple cores, they wouldn't employ them on the possibly generally less processor-intensive (I could be wrong) Lightroom operations. And yes, we have a client that does this type of work, and your right that you cannot just click a checkbox and have the application magically employ multiple cores. Those who say that the processors go about 100% are right though, and I wondered the very same thing 15 minutes ago. I did as Paul Grant suggested, and exported to XMP, then deleted all my catalog items, and am currently re-importing them. After this, I think I can select all non-DNG files, go to the Library menu, and convert from their existing format to *.DNG. Anyway, we'll see...

Here's a screen cap of my current processor load. Perhaps the >100% load means that the processor is backlogged to 234%, but only outputting shy of 100% of one core effort.

Photo of dmeephd

dmeephd

  • 300 Posts
  • 73 Reply Likes
Good luck.  Since 7.5, LR has had a nasty habit of producing duplicate and triplicate dng files—not consistently, but true nonetheless.  Before LR 8, if you attempted to remove one of the duplicates from the catalog, they all went bye-bye.  After LR 8, only the 'ghost' dngs are removed.

Physically, on the hard drive, there were never any dupes.  These are ghosts.  However, it is a PITA to waste time removing them.  Their creation seems to be more prone when converting jpgs to dngs.  No idea why.

Performance has always been an issue with Lightroom, and I really wish Adobe would spend some time and effort to effectively take advantages of todays processors.  I mean really, if AutoDesk can do it for AutoCAD on the Mac, then there is no reason why Adobe cannot.

Instead, Adobe keeps tossing out useless features for the iphone crowd and neglects those users who earn a living from photography.
Photo of David Arthurs

David Arthurs

  • 10 Posts
  • 1 Reply Like
Everything worked perfectly, and I fully retained my new directory structure. There was one situation where the *.mov files could not be read, ostensibly due to a dynamicmediaserver issue, so I quit Lightroom, force quit dynamicmediaserver, and all was well after that. 86,079 happy files now. Thanks again to you Paul Grant and dmeephd for your constructive help. This newbie learned a ton about several aspects of Lightroom over the past few hours! :)

There was actually one issue, and that was that I could not see thumbnails for all my DNG files, nor could I get any QuickLook views. I have implemented a solution over the past few minutes, which works perfectly, effected by an add-on module called Ardfry.com's "DNG Suite". The software is offered through an OS-X preference pane, and provides all functionality I was targeting. $10 is the cost, which I intend to pay after the 15-day trial period expires.
(Edited)