Lightroom: Give warning when opening backup (with Automatically Write Changes set)

  • 3
  • Idea
  • Updated 6 years ago
  • (Edited)
I recently opened a backup of my Lightroom catalog. It was only after I opened it that I realized that, because I have "Automatically write changes into XMP" set, it was silently overwriting metadata in the tens of thousands of images on my disk. Once I figured this out, I immediately closed the backup, opened the most recent copy of my catalog, and made sure that the data in that catalog was written out to my files.

But this is a disaster waiting to happen to anyone who opens a backup database.

Adobe Lightroom should save in all backup files a status bit noting that they are backup files, and when these files are opened, give the user a warning when "Automatically write changes into XMP" is set, allowing one to unset this value before any metadata is actually written out to the files.

Alternatively, Lightroom could remember which was the latest catalog opened, and any time that the user opens a catalog that was not the latest, in which "Automatically write changes into XMP" is set, warn the user that they might be changing values on disk that they didn't intend to change.
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
  • worried

Posted 6 years ago

  • 3
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4128 Posts
  • 1472 Reply Likes
Yep, it's a feature that I think is really needed too. I hadn't considered the write to xmp scenario, but I often see people accidentally open a backup and wonder where all their settings have gone.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
Are you sure LR was actually overwriting current settings in the XMP files with older settings from the backup catalog? Have you compared two versions of an XMP to see what has changed?

Maybe things don't work this way, but if there were XMP files already exiting from your work in the current LR database, I would have expected you to get a metadata conflict indication if the settings in the XMP were newer than the settings in the older LR database, instead of a silent overwrite with older settings, which I agree would note be safe.
--
If for some reason there were no XMP files at all for some photos, then the old catalog may have created some with its older settings, which would get overwritten with current settings whenever XMP files were written with your current catalog open, either automatically, or manually, depending on if auto-XMP-writing occurs when you're working in your current LR database.

If a backup/older LR database overwrites XMP files with older settings despite having newer settings existing from a newer LR database, then this is a problem, but it could still be handled by looking in each XMP file to see the date it was last updated comparing to the date the LR database settings were last written, and not doing any auto-XMP-writing if the LR database metadata was older than the data in the XMP file.

I suggest this per-file-using-dates metadata conflict detection, rather than on a catalog-wide basis, because things can be more complicated than just current and backup-copies. You might export a subset of photos for a client and that copy is not a backup of any catalog, but still a duplicate reservoir of image settings and would cause the same problem.

I agree that it should be safe to open a catalog with auto-XMP set and not overwrite newer settings written by another catalog without explicitly being instructed to do so and possibly being warned about metadata conflicts, I am just questioning if this is what happened, or if you're just assuming this is what happened after seeing evidence of file activity related to XMP files.
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
In reply to Steve, no I haven't tested it completely. The evidence I have is listed at http://forums.adobe.com/message/48307.... It would take a few hours of my time to figure this out, and right now I have Lightroom saving the changes from my latest database (an all-day process), because I don't trust Lightroom to always indicate whether there are metadata conflicts vis-a-vis my latest database. (How could it know whether there are conflicts for 10's of thousands of files without checking every one? How long does this take? Am I sure that Lightroom has finished checking, or is it still looking? There is no user feedback for this).

So I couldn't even begin to test this until Lightroom finishes its current process.

In reply to Victoria--thanks for the confirmation.

A
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
I was all set to provide a clever work-around by editing the .agprefs file, but alas: the setting obviously named to do the dirty deed (AgMetadata_autoWriteXmp) didn't do it.

I have no idea where the auto-write xmp pref is stored, but on my box, it's *NOT* in:

Lightroom 4 Preferences.agprefs

(which is where most Lr preferences are stored)

There is no other setting to account for it either - I did byte-by-byte comparison of prefs file saved with auto-write xmp on, & off.

??????????????? - maybe stupid mistake - "happy" to be corrected...

Rob
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
The Auto-write-XMP setting is catalog-specific (in the catalog settings), stored in the catalog file, so the one you're seeing in the Prefs file would be in effect for all catalogs, and must only be the default when creating a new catalog, right?
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Steve - I think you are right. But it still begs the question: where is the current pref stored?
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
In some database table in the .lrcat file along with any other catalog-specific settings. I'm sure it's not too hard to find in you're proficient in SQL.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Perhaps I'm not that proficient.

I was able to find the data which reflects the state of the setting, but I've not been successful at controlling the setting by changing it's value.

Table: Adobe_variablesTable
Cell: AgMetadata_autoWriteXmp

I was hoping for a simple work-around solution: turn auto-write xmp off before opening backup catalogs, but so far: no go.

Maybe one of the SQL heavies will chime in.

R
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
From reading the back-and-forth in the forum message you linked-to, above, the situation you describe could happen if having the old LR catalog open CREATED new XMP files for images that didn't have any, and then your current LR warned for those, which would be the correct behavior, but no settings would be silently overwritten in this case, because there were no XMP settings existing from adjusting the files in your current LR catalog.

It would only take a minute to validate if LR really has a problem or not, whenever you are finished updating XMP files for your entire catalog.

Victoria's reply is not specific enough to confirm anything, especially if XMP files had not been considered as the source of the percieved problem, meaning the situation being remembered has not been analyzed sufficiently to actually understand the source of the confusion.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Lightroom has forever been buggy in determining whether xmp has changed and should be written (and properly attending to problems writing it). In versions 1 & 2, Ctrl/Cmd-S would save *only* changed files, except sometimes it screwed up, and didn't save xmp when it should have. @Lr3, Ctrl/Cmd-S saves xmp for all selected files - problem solved (ha-ha). Some people like to argue that auto-write xmp is good and should be the default... - I say: "it's evil" - *never* enable auto-write xmp (see above), and you'll never have this kind of problem, or any other problems due to relinquishing control of your metadata to an incorrigible software.

Learn to save xmp, when it is time, and until then, don't save it, nor let Lightroom save it - that's my advice.

Feel free to disregard my advice if you think you know better - maybe you do...

Disclaimer: sorry if this seems over-dramatic, but I do feel strongly about my preference. Some people use auto-write xmp and have never had any problems with it (that they know about ;-}).

Cheers,
Rob
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
Steve

If you look at the linked post now, you will see a test that says that the situation is not as dire as I had thought. However, the reason I panicked was because (a) Lightroom said "I haven't finished writing metadata" when I tried to close the backup copy, and (b) Lightroom said "Metadata has been changed by an external program for some files" when I opened the original (latest) catalog. This was enough for me to believe that changes were being applied.

I'm still stuck with Lightroom writing changes (and will be for the rest of the day, I think).

Thanks for you thoughts.

Re Rob's comments, autowrite has always worked (pretty well) for me. I am forever changing metadata using a variety of programs, and I feel better if I know (suspect) that my metadata is always "currrent". And I don't understand your advice--do you think that Lightroom knows what files need to be updated (the metadata status is correct, but there is a problem writing the metadata?), or do you think that the problem at the heart is that Lightroom doesn't always know if metadata needs to be written? If the latter, then the only way to be sure that your changes are written is to Save metadata for your entire catalog, which is a day-long process for me. (I'm doing it right now, exactly because I don't trust Lightroom to know which files are correct and which are not). Data concurrency is a big issue, not just in Lightroom.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Hi Alan.

Honestly, I don't understand all the phenomena I've seen around metadata in Lightroom, w.r.t. saving metadata in xmp I mean.

But I've seen enough unexplained (to me) behavior that I do not trust Lr in this regard.

For example, I have files now that I save metadata for, and it seems to work, but then it thinks it needs to save it again (down arrow). What happens if autowrite-xmp is on? I dunno, since I leave it off, but my suspicion is that it would just keep saving it over-n-over-n-over again, since I have no good reason to believe the root of the problem is tied to manually saving.

What I know is:
* Lightroom doesn't always know (correctly) if it needs to save xmp metadata or not.
* When it does attempt to save xmp metadata, it doesn't always work correctly and/or it doesn't always know if it worked correctly, and in some cases anyway it thinks it needs to do it again shortly thereafter. And when these things occur - the user is not notified (other than by the thumbnail decoration).

Disclaimer: I use a lot of plugins which have background tasks constantly running, which can update the private metadata of a photo. These things are not saved in xmp, and therefore shouldn't cause a photo to be marked as dirty (unsaved changes), but as I said - there is some unexplained behavior...

Cheers,
Rob
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
Unlike Rob, I usually have auto-write-XMP turned on all the time, as I've not had a problem with it, but like Rob I also do an explicit writing of metadata to XMP files.

I always at least manually save metadata to XMP when I offload my oldest work to separate archive catalogs and remove the images from my working catalog. I do this so I have both a LR copy and an XMP copy of my settings.

My approach works well because I usually only modify images for a particular period of time, and then not after that. I could see things being more of a problem if I was more ad hoc about which images I was working on, and could never predict how old something might be that I was changing. For more ad hoc work I might always write XMP to files, just to avoid the warning when I exit.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 379 Reply Likes
Different strokes for different folks - that's why it's an option.

Personally, I rarely save xmp using Ctrl-S directly.

Instead, I lock my files after editing, which:
* saves xmp metadata
* compares saved xmp metadata to catalog
* makes xmp file read-only.
* assigns a custom label so locked files are easily identified.

Via this approach, I have achieved confidence in the xmp metadata that is saved. I realize this is not appropriate for everyone (requires a plugin...), but maybe Adobe will take the hint - it is my hope that Lr5 (or 6) will include an integrated locking mechanism, and also fortification/robustening of xmp/metadata handling.

Rob
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
Hi

I wish I could edit my original post to reflect my current understanding, but I can't on this forum....

Steve is right—further testing indicates Lightroom will not overwrite external metadata when you open a backup that has different metadata than the latest version of your database.

What must have happened to me is that I made a backup while Lightroom was in the process of changing external metadata in order to make it consistent with the values in the database. (I don't know this, but it is consistent with what I am seeing).

If you backup a database with pending changes, then LR will try to apply those pending changes upon opening the backup database. However, usually those pending changes will already have been applied when you opened your main database after making the backup (and gave LR enough time to apply the pending changes).

As LR tries to apply these pending changes that it finds in the backup database, it will discover for each file that the values in the database are inconsistent with the values on disk (because it thinks that the images still have the before-pending-changes values, but they really have the after-pending-changes values + any other changes applied subsequently). And when it discovers the inconsistency, it will not apply the pending changes, or any other changes, without manual intervention.

However, if you close your backup before LR has finished trying to apply the pending changes in the backup, you get a message that "Adobe Lightroom has not finished writing metadata changes into files..."). This caused me to panic, because I thought that LR was writing old values from the backup database into my files. However, all the message indicated was that it was trying to write these values, but not doing so whenever it found a conflict.

Thanks, Steve, for your level-headed response.

I still think that a warning message on opening a backup could be useful, but now I am not worried that opening a backup will cause irreparable harm.

Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
Just as a follow-up, it turns out that the above message comes up in situations different than I thought, completely contrary to the language used in the dialog.

I don't understand it, but if I make a backup database, then make some changes to the photos on the disk (I have "Automatically write changes into XMP" set, and then open the backup database, then when I try to close the backup database I always get the above warning message, even when I have made no changes to metadata using the backup database. Eventually, I can close the database and not have the dialog displayed, but it takes minutes to hours before that happens.

I think that this dialog is being displayed when Lightroom has not read the metadata from all files, but I wouldn't know how to test this hypothesis.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
What would the wording of the backup warning message say?
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
"You are opening a backup of a Lightroom database that may not reflect later changes made to your images. Be sure to synchronize folders and check the metadata status of files if you want to restore the backed-up values to your files."

This (a) helps people understand the relationship between a backed-up database and the current presence/absence/XMP values of files on disk, and (b) helps people like me be reassured that the backed-up values are not going to be written willy-nilly to the files. (I note that (a) LR gives no feedback when it is applying deferred changes, and (b) the metadata status flag often takes a long time to show the actual metadata status of images for large catalogs. So I think a warning message could be very helpful).

I would only show this message when opening a backup of a database in which "Automatically write changes into XMP" is set, as when this flag is not set the difference between values in the database and values on disk is clear (or should be) to the user.
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
PS, A long time ago I made the suggestion that Lightroom create a "task" (progress bar) for changes to metadata, just like it does for creating previews or changing develop settings. It is really much easier to understand what is going on, than to have the dialog above just show up when quitting the program. But that is another suggestion, which seems to have been rejected by the LR team.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
The first part of the proposed wording is reminding the user what a backup copy means and was not an issue in your situation--you knew you were opening a backup copy.

For the second part of the warning, "If you want to restore the backed up values to your files" means what? And is the context of the two suggested tasks of synchronizing folders and checking metadata status, when I am in the older backup copy or back in the newer current database? The message should be clear and concise and not generate support calls to Adobe wondering what they are supposed to do.
--
I wouldn't mind if a metadata progress bar or ETA timer placed somewhere in the UI so I know how long I have to wait for it to finish, because, as the warning message suggests, I would rather not wait for things to slowly get written out, next time.
Photo of Alan Harper

Alan Harper

  • 412 Posts
  • 82 Reply Likes
I'm sorry, are we having this discussion because it might result in Adobe listening to and accepting my suggestion, or is it just the two of us talking pointlessly? If the latter, we can agree to disagree. If the former, I'll put some effort into addressing your comments.

Not trying to be annoying, but not wanting to waste my time either
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2611 Posts
  • 333 Reply Likes
Adobe does read things over here.