Lightroom: Prevent loss of Edit Histories when Reimporting Photos

Edit history isn't in the XMP because plenty of people - me included - simply wouldn't want it there.

If catalogue corruption is the real problem, it's that problem that needs to be resolved. eg by the user using a backup (which contains edit history), or by Lightroom's backup being smaller (ie a transaction log allowing rollback/forward)

John

TK said: "Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help."

Whoa. Yeah, people act as if backups are a panacea, but that assumes your backups aren't wonky.

I've known that edit history is only in the catalog for a long time, so it wouldn't have been a surprise, but it was a surprise the first time... - I feel for ya.

So, I take it the catalog integrity check that I assume you were smart enough to perform before the backups (if not, I've heard confession is good for the soul...) was passing, despite the problem? I hope you've sent the funky catalog to Adobe for inspection(?)

Anyway, if the catalog is mostly OK, it may be possible to reconstruct the missing pieces using an SQL client. Or at least someone at Adobe maybe could, I'm not sure I know enough, or you...

One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue. As someone who has almost deliberately corrupted catalogues (eg by running SQL) I've found that importing into a clean catalogue can purge awkward problems. A nice bloke at Adobe sometimes helps fix serious cases of catalogue corruption too. Have you posted elsewhere about the corruption?

Overall, I am not blindly against saving history steps to XMP - though XMP does have the adjustment settings so you're only losing the "route". I do think there should be an option (haven't we had this conversation before?) to save history to XMP, but it should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.

On the other hand, I wouldn't want Adobe to see this as anything more than a very secondary level of backup.

John

TK,

John's suggestion may just fix you right up. If not, if you send me both catalogs + a list of missing info, I'll transfer what I can, which may be nothing...

R

Good news:
I discovered that the corrupted catalogue had only one corrupt entry. It was an image whose "root folder" was "nil".

I exported all photos in the corrupted catalogue -- except for the one problematic one -- to a new catalogue and, surprise, surprise -- the new one shows all the missing subfolders again!

So LR stumbled across one corrupt database row, causing it to drop a number of library folders from the display. I also noticed that LR became very slow with the corrupted catalogue in some operations (displaying "All Photographs").

Maybe (wild speculation) corrupted catalogues can sometimes be the source of some "performance" problems?

That is good news!

One time, simply optimizing my catalog improved performance by A LOT, meaning it went from buggy slow to normal.

Question (to the staff): Should I post this as a feature request? It seems it is not really a bug because the loss of the edit histories is "as designed". But it seems that there might be cases when saving editing histories in or with photo files would be desirable.

It seems that I cannot convert this bug report into a feature request anymore, so should I create a new one?

Personally, I'd like anything that can be stored in XMP to be stored there. I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup). Basically, the catalogs seems fragile to me and the XMP data is not so I rely on the far more robust XMP backup strategy instead of the fragile catalog backup strategy. The bad news is that I can't use many of LR's features including collections, stacks, VCs and flags because they aren't stored in the XMP data. Personally, losing the Develop history is the least important to me as I rarely use it for anything anyway.

Wish I could work with one hand tied behind my back too....

In any case, LR catalogues are impressively robust and we have very few reports of corruption. That's why they are worth backing up.

Two thoughts on this: I've generally been of the opinion that significant metadata should (eventually) find its way into XMP since XMP can be used as a kind of distributed catalog backup. The main reason I could see not to have history in the XMP is that it'd be too big, but if it were stored smartly as a series of deltas against the final settings, it's should be fairly small in most cases.

Which reminds me of another mini-feature I've wanted for the history panel: a "minimize" option that elides multiple (even non-contiguous) adjustments of the same slider into a single step. It would turn a messy history list into a simplified summary of all the actions taken to get from original to final.

The reason that's possibly relevant here is that minimize could probably be implemented in such a way that it could compare the default settings with the current settings which would mean an image imported without history would get a concise summary of the changes. Not as complete as the original, but it provides at least some of the utility.

@John I think we're confusing two things here.

The minimize feature would definitely lose the complete path from start to finish as that isn't the intention. For me, I never (ever) care about the whole path. I care about a summary of the set of adjustments I made (that is, what path would I have taken if I'd known where I was going and made no mistakes). That's a distinct way to view history.

On the other hand for storing complete history in XMP, even if you store the deltas, you can reconstitute any step along the way (source control systems use this trick for efficient storage). It is undiluted, just very smartly compressed.

As an example of the reason for using deltas, I once was sent a corrupted catalog that was 7 GB. The size was dominated by the data for a small handful of photos. The history for the largest of them was was 450 MB (it had been heavily retouched with localized corrections). As an experiment, I blasted that data (history step by history step) into a Mercurial (source control) repository and committed each step. The repo at completion compressed to just under 700KB (and was only a couple of MB total, only about half again or maybe double the size of the XMP file containing only the final settings which were about 1.6MB).

All that is to say, while there are reasons not to store history in XMP, size is not one of them unless the implementation is naive.

As you know this started as a bug report. If I could edit my original post, I'd update the text to read as a feature request. Maybe a staff member could replace the text of the first post with the following?

=== Feature request ===
LR catalogues store full edit histories. However, when changes are saved to image files (DNG, RAW+XMP, JPG) only a set of instructions to recreate the final rendering is stored, all of the edit history is lost.

When importing such image files one only gets a "Imported..." entry, instead of the full list of edits.

I suggest that users have the following options regarding data stored in image files:
a) no history stored (current situation)
b) essential history stored
c) full history stored

An "essential history" only contains those steps that are necessary to create the final rendering. For instance, all "back and forth" settings to a single parameter are replaced by a single parameter change. It would actually be nice if users were given the option to reduce the in-catalogue histories to "essential histories" as well.

The motivation for having histories outside catalogues is twofold:
1. It would simplify backup strategies in that one would only need to back up image folders to retain final renderings with their edit histories.

2. It would create a safety net against catalogue corruption. If a catalogue develops a problem one may not immediately notice and thus create multiple backups with the same problem. Instead of being required to rebuild a clean catalogue from (potentially very old) clean backups, one should have the option to just create a new catalogue from the image files.

The size of histories stored outside of catalogues could be addressed by efficient storage schemes (e.g,. binary compression).

Just for giggles, I compressed a LR catalogue with WinZip. A 56.3 MB catalogue was reduced to 5.86 MB. That's just a fraction over 10% of the original size.

I though it was worth a new feature request to suggest that catalogues are stored in compressed form.

If the purpose is just for backup/transfer/security, perhaps I could throw another alternative into the net - how would it work for people if there was a LR-specific sidecar option, which could contain absolutely everything including History, flags, etc. Something like XMP sidecars, but without interfering with the XMP spec and without having to worry about it breaking other programs that use XMP.

Dan,

The default should be Off, just as it is with Photoshop, which has had the ability to save history to XMP and/or a text file for years. Actually, take a look at how it done in Photoshop.

Also, be warned, set default to On and there will be h.ll to play!