Skip to main content
Adobe Photoshop Family

513 Messages

 • 

11.1K Points

Thu, Apr 7, 2011 5:33 AM

6

Lightroom: Prevent loss of Edit Histories when Reimporting Photos

When importing DNGs with stored edits (included XMP data) then the history of the photo just shows "Imported..." instead of the list of edits.

I have a corrupt catalogue. (I did nothing to cause the correction :()
The catalogue contains photos which are not associated to folders in the library module. When I choose "Got to folder in Library module" from the context menu for such photos, nothing happens. I imported them just like any other photos, but somehow the corresponding library folder wasn't created or lost.

I tried synchroning the parent folder but the missing subfolders are not created again.

That's why I decided the only way forward is to create a new catalogue. However, the new catalogue doesn't have any of the edit history. The rendering is OK and I can reset it to see the original version of the photos but I cannot see the edit history anymore.

Why is the edit history not recreated? The essence of it must be available because otherwise the correct final rendering could not be created.

I believe edit histories should be available for JPGs, RAW and DNG files. When I decided to use DNG files vs RAW files with sidecar (XMP) files, I didn't know that I'd lose the history with a fresh import of a DNG file. I suppose that if I had XMP files, I could copy these and still had my edit histories.

Responses

1.3K Messages

 • 

22.5K Points

10 years ago

Edit history isn't in the XMP because plenty of people - me included - simply wouldn't want it there.

If catalogue corruption is the real problem, it's that problem that needs to be resolved. eg by the user using a backup (which contains edit history), or by Lightroom's backup being smaller (ie a transaction log allowing rollback/forward)

John

513 Messages

 • 

11.1K Points

I see your point and concede that if the catalogue corruption would not have occurred, I would not have reported the lack of edit histories. However, now with a corrupt catalogue, I'm left between a rock and a hard place. Either I use the corrupt catalogue and cannot access some of the images in it properly, or lose all edit histories.

Note that some people may prefer to have all information that is needed to recreate a LR catalogue in DNG or RAW+XMP files. That would alleviate the problem of having to backup catalogues separately.

Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help. I guess LR never added the library folders properly (just added the photos to the catalogue as orphans) and hence there is no "good" version. The only option would be to go back to an intact but incomplete catalogue and try to build it up again. I'd rather prefer a synchronise or repair operation to succeed.

4.5K Messages

 • 

76.3K Points

10 years ago

TK said: "Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help."

Whoa. Yeah, people act as if backups are a panacea, but that assumes your backups aren't wonky.

I've known that edit history is only in the catalog for a long time, so it wouldn't have been a surprise, but it was a surprise the first time... - I feel for ya.

So, I take it the catalog integrity check that I assume you were smart enough to perform before the backups (if not, I've heard confession is good for the soul...) was passing, despite the problem? I hope you've sent the funky catalog to Adobe for inspection(?)

Anyway, if the catalog is mostly OK, it may be possible to reconstruct the missing pieces using an SQL client. Or at least someone at Adobe maybe could, I'm not sure I know enough, or you...

513 Messages

 • 

11.1K Points

Unfortunately, the corrupt catalogue passed and passes the integrity test. If it is just an SQLite integrity check then it wouldn't mind about orphan photos (photos who lost their direct folder association), would it?

I'd be happy to send the catalogue to Adobe but won't do it unsolicited. I seem to remember some bug like this being discussed in the U2U forums.
IIRC, such corruption can occur when you move folders within LR. Seems that moving them with the OS and then finding them again in LR is safer. But the latter issue could be a separate one to what I was experiencing because some of the missing folders were never moved; just imported "into a subfolder" as I always do.

4.5K Messages

 • 

76.3K Points

I dont know what all the integrity check does, but I now know one thing it does not do. Do you know any SQL, TK?

1.3K Messages

 • 

22.5K Points

10 years ago

One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue. As someone who has almost deliberately corrupted catalogues (eg by running SQL) I've found that importing into a clean catalogue can purge awkward problems. A nice bloke at Adobe sometimes helps fix serious cases of catalogue corruption too. Have you posted elsewhere about the corruption?

Overall, I am not blindly against saving history steps to XMP - though XMP does have the adjustment settings so you're only losing the "route". I do think there should be an option (haven't we had this conversation before?) to save history to XMP, but it should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.

On the other hand, I wouldn't want Adobe to see this as anything more than a very secondary level of backup.

John

513 Messages

 • 

11.1K Points

"One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue.": Thanks a lot for this suggestion. I thought about doing that but wasn't sure what end result to expect from it. I'll try it. Thanks again.

The only other place I reported about the corruption is when I suggest that a folder synchronise should bring back / restore missing subfolders.

I agree that having options regarding what is saved in XMP files / DNG metadata would be desirable.

164 Messages

 • 

5.2K Points

"File > Import as Catalog" isn't panacea either. I've done this before to try to fix a catalog and only realized much later that I had lost many of my collections and web galleries. That one still stings...

4.5K Messages

 • 

76.3K Points

10 years ago

TK,

John's suggestion may just fix you right up. If not, if you send me both catalogs + a list of missing info, I'll transfer what I can, which may be nothing...

R

513 Messages

 • 

11.1K Points

Rob, thanks a lot for the offer! I'll see what I can do myself first. I hope I won't have to bother you with this.

4.5K Messages

 • 

76.3K Points

OK. The Lr database is simple and straight-forward in some respects, and a bit complicated and intimidating in other respects. Some things may be readily transferable, other things - not so much...

513 Messages

 • 

11.1K Points

10 years ago

Good news:
I discovered that the corrupted catalogue had only one corrupt entry. It was an image whose "root folder" was "nil".

I exported all photos in the corrupted catalogue -- except for the one problematic one -- to a new catalogue and, surprise, surprise -- the new one shows all the missing subfolders again!

So LR stumbled across one corrupt database row, causing it to drop a number of library folders from the display. I also noticed that LR became very slow with the corrupted catalogue in some operations (displaying "All Photographs").

Maybe (wild speculation) corrupted catalogues can sometimes be the source of some "performance" problems?

11 Messages

 • 

270 Points

Thanks for the update. Glad to hear things are working again!

4.5K Messages

 • 

76.3K Points

10 years ago

That is good news!

One time, simply optimizing my catalog improved performance by A LOT, meaning it went from buggy slow to normal.

513 Messages

 • 

11.1K Points

10 years ago

Question (to the staff): Should I post this as a feature request? It seems it is not really a bug because the loss of the edit histories is "as designed". But it seems that there might be cases when saving editing histories in or with photo files would be desirable.

It seems that I cannot convert this bug report into a feature request anymore, so should I create a new one?

11 Messages

 • 

270 Points

It should be showing as an "idea" now.

513 Messages

 • 

11.1K Points

Yes, it does. Thanks Chad! It doesn't really read like a feature request, though. If it is clear what I'm after then that's OK with me. I'd be happy to rephrase the original bug report as a feature request, if this would be useful.

946 Messages

 • 

13.8K Points

10 years ago

Personally, I'd like anything that can be stored in XMP to be stored there. I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup). Basically, the catalogs seems fragile to me and the XMP data is not so I rely on the far more robust XMP backup strategy instead of the fragile catalog backup strategy. The bad news is that I can't use many of LR's features including collections, stacks, VCs and flags because they aren't stored in the XMP data. Personally, losing the Develop history is the least important to me as I rarely use it for anything anyway.

513 Messages

 • 

11.1K Points

Lee, I perhaps wouldn't call the catalogues "fragile" but I agree with your overall sentiment. I think you should press the "like" button for the feature request. :)
Still not sure whether the former bug reports works as a feature request. Shall I create a new one?

11 Messages

 • 

270 Points

Feel free to rephrase this as a feature request in a new thread if you'd like to make it a bit more concise.

513 Messages

 • 

11.1K Points

I think I waited a bit too long to create a new thread. There are so many contributions in here that it would be suboptimal to start afresh. However, maybe my first post could be edited to feature the rewritten feature request I posted here as a comment.

1.3K Messages

 • 

22.5K Points

10 years ago

Wish I could work with one hand tied behind my back too....

In any case, LR catalogues are impressively robust and we have very few reports of corruption. That's why they are worth backing up.

513 Messages

 • 

11.1K Points

John, I had quite a number of backups of the catalogue but they didn't help one bit in this instance because the error was a silent one and I didn't discover it before several generations of corrupt backups. Going back multiple generations to one that is still clean means that a lot of work has to be redone. This is not optimal.

Employee

 • 

166 Messages

 • 

3K Points

10 years ago

Two thoughts on this: I've generally been of the opinion that significant metadata should (eventually) find its way into XMP since XMP can be used as a kind of distributed catalog backup. The main reason I could see not to have history in the XMP is that it'd be too big, but if it were stored smartly as a series of deltas against the final settings, it's should be fairly small in most cases.

Which reminds me of another mini-feature I've wanted for the history panel: a "minimize" option that elides multiple (even non-contiguous) adjustments of the same slider into a single step. It would turn a messy history list into a simplified summary of all the actions taken to get from original to final.

The reason that's possibly relevant here is that minimize could probably be implemented in such a way that it could compare the default settings with the current settings which would mean an image imported without history would get a concise summary of the changes. Not as complete as the original, but it provides at least some of the utility.

946 Messages

 • 

13.8K Points

Yes, Dan, I'll take both of those with a side of fries, please. ;-)

1.3K Messages

 • 

22.5K Points

If one's to have an *option* to write the edit history in the XMP, Dan, I'd rather have it undiluted. History's about how one got to the final result, and restoring only the differences would mean people wouldn't be able to go back to a certain stage of work or use it for Before/After comparison. How could one get that value from the differences from default?

And you don't even need to store the differences in the XMP - LR could already compute them from the adjustment values already in the XMP. So again, where's the value?

946 Messages

 • 

13.8K Points

"And you don't even need to store the differences in the XMP - LR could already compute them from the adjustment values already in the XMP."

That's not true, John. Remember, due to the Camera Raw defaults and the application of presets, the starting point isn't always fixed.

1.3K Messages

 • 

22.5K Points

Application of presets? You mean upon import? I'd say they form part of the difference. But that's arguable. Some will want them in, some out, and as there's not a lot of value from having simplified history....

946 Messages

 • 

13.8K Points

We have a simplified history already. It just doesn't look like it. The numbers are cumulative so they aren't quite the same as those in the PS history palette.

Employee

 • 

166 Messages

 • 

3K Points

10 years ago

@John I think we're confusing two things here.

The minimize feature would definitely lose the complete path from start to finish as that isn't the intention. For me, I never (ever) care about the whole path. I care about a summary of the set of adjustments I made (that is, what path would I have taken if I'd known where I was going and made no mistakes). That's a distinct way to view history.

On the other hand for storing complete history in XMP, even if you store the deltas, you can reconstitute any step along the way (source control systems use this trick for efficient storage). It is undiluted, just very smartly compressed.

As an example of the reason for using deltas, I once was sent a corrupted catalog that was 7 GB. The size was dominated by the data for a small handful of photos. The history for the largest of them was was 450 MB (it had been heavily retouched with localized corrections). As an experiment, I blasted that data (history step by history step) into a Mercurial (source control) repository and committed each step. The repo at completion compressed to just under 700KB (and was only a couple of MB total, only about half again or maybe double the size of the XMP file containing only the final settings which were about 1.6MB).

All that is to say, while there are reasons not to store history in XMP, size is not one of them unless the implementation is naive.

513 Messages

 • 

11.1K Points

"I care about a summary of the set of adjustments I made": Yes, I do too. I'd call that "essential history". It would be nice if one could compress/minimise one's histories into essential histories.

Regarding your Mercurial experiment: Certainly version control systems can store the latest version of a document efficiently as a delta against a base. However, given that you need the base as well, I reckon that most of the compression you saw (the couple of MB total vs the original 450MB) came from a compressed (à la .zip files) storage.

142 Messages

 • 

3.7K Points

FWIW, I've first logged that "collapse history" function as a feature request in the 1.x days. So count me in! I don't think it needs to be anything fancy UI-wise, a menu item would do the trick. Needs to work on all selected photos.

513 Messages

 • 

11.1K Points

10 years ago

As you know this started as a bug report. If I could edit my original post, I'd update the text to read as a feature request. Maybe a staff member could replace the text of the first post with the following?

=== Feature request ===
LR catalogues store full edit histories. However, when changes are saved to image files (DNG, RAW+XMP, JPG) only a set of instructions to recreate the final rendering is stored, all of the edit history is lost.

When importing such image files one only gets a "Imported..." entry, instead of the full list of edits.

I suggest that users have the following options regarding data stored in image files:
a) no history stored (current situation)
b) essential history stored
c) full history stored

An "essential history" only contains those steps that are necessary to create the final rendering. For instance, all "back and forth" settings to a single parameter are replaced by a single parameter change. It would actually be nice if users were given the option to reduce the in-catalogue histories to "essential histories" as well.

The motivation for having histories outside catalogues is twofold:
1. It would simplify backup strategies in that one would only need to back up image folders to retain final renderings with their edit histories.

2. It would create a safety net against catalogue corruption. If a catalogue develops a problem one may not immediately notice and thus create multiple backups with the same problem. Instead of being required to rebuild a clean catalogue from (potentially very old) clean backups, one should have the option to just create a new catalogue from the image files.

The size of histories stored outside of catalogues could be addressed by efficient storage schemes (e.g,. binary compression).

513 Messages

 • 

11.1K Points

10 years ago

Just for giggles, I compressed a LR catalogue with WinZip. A 56.3 MB catalogue was reduced to 5.86 MB. That's just a fraction over 10% of the original size.

I though it was worth a new feature request to suggest that catalogues are stored in compressed form.

Employee

 • 

166 Messages

 • 

3K Points

Surprisingly, no. Zip compression of the 450 MB develop history only reduced it to a third of its size or so (IIRC), which is beaten by a factor of nearly 100 by deltas. Local corrections develop history compresses really badly with the zip algorithm. A RAR based compressor did much better (making the whole 7 GB catalog only 130 MB), but was MUCH slower to perform.

The 10:1 compression ratio comes (I think) from the metadata table's XMP content, which has its own efficiency issues.

513 Messages

 • 

11.1K Points

Thanks a lot for your comments, Dan. It is very good to see that you have looked into compression options already at some point.

It seems that there is little contention about compressing (older) backups. I'll do that manually for now, but it would be nice if some automation would be available in the future. The latter could include moving backups from a (quick working) drive to another (archival) drive once backups have reached a certain age.

Champion

 • 

5.8K Messages

 • 

102.5K Points

10 years ago

If the purpose is just for backup/transfer/security, perhaps I could throw another alternative into the net - how would it work for people if there was a LR-specific sidecar option, which could contain absolutely everything including History, flags, etc. Something like XMP sidecars, but without interfering with the XMP spec and without having to worry about it breaking other programs that use XMP.

1.3K Messages

 • 

22.5K Points

Not another format, another type of sidecar!

TBH storing extra fields would not "interfere with the XMP spec" or trouble other programs. It's the X in XMP - extensible by you, me or Adobe. Adding extra fields is what it's all about, and other programs simply pass over fields which they aren't set up to read. OTOH a new format or sidecar would make it more complicated to update other programs as they'd have to have logic for a new file type and data format, rather than just being told to read a new XMP field.

4.5K Messages

 • 

76.3K Points

Would it really be necessary to invent a new file format, just to add a few blocks to the xmp?

Champion

 • 

5.8K Messages

 • 

102.5K Points

But John, you just said you didn't want that extra data in XMP....

Putting it in the normal XMP files would certainly be my first choice.

1.3K Messages

 • 

22.5K Points

I've said I don't see much value in Dan's summarised history, but I did say "I do think there should be an option... to save history to XMP, but it [the option] should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.". I'd probably switch off the history option though, and think off should be the default.

Employee

 • 

166 Messages

 • 

3K Points

If we could ensure the storage design was efficient enough to not be an undue burden (bloated size) and XMP was extended to be more complete with respect to the catalog, I would say the default behavior should be to preserve all data and let people optionally choose to exclude the data they don't mind losing rather than making presumptions about what is important and what is not.

Champion

 • 

28 Messages

 • 

532 Points

10 years ago

Dan,

The default should be Off, just as it is with Photoshop, which has had the ability to save history to XMP and/or a text file for years. Actually, take a look at how it done in Photoshop.

Also, be warned, set default to On and there will be h.ll to play!

4.5K Messages

 • 

76.3K Points

The default only matters if you don't know you have options, or where to go to change. Consider a warning dialog - first time save: warn about the fact that not everything is in, or it includes the kitchen sink..., then the user knows what to expect, can decide to do it differently, knows where to find the setting to change mind in future...

Employee

 • 

166 Messages

 • 

3K Points

I'd have to look at the specific trade-offs involved in the Photoshop design. Lightroom and PS are different products that may justify different choices.

Employee

 • 

166 Messages

 • 

3K Points

I'd have to look at the specific trade-offs involved in the Photoshop design. Lightroom and PS are different products that may justify different choices.