Lightroom/Camera Raw: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

  • 22
  • Idea
  • Updated 7 months ago
  • (Edited)
Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
  • frustrated

Posted 6 years ago

  • 22
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 394 Reply Likes
I agree 100%.

I also acknowledge that the problem is more complicated than it seems at first.

As currently defined:

MyFile.RAW and MyFile.JPG and MyFile.TIF.. would have the same xmp filename.

I realize the problem is solvable, but due to it's impact on other apps etc, would be a can of worms.

Bottom-line: Adobe screwed up (yeah: that's my opinion) by not including the original extension in the xmp filename, but it's a screw up which would be hard to rectify at this point. Easy I mean, in a closed/self-contained environment, but legacy compatibility must also be maintained, so other apps aren't broken.

Don't get me wrong, I still vote for a remedy, and in another decade (probably two) we'd all laugh about the problems it caused at first..

That said, I think the fact that Adobe chose NOT to include the original extension in the filename represents a firm commitment to go with embedded xmp in all cases where it's supported, and think of sidecars as an unfortunate exception. If I'm right, then the fight for sidecars is a losing battle - hope I'm wrong.

FWIW: there *are* some apps which use sidecars for all file-types, and *do* include the extension in the filename. It works rather well, but these non-standard sidecars do not interoperate with apps which only support standard sidecars (like Adobe's). Note: these apps do NOT support embedded xmp.

When both embedded and non-embedded must be supported, the problem is exacerbated - for example: there would not only be the potential issue of metadata conflict between xmp and Lr catalog, but embedded xmp and non-embedded xmp too...

I can see why Adobe would not want to touch this problem with a 10-foot pole.

Rob
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
I know the price of backward compatibility in software devloppement.

But some time we need a breakthrough to stay connected with current technology.
Now a days cloud save is very widespread and accecible.

XMP is a xml style file and a simple new tag "" could make the job and correct the colision of MyFile.RAW and MyFile.JPG and MyFile.TIF that now could have xmp file that doesn't named MyFile.XMP.

With this kind of tag older engine can ignore this function and continue to use the file.

For some exportation, the xmp external would be embedded back inside like synchronize meta data function in lightroom.

This feature will be very appreciated for backup efficienty (low drive writeback, low delta data, original file untouched since the external declaration)

Benoit
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 394 Reply Likes
So if you have MyFile.JPG and MyFile.TIF, both to have their xmp in a sidecar, what would be the name of the file into which you put this new tag?

Can't be MyFile.XMP, since well 2 different files can't have the same name.

If MyFile.JPG.XMP and MyFile.TIF.XMP would be the filenames, then would you really need to have a new tag inside?

Anyway, this is probably all academic since it is highly unlikely that Adobe will remedy for us, or so I predict - hope I'm wrong.

Rob
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
Use the extention as filename part can also do the job.

I'm hope your are wrong to.
Now I have swiched back to my Native camera format until they find a solution.

Benoit
Photo of john beardsworth

john beardsworth

  • 1356 Posts
  • 374 Reply Likes
Rather than switch back to the native format, review your backup strategy.

DNGs only need backing up a single time when they are created, not continually. Once you have these "virgin" backup copies, it doesn't matter if Lightroom saves metadata to its "working" copies of the DNGs. The backup of your adjustment and other work is the backup of your Lightroom catalogue, and this contains 100% of what you've done to these images (which xmp doesn't).

However, I wouldn't disagree with Rob's forecast.
Photo of Steve Brown

Steve Brown

  • 95 Posts
  • 17 Reply Likes
This reply was created from a merged topic originally titled Lightroom: Option to save all metadata to XMP file instead of PSD or TIFF etc..

Currently, when the "Automatically write changes to XMP" option is checked this ONLY writes metadata to an XMP sidecar file if the file being worked on is a proprietary format - eg. .CR2, .NEF etc.; if you're working on a .PSD (or .TIFF) file (eg. after editing it in Photoshop) then any develop module or keyword changes are written directly into the (often very large) .PSD file with no sidecar created. 

This is OK if you never back up your PSD files to a non-local device, as copying the changed files is pretty quick, but when backing up to a NAS or the Cloud etc., rewriting 100 huge .PSD files just because you added a keyword or changed the output sharpening setting is a huge overhead! 

The solution would be to add an option to make Lightroom store ALL metadata in sidecar files when the "Automatically write changes to XMP" option is checked. The files could be given an extension of xyz.psd.xmp etc. to differentiate them from the XMP files generated for same-named RAW files. 

With this option checked NO metadata would ever be written directly into files such as PSDs, TIFFS etc. and backups of changed metadata to network storage would only have to copy the relatively tiny .XMP files as is already the case for proprietary files. 

Clearly the code to create such .XMP files is already in place for the other base file types so enabling it for .PSDs and TIFFs etc. should be a relatively trivial matter that would save users literally HOURS of backup time and gigabytes of network/cloud bandwidth. 
Photo of M Marsman

M Marsman

  • 1 Post
  • 0 Reply Likes
Agree to the former, I would very much like this feature. Backward compatibility would not be too hard to realize IMHO.

Currently I shoot RAW only (Olympus .ORF) so I am happily using .XMP sidecar files. The small files and changes are uploaded quickly through my Online backup software. However others supply JPG files and my older pictures are all in this format as well.

Just reorganizing my keyword sets changed all these JPG files. This is not a big problem for my backup to a secondary HDD. But for the online backup, my computer will be busy during a full day to upload 250GB. Alternative would be to use only the catalog files to store XMP, size is less and so quicker upload. However one can find may examples that this introduces a single point of failure which does fail...
Photo of John R. Ellis

John R. Ellis, Champion

  • 5167 Posts
  • 1476 Reply Likes
While I agree it's unlikely Adobe will ever change this, allowing sidecars for non-raw files while maintaining backward compatibility is quite straightforward.  

Given a file "f.ext", if the file is raw, its sidecar continues to be named "f.xmp", whereas if it's not raw, the sidecar is named "f.ext.xmp".   There would be two additional options in Catalog Settings > Metadata:

[] Write metatadata for non-raw photos into the file
[] Write metadata for non-raw photos to .xmp sidecars

(The careful reader will wonder what happens when there are two raws in a folder with the same basename, e.g. "f.cr2" and "f.nef".  In that case, LR currently overwrites "f.xmp" with the metadata of whichever photo last has its metadata saved. This proposal doesn't change that behavior.)
Photo of Richard L Hess

Richard L Hess

  • 6 Posts
  • 2 Reply Likes
I agree that this is a critical issue for keeping backup simple, but there is another issue. When one makes changes to an original image file, there is a small possibility of corrupting that file. While it has never happened to me with image files, it has happened to me with audio files.

In addition, if, at the file-system level, we create file checksums to check for random changes (file fixity), changing the metadata within that file will, rightfully, change the checksum. While there are audio solutions that only generate checksums for the audio chunk within the WAV file, these have so far proven to be non-robust solutions. I am unaware of that utility for TIF files.

I am spoiled as most of my new files are NEF files (Nikon Camera Raw) so these have the XMP side cars. I am not pleased that I don't have the option of having XMP side cars for my 50,000 TIF images made mostly with my Nikon Coolscan 5000ED.

I did find the ability to turn off the automatic following of the XMP data within the TIF files within Camera RAW, so my original complaint about this has been solved, but the ability to maintain fixity confidence while using the TIF files is a major frustration.

Just as an aside, the volume of backup is actually somewhat minimized by RSYNC which does do partial file backups and that function is used within Dropbox, as I understand it--I don't know about the other cloud storage providers, but even with that said, the fixity monitoring issue remains.

Thanks!

Cheers,

Richard
(Edited)
Photo of Robert Ripps

Robert Ripps

  • 155 Posts
  • 13 Reply Likes
I like that I can set LR (Classic CC 2019 for Mac) to automatically write changes made on raw files in LR or CR (Camera Raw) to the xmp files. What I don't like is if I check that preference, LR also writes any changes made to jpeg files- like ratings changes or flags. It would be great if LR had a few more options, such as don't write changes to PSD, TIFF or JPG files, but instead keeps those internal in the LR database. No reason to rewrite a jpeg, and change its modification date just because I added a star!

And speaking of suggestions, it would be great if I could sort a jpeg by its creation date, not its modification date, which keeps changing (see above).

Thanks.
(Edited)
Photo of David Terry

David Terry

  • 4 Posts
  • 4 Reply Likes
This reply was created from a merged topic originally titled Lightroom Classic: Is it possible to use XMP files with DNG?.

It was suggested that I bring this discussion over here from another forum, so I apologize if you've seen this before.

I have used XMP files since the very beginning of Lightroom. It's a handy way of having an "instant backup" of changes just in case something ever happens to my Lightroom Catalog.  I love the safety net that XMP files give me.

A year and a half ago (after 15 years with Canon) I switched to the Sony a7iii.  The biggest downside to that is that Sony's ARW files are 47mb each.  But if I use the Adobe DNG Converter, I can losslessly compress them down to just 27mb each.  As I shoot thousands upon thousands of images each month, that 20mb per image savings is huge.  Especially when you take into account backups.  

However, there is one big thing I hate about using DNG files.  And that is that every time I make a change in Lightroom (change a star rating, adjust exposure, crop, etc) that 27mb DNG file needs to be backed up.  AGAIN.  Because unlike my Canon raw files, with XMP sidecar files, the DNG files are written back to whenever I make any changes.  

Is there any way to change Lightroom to write to a tiny XMP file every time I make a change, rather than modifying my 27mb DNG files?  I would much rather backup and rebackup tiny little XMP files.  It makes no sense to have to backup an entire 27mb file just because a few bytes inside of it have changed.  And it costs me money (I have to pay extra to Internet provider as I go over their imposed limits). 

For a real world example, let's say I shoot 64GB worth of files for a wedding.  It backs up 64GB worth of files right after I get home and import the files into my system.  It'll be a few weeks before I get to the wedding, but when I do, I'm going to rate all of the images.  Now they all get backed up again (that's an extra 64GB needlessly being backed up).  I probably won't finish all of my edits in one day, so the next day as I continue editing... any remaining files that get changed will be backed up again.  And again.  And again, for as long as I keep making changes inside of Lightroom.  See how this is suboptimal?  Certainly it's not a lot of money, but it all adds up and is costing me extra for my Internet usage because it is metered.  Whereas if it was using XMP files, then only tiny (what, 10K?) files change as I make my LR edits and that is all that has to be backed up, with XMP files the much larger raw files remain unchanged.

So for me ... DNG storing internally any changes is expensive.  I would love to have an XMP option if it were available.  It would reduce how often my local drives get hit with backups as well as the my internet cost for backing them up. 
Photo of John R. Ellis

John R. Ellis, Champion

  • 5167 Posts
  • 1476 Reply Likes
I too would like the option to use .xmp sidecars with all photos, raw and non-raw. I backup the catalogs regularly. But backup procedures can fail (more often than you think), and I like having the sidecars as a secondary backup. Also, about once a year I accidentally delete a photo from disk, and the sidecar makes it more likely I'll be able to recover the most recent edits and metadata changes.
Photo of David Converse

David Converse

  • 966 Posts
  • 281 Reply Likes
XMP has a tag that tells what filename is linked to that sidecar.
Photo of Vinelander

Vinelander

  • 8 Posts
  • 7 Reply Likes
Yes, Adobe, please provide an option to force creation of XMP sidecar files for ALL file types, including DNG. I just have to change or add one single keyword for a larger number of photos (an operation which may not take more than a split second), and because they are DNG, instead of a few megabytes (at most) my next backup is hit with dozens or even hundreds of gigabytes of data which is getting needlessly re-copied.

The capability is already there (at least on Macs): if you lock a DNG file in Finder, Lightroom immediately starts writing metadata changes to an XMP file for that DNG, just like it would with a proprietary raw file. Unfortunately, locking DNGs manually is not remotely practical for normal usage; but it shows it must be nearly trivial to offer such an option. So, please, please do so.
Photo of john beardsworth

john beardsworth

  • 1352 Posts
  • 373 Reply Likes
I couldn't care less what your current backup solution looks like, but it's clearly not optimized for DNG if you are repetitively backing up files. Flabby backup procedures aren't a reason for adding what isn't just a simple option but one which has other consequences, since LR and other apps will no longer know whether to prefer embedded or sidecar metadata. And while development resources may or may not be a zero-sum game, time spent on an unnecessary option will not be available to work on things one does appreciate. So no, no option thank you.

Photo of Richard L Hess

Richard L Hess

  • 6 Posts
  • 2 Reply Likes
John, I don't know what environment you work in, but I would suspect many users here are lucky to be able to run simple NAS hardware and at least some are using arrays of HDDs. The simplest implementation involves using OS-based tools that are available for Windows and Macs that have a proven track record and do not create custom files.

For me, it is important that all the backups are a replica of the tree and accessible to the individual file level without special software.

That works for NAS units and local HDDs and is mimicked in Backblaze and Dropbox.

Most OS-level tools (unlike Backblaze and Dropbox) that I am aware of do not do block level compare/copy, but rather just copy the entire file.

But besides that, I am not pleased at risking my original archival "negative" files being written to on a regular basis.
Photo of Brian Kimball

Brian Kimball

  • 23 Posts
  • 6 Reply Likes
Lightroom should behave nicely with common and ubiquitous consumer-level backup solutions.  Writing metadata changes & develop settings into a DNG that is 20-100 MB per image is not good design and does not play well with others.

Lightroom and the DNG spec do not exist in a vacuum nor in a perfect world.  It's time Adobe create an option for those who want XMP sidecars with DNGs that aren't re-written every time they're edited.
Photo of john beardsworth

john beardsworth

  • 1352 Posts
  • 373 Reply Likes
"But besides that, I am not pleased at risking my original archival "negative" files being written to on a regular basis."

Sure, and in a DNG workflow, when you choose to write only some of your work to the files, the DNGs recorded in Lightroom gain a more cache-like status. In any case, their backup value is diminished because of xmp doesn't contain large areas of the work you've done on them in LR. The original archival "negative" files are the copies which backed up when the DNGs were first created.

"Lightroom should behave nicely with common and ubiquitous consumer-level backup solutions. "

And I think that's done better by other means. For example, incremental backup of the catalogue / transaction logs would not break Adobe and third party standard practices for metadata location.
Photo of Edmund Gall

Edmund Gall

  • 157 Posts
  • 55 Reply Likes
Frankly speaking, I don't think John is the person you need to convince, folks - it's the guys at Adobe. So, if John or any non-Adobe staff doesn't understand your points, that's okay...
Photo of Johan Elzenga

Johan Elzenga, Champion

  • 2969 Posts
  • 1270 Reply Likes
“See my comment above. This whole discussion presumes you have "Automatically write changes to XMP" enabled for your catalog. Otherwise locking DNG files will of course not cause LR to produce XMP files since all changes will be written to the catalog

Changes are always written to the catalog, even if you have this option turned on. Writing changes to XMP is additional, it does not replace the catalog entry. That may explain why this does not work in Lightroom, while apparently it does work in Bridge.
Photo of Vinelander

Vinelander

  • 8 Posts
  • 7 Reply Likes
"Changes are always written to the catalog, even if you have this option turned on."

That is correct, and I didn't mean to imply otherwise. (Also, see my correction concerning locked DNG files above.)
Photo of Richard L Hess

Richard L Hess

  • 6 Posts
  • 2 Reply Likes
The good news is that many cloud services do incremental backups if that is your method of backing up.

I also keep two NAS units. Updates are made to the main NAS, but for certain file types (WAV, TIFF, JPG) updates are not propagated to the backup NAS. Deletes are not propagated to the backup NAS. Updates and deletes are propagated to the backup machine that feeds Backblaze.
Photo of John R. Ellis

John R. Ellis, Champion

  • 5167 Posts
  • 1476 Reply Likes
Note that LR uses .xmp sidecars for HEIF files (what Apple calls HEIC), even though the format specifies where to store EXIF and XMP metadata inside the files.
Photo of Robert Ripps

Robert Ripps

  • 155 Posts
  • 13 Reply Likes

DNG files aside, I still wish there was a way to make changes to a file’s rating or flag, without Lightroom changing the files CREATION DATE. I understand that it will write the changes in a rating to the jpeg file (although I am not happy about that) which changes modification date, but why does Lightroom change the creation date as well? That makes no sense! Interestingly, Lightroom does not change the creation date of an xmp file when I make a rating change, only the modification date.

Why can’t Lightroom give us better choices in settings to including not write changes in ratings, flags, etc. to jpegs, while still allowing changes made in Lightroom (or Camera Raw) to raw files to be recorded to the corresponding xmp files?

Photo of John R. Ellis

John R. Ellis, Champion

  • 5167 Posts
  • 1476 Reply Likes
"I still wish there was a way to make changes to a file’s rating or flag, without Lightroom changing the files CREATION DATE. I understand that it will write the changes in a rating to the jpeg file (although I am not happy about that) which changes modification date, but why does Lightroom change the creation date as well?"

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of_image_files

To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

Photo of Edmund Gall

Edmund Gall

  • 157 Posts
  • 55 Reply Likes
I think the guys at Adobe (and anyone who knows how incremental cloud backups work) understand what you're asking for, David Terry & Vinelander. DNGs operate like a mix of JPEG with raws - you get more pixel-related data than a JPEG (but less than a raw) and metadata is stored within the file (just like JPEGs). The apparent trade-of was always there: you're saving local disk space but if you use cloud backups, then bandwidth usage will increase as changed DNG data gets transferred after any modification (in addition to changed LR catalog data).

I want to make 3 points that haven't been mentioned.

First, if I understand David's and Vinelander's posts above (and Benoit Malrat's original post 5 years agoe), the concern is that if a simple metadata change (e.g. change in flag rating) is made, they think the entire DNG is re-uploaded to their cloud backup. Thus, they conclude that by transferring metadata from DNGs to sidecar XMPs, the amount of data that gets re-transferred to cloud backups drops significantly. I recommend that they confirm with their cloud backup service provide whether that's the case today (as Richard Hess hinted above both 3 years ago and yesterday).

For e.g., I use Crashplan who use a block-based method for their incremental backups. Basically, a file occupies a set of blocks on disk; blocks are a much smaller unit of size (e.g. a 1MB file could comprise 1000 blocks if each block is 1KB in size). When Crashplan finds a new file, it copies the entire file to cloud storage. After that, it uses incremental backups: Crashplan compares each block for that file on local disk with that already within the cloud backup, and only transfers the blocks that were changed – not the entire file. This is why, even after a long editing session, Crashplan does not need to re-transfer my entire multi-GB LR catalog file to cloud backup. See: https://support.code42.com/CrashPlan/6/Backup/How_backup_works

If you don't use Crashplan, see if your service provider's website has a similar explanation, or contact their support team. What this means is – provided LR does not recreate DNG files for the simple changes you're concerned about (and I think it doesn't) – your assumption that the entire DNG file is re-transferred to cloud backup may not be accurate. If your cloud backup does not use a similar bandwidth-efficient means for incremental backups, then perhaps you need a new service provider.

My 2nd point is to consider re-evaluating your use of DNG (rather than changing the way DNGs work). With storage prices dropping each year per unit, if you are in some type of crazy Internet service plan that still limits your bandwidth and charges for excess, then – as counterintuitive as it may sound – maybe your overall cost for storage, backup and Internet would reduce if you switch back to raw files (and/or get a better Internet plan).

My 3rd point is, if you've re-evaluated and it's still better for you to stick with DNG, then you can also check when you convert to DNG in your workflow. Some years back when I was consuming as much training about LR before adopting it, a highly-experienced photographer explained he used DNG at the end of his workflow as part of his long-term archiving strategy. In other words, he used proprietary raws during his initial stages of editing a photo. Once he was sure he won't need to do any more processing, then he'd convert the image to DNG, link the DNG to the editing metadata in LR (by swapping the raws with DNGs) and then move that folder to his archive storage (NAS, cloud or whatever). So, if you're concerned about the Internet/backup overhead from modifying DNGs, maybe you can re-examine whether you can convert to DNGs much later in your workflow.

Hope this helps...

[ As a side note, if Adobe enables the storage of metadata within XMPs instead of DNGs, they have to consider the wider consequences. For e.g., there would be additional administrative overhead when Adobe was aiming for simplification (even if implemented via an optional control in Preferences as John Ellis mooted above). The more 'moving parts' you introduce into the workflow (i.e. an image's data is now spread over the catalog, DNG and sidecar), the greater the risk of something going wrong. You may think that's fine for you because you're an experienced user with stronger technical nous than the average customer, but it can create work (read: trouble) for inexperienced users who just need a solution that reduces their local storage usage (relative to proprietary raw formats) and provides a backup (albeit imperfect one) to what's stored in the LR catalog in a single file. If I look back at the changes Adobe made over the last five years, they're targeting more non-professionals: so a lot of the complexity has to be hidden or removed. You read this forum for a year and you'd realise how many newbies get into trouble because they just jump straight into using LR without any training, because Adobe's marketing makes them think it's as simple as iPhotos. Any new feature we customers ask for has to be looked at through this prism. ]
Photo of Robert Ripps

Robert Ripps

  • 155 Posts
  • 13 Reply Likes

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of_image_files

I thought I was by posting to this forum, over and over...;).


To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

I think I understand what you are saying & why Lightroom does what it does, but I don’t like it. However, I am not an engineer, or even a software expert. I am a photographer who is relying on some basic software tools to be able to sort some of my cataloged images (jpegs) based on the dates the jpeg was created, not the date I changed a rating on that image. I need a simple way to preserve the date I created that jpeg (and not the earlier date an underlying image used in that master Photoshop file that this jpeg was saved from), and I need Lightroom to be able to sort by the date I created the jpeg from my PSD file- seems simple enough. When the modification date the OS presents to me, and to Lightroom is consistent with the date I created the psd, then saved as a jpeg, it doesn’t matter if the actual file is really an identical copy that underneath has a new creation date.

If Adobe doesn’t think preserving the creation date of the original file and writing in the new file so it can be read by the MacOS, but instead thinks that the most recent modification date should be displayed as the creation date is the way to go, even though the real creation date is saved somewhere in the file/OS, then why doesn’t Lightroom offer a way to see that date, and use it to sort by date without me having to modify things to change dates? Seems unfair- Lightroom can change the creation date when it wants, but then it forces you to use that date when sorting your files, even though it knows it is not the real creation date. Also, the capture date of a raw image is not necessarily the creation date of a jpeg from an image created in Photoshop- to me, the creation date is the date that I saved a jpeg from a finished psd file, not the dates of one of the images used to create that Photoshop file.

Why do changes in xmp files, which seems the same principal as changes to jpegs, not change the creation date as well? And why do Photoshop & Bridge not behave this way (from what I have read in this thread)?

And, if Lightroom offered a way to write changes to xmp files only, AND not write them to jpegs or dng files, this would not be an issue.

I will try you suggestion to set the captured time to the file creation time when I import the next batch of jpegs and see if that works for my issue.