Lightroom/Camera Raw: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

  • 13
  • Idea
  • Updated 2 years ago
  • (Edited)
Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
  • frustrated

Posted 4 years ago

  • 13
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
I agree 100%.

I also acknowledge that the problem is more complicated than it seems at first.

As currently defined:

MyFile.RAW and MyFile.JPG and MyFile.TIF.. would have the same xmp filename.

I realize the problem is solvable, but due to it's impact on other apps etc, would be a can of worms.

Bottom-line: Adobe screwed up (yeah: that's my opinion) by not including the original extension in the xmp filename, but it's a screw up which would be hard to rectify at this point. Easy I mean, in a closed/self-contained environment, but legacy compatibility must also be maintained, so other apps aren't broken.

Don't get me wrong, I still vote for a remedy, and in another decade (probably two) we'd all laugh about the problems it caused at first..

That said, I think the fact that Adobe chose NOT to include the original extension in the filename represents a firm commitment to go with embedded xmp in all cases where it's supported, and think of sidecars as an unfortunate exception. If I'm right, then the fight for sidecars is a losing battle - hope I'm wrong.

FWIW: there *are* some apps which use sidecars for all file-types, and *do* include the extension in the filename. It works rather well, but these non-standard sidecars do not interoperate with apps which only support standard sidecars (like Adobe's). Note: these apps do NOT support embedded xmp.

When both embedded and non-embedded must be supported, the problem is exacerbated - for example: there would not only be the potential issue of metadata conflict between xmp and Lr catalog, but embedded xmp and non-embedded xmp too...

I can see why Adobe would not want to touch this problem with a 10-foot pole.

Rob
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
I know the price of backward compatibility in software devloppement.

But some time we need a breakthrough to stay connected with current technology.
Now a days cloud save is very widespread and accecible.

XMP is a xml style file and a simple new tag "" could make the job and correct the colision of MyFile.RAW and MyFile.JPG and MyFile.TIF that now could have xmp file that doesn't named MyFile.XMP.

With this kind of tag older engine can ignore this function and continue to use the file.

For some exportation, the xmp external would be embedded back inside like synchronize meta data function in lightroom.

This feature will be very appreciated for backup efficienty (low drive writeback, low delta data, original file untouched since the external declaration)

Benoit
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
So if you have MyFile.JPG and MyFile.TIF, both to have their xmp in a sidecar, what would be the name of the file into which you put this new tag?

Can't be MyFile.XMP, since well 2 different files can't have the same name.

If MyFile.JPG.XMP and MyFile.TIF.XMP would be the filenames, then would you really need to have a new tag inside?

Anyway, this is probably all academic since it is highly unlikely that Adobe will remedy for us, or so I predict - hope I'm wrong.

Rob
Photo of Benoit Malrat

Benoit Malrat

  • 9 Posts
  • 0 Reply Likes
Use the extention as filename part can also do the job.

I'm hope your are wrong to.
Now I have swiched back to my Native camera format until they find a solution.

Benoit
Photo of john beardsworth

john beardsworth

  • 1041 Posts
  • 239 Reply Likes
Rather than switch back to the native format, review your backup strategy.

DNGs only need backing up a single time when they are created, not continually. Once you have these "virgin" backup copies, it doesn't matter if Lightroom saves metadata to its "working" copies of the DNGs. The backup of your adjustment and other work is the backup of your Lightroom catalogue, and this contains 100% of what you've done to these images (which xmp doesn't).

However, I wouldn't disagree with Rob's forecast.
Photo of Steve Brown

Steve Brown

  • 95 Posts
  • 15 Reply Likes
This reply was created from a merged topic originally titled Lightroom: Option to save all metadata to XMP file instead of PSD or TIFF etc..

Currently, when the "Automatically write changes to XMP" option is checked this ONLY writes metadata to an XMP sidecar file if the file being worked on is a proprietary format - eg. .CR2, .NEF etc.; if you're working on a .PSD (or .TIFF) file (eg. after editing it in Photoshop) then any develop module or keyword changes are written directly into the (often very large) .PSD file with no sidecar created. 

This is OK if you never back up your PSD files to a non-local device, as copying the changed files is pretty quick, but when backing up to a NAS or the Cloud etc., rewriting 100 huge .PSD files just because you added a keyword or changed the output sharpening setting is a huge overhead! 

The solution would be to add an option to make Lightroom store ALL metadata in sidecar files when the "Automatically write changes to XMP" option is checked. The files could be given an extension of xyz.psd.xmp etc. to differentiate them from the XMP files generated for same-named RAW files. 

With this option checked NO metadata would ever be written directly into files such as PSDs, TIFFS etc. and backups of changed metadata to network storage would only have to copy the relatively tiny .XMP files as is already the case for proprietary files. 

Clearly the code to create such .XMP files is already in place for the other base file types so enabling it for .PSDs and TIFFs etc. should be a relatively trivial matter that would save users literally HOURS of backup time and gigabytes of network/cloud bandwidth. 
Photo of M Marsman

M Marsman

  • 1 Post
  • 0 Reply Likes
Agree to the former, I would very much like this feature. Backward compatibility would not be too hard to realize IMHO.

Currently I shoot RAW only (Olympus .ORF) so I am happily using .XMP sidecar files. The small files and changes are uploaded quickly through my Online backup software. However others supply JPG files and my older pictures are all in this format as well.

Just reorganizing my keyword sets changed all these JPG files. This is not a big problem for my backup to a secondary HDD. But for the online backup, my computer will be busy during a full day to upload 250GB. Alternative would be to use only the catalog files to store XMP, size is less and so quicker upload. However one can find may examples that this introduces a single point of failure which does fail...
Photo of John R. Ellis

John R. Ellis, Champion

  • 3690 Posts
  • 963 Reply Likes
While I agree it's unlikely Adobe will ever change this, allowing sidecars for non-raw files while maintaining backward compatibility is quite straightforward.  

Given a file "f.ext", if the file is raw, its sidecar continues to be named "f.xmp", whereas if it's not raw, the sidecar is named "f.ext.xmp".   There would be two additional options in Catalog Settings > Metadata:

[] Write metatadata for non-raw photos into the file
[] Write metadata for non-raw photos to .xmp sidecars

(The careful reader will wonder what happens when there are two raws in a folder with the same basename, e.g. "f.cr2" and "f.nef".  In that case, LR currently overwrites "f.xmp" with the metadata of whichever photo last has its metadata saved. This proposal doesn't change that behavior.)
Photo of Richard L Hess

Richard L Hess

  • 3 Posts
  • 0 Reply Likes
I agree that this is a critical issue for keeping backup simple, but there is another issue. When one makes changes to an original image file, there is a small possibility of corrupting that file. While it has never happened to me with image files, it has happened to me with audio files.

In addition, if, at the file-system level, we create file checksums to check for random changes (file fixity), changing the metadata within that file will, rightfully, change the checksum. While there are audio solutions that only generate checksums for the audio chunk within the WAV file, these have so far proven to be non-robust solutions. I am unaware of that utility for TIF files.

I am spoiled as most of my new files are NEF files (Nikon Camera Raw) so these have the XMP side cars. I am not pleased that I don't have the option of having XMP side cars for my 50,000 TIF images made mostly with my Nikon Coolscan 5000ED.

I did find the ability to turn off the automatic following of the XMP data within the TIF files within Camera RAW, so my original complaint about this has been solved, but the ability to maintain fixity confidence while using the TIF files is a major frustration.

Just as an aside, the volume of backup is actually somewhat minimized by RSYNC which does do partial file backups and that function is used within Dropbox, as I understand it--I don't know about the other cloud storage providers, but even with that said, the fixity monitoring issue remains.

Thanks!

Cheers,

Richard
(Edited)