Camera Raw: Embed Sidecar documents inside the raw file

  • 1
  • Idea
  • Updated 3 years ago
  • (Edited)
Why do we have to see sidecar files with Adobe Camera RAW? all it does is save information to edits we do in camera raw. So why not set it up like dng's do???? Just embed those files inside the raw file itself? Yes i know there is a database we can use, but from what I understand if you trash a raw image, the sidecar file still remains. and when going through images in my finder, going from raw image to sidecar to raw image to sidecar, is annoying. Just embed the sidecar into the RAW file. It only makes sense!
Photo of Jerry Fictum

Jerry Fictum

  • 2 Posts
  • 0 Reply Likes
  • annoyed

Posted 3 years ago

  • 1
Photo of Jeffrey Tranberry

Jeffrey Tranberry, Sr. Product Manager, Digital Imaging

  • 14651 Posts
  • 1950 Reply Likes
Raw files are read only. There is no way to embed a file in another file. That's why we developed the DNG file format.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
Hi Jeff,

I think you are being a wee bit disingenuous in your reply. ;)

Raw files are not necessarily read-only! Nikon, for example, has been writing editing data to its raw nefs for years (although it is now using a sidecar file with the latest Nikon software). It also embeds various jpgs in the raw file, so files can be embedded in raw files because the raw image data, metadata, and jpgs are just contained in a tif-like container - the nef. Other programs can also write to raw files.

So although Adobe likes to treat raw files as read-only, it does allow capture times to be altered and written to the raw files, showing that they are NOT read-only in reality, just in Adobe's mind-set.

I don't think using another tif-like container (called a dng) for the raw image data makes the raw data any safer. The main advantage of dngs would be if all camera manufacturers standardized on it; then there would be big advantages to Adobe in not having to crack all the different types of raw files, and having all the camera info presented to them on a (dng) plate!!

Bob Frost
Photo of Pedro Marques

Pedro Marques

  • 163 Posts
  • 25 Reply Likes
That's why we use dng instead.
The metadata and also the camera color profile from each studio is also embedded.
And the DNG is 15% lighter from the native camera raw format.
It has simplified the productivity and reduced the files quantity.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
"And the DNG is 15% lighter from the native camera raw format."
..................................

I find that rather worrying; it probably means that info that the camera maker thought should be in the raw file has been discarded when it was transformed into a dng.

Bob Frost
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4513 Posts
  • 1703 Reply Likes
No, it just means the computer has much more processing power available to do lossless compression calculations.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
Mmmm. There are various reports that Adobe does discard some info from raw files that they can't or don't want to use. Are they not true?

I realise that if I included the original raw file in a dng then I would have later access to everything if I wanted it, but there seems to be doubt about the full content in the normal dng.

Bob frost
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4513 Posts
  • 1703 Reply Likes
Google for DNG Makernotes and you can read the techie details of what may not be copied in very specific scenarios.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
Thanks Victoria.

I found this article - http://www.barrypearson.co.uk/article... -
to be very informative, with links to discussions on the subject with the likes of Thomas Knoll and Eric Chan and summaries of them.

I liked Barry Pearson's concluding paragraph, especially this bit -

"Anyone who is obsessed with every last bit of information for the infinitesimal fraction of available images they actually managed to capture has lost the plot! One of my biggest obstacles to delivering world-beating pictures is that I have diverted too many resources away from capturing new pictures towards playing safe with the ones I actually have captured. "

Bob Frost
Photo of John R. Ellis

John R. Ellis, Champion

  • 3895 Posts
  • 1035 Reply Likes
I think a core issue is that each manufacturer defines their own proprietary, more-undocumented-than-not, raw format. LR/ACR does have to understand those formats to be able to read them, but there is a higher level of risk writing back to these formats. In principle, Adobe could do more reverse engineering to reduce that risk -- Exiftool does a pretty good job of that. But every dollar of license revenue that goes to such reverse engineering is a dollar not spent on some other part of LR. In my own opinion, I think LR's approach of using sidecars is a good compromise.

Note that Nikon's own ViewNX software, in at least two different versions, over several years, corrupted their own NEF files by modifying them as they were transferred off of the camera cards. Users had to resort to a utility provided by the Exiftool author to recover them. This is an example of the kind of risks involved in writing to proprietary, undocumented, always evolving file formats.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
"Note that Nikon's own ViewNX software, in at least two different versions, over several years, corrupted their own NEF files by modifying them as they were transferred off of the camera cards. Users had to resort to a utility provided by the Exiftool author to recover them. This is an example of the kind of risks involved in writing to proprietary, undocumented, always evolving file formats. "
...............................................

This is only a problem if you use old versions of ViewNX written before the newer cameras were invented. Nikon provides the latest version of its software with its cameras, so there is no reason to use old outdated software. It also updates itself unless you ignore the updates.

Somewhat similarly, it is not a good idea to use old versions of the dng converter software. Some of them had bugs, and didn't know about the latest cameras and raw files.

I've always used Nikon's NikonTransfer(2) software (part of Nikon ViewNX) because it was more reliable than LR in earlier versions. I've never had a problem despite changing cameras frequently. But I do keep my software up-to-date. And I don't delete images on my cards until I know they have been transferred and then imported into LR successfully. That is the important safeguard, whichever software you use.

Bob Frost
Photo of Jerry Fictum

Jerry Fictum

  • 2 Posts
  • 0 Reply Likes
I don't delete ftom my card till they are backed up on other drives. If you just import to Lightroom and something goes haywire, your screwed, especially if you have deleted your images from the card
Photo of John R. Ellis

John R. Ellis, Champion

  • 3895 Posts
  • 1035 Reply Likes
"That is the important safeguard, whichever software you use."

Totally agree. I called this out as an example of the additional product engineering Adobe would surely want to handle if LR routinely wrote XMP into the sidecars. They can't assume that users will keep their software up-to-date, so they have to be more careful (as Nikon wasn't) to protect against semi-documented, proprietary, evolving file formats.

DNG's ability to embed the original raw file is an example of this additional engineering helping to mitigate such risk.
Photo of Robert Frost

Robert Frost

  • 406 Posts
  • 63 Reply Likes
"DNG's ability to embed the original raw file is an example of this additional engineering helping to mitigate such risk."
.......................................

It sounds nice in theory, but here is a thread where someone can't get the raws out of the dng using Extract.

https://forums.adobe.com/thread/55356...

Makes me wary! Simpler to just keep the original raws as a backup.

Bob Frost