Why can't JPEG originals be exported with an XMP Sidecar?

  • 1
  • Question
  • Updated 1 week ago
I'm not sure where the limitation exists here. The use case is, I'd like to be able to export my JPEG originals + settings from CC and import them with adjustments into Classic (or vice versa), but I'm not able to do that, I'm presuming because Lightroom does not give them XMP sidecars. I'm mostly just curious why and where that limitation exists. But also, I guess, if it's possible, I'd love to be able to include XMP sidecars and/or be able to export adjustments from my JPEGS.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes

Posted 9 months ago

  • 1
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
Side car files are intended for edits to be placed OUTSIDE proprietary raws. The same data can and is saved inside non proprietary files like TIFF, JPEG, PSD and DNG. IOW, there's very little reason for side car files in such documents and side car files were produced by Adobe because they will not edit proprietary raw data files. That data exists inside your JPEGs.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
But that data does not transfer with my JPEG. When I export the original + settings for any JPEG within Lightroom CC, it carries the tags, stars, flags, etc. but not the adjustments (light, color, etc.). Whereas when I export an RAF or DNG it imports with those adjustments.
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
When you export from LR, you create a new JPEG in this case, edits are “burned” into the new pixels and resulting exported JPEG.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
That would be saving as a JPEG and would carry over destructive edits. I am trying to carry over the non-destructive edits.
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
Nope. You are still introducing errors into the data, there is no free lunch.

The JPEG has metadata edits if made within LR/ACR. Go ahead and apply some very radical visual edits on a JPEG in either product. Now find that JPEG and open it in Photoshop proper. You will see NO edits. Because they are metadata and exist inside that original JPEG header as text. All you did was edit instructions. You want to actually SEE the edits now outside of LR/ACR? You have to export a NEW JPEG (or TIFF whatever) and a new image is created from old (and metadata) and boom, there's the data loss. It absolutely isn't non destructive. 

The only 'true' non destructive editing is when you do the work from the raw and render a new, virgin set of pixels from that raw data plus your metadata instructions. 

Oh, some think layers are non destructive editing. Not really so either. Like the JPEG example above, at some point you must flatten the layers if you hope to see the image outside of Photoshop or print the darn thing. Then the edits are applied and it's not non destructive to the data. It may appear to be non destructive because you can always undo the edits from the layers but the same could be said of using 'save as' to apply edits and spin off an iteration without applying the edits to the original (hence the 'idea' of non destructive). But that isn't the case when one applies edits to any existing pixel data. There IS data loss. In high bit, it's moot; you can't see it. Here's why:

http://digitaldog.net/files/TheHighBitdepthDebate.pdf
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
I don’t want to view it outside LR. I want to be able to move the edits between LR Classic and LR CC, as I explained in the question.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
Does not matter!
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
It does matter, because Adobe should be able to read it’s own metadata. If it can sync the edits between the two, there’s no reason they couldn’t write the software to allow for manual transfer of the edits between the two.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
And Adobe can! It doesn’t matter as you have been repeatedly told.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
It may not matter to you, but it does to me. I don’t want to make those edits twice.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
It would matter if you understand the request you made and why it's utterly unnecessary nor why you'll see it implemented. Concentrate on the bug/user error not the need for sidecar files for non proprietary data files. 
(Edited)
Photo of Jim Tillman

Jim Tillman

  • 1 Post
  • 0 Reply Likes
It does matter. I have middle school students that spent quite a lot of time working on files over the course of a trimester only to find that when their school accounts are disabled and they try to migrate their files to a new account of their own, that the edits will not make the migration since they shot jpegs.
Just because it doesn't matter to YOU doesn't mean it doesn't matter. Right?
Photo of Just Shot Me

Just Shot Me

  • 324 Posts
  • 91 Reply Likes
I don't work with the CC, cloud based, version of LR but with the classic version when you export any image, RAW/DNG, PSD, TIFF or JPG that you have made edits to LR C CC Burns those edits into the newly created exported file (that is unless you select Export Original and then it does just that with No edits included).
With JPG, TIFF, DNG, PSD and I think PNG files the edits you make are saved to the catalog file and if you have the option checked to Auto Write changes to XMP the edits also get stored in a special area of the actual file. That is a Feature of the File Format. Camera makers proprietary RAW file format do not have that ability so LR and ACR store the edits in a XMP sidecar file.
In any event the only program that can read and display the edits made with either LR or the ACR plugin to any file format is either LR or ACR. No other RAW converter can read and display LR/ACR edits. They are just ignored by other programs.
So I don't see the point of exporting JPGs with a Sidecar XMP file.

If you edit a JPG in LR and you have the option checked to Auto write changes to XMP you don't need to Export that JPG. All you have to do is Copy it as all the edits are in the file already. Sending that file to some other user that uses LR or ACR will see the edits when viewed in their copy of LR or ACR.
(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
When I export an RAF or DNG from CC using the “Original + Settings” format, I can then import it to Lightroom Classic and all the adjustments (light, color, etc.) come with it. When I do the same for a JPEG, the stars and tags come with it, but the adjustments do not. If I were to export the JPEGs as JPEGs (instead of as original), then those adjustments would become destructive. The point, as I said in my post, is to move a file from Lightroom CC into Lightroom Classic, and not lose my non-destructive adjustments. At the moment, there appears to be no way to do that (without using sync between the two which is not supported, and was causing me to lose files, so is no longer an option for me).
Photo of Jim Wilde

Jim Wilde, Champion

  • 350 Posts
  • 131 Reply Likes
It should work, and it does work for me. Make sure that you're working with a jpeg that has definitely been edited in LRCC (e.g. make an obvious change such as converting to B&W) then try again.

I'm not sure why you think syncing files between LRCC and LR Classic is not supported, as that's the obvious thing to do to get files in LRCC added to Classic.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
When I first migrated to Lightroom CC (a couple months ago), Adobe had all these warnings saying that using both CC and Classic concurrently were not recommended and that errors could occur. When I first migrated, CC and Classic went into a syncing war, and dozens of my photos were lost with “sync errors” that could only be resolved by deleting and re-importing the original file, then re-adding each to all the relevant albums. It kept happening, and to avoid having to do that to hundreds of photos, and potentially losing all my work in organizing them, I turned sound off in Classic. Are you saying that this is no longer the case and that Adobe does support syncing back to my original Lightroom catalog? And if so, I don’t even know where to start.

When you say it works for you, are you talking about the changes syncing back, or have you tried exporting a JPEG from LRCC using “Original + Settings” and importing it into Classic and the adjustments to contrast, highlights, etc. get carried into Classic?
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Okay, this is so strange. I just did a big export of JPEG originals from LRCC to Classic, and this time the adjustments did carry over... I guess I need to go back and try to find some of the JPEGs where they didn’t carry over, and figure out what was different abou those...
Photo of Jim Wilde

Jim Wilde, Champion

  • 350 Posts
  • 131 Reply Likes
Just make sure that the files you select in LRCC have actually been edited in LRCC. That's about the only thing that might explain your initial findings, but as you've discovered it will work as expected when edits have definitely been applied before the export.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Yeah, so I went back to test this just to make sure it was not user error or maybe I just missed something. I just exported two separate JPEGS from LRCC using “Original + Settings.” Both were taken with the same camera (Fujifilm X-T2) in JPEG. Both were edited in LRCC before being exported and imported to LR Classic. One of them imports with the adjustments every time, the other does not, every time. The only difference I can think of is that one of the photos was imported directly into LRCC on my computer, the other was imported on my iPad Pro (so via Apple Photos first, then into LR Mobile), however, I just repeated that process with a new photo that I just imported in through LR Mobile and it transferred with the adjustments. I guess I found an actual bug here.
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
But the bottom line, bug or user error, there is NO reason for sidecar files for JPEGs or non proprietary formats.Consider what happens when sidecar files disappear, get corrupted, etc instead of having that data embedded into the original file format. It's not a feature. It's a way in which Adobe can (for mostly legal issues), avoid writing that metadata into proprietary raws. Other files, no problem. Proprietary raws are treated as 'read only' by Adobe. Other's may not. 
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Sidecar, no sidecar, I don’t really care so long as the non-destructive edits come over to LR Classic or vice versa with my JPEGs.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
Once you understand what non destructive editing really is, you'll see that sidecar files have absolutely nothing to do with anything you're asking about. 
Photo of Robert Frost

Robert Frost

  • 426 Posts
  • 72 Reply Likes
"Camera makers proprietary RAW file format do not have that ability so LR and ACR store the edits in a XMP sidecar file."
.......................................................................

My raw nefs DO have that ability; it is just a 'political' decision on the part of Adobe only to use that ability for capture time edits. And even then they only put the new capture time into the xmp part of the raw file; they don't edit the actual capture time in the Exif part of the metadata. So you end up with two different capture times in your raw file. They made up the tale about raw files being 'untouchable'. Rubbish; Nikon software edits them, so do Photomechanic and Exiftool, and probably others I haven't tried. Adobe want everyone to use their Dng, so they disparage editing other types of raw files.

Bob Frost 
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
Bob is correct that Adobe doesn’t write to proprietary raws for “legal” (he preferred political) reasons. It isn’t a technical limitation.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
Jon wrote:

"When I export an RAF or DNG from CC using the “Original + Settings” format, I can then import it to Lightroom Classic and all the adjustments (light, color, etc.) come with it. When I do the same for a JPEG, the stars and tags come with it, but the adjustments do not."

This is clearly a bug in LR Classic. The develop settings should come through with JPEGs as well as with raw and DNG.  LR CC is writing the develop settings to the JPEGs (in the XMP metadata section), but when they get imported into LR Classic, the effect of the settings don't show in Library.  However, if you open those imported JPEGs in Develop, the effect of the settings does show, and the sliders are set correctly.  Doing Library > Previews > Build 1:1 Previews has no effect.  However, exiting LR, deleting the Previews.lrdata file, and restarting LR, the imported JPEGs show the settings correctly in Library.

Try that out with your LR Classic catalog. Import one of the JPEGs saved from LR CC and go into Develop -- do you see the effect of the settings and sliders correctly?  If so, then go to Catalog Settings > General and click Show to open the catalog folder in Finder / File Explorer.  Exit LR, and then delete the Previews.lrdata, which appears as a file in Finder and a folder in File Explorer. Restart LR and wait for it to rebuild previews.

(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
I just commented on your previous thread. TL;DR, the develop adjustment levels are not showing up in some but not all exported images. I have not found a pattern to which it works for and which it does not yet.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
As a test, I took one of the photos for which the develop settings were not transferring from LRCC to Classic, and made some changes in Classic and imported it the other way. The changes did transfer back to CC.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Okay, one more test, took an image where the develop settings were not transferring, and changed one setting (increased contrast from 0 to +1) in LRCC, and this time when exporting to Classic all the develop settings transferred with it.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
In your tests,

1. Are you going to Develop in LR Classic to examine the photo there? 

2. Have you tried the workaround of exiting LR Classic, deleting the Previews.lrdata, and restarting LR Classic?
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
1. Yes, see video posted below.

2. I have not yet, will that affect the settings bars in the Develop Settings?
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
"2. I have not yet, will that affect the settings bars in the Develop Settings?"

Deleting the Previews.lrdata file is harmless and doesn't affect the contents of your catalog. The only effect is to cause LR to rebuild the previews it uses for displaying photos in Library (which could take a very long time depending on the size of your catalog).  So I am 99.9% confident that it won't affect the settings you see in Develop (it would be an extremely wierd-science bug if it did).
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Okay, I confirmed that deleting the previews.lrdata file does not fix the issue.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
Steps to reproduce the bug:

1. On an iPhone, check the setting Settings > Camera > Formats > Most Compatible (to force it to generate a JPEG). 

2. Use the Camera app to snap a photo.

3. Go to iOS LR CC and wait for it to sync the photo to the cloud.

4. Start desktop LR CC, select the photo, and set Exposure = -3.

5. Select the photo and do File > Save To, Original + Settings.

6. Import that saved JPEG into LR Classic with the Import > Add option.

7. Observe that the Library thumbnail does not show the effect of Exposure = -3.

8. Go into Develop and observe that Develop does show the effect of Exposure = -3, and the slider is set appropriately.

9. In Library, do Library > Previews > Build 1:1 Previews and observe the thumbnail doesn't change.

10. In Library, do View > Toggle Loupe View and observe that the photo still doesn't show Exposure = -3.

11. Exit LR Classic, delete the Previews.lrdata file/folder, and restart LR Classic.

12. Observe that Library shows the effect of Exposure = -3 correctly.

Tested with LR Classic 7.5 / desktop LR CC 1.5 / mobile LR CC 3.4.0 / Macos 10.13.6 / Ios 11.2.1.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
I posted this as a separate, clean bug report so that Adobe will notice it:
https://feedback.photoshop.com/photoshop_family/topics/jpegs-saved-from-lr-cc-dont-import-correctly-...
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
The bug I've encountered appears to be different than the bug you encountered, as the settings do not show in Develop.
Photo of Jim Wilde

Jim Wilde, Champion

  • 351 Posts
  • 133 Reply Likes
John, I'm not able to reproduce this issue on either my MacOS or Windows10 systems. In all cases the develop settings are correctly included in the exported jpeg, and seen in Library when importing them into Classic.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
Jim, after iterating a bit with Rikk Flohr, I narrowed down the symptoms I observed to a confusing bug in the import option Build Previews: Embedded & Sidecar:

https://feedback.photoshop.com/photoshop_family/topics/lightroom-importing-with-embedded-sidecar-pro...

If you followed my recipe using Build Previews: Standard, you wouldn't observe the bug.  

But John Anscher's problem is something different, as already discussed.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
How's this for evidence? Note there was something weird where auto import did not import the photo the second time. Not sure what happened there.


(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Okay, so I figured out what happened with the 2nd import. It apparently went into the Testing folder I made when I imported the 2nd photo. I probably should have deleted rather than just moving it, but this does show the develop settings not showing. I'll try deleting the previews.lrdata first, then re-test this again but delete the photo after the first import instead.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
I ran another test where I did not use auto import, and I deleted the photos between. I replaced the video above, with that, so you can see it happening on my machine.
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
"Okay, I confirmed that deleting the previews.lrdata file does not fix the issue." 
"
The bug I've encountered appears to be different than the bug you encountered, as the settings do not show in Develop."

Most likely, we're seeing different manifestations of the same underlying bug, which we don't completely understand yet. (That's the nature of bugs.)

Can you take a LR CC-saved photo that still won't import correctly into LR Classic, upload it to Dropbox or similar, and post the sharing link here? I'd like to examine the XMP metadata of the file, which encodes the Develop settings. The more detail we can hand to Adobe, the more likely the bug gets fixed sooner.


Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
Great, I'll take a look at this in-depth in a couple of hours.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Thank you for taking the time to look into this. Let me know if I can provide any other testing or information.
(Edited)
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
"Here is one: http://jonanscher.com/resources/DSCF9619.JPG."

When you look at that photo as it currently exists in LR CC, are any of the setting sliders set at something other than 0?
(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Yep. Just like all the others near it.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
I’m not near my computer at the moment, but these are from my iPad, the edits on that photo.



(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
And again, if I make a single adjustment to that photo, then save it, it will import with all those adjustments. (I’m happy to start a fresh bug request by the way, now that I know this is a bug and not a feature issue.)
(Edited)
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
The problem with DSCF9619.JPG is that LR CC appears to have incorrectly saved it with this develop setting:
$ exiftool -G -xmp:alreadyapplied DSCF9619.JPG 
[XMP] Already Applied : True
The AlreadyApplied field is supposed to be True when all the develop settings recorded in the XMP metadata have already been applied to the underlying JPEG image. For example, when you export a JPEG from LR Classic, the underlying image will have all the develop settings "baked in", so the XMP metadata will include AlreadyApplied = True to indicate that.

But in this case, the settings clearly haven't been applied to the underlying photo. But LR Classic thinks that they are due to AlreadyApplied = True, so it ignores them. When I use the Exiftool utility to change the value to False and then import into LR Classic, the settings come in properly.

Do you remember the history of this photo?  The metadata indicates it was edited with LR Classic 7.5 as well as LR CC 1.5. I've tried unsuccessfully reproducing this bug both with migrating a Classic catalog and with importing directly into LR CC.

I think you should post a new bug report with the screenshots and the link to the saved file.  Also include screenshots of the Basic panel in Develop after you've imported the file into LR Classic, showing the sliders are all zero.   Or a video showing the sliders in CC, saving the photo, importing into Classic, and then Develop sliders in Classic. 
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Unfortunately, I don't remember the exact history of that photo. Generally, since I migrated to LR CC I only edit them in LR Classic if they are merged panos or HDR. It's possible that when I grabbed the photo, I grabbed the version that was imported into Classic. I'll make sure I put a clean file when I repost this as a bug.
Photo of Andrew Rodney

Andrew Rodney

  • 761 Posts
  • 149 Reply Likes
So you can not replicate this bug on all JPEGs?
Photo of John R. Ellis

John R. Ellis, Champion

  • 4361 Posts
  • 1160 Reply Likes
Also please copy the first ten lines of Help > System Info from both LR Classic and LR CC, so we can see the exact configuration you're running.
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
LRCC:

Lightroom Classic version: 7.5 [ 1186149 ]License: Creative Cloud
Language setting: en-US
Operating system: Mac OS 10
Version: 10.13.6 [17G65]
Application architecture: x64
Logical processor count: 8
Processor speed: 2.6 GHz
Built-in memory: 16,384.0 MB
Real memory available to Lightroom: 16,384.0 MB
Real memory used by Lightroom: 2,430.8 MB (14.8%)
Virtual memory used by Lightroom: 4,888.3 MB
Memory cache size: 1,922.8MB
Internal Camera Raw revision: 1001
Maximum thread count used by Camera Raw: 5
Camera Raw SIMD optimization: SSE2,AVX,AVX2
Camera Raw virtual memory: 365MB / 8191MB (4%)
Camera Raw real memory: 366MB / 16384MB (2%)
Displays: 1) 2560x1440

Graphics Processor Info: 
Metal: NVIDIA GeForce GT 750M
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes
Lightroom CC version: 1.5 [ 20180807-1548-54800d7 ] (Aug  8 2018)Operating system: Mac OS 10
OS Version: 10.13 [6]
Application architecture: x64
Logical processor count: 8
Processor speed: 2.6 GHz
Built-in memory: 16,384.0 MB
Real memory available to Lightroom CC: 16,384.0 MB
Real memory used by Lightroom CC: 806.6 MB (4.9%)
Virtual memory used by Lightroom CC: 5,696.3 MB
Memory cache size: 3,829.8 MB
Internal Camera Raw revision: 1001
Maximum thread count used by Camera Raw: 5
Camera Raw SIMD optimization: SSE2,AVX,AVX2
Camera Raw virtual memory: 1933MB / 8191MB (23%)
Camera Raw real memory: 1934MB / 16384MB (11%)
Displays: 1) 2560x1440

Graphics Processor Info: 
Metal: NVIDIA GeForce GT 750M
(Edited)
Photo of Jon Anscher

Jon Anscher

  • 202 Posts
  • 32 Reply Likes