Lightroom 6: Invalid XMP metadata written to exported JPEGs, tripping up reading software

  • 10
  • Problem
  • Updated 3 years ago
  • Solved
Lightroom 6 often writes invalid XMP metadata in exported JPEGs, including for photos containing lots of brush strokes made with the adjustment brush. This trips up Google Photos, preventing it from showing any of the EXIF metadata. It may well trip up other software.

To reproduce:

1. Start with any image.

2. Use a small adjustment brush with Exposure = 100.

3. Make lots and lots of brush strokes (see the example pic below).

4. Export the image as a JPEG, including all metadata.

5. Load the image to Google Photos and observe that it doesn't show any EXIF metadata.

6. Delete all of the XMP develop settings with:

exiftool -xmp-crs:all= file.jpg

7. Upload that modified file to Google Photos and observe that it now shows the EXIF metadata.

Here's an example pic exported from step 4:

https://dl.dropboxusercontent.com/u/2...

If you extract the XMP metadata with:

exiftool -a -m -b -xmp file.jpg

you'll see that LR has recorded all of the develop settings twice, including all the brush strokes.

Worse, if you examine the file layout with:

exiftool -m -htmldump file.jpg > file.htm

you'll see that the APP1 Extended XMP segments recording the second copy of the develop settings have incorrect segment offsets, with the first segment of the duplicate settings having offset 0, and the following segments with offsets based on that. That's not allowed by the XMP standard, which requires all the segments to have linearly increasing offsets with no gaps. It's easy to imagine how this might trip up a program trying to read the metadata.
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes

Posted 3 years ago

  • 10
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes
See this thread for users encountering this problem: https://forums.adobe.com/message/7502...
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes
A workaround is to use the Metadata Wrangler plugin on your exports: http://regex.info/blog/lightroom-good...

Choose the option to remove the XMP "crs" block (which contains the develop settings).
Photo of Mark Swaisland

Mark Swaisland

  • 10 Posts
  • 1 Reply Like
I'm one if the users affected by this issue in the thread referenced by John.

This is causing me a bug problem and I hope it gets fixed quickly; a DAM application generating invalid XMP metadata is unacceptable.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
I'm having the same problem with my Phorganizer Android app. It says it can't find any EXIF data with JPGs exported with LR6. But it can find the EXIF data with JPGs exported with LR5.7 Also with JPGs exported with LR6, the Gallary Android app slideshow function doesn't display the photos in shot date order anymore. It displays the photos in file "date modified" order.

Hope Adobe fixes this soon!
Photo of Rudi

Rudi

  • 9 Posts
  • 1 Reply Like
Same problem here (among many others)! Culling my photos with Photo Mechanic and having set most metadata there when I import them to LR6 metadata are invalid for jpgs.
Photo of Dawid Ciecierski

Dawid Ciecierski

  • 9 Posts
  • 1 Reply Like
Came across this issue when Picasa would not see some of the EXIF metadata stored in files (photo location and exposure details among others).

Crazy thing to see such an important aspect of the software goofed up. Hoping for a point release sometime soon!
Photo of Phil Harvey

Phil Harvey

  • 11 Posts
  • 0 Reply Likes
Actually, this is allowed by the XMP specification, which states:

"The GUID is also stored in the StandardXMP as the value of the xmpNote:HasExtendedXMP property. This allows detection of mismatched or modified ExtendedXMP. A reader must incorporate only ExtendedXMP blocks whose GUID matches the value of xmpNote:HasExtendedXMP"

The GUID of the second extended XMP doesn't match, so technically it should be ignored. ExifTool doesn't ignore it though.

- Phil
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes
Good catch, I didn't notice the different GUIDs, thanks.

So this issue now falls into the same category as the other JPEG format issue: http://feedback.photoshop.com/photosh.... While technically conforming to the industry standard, the atypical layout of the XMP information, surely not intended by the developers, will cause LR's users significant practical problems:

1. Software like Google Photos/Picasa chokes on the XMP metadata and refuses to show any metadata at all. If Exiftool, the widely acknowledged authority on metadata, didn't properly handle LR's layout of XMP metadata, what's the likelihood of the typical mediocre software app handling it properly?

2. Duplicating the XMP-crs information twice can add a lot of bloat to files containing lots of adjustment-brush strokes. For example, it could take 1.5 MB or more to represent the strokes in a photo that's had a lot of brush work. This bug causes that to be duplicated, adding another 1.5 MB to the file size needlessly. This could be significant when publishing Web-quality photos. E.g. a high-quality 2400-wide JPEG might take 1 MB without metadata; with the XMP metadata stored properly, that would expand to 2.5 MB; with the XMP metadata stored by LR 6, it would expand a further 60% to 4 MB. Telling a user not to export any metadata isn't always an option, because she may well want EXIF camera info, keywords, captions etc. exported, and LR natively doesn't provide any mechanism for easily suppressing the develop settings only.
Photo of John Ellis

John Ellis

  • 117 Posts
  • 12 Reply Likes
Yes, if you don't do step 6 (remove all the XMP-crs metadata), then you can't see any of the EXIF metadata in Google Photos.
Photo of David Franzen

David Franzen, Employee

  • 100 Posts
  • 20 Reply Likes
Thanks -- I sort of misread your original post.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thanks David for looking into this. And thanks again John for your work pin-pointing this situation. Hope this gets corrected soon.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thanks John! As you pointed out, this was "surely not intended by the developers".
Photo of Tobias Frenzen

Tobias Frenzen

  • 5 Posts
  • 0 Reply Likes
Same problem! Thought I was going mad! My website relies on the EXIF data of uploaded images and it displays garbled up and useless information since the update to Lighroom 6! This needs to be fixed ASAP as it's a KEY feature for most Photographers to have accurate EXIF data in their images!

Some examples:

Camera Make (Manufacturer) : Empty (used to display NIKON CORPORATION)
Camera Model: CORPORATION (used to display NIKON D750)
Photographer: xx xx:xx:xx (used to display Photographer's name)
Copyright: shows Photographer's last name from field above (used to display camera copyright text)
Photo of John R. Ellis

John R. Ellis, Champion

  • 3772 Posts
  • 985 Reply Likes
I suspect you're encountering a different problem. Does this problem occur for photos that don't have a lot of brush strokes from the adjustment brush? If so, then it's the other problem: http://feedback.photoshop.com/photosh....
Photo of Tobias Frenzen

Tobias Frenzen

  • 5 Posts
  • 0 Reply Likes
Thanks, didn't realise there were two bugs! That other link looks more likely as there are no brush strokes used here, just standard adjustments.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3779 Posts
  • 987 Reply Likes
A workaround is to set the export option Limit File Size To to be a very high number, e.g. 30000 K (30 MB). This was suggested by an Adobe employee in another bug report that might have the same cause as this issue:

http://feedback.photoshop.com/photosh...

I've tried a couple of exports and it seems to work.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
First off, thank you so much John for all your hard work. I know it takes a lot of time & effort to figure out this stuff. Adobe should put you on their payroll.

So I tried some cursory tests with this Limit File Size workaround. Now the Android Gallery Slideshow function seems to work better. But still there are some photos that are out of sequence when they were taken within the same second (used to sequence down to Sec./100 I think, no time to research this in detail right now). Also, now one export process runs at 100% CPU. I used to run three parallel exports to use up CPU and still had a little CPU left. Now it's too frustrating getting LR6 to respond quickly enough so I can get the other exports started.

Unfortunately, I don't have the time to do more in-depth research and testing on this. I need to get my work out the door and don't want to redesign/change my work flow processes because of this LR 6 issue. So I back to using LR 5.7.1 even though it doesn't support my GPU & 4K monitor well. Nevertheless, it would be nice to have LR 6 working like LR 5.7.1 EXIF data wise.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3779 Posts
  • 987 Reply Likes
"Also, now one export process runs at 100% CPU. I used to run three parallel exports to use up CPU and still had a little CPU left. Now it's too frustrating getting LR6 to respond quickly enough so I can get the other exports started."

LR 6 uses all your processors to do an export, making each export go faster, so you don't need to split a large export into smaller pieces and fire them off manually, which is what people used to do in LR 5. But I've seen a lot of reports that LR is unusable during an export now, which suggests it's being too aggressive at using all available CPU for the export.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thanks John for your reply. I was unaware that LR 6 uses all my processors at 100% for exports compared to LR 5. I've only run one or two large export batches using LR 6 and didn't see the 100% CPU until my "Limit File Size" test. And I guess it would be much easier to start one large batch export anyway.

However, since realizing the EXIF data problem a month ago (which messed up my work flow processes with my customers), I haven't been using LR 6 at all. There is much less risk running LR 5.7.1 for my production work since I know it works within my work flow.

Moreover, using the "Limit File Size" workaround, any ideas about photos taken within the same second? I think the IPTC "Date Created" goes out to 1/100 sec. and maybe LR 6 is truncating the 1/100 sec. part (haven't had time to looking to this). Any help would be appreciated.

All in all, I would love to use LR 6 for its use of my GPU and increasing its response when using my 4K monitor. But right now, I can't trust it for my production work.

Anyway, Adobe should hire you. You are great. Thanks again!
Frank
Photo of John R. Ellis

John R. Ellis, Champion

  • 3779 Posts
  • 987 Reply Likes
"any ideas about photos taken within the same second? I think the IPTC "Date Created" goes out to 1/100 sec. and maybe LR 6 is truncating the 1/100 sec. part (haven't had time to looking to this)."

I just did a quick test, and LR 6's export does preserve fractional seconds in capture time. Note that the EXIF and XMP capture-time fields can store fractional seconds, but IPTC cannot. So when you export a photo whose capture time has fractional seconds, LR will properly set EXIF:DateTimeOriginal, EXIF:SubSecOriginal, and XMP:DateCreated to have the fractional seconds. But it will set IPTC:TimeCreated to omit the fractional seconds.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thanks John for verifying this for me. However, given my cursory test, the sequencing in Android's Gallery Slideshow is off when displaying photos taken within the same second.

Never-the-less, once I get a chance, I think I have a test in mind to verify the sequencing in Android's Gallery Slideshow. I'll upgrade one of my smaller LR 5.7.1 catalogs to LR 6. Run an LR 6 export with the Limit File Size workaround. Then see if Android's Gallery Slideshow sequences the same sub-second photos the same way as photos exported with LR 5.7.1.. I hope to do this test next week, maybe Thursday or Friday. I'll get back to you with my result.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
I just notice that the LR6.1 June update is ready for download, but I haven't tried it yet. Has anyone tried it and if so, does it fix this issue? The "Bug Fixed" list doesn't seem to reference this issue ( https://blogs.adobe.com/lightroomjour... ). However, I hopeful that it does and they just left it off the list.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3779 Posts
  • 987 Reply Likes
The bug still exists in LR CC 2015.1. I haven't tested it in LR 6.1, but it's safe to assume it isn't fixed there either.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thank you so much John for all your help and hard work.

But instead of me spending a lot of time trying to find the exact cause for my Android Gallery Slideshow app sequencing issue and going crazy doing it, I found a better Android app to do my customer slideshows. So I changed my workflow to use the Android QuickPic Gallery app. (I really didn't want to change my workflow, but all-in-all this change is the best option for my situation.) This app allows me to easily select, in a number of ways within a folder, a slideshow's sequencing order and the by "file name" is the selection I'm using. (You would think that the Android Gallery app would have a by file name order option, but it doesn't.). And since my file names are already named by yyyy-mm-dd-hh-mm-ss and trailing sub-second suffix if needed for uniqueness, I'm "home free" and in control of the slideshow's order no matter when this LR6 XMP issue is fixed.

Never-the-less, hope Adobe fixes this LR6 XMP issue for you soon. Good Luck!
Photo of John R. Ellis

John R. Ellis, Champion

  • 3772 Posts
  • 985 Reply Likes
Well, I'm glad you have a workaround.
Photo of David Franzen

David Franzen, Employee

  • 100 Posts
  • 20 Reply Likes
John,

Can you try again to reproduce your original issue: no Exif appearing in Google Photos? I tried to today with your "example from step 4" photo today and it displays the basic Exif information. Perhaps Google Photos has fixed support for extended XMP on their end.

Photo of John R. Ellis

John R. Ellis, Champion

  • 3772 Posts
  • 985 Reply Likes
Agreed, it appears that Google Photos does correctly show the EXIF fields for that example photo (and a couple of new ones created with the same recipe).

However, the most recent desktop Picasa for Mac (3.9.139.218) and for Windows (3.9.139.218) still trip up on them. (I don't think Google is very actively maintaining Picasa?)

I don't have access to Android devices and thus can't test out whether they've fixed the issue in Google's Android gallery (which I think some people were complaining about).

Note that LR CC 2015.1 is still writing the XMP-crs info twice into the file, adding an extra megabyte or two for extensive brush strokes.

Thanks.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Just to report back, I have both Google's Android Gallery 5.0.1 "Lollipop" (the most up-to-date version) & 4.4.2 "Kitkat" (the previous version). Both still have the problem I described a month ago with its slideshow function. Also, many other Android photo apps (e.g., Phorganizer) I think use the same "parts" (data structure, shared code, ... ) that Gallery uses to read the .jpg file meta data so they will and do error out too. And I guess these Android apps could eventually update their code to conform with LR 6.x's .jpg meta data.

But more importantly, does Adobe considers this to be a problem with LR 6.x that needs to be fixed and will fix it? Or are we just out of luck and just need to live with it & find workarounds? My hope is that Adobe will implement a fix soon (maybe a check box within export to write the meta data like LR 5.7.1.)

Frank
Photo of David Franzen

David Franzen, Employee

  • 100 Posts
  • 20 Reply Likes
John,

I have not been able to reproduce the problem of the XMP repeated twice. With the step 4 JPEG I do see that the JPEG has several extended JPEG blocks. It might help me if you could post a raw, DNG or JPEG file from step 3 in your original steps, but without those adjustments applied to the image yet and "baked into" the pixels. Then I could try exporting that file to see if the export produces a JPEG with multiple copies if the XMP. So far I've not seen that happen, but if you have a file that, when exported, consistently produces the result when doing an export, that would help.

Thanks,
David
Photo of David Franzen

David Franzen, Employee

  • 100 Posts
  • 20 Reply Likes
John -- nevemind. I've found a way to reproduce the problem. Thanks.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3772 Posts
  • 985 Reply Likes
OK, good.
Photo of David Franzen

David Franzen, Employee

  • 100 Posts
  • 20 Reply Likes
One workaround for now would be to go back to your original file and re-export a new JPEG. That new JPEG might have extended XMP (based on the amount of metadata and the metadata export options), but it should not have duplicate copies. After export, do not make any additional metadata edits to the exported file, if it contains extended XMP.
Photo of David Hovi

David Hovi

  • 1 Post
  • 0 Reply Likes
Same here. Problems with Exif data in various forums and websites, including my own Wordpress site with Exif plugin installed (tried all available through WP repository). Simply seems the 'industry standard' is not so standard if every plugin under the sun used online does not conform to it. Back to old version, though I frankly find it unacceptable considering we're now essentially paying for something that fails to deliver a very basic functionality for photographers on the premiss that Adobe does it right, and everyone else should readjust. If a third party LR plugin can solve, or at least circumvent the problem, why is it such a big deal to push a hotfix addressing the issue?
Photo of Dawid Ciecierski

Dawid Ciecierski

  • 9 Posts
  • 1 Reply Like
"though I frankly find it unacceptable considering we're now essentially paying for something that fails to deliver a very basic functionality for photographers on the premiss that Adobe does it right, and everyone else should readjust"

Could not agree more. I see this plus silence on the issue (i.e. "yes this is a bug we're going to fix it in the next point release" or "sorry we are not fixing this stick to LR5") as both disrespectful of users.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
I totally agree. The silence from Adobe is "deafening".
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2648 Posts
  • 339 Reply Likes
On July 7 an Adobe employee said he could reproduce the problem.

That is as much feedback as you'll probably get until the next release.

It is not disrespectful to us customers if we aren't privy to Adobe's internal project management and bug tracking systems.
Photo of Dawid Ciecierski

Dawid Ciecierski

  • 9 Posts
  • 1 Reply Like
This bug is not a minor inconvenience but something that trips up pretty much everything outside of LR6, including our clients' software. If you're a photographer with little technical skill or not willing to fiddle with EXIFtools this is a show stopper.

All I was saying was that I see this issue as serious enough to warrant at least an acknowledgement ("yes it's a bug not feature") and confirmation this is worked on... or a "no sorry it's not getting fixed go elsewhere".

All we got in the past *three months* was "oh yes, the software does behave as you guys say it does". I expect more from a tool marketed as one for professionals who depend on it in their job.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2648 Posts
  • 339 Reply Likes
The Adobe employee's exact phrase, above, is:

"I've found a way to reproduce the problem."
Photo of Chris Cox

Chris Cox

  • 20280 Posts
  • 814 Reply Likes
If they can reproduce the problem, then they're also working to solve the problem.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Thank you Chris for your official acknowledgement. Hopefully we'll see this problem fixed in LR6.2.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3779 Posts
  • 987 Reply Likes
This problem appears to be fixed in LR 6.1.1 / CC 2015.1.1.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Great, thanks John! I'll try it out once I get a chance.
Photo of Frank Chan

Frank Chan

  • 47 Posts
  • 5 Reply Likes
Did a quick test. Looks like it's fixed. Thanks