Lightroom: Wrong timestamp stored in catalog causing wrong metadata status (all Windows versions)

  • 5
  • Problem
  • Updated 19 hours ago
  • Acknowledged
  • (Edited)
Hi,

The problem I describe below is not new (I observed this at least since version 3) but this time I took the time to investigate more deeply...

From time to time, LR tells me that the XMP file of a given image is no longer in sync with the metadata in the catalog. Most often, this is correct because I made changes without recording the XMP file (Ctrl-S - I'm not using the automatic XMP updating mode). But very often, this information is simply wrong. I didn't change anything to the image and suddenly, the little down arrow appears in the upper right corner of the thumbnail.

Hitting Ctrl-S may or may not fix the problem. Sometimes, the little down arrow reappears after a few seconds or minutes although I didn't do anything (hands away from keyboard and mouse).

I recently did the following test for multiple images unduly displaying the "metadata status changed" flag. I compared the following values :

1. Windows "last modified" timestamp for the XMP file.
2. Value of the xmp:MetadataDate field in the XMP file.
3. touchTime column value for that image in the Adobe_images table of the catalog (which is a SQLite database).

The touchTime value is stored in a special format, so I had a hard time converting it to a readable date/time value. However, I will not explain this and how I navigated the database in order to access this timestamp (this requires some knowledge about databases).

Result:
For all the images tested, values #1 and #2 were always strictly identical. The touchTime value was always off (sometimes about 10-15seconds, sometimes much more). So no wonder that LR thought that the XMP file and the metadata in the catalog were not in sync.

Moreover, the difference in time can be negative of positive. So LR displays the up or down arrow accordingly (meaning that the XMP file is older or newer than the metadata in the catalog, respectively - which is wrong in both cases).

I explained above that sometimes, the image reappears as "not in sync" just a few seconds or minutes after I did a Ctrl-S. In that case, a quick look at the database showed me that the touchTime field had not been updated. So the time difference causing the image to be flagged as "not in sync" was still there. In that case, the problem can be fixed by reading the metadata from the XMP file which was actually correctly updated. This operation updates the catalog and everything is in sync again.

Anyway, there's something very wrong in the computation of the touchTime value of the Adobe_images table. That seems to be obvious. This wrong timestamp generates in turn a wrong metadata status.

I made another interesting test :

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

So now I know what's going wrong but I'd like to have this problem fixed after all these years.

Thanks in advance.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
  • frustrated

Posted 3 years ago

  • 5
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
When I say, "all Windows versions", I mean "all Windows versions of Lightroom". I didn't check what happens on a Mac.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
> the little down arrow appears in the upper left corner

upper right corner.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
Up!

Not even a single answer after one year. Bug lasting since years but still not fixed.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
This reply was created from a merged topic originally titled Wrong timestamp stored in LR catalog generating wrong metadata status (all Window....

[Restarting this bug report which has been totally ignored since one year - bug lasting since 5 years]

Hi,

The problem I describe below is not new (I observed this at least since version 3) but this time I took the time to investigate more deeply...

From time to time, LR tells me that the XMP file of a given image is no longer in sync with the metadata in the catalog. Most often, this is correct because I made changes without recording the XMP file (Ctrl-S - I'm not using the automatic XMP updating mode). But very often, this information is simply wrong. I didn't change anything to the image and suddenly, the little down arrow appears in the upper right corner of the thumbnail.

Hitting Ctrl-S may or may not fix the problem. Sometimes, the little down arrow reappears after a few seconds or minutes although I didn't do anything (hands away from keyboard and mouse).

I recently did the following test for multiple images unduly displaying the "metadata status changed" flag. I compared the following values :

1. Windows "last modified" timestamp for the XMP file.
2. Value of the xmp:MetadataDate field in the XMP file.
3. touchTime column value for that image in the Adobe_images table of the catalog (which is a SQLite database).

The touchTime value is stored in a special format, so I had a hard time converting it to a readable date/time value. However, I will not explain this and how I navigated the database in order to access this timestamp (this requires some knowledge about databases).

Result:
For all the images tested, values #1 and #2 were always strictly identical. The touchTime value was always off (sometimes about 10-15seconds, sometimes much more). So no wonder that LR thought that the XMP file and the metadata in the catalog were not in sync.

Moreover, the difference in time can be negative of positive. So LR displays the up or down arrow accordingly (meaning that the XMP file is older or newer than the metadata in the catalog, respectively - which is wrong in both cases).

I explained above that sometimes, the image reappears as "not in sync" just a few seconds or minutes after I did a Ctrl-S. In that case, a quick look at the database showed me that the touchTime field had not been updated. So the time difference causing the image to be flagged as "not in sync" was still there. In that case, the problem can be fixed by reading the metadata from the XMP file which was actually correctly updated. This operation updates the catalog and everything is in sync again.

Anyway, there's something very wrong in the computation of the touchTime value of the Adobe_images table. That seems to be obvious. This wrong timestamp generates in turn a wrong metadata status.

I made another interesting test :

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

So now I know what's going wrong but I'd like to have this problem fixed after all these years.

Thanks in advance.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3709 Posts
  • 968 Reply Likes
I copied this version of your post to the first post, correcting the formatting bungled by the forum software.
(Edited)
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
Thanks very much, John.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
This annoying issue is still waiting to be fixed. It's not hard to reproduce. At least, an acknowledgment would reward all the time I spent tracking down this bug lasting since the beginning of Lightroom.
Photo of Tom Mickow

Tom Mickow

  • 288 Posts
  • 95 Reply Likes
Perhaps there aren't any comments / acknowledgement because others aren't experiencing the same problem you are?  Doesn't mean you're not seeing it, but I for one am not.
Photo of Patrick Philippot

Patrick Philippot

  • 349 Posts
  • 62 Reply Likes
Hi,

Maybe others are not seeing the problem for any reason (e.g. not using LR the same way as I use it) but it's easy to check what I have explained above, that is, the creation of wrong time stamps in the catalog. Any developer capable of opening the LR SQLite database can check this.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3709 Posts
  • 971 Reply Likes
There's been a steady stream of occasional complaints over the years about incorrect metadata status.  Some of the causes have been resolved in earlier versions, but clearly not all.  For example, see this recent post: https://feedback.photoshop.com/photoshop_family/topics/problem-saving-metadata-if-certain-adjustment... .  Adobe employee Rikk Flohr has been involved in that topic, and I've replied there, referencing this topic.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3709 Posts
  • 971 Reply Likes
Patrick, great detective work. You've observed that LR is using one time value for file last-modified date and XMP:MetadataDate, and another time value for "touchTime". It's very possible that LR grabs the current time for XMP:MetadataDate, then writes the metadata to disk, then updates "touchTime" with the new current time. Usually, that happens very quickly, so the two times don't differ. But if there is a delay in writing the metadata to disk due to operating-system or disk vagaries, they could differ by a lot, especially when writing large JPEGs. (LR sometimes or often rewrites the entire JPEG when it updates the metadata.)
Photo of Chris

Chris

  • 50 Posts
  • 4 Reply Likes
I wonder how similar the two issues are, though.

However, this would certainly explain the problems I was having a fair while ago - where doing the 'read' after a 'save' would indeed fix the status.  Even more on point, was that I used to access the image files on a USB hard drive which I would then map as a network drive, in order to be able to disconnect the images and work on just the smart previews.  (Before LR gained that preference item to do so.)
     That issue became far less prevalent after moving the images onto an internal sata drive (but still mapped as a network drive).  Although, IIRC, I may still have encountered a couple/few where the save-read would resolve status.  But I haven't encountered any of these since removing the mapped reference and directly accessing the images, albeit it hasn't been that long.
(It may also be worth noting that I've always used DNG, where I suppose it may be more possible for the two above timestamps to differ..  Maybe similar to jpg, as John notes.)

The issue at the other thread is not resolved by the save-read.  And is only resolved after removing the problem brushing from the currant settings, as well as in any snapshots.  I'm not saying they couldn't still be related somehow, but the workaround is different, on top of the brushings 'cause'.  Maybe that's an avenue that Rikk/Adobe could explore.
(Edited)
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
However, this would certainly explain the problems I was having a fair while ago - where doing the 'read' after a 'save' would indeed fix the status.
This is probably related. When using Ctrl-S to save the metadata to the XMP file and when the little down arrow re-appears after a while although no parameter has been changed for that image, one may wonder whether the metadata have been correctly written to the XMP file. I have checked this and I can confirm that the data were correctly written. This is why the "read metadata after a save" trick works. At least for a while. The problem can quickly re-appear, though. For example, if the preview for this image has been discarded, just browsing that image's folder can make the problem re-appear.

I now have a smart collection named "Unsaved metadata" which uses that simple rule : Metadata status, is not, Up to date. This allows me to see more quickly when images are suddenly assigned a wrong status.
Photo of Chris

Chris

  • 50 Posts
  • 4 Reply Likes
I think I have all the possible statuses in a collection set.  However, with the recent 'offensive brush' problem (for which the save-read doesn't resolve) - I'm still in the process of deciding which way to go...  Either just leave the offensive brush in there as current rendering and/or in snapshots, or keep them in VC's only.  The latter will allow the status to resolve.  In the meantime, I've created a keyword "snapshots-as-VC" and also enter that into the user comments XMP field, for both masters and VC's.  And for some of status smart collections, I've now excluded ones with that keyword.

With the timestamp problem - almost certainly, as there would have been more latency with an external USB drive mapped to a network drive letter, than an internal drive..  Therefor allowing more chances of having the timestamp differences when writing it out to the DNG. What I wonder about now, is if the timestamp problem also happens with proprietary raws - since the XMP file is much smaller.  I wouldn't know though, as all I use is DNG.

It IS amazing, though, that this timestamp issue hasn't been fixed in all these years.
The offensive-brush problem I can see not being fixed yet - as it's only recently started happening in maybe the last 1 to 2 versions or so.  (For posterity, that's LR 6/2015.8 or .7)

However, for either issue/cause, it is certainly unsettling to those of us that like the reassurance of having the belt-and-suspenders backup approach.  :-/
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
What I wonder about now, is if the time stamp problem also happens with proprietary raws - since the XMP file is much smaller.
It happens with any file type including TIFFs  and JPEGs.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3709 Posts
  • 968 Reply Likes
Have you observed that it happens less frequently or not at all with .xmp sidecars?  JPEGs, TIFFs, DNGs, and PNGs all store metadata inside the file, and LR sometimes or often rewrites the entire file when it makes a change to the metadata.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3709 Posts
  • 968 Reply Likes
...and PSDs too.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
I don't see any difference between the various file types.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
Bump!
Photo of Matthieu Kopp

Matthieu Kopp

  • 1 Post
  • 0 Reply Likes
For your information, the timestamps are stored as a time interval (float) that measures the number of seconds elapsed since 2001-01-01T00:00:00Z, that is since the 1st of January 2001 at midnight in UTC time. You can use https://www.epochconverter.com/coredata to do conversions.
Photo of Sean Ruddy

Sean Ruddy

  • 2 Posts
  • 0 Reply Likes
This reply was created from a merged topic originally titled Lightroom Classic: Metadata conflict.

"the metadata for this photo has been changed by both lightroom and another application"
win 10 on a smart raided drive.  System is about 3 years old.  last few months started getting this.  I keep up to date on all software involved. 
No other applications that I am aware of have accessed the files.
I have 30,000 dng files from the last few months with this error randomly mixed into them.  Is there some way to do a mass overwrite from catalog?  Selecting each one individually is extremely tedious.
I can not determine a difference in the files; but I know they have not been edited with other applications.
Photo of Chris

Chris

  • 46 Posts
  • 3 Reply Likes
I've been putting up with this bug just about since I started learning LR in ~2012.  For these, the  save-read trick resolves them.  However, the problems with the offensive brushings is more recent than that, and is NOT resolved by a save-read.

Very irritating, indeed.

As far as not having to select them individually, there are metadata-status criteria for smart collections which will consolidate them.  For the ones that aren't resolved by a save-read, I then added keywords of something like 'metadata problems', and then did a 'save' and created additional smart collections that filtered out those that were keyworded.

In all cases that I've noticed, it seems that LR actually DOES save it properly - it just won't understand that it did.  So you'll have to be careful to manually save metadata for those you may have keyworded as having problems.  ...  And/or periodically select the smart collection that doesn't filter out the keyworded ones, and do a manual save every once in a while to make sure.

I really wish that Adobe would fix this, one of these decades.
(Edited)
Photo of Patrick Philippot

Patrick Philippot

  • 260 Posts
  • 36 Reply Likes
Chris,

> In all cases that I've noticed, it seems that LR actually DOES save it properly - it just won't understand that it did.

I would say : it just won't understand when it did :-) .
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
This reply was created from a merged topic originally titled Lightroom Classic: Metadata conflict (merging).

Hi Sean,

This is a very old bug that has never been fixed although it has been thoroughly documented and analyzed. The problem is a wrong timestamp stored in the catalog. Please read the following threads :

https://feedback.photoshop.com/photoshop_family/topics/wrong-timestamp-stored-in-lightroom-catalog-c...

 (the first post is badly formatted due a forum issue. Just ignore it. It has been rewritten in a subsequent post)

https://feedback.photoshop.com/photoshop_family/topics/problem-saving-metadata-if-certain-adjustment...

The only known way to (temporarily) get rid of  this issue is to select all affected images, hit Ctrl-S to save the metadata and then use the Metadata | Read from file command to update the catalog with the corect timestamp. But the problem will re-appear later anyway.

Note: This conversation was created from a reply on: Lightroom Classic: Metadata conflict.
Photo of Sean Ruddy

Sean Ruddy

  • 2 Posts
  • 0 Reply Likes
This reply was created from a merged topic originally titled Lightroom Classic: Metadata conflict (merging).

Wow looks like you have been looking at this bug for a long time.  Thanks for all your effort!  Seems odd that it A) just started on my system B) no particular pattern as to what files are affected.

The inconsistency bugs me.

One thing that is different in my workflow is that I moved homes so I let images stack up without culling and processing... hence I have so, so many files to get through.

Note: This conversation was created from a reply on: Lightroom Classic: Metadata conflict.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
Please do something about this one. It's incredibly disturbing, given my workflow. It obviously doesn't have the correct priority level in the bug log. This should be fixed since years.

Just a few minutes ago, about 50 images spontaneously and unduly got the "Metadata Status is not up to date" flag and re-appeared in the smart collection I created to collect such items. This is happening every day I'm using LR since years.
Photo of Marco Klompalberts

Marco Klompalberts

  • 27 Posts
  • 4 Reply Likes
With all the detective work Patrick has done, you would think that Adobe would at least have a clue where to look for it.
Having the same issue here.
Once in a while, I will save to file, to make sure that everything is in sync.
I will usually select (with smart collections) all photos from 1 year and Save To File.
But I am hesitant to Read From File, because what if the save didn't go as planned?

Really hope that this will be fixed.

I am on a Windows 10 PC with the most recent update of Lightroom CC Classic installed.
(Edited)
Photo of Marco Klompalberts

Marco Klompalberts

  • 27 Posts
  • 4 Reply Likes
By the way, I noticed you can search for (and have smart collections with) metadata status.
It would be handy if you could also search for metadata date. That way I would be able to Save To File and afterwards Read From File if the metadata date is after the last save time
(hope that thought makes sense ;-) )
Photo of Patrick Philippot

Patrick Philippot

  • 349 Posts
  • 62 Reply Likes
Hi Marco,

This bug exists since the beginning of Lightroom and I have been experimenting with the Save/Read workaround a lot. AFAIK, this procedure is safe. Never had a problem with this. However, this fixes the problem only for a (sometimes very short) while. As soon as the files are accessed again (for example when rebuilding a discarded preview), the problem may appear even if you didn't make any change. Just viewing the images is enough.

I am using a single rule dynamic collection (Metadata Status is no up to date) to regularly fix this mess. Open the collection / Select All / Save.

As for a fix by Adobe, give up any hope. The usual rule applies : once a bug has survived one or two years, you can be assured it will never be fixed. They just don't care and this is a demonstration that the bug  doesn't  prevent the product from being sold anyway. High code quality and professional conscience are no longer a preoccupation at Adobe.
Photo of avpman

avpman

  • 96 Posts
  • 63 Reply Likes
IMHO, the LR dev team and their processes (QA in particular) is no where near the quality of the PS development team. Adobe have the capability but seem content on collecting their monthly fees and not delivering what was promised. I am still confounded why no one has not yet brought a class action law suit against them. Someone who owns a business impacted by these bugs surely would be a good lead plaintiff.
Photo of Patrick Philippot

Patrick Philippot

  • 351 Posts
  • 62 Reply Likes
Still not fixed in 7.5 but I would be inconsistent if I said that I am surprised. :--((