Lightroom: Photos incorrectly considered as "changed" by republishing and smart collections

  • 17
  • Problem
  • Updated 6 months ago
  • Acknowledged
  • (Edited)
[Update from John R. Ellis:

At least some instances of this bug are caused by the new develop settings added by CC 2015.6 for the Guided Upright tool (2015.6 was released 6/8/2016). When 2015.6 or later first renders a photo at 1:1 zoom that had been imported by 2015.5 or earlier, it adds those develop settings before rendering. Then it compares those develop settings with the old, notices they are different, and incorrectly marks the photo to be republished.

Here's how to work around this instance of the problem: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

See here for a detailed recipe to reproduce the bug, along with an analysis of the problem and suggested fix: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis... ]
-----------------------------------------------------------------------------------------------------------------

Sometime photos in my published collections began to randomly mark themselves as modified to republish.   Photos I haven't touched in years, in galleries I haven't recently changed, all of a sudden appear under 'Modified Photos to Re-Publish.'   If I even scroll through a collection, dozens of the images begin to jump up to 'Modified Photos'. I can select the photos and send them back to 'Published' with 'Mark as Up-to-Date,' but then more immediately jump up to Modified.
Photo of Mauro Iannicelli

Mauro Iannicelli

  • 11 Posts
  • 0 Reply Likes
  • Frustrated

Posted 2 years ago

  • 17
Photo of Martin Meßmer

Martin Meßmer

  • 11 Posts
  • 1 Reply Like
That sounds reasonable and it explain why I don't see this effect when I zoom-in photos that were last edited w/ the latest Lr version.

I using the Photo StatLr plugin (upload to Synolgy Photo Station) which was written by me.

The publishServiceProvider.metadataThatTriggersRepublish()  function returns { default = true}, i.e. any metadata change will trigger re-publish. I think, most of the Publish plugin will use that setting.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3292 Posts
  • 820 Reply Likes
"The publishServiceProvider.metadataThatTriggersRepublish()  function returns { default = true}, i.e. any metadata change will trigger re-publish. I think, most of the Publish plugin will use that setting."

I'm not expert on implementing publish services, but I think that changes to Develop settings always trigger a republish, regardless of what that function returns.  I say that because this is what all of Friedl's publishing services show:


I think Friedl designed the LR SDK's publish-service architecture, under contract to Adobe, so he would know (only 85% confident about that).
Photo of Martin Meßmer

Martin Meßmer

  • 11 Posts
  • 1 Reply Like
That's exactly how I read the API documentation on this callback:
It allows the plug-in to specify which metadata should be considered when Lightroom determines whether an existing photo should be moved to the "Modified Photos to Re-Publish" status.
No develop settings mentioned here. That means that any publish service will suffer from this (d)ef(f)ect.

I think we now can hand this bug to Adobe on a a silver platter, can't we?
Photo of John R. Ellis

John R. Ellis, Champion

  • 3292 Posts
  • 820 Reply Likes
I just posted a precise recipe for reproducing the bug and updated the original post with a link to the recipe. This is often required to get Adobe to add the bug to their bug-tracking, but it doesn't guarantee a fix.
Photo of Martin Meßmer

Martin Meßmer

  • 11 Posts
  • 1 Reply Like
Thanks a lot for your support!
Photo of John R. Ellis

John R. Ellis, Champion

  • 3488 Posts
  • 888 Reply Likes
At least some instances of this bug are caused by the new develop settings added by CC 2015.6 for the Guided Upright tool. When 2015.6 or later first renders a photo at 1:1 zoom that had been imported by 2015.5 or earlier, it adds those develop settings before rendering. Then it compares those develop settings with the old, notices they are different, and marks the photo to be republished (incorrectly).

Here's a recipe for reproducing the bug:

1. In CC 2015.5, create a new catalog with Catalog Settings > File Handling > Standard Preview Size = 1440 pixels. Uncheck the option Catalog Settings > Metadata > Automatically Write Changes Into XMP.

2. Make a folder with 50 copies of this 5472 x 3648 raw file: https://dl.dropboxusercontent.com/u/21811200/DSC05456%20copy%201.ARW

3. Import the folder into LR and wait until the standard previews have built.

4. Set up the Flickr publish service with default settings, except Resize To Fit: Width & Height = 1000 x 1000 (to make the uploads go faster).

5. Make a Flickr publish album containing all 50 pics and publish it.

6. Exit LR CC 2015.5 and start LR CC 2015.7 on the same catalog.

7. Using the plugin SDK, verify that all of the photos have the develop setting UprightVersion == nil (this is one of the settings added for Guided Upright in 2015.6):

        

This plugin is handy for examining the internal metadata and develop settings of a photo as exposed by the plugin SDK.

8. While viewing the published Flickr album in grid view, select one of the photos, go to Loupe, and then zoom 1:1.  The photo will show up in Modified Photos To Re-Publish. And often the next photo in grid view will also show up, presumably because of LR's pre-rendering policy.

9. Repeat step 8 a number of times.

10. Verify that the photos that appear in Modified Photos To Re-Publish are exactly those with the develop setting UprightVersion ~= nil:

        
        
----------------------------------------------------------------------

Examining the before and after develop settings of a photo, it's exactly the settings added for the Guided Upright tool that appear after the 1:1 zoom but not before:
PerspectiveX = 0, 
PerspectiveY = 0,
UprightCenterMode = 0, 
UprightCenterNormX = 0.5, 
UprightCenterNormY = 0.5, 
UprightFocalLength35mm = 35, 
UprightFocalMode = 0, 
UprightPreview = false, 
UprightTransformCount = 6, 
UprightVersion = 151388160, 


It appears that when
a photo imported into the catalog by CC 2015.5 is zoomed 1:1 for the first time
in CC 2015.7, LR adds the Upright settings with default values to the develop
settings, then it renders the photo.  It
then compares the new develop settings with the old, notices that the Upright
settings are different, and marks the photo for republishing.

 The fix is to treat
a nil value for those settings the same as the default settings, thus avoiding the spurious conclusion that the photo has changed.
(Edited)
Photo of Fredy Benavides

Fredy Benavides

  • 3 Posts
  • 2 Reply Likes
I found a way to fix this issue in all the photos that I have in a Lightroom Catalog. My Lightroom version is CC 2015.7

Follow these steps:

1) Select all your Pictures in your Catalog.
2) Save metadata to file
3) Read metadata from file 
4) Go to your published services Folders and select all the pictures
5) Mark all the pictures as up-to-date.

After doing these steps published folders work properly. No more random pictures appear under "Modified Photos to Re-Publish". 


Enjoy Lightroom again!
Photo of John R. Ellis

John R. Ellis, Champion

  • 3473 Posts
  • 885 Reply Likes
Excellent!

I verified that your steps work around the particular instance of the problem described here: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

For others reading this who aren't as familiar with LR commands, the steps are:

1. In the Catalog panel on the left, click on All Photographs.

2. Do Edit > Select All.

3. Do Metadata > Save Metadata To File.

4. Do Metadata > Read Metadata From File.

5. In the Publish Services panel, click on a published collection that contains Modified Photos To Re-Publish.

6 Right-click one of the select pics and do Mark As Up-To-Date.

7. Repeat steps 5-6 for every published collection affected by the bug.
Photo of seancho

seancho

  • 3 Posts
  • 0 Reply Likes
Well I was quite happy about this workaround until I realized that writing/reading the metadata breaks the Develop module before/after functionality. Not so good.

Writing/reading the metadata apparently resets the develop module before state to when the metadata file was read into the catalog. Previous edit states still exist and can be accessed, but 'y' and'\' no longer show the original imported image.

Images in published collections jumping to Republish was driving me insane, but now I have 1000 images that I can't use the '/' and 'y' key to check before and after. I'm not sure I understand what the read/write workaround is doing, but is there a way to correct the metadata without breaking before/after? And is there a way to repair my metadata to restore correct before/after?
Photo of seancho

seancho

  • 3 Posts
  • 0 Reply Likes
OK, if anyone is in the same position, there is a way to repair the before state. Right-click the 1st 'import' history step, and select 'Copy history step settings to before.' That resets the before state to the original import. Of course, it's only one image at a time, but better than nothing.
(Edited)
Photo of seancho

seancho

  • 3 Posts
  • 0 Reply Likes
Since it appears that the bug is triggered by new versions of LR viewing images published by older LR versions, and re-importing the metadata from file messes with the Develop module 'before' state (and creates many unnecessary files), I think I've found a solution that works better for me - just edit all published images in the new LR version. This will update the metadata for all published images, preventing the Republish bug from being triggered.

This is what I did. One caution... Since this procedure temporarily modifies develop settings of all of your published images, the safest plan is to back up your catalog before starting. If LR crashes while applying edits in Autosync, you could end up with unwanted changes to all your published images. 'Undo' will reverse Autosync changes, but not after LR quits. So...

1. Select all the published collections in Publish Services.
2. Select all the images in grid view.
3. Go to Develop module
4. Engage Autosync (Be very careful with this setting: With Autosync engaged, every edit you make is applied to all selected image at once)
5. Carefully make one small change to a setting not used for any published image. (I chose Grain) If you are editing many published images, you may have to wait while the edit is applied to all the images. Look for the progress bar, upper left.
6. Immediately reverse the edit you just made, leaving the setting as it was before. Wait for change to apply to all the selected images. So now all published images have been updated to the new LR version.
7. Disengage Autosync! (very important. If you forget, future edits could be accidentally applied to all your published images)
8. Return to Library module
9. Go through each published collection and mark all images as up to date.

This seems to have solved the problem for me.

Incidentally, in my case (Smugmug LR plugin) published images were jumping up to Republish without even viewing them 1:1. All I had to do was scroll through the collections in grid view and images would be marked to republish and jump to the top. I couldn't even reliably select images in the collections, because the grid was always changing so quickly. Really frustrating! Nice work by the folks posting to this thread in diagnosing the problem. Adobe really needs to fix this republish bug ASAP.
(Edited)
Photo of Margie

Margie

  • 3 Posts
  • 1 Reply Like
Thank you soooo much! This seems to have fixed the issue for me. I've been putting up with it for months (I too was experiencing the issue when simply scrolling through grid view), but since following your nice clear instructions a couple of weeks ago, I haven't had any images mysteriously marked to republish. Such a relief!
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
This reply was created from a merged topic originally titled Lightroom: Smart Collections - "Edit Date" rule buggy ?.

Hi,

Upon installation, Lightroom creates a set of sample smart collections. Among them the "Last 7 days edited images" and the "Last month edited images".

I'm wondering what "edited" means in that context since I can see in these collections thousands of images that I have not touched since years.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
See the previous post by Paige Miller with very similar symptoms: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

It's very possible that the issue with Guided Upright settings added in 2015.6, described here:  

https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

is also causing LR to mark the photo as "edited" when it adds the new develop settings to the photo.  Or it could be a separate cause.
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
I further investigated this case. I looked at the data related to an image unduly added to the "Last 7 days edited images". This is what I found :

Catalog
Adobe_Images table  / touchTime  field :  2017/02/12    17:32:41
AgLibraryFile table     / modTime field :      2015/05/16     11:57:04

XMP file
xmp:ModifyDate="2015-05-16T13:57:04.46"
xmp:CreateDate="2015-05-16T13:57:04.46"
xmp:MetadataDate="2017-02-16T09:48:06+01:00"

Only 2 time stamps could justify the addition of this image to the collection. So I have successively reset these fields to an older date (this requires conversion to the Cocoa date format). This had absolutely no effect. The image was still added to the collection.

Now I'm wondering which data LR is looking at when determining the last "edit" date.

I also tried to change the "last modified" Windows time stamp of the XMP file itself, resetting it to an older date. No change. The image is still in the collection.

Really strange.

PS : Those interested in playing with the time stamps contained in the LR database will have to convert to and from the Cocoa date format. You'll find an online converter here : https://blog.paddlefish.net/?page_id=90 .
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
This appears to be a very old bug, indeed :

https://forums.adobe.com/thread/1399404
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
"Now I'm wondering which data LR is looking at when determining the last "edit" date. "

The SDK API provides access to a per-photo field "lastEditTime".  This appears to correspond to the Adobe-images.touchTime field in the database.  To test that, I took a several values of "lastEditTime" and searched for them in a text SQL dump of the database, and they showed up in the "touchTime" field.

You might find this plugin handy for seeing all the photo fields that are available to plugins: https://dl.dropboxusercontent.com/u/21811200/showmetadata.1.1.zip
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
Thanks, John. I'll have a look at this plugin. However, as explained above, I already tried to set the touchTime field to a date value that should exclude the corresponding image from the smart collection. This had no effect.

Maybe I made a mistake. I'll try again and let you know.
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
Maybe I made a mistake. I'll try again and let you know.
No mistake. Changing the touchTime field has no effect on the membership to the smart collection.

There's another table, AgLibraryImageChangeCounter,  that has a date field : changedAtTime. It is not stored in the Cocoa format as the other time stamps in the database but as a string.

Not many images in that table and I can't make any connection with the image I'm testing. There's an image field that is obviously there to identify the image but I can't connect it to the rootFile field in Adobe_images or to the id_local field in AgLibraryFile.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
Mysterious.  "touchTime" maps to "lastEditTime" in the API but not "Edit Date" in smart collections. We're clearly missing something.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
My curiosity got the better of me and did some tests. After first, I observed the same thing you did -- changing "touchTime" of a photo didn't change the contents of an "Edited Today" smart collection.

I was making the change to "touchTime" using Database Browser For SQLite (Mac, 3.3.1). But when I started making the change with "sqlite3", it worked exactly as expected, with the value of "touchTime" corresponding exactly to "Edit Date" in smart collections.

Then I noticed that when I changed "touchTime" with Database Browser and then dumped the database with sqlite3's .dump command, the value would be quoted, e.g.

'509059250.086592'

But when I changed "touchTime" with "sqlite3" and dumped the database with .dump, the value was not quoted, e.g.

509059250.086592

My understanding was that SQLite represented everything internally as strings, so I'm confused as to what's happening with Database Browser.  But using "sqlite3" confirms that "touchTime" really is "Edit Date" in smart collections and "lastEditTime" in the SDK API.
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
Hi John,

I did all the data manipulations described above with SQLite Expert Personal 4. So, it may have yet another behavior when manipulating the touchTime field. If I look at the Design tab for the Adobe_images table in SQLite Expert, the touchTime field has no specified type. So I guess it's a string.

Anyway, this morning, my "Last 7 days edited images" collection has much less images in it. So I guess that something happened during the past week that caused a lot of images to be incorporated. This event probably spanned over several days, otherwise I wouldn't still have thousands of images in the collection.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
I refreshed my understanding of SQLite's types. Columns are dynamically typed, and any type can be stored in any column.  The type declarations of columns are hints ("affinities") about how values should preferably be stored. Adobe_images.touchTime is missing a type declaration:

CREATE TABLE Adobe_images (...    touchTime NOT NULL DEFAULT 0)

so according to the documentation it should have type NUMERIC, where values are stored as integer or real if possible. 

DB Browser for SQLite 3.9.1 is buggy and by default stores values in "touchTime" as text rather than numeric values.  According to the documentation, that shouldn't matter because the text value will be converted to numeric in expressions that require numeric values.  But likely there's some subtlety here that causes LR to treat text values in "touchTime" representing numbers differently than numeric values.

Most likely, SQLite Expert Personal 4 is doing something similar, storing the values you enter for "touchTime" as texts rather than numbers.

So I think any further experiments need to be done with "sqlite3".
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
"So I guess that something happened during the past week that caused a lot of images to be incorporated. This event probably spanned over several days, otherwise I wouldn't still have thousands of images in the collection."

I repeated the recipe I posted earlier for showing that the introduction of Guided Upright in CC 2015.6 caused photos to be marked for republishing spuriously: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis... In that situation, LR is adding new develop settings to the photo for Guided Upright whenever you first zoom in on the photo, fooling publish services into thinking the photo has changed.

But the "touchTime" and "touchCount" of these photos isn't changing when that happens.  So I don't think Guided Upright / Zoom is the cause of spurious changes to Edit Date as viewed by smart collections.
Photo of Patrick Philippot

Patrick Philippot

  • 309 Posts
  • 46 Reply Likes
Got it! Actually, it's a side effect of the bug reported here which is lasting since version 1 :

https://feedback.photoshop.com/photoshop_family/topics/wrong-timestamp-stored-in-lightroom-catalog-c...

I just remembered that I have been recently hit by this bug and I found myself with a lot of images for which the metadata status was wrong. So I used the write/read XMP method to fix the problem. Unfortunately, LR considers this as an editing operation although no setting has been actually changed.

Conclusion : bugs that are not timely handled are never fixed by themselves. They just generate more trouble with time. We have now version 6 and the time stamp bug is here since version 1. Time to do something.
Photo of John R. Ellis

John R. Ellis, Champion

  • 3481 Posts
  • 888 Reply Likes
Very much agreed.
Photo of Bhakti Ramnani

Bhakti Ramnani, Employee

  • 5 Posts
  • 2 Reply Likes
Thanks for reporting the issue. We are looking into it.
Photo of Dan Hartford

Dan Hartford

  • 120 Posts
  • 49 Reply Likes
I'm having same problem.  But, just to throw a wrench into this issue, For me it seems to have started when I started putting images into a dumb collections that are synced to LR Mobile.   I DO NOT modify images on any mobile device, only on my desktop.  But I do use LR Mobile on my smart phone to view images.   Could the sync process to LR Mobile or the fact that I view synced images on a mobile device trigger the "modified to republish"  in Publish Services that those images also appear in?
Photo of Rick Spaulding

Rick Spaulding, Champion

  • 564 Posts
  • 219 Reply Likes
As usual, you, our users, have done a fantastic job of addressing this issue and bringing it to our attention. 

Due to it’s obscure nature and that a solid workaround has been discovered we will not be expending any time or resources to fix it. It will only affect those users who are planning to migrate from a version <6.6 to a version >= 6.6 so that number is extremely small. 

Huge thanks to all those who helped sleuth this out and especially to John and Martin. You guys are amazing! 

And John, thank you very much for writing such clear, concise instructions for the workarounds.

Great work!

Photo of Patrick Philippot

Patrick Philippot

  • 305 Posts
  • 46 Reply Likes
> Due to it’s obscure nature and that a solid workaround...

I beg your pardon ? How you dare writing this ? I have spent hours investigating this bug and I have clearly explained what was wrong in the database. You have the necessary information to definitively fix this issue. If the Adobe developers are not able to correct this problem with all the information given here, they must return to school or look for another kind of job. I'm afraid, the real cause is : "We don't care. Pay your subscription and shut up.".

The write/read metadata trick is not a solid workaround because the problem comes back anyway.

This post is pure provocation. Really. I have never seen so much arrogance.
Photo of Patrick Philippot

Patrick Philippot

  • 305 Posts
  • 46 Reply Likes
> I have never seen so much arrogance.

Wrong! At IBM, just before their fall.
Photo of Patrick Philippot

Patrick Philippot

  • 305 Posts
  • 46 Reply Likes
Look again at this thread

https://feedback.photoshop.com/photoshop_family/topics/wrong-timestamp-stored-in-lightroom-catalog-c...

Both problems are related and the bug described there is still not fixed AFAIK. As long as is is not, all related problems are not fixed either.
(Edited)
Photo of Matthew Weinel

Matthew Weinel

  • 4 Posts
  • 2 Reply Likes

I'm absolutely disgusted by Adobe's response to this issue. We are not asking for a favour. We are paying customers.

Unfortunately many people foretold that the subscription model would lead to this kind of behaviour. It seems they were right.

Photo of John R. Ellis

John R. Ellis, Champion

  • 3488 Posts
  • 888 Reply Likes
There might be a new instance of this bug in LR 7.0.1 caused by the introduction of the Range Mask develop setting.  See this user's report of spurious republishing for SmugMug: 
https://forums.adobe.com/message/10026416#10026416

I've confirmed that after upgrading from LR 6 to LR 7, LR adds the develop setting PaintBasedCorrections.CorrectionRangeMask on the fly as you access photos in your catalog.  This could be triggering the spurious republishing, just as the introduction of the Guided Upgright Tool did in 2015.6.
Photo of Adam Paris

Adam Paris

  • 4 Posts
  • 0 Reply Likes
I am also having this issue with my catalog over 4500 published images and a catalog of 200K photos. A "work-around" is not what I'm looking for from a company like this. A true FIX should be made.
(Edited)