Lightroom Classic CC 7.3.1 Corrupts Catalog

  • 1
  • Problem
  • Updated 2 months ago
  • (Edited)
I have experienced catalog corruption error messages when trying to backup the catalog. I have posted information in the Adobe Lr Community forum here (https://forums.adobe.com/thread/2484945 )Problems definitely exist in 7.3.1 that did not exist in 7.3 (which I have rolled back to). Mac OS 10.13.4 (High Sierra)
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes

Posted 3 months ago

  • 1
Photo of Rikk Flohr

Rikk Flohr, Official Rep

  • 3439 Posts
  • 618 Reply Likes
Do you have any text from the error messages? Any screen captures? 
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Hi Rikk--

Yes, I do. Here is a sequence of four. The first is after the backup process started...



Then, after clicking on OK...



Then, after clicking on Repair Catalog...



and shortly thereafter...




I then just clicked on Quit. As I said in one of my posts, after all this I was able to double-click on "Lightroom Classic Catalog_2018-04-20.lrcat" and Lightroom opened it without a complaint. So, it seems that either it wasn't corrupt or it was repaired (or maybe something else).

Thanks,
Dan
Photo of Sumeet Choudhury

Sumeet Choudhury, Employee

  • 29 Posts
  • 10 Reply Likes
Hi Dan,

Thanks for reporting. 

- Just wanted to know since you are saying you rolled back to 7.3, so did you open the catalog second time in 7.3 or in 7.3.1 ?
- Is your issue resolved currently ?
- Can you try this again with another catalog(after taking a manual backup) ?

Thanks,
Sumeet
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Hi Sumeet

Yes, I have been able to open the catalogs (more than one) in both 7.3 and 7.3.1. I have tried it with a couple of catalogs. Please see my post to the Lightroom Community forum referenced in my original post here..

Do I think this issue is resolved? NO. 7.3 has a number of issues that 7.3.1 attempted to fix, so that’s not a preferred version to use. I would prefer to use the most recent version but it now has a bug that it “corrupts” catalogs upon trying to back them up. How do I know I don’t have a database integrity issue like the error messages say? Has 7.3.1 somehow corrupted my database and there is something lurking in there?

Are you saying for me to use 7.3.1 and just do manual backups? Is 7.3.1 OK to use except for backing up?

Thanks,
Dan
Photo of Sumeet Choudhury

Sumeet Choudhury, Employee

  • 25 Posts
  • 8 Reply Likes
Hi Dan,

Rest assured, we are investigating the issue.

And as for your question - Is 7.3.1 OK to use except for backing up? 
- Absolutely yes.

Thanks,
Sumeet
Photo of Sumeet Choudhury

Sumeet Choudhury, Employee

  • 25 Posts
  • 8 Reply Likes
I tried the following steps on MacBook Pro (15-inch, 2017), High Sierra 10.13.1 with Lr Classic 7.3.1

- Created a new catalog on Lr Classic 7.3.1. Imported photos.
- Took a backup of my catalog
- Opened it directly from the Backup location. Opened fine.
- Copied it to the default location after deleting the original catalog. Opened it from this default location. Opened fine without issues.

Am i missing any steps ? Let me know if you want me to try out something else.

Thanks,
Sumeet
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
I encountered the same issue. It appears backups made using 7.x can be corrupted, but not reported during the backup  integrity check. Both the original catalog file and backup file may in fact appear to be working fine when unzipped and used as the working catalog, but fail integrity checking.

I was able to "fix" my catalog corruption by installing LR 7.3.1 and running the catalog repair process again. No more corruption and it now passes the integrity check and can be backed up. At the LR Forum post it appears you've already tried catalog repair with 7.3.1. Is that correct?
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Todd, how did you run the catalog repair process in paragraph 2?

I am guilty of not thoroughly reading this post the first time I saw it.
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
I ran the Integritiy check on launch (Alt/Option key) and then chose the option to let LR repair the catalog, which was  successful and opened automatically after the repair.
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Well, 7.3.1 is where the trouble started. Since 7.3.1 is where the problem exists for me, there’s no way I know of to fix it with 7.3.1. Nothing passes the integrity check. I have gotten the error messages above on several catalogs using 7.3.1. No problems with 7.3 or 7.2 for that matter. As I said earlier, I am using Mac OS 10.13.4. From what someone wrote in the Lr Forum, it was Windows OS that had the problem in 7.3.
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
It sounds like your current working LR Catalog file is OK and It's just the catalog backup files that fail the Integrity Check. Is that correct? If so close LR, hold down the Alt/Option key, and restart LR. In the popup check the option for 'Test integrity of this catalog' and make sure your current working LR 7.2 catalog file is selected. Does that pass the Integrity check? Try the same with your previous catalog backup files created with LR 7.3 or earlier that were NOT opened in LR 7.3.1. Do they pass?
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
No. The "Current" catalog, which has seen 7.3.1, and a couple of manually backed up catalogs from before 7.3.1 (but probably from 7.3) fail the Option-Restart integrity test. I don't have any manually backed up catalogs from 7.2.
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
This is a long post. Hopefully it is read all the way through. I know it might be confusing, so I apologize in advance, but the amount of testing with different catalogs and different versions of LR made it so.

Hmm. Well, initially I was grateful you pointed out how to run an integrity test outside of the backup process. Now, not so much. :-) Before I forget, is there a way to find out when I first installed 7.3?

It seems as though I have nothing but a bunch of bad catalogs and no way to fix them. Here's a run-down of what I've done today:

At the start of this run-down, I am running 7.3
1. My "Current Catalog" (has seen 7.3.1, the catalog that initially failed the integrity test but would still open in 7.3.1 and 7.3) opened OK. I then closed LR, held down Option key and restarted LR, selected "test integrity" and it proceeded to fail integrity test.

2. Went back to a catalog that had not been opened in 7.3.1(one that had been manually backed up), but had been opened in 7.3. Opened fine in 7.3 again. Quit LR, Option-Restart, checked test integrity, and failed. Tried to repair, but no go.

3. Rolled back to 7.2.

4. Double-clicked on catalog that had been opened successfully in 7.3 (not tested in 7.3.1) and 7.2 said it was corrupted from the get-go. This was a catalog from 4/15/18 that was backed up by LR.

5. I've gone back to a LR-backed up catalog from 4/9/2018 and tried to open it in 7.2 and it won't open (it's been trying to open it for 15 minutes). I'm not sure what version of LR I was using back on 4/9, so I don't know what it was backed up with. Do you know of a way to tell?

So, it looks like I am dead in the water. I seem to have catalogs that are corrupted in some way that will open in Lightroom, but can't pass an integrity test. I have older LR-backed-up catalogs, but Don't want to use them (the first one I tried LR doesn't appear to want to use either) because I have created a book since then (spending days and days) and that is saved in the catalog. 

I have two catalogs that seem to work: the "Current" catalog and one from 4/20/18 that is manually saved on two separate external hard drives. I am pretty sure I was using 7.3 when I manually saved them. None of these pass an integrity test, and I don't think any of them can be repaired.

As I was writing this and waiting for 7.2 to open the 4/9 catalog (which I gave up on), I saw another LR backup folder dated 4-30. This was created using 7.3 yesterday when I was testing things. It is a copy of my "Current" catalog. Since it was a part of a zip file, I know it was created by LR, and it had to pass an integrity test to do that. I unzipped it and moved it to another folder. It opened in 7.2. I quit LR and Option-started, selected the catalog, and checked test integrity. It failed. I clicked to allow it to repair. It failed. Will a catalog created in a later version of LR (7.3) fail an integrity test using an earlier version of LR (7.2) just because 7.2 can't understand some of it?

After this, I'm still not sure I have any good catalogs, and it appears as though I am toast. Quite an unhappy camper. Sure, I can run LR, but am I just laying the groundwork for something bad down the line? I don't really want to put a lot of work into something and have it all go down the tubes.

@Sumeet, if you've read this, do you follow? While writing this I went and click don the link Todd included in one of his posts. It seems as though I am caught up in the same issue that johnshore uncovered and documented in his post. I have backups created by LR that are corrupt. How could Adobe let this happen and not warn people back in February? How am I going to get one of my catalogs repaired?
(Edited)
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
This all sounds very similar to my experience with corrupted LR 7.x catalog files, but mine were repairable using LR 7.3.1. LR 7.2 failed the repair process with all catalog files that had integrity check errors. The fact that you can still open and work with them in LR is promising and means the catalog probably can be repaired. Perhaps an Adobe SQlite Database Engineer can take a look at it.

I ran my catalog through a free trial copy of SQLite Expert Professional and there were some missing develop settings history step indexes. It was also able to successfully repair the catalog same as LR 7.3.1. If you can post a working catalog.lrcat file to a file sharing site we can take a look at it. I can't promise anything, but at least it should tell us what's corrupted or missing. Use your current working catalog file and ZIP a copy manually outside of LR then upload the ZIP file. Make sure you have all of your catalog files copied to a separate backup drive. I'm sure you already know this, but better safe than sorry.
(Edited)
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Hi Todd-
I have a catalog that was backed up successfully by 7.3. I tried opening it in 7.2 and it opened fine. I then moved up to 7.3 and it opened successfully. I quit 7.3 and copied that catalog to a new folder. Option-restarted 7.3, selected catalog and it failed integrity test. Selected Repair and repair failed and said it was corrupted. I then doubled-clicked on that "failed-corrupted" catalog and it opened in 7.3 without a problem.

I then moved up to 7.3.1. I started 7.3.1 and it opened the same "repair-failed-corrupted" catalog without a problem. I quit 7.3.1 and Option-restarted and selected the "repair-failed-corrutped" catalog. It failed the integrity test and failed to be repaired and Lr said it was corrupted. I started 7.3.1 again and selected the double r-f-c catalog and it opened without a problem (did not select integrity test be run).

So, even though the integrity test at open says it has problems and can't be repaired and is corrupted, various versions of Lr can open it. So maybe, somehow, it can be repaired by someone. Unfortunately, I wasn't as "lucky" as you to get 7.3.1 to fix it.

I have made a copy of it and compressed it using Finder and uploaded it to my Creative Cloud Assets area. How do I get it to someone who can look at it? Do i just post a link to the file here or do I send a link to someone via email, or what?

If it would be better for me to upload a file that hasn't been backed up by Lr, I can do that too. I have one from 4/20 that has manually be backed up to a external hard drive that I can zip and upload. 

Thanks so much for your support.
(Edited)
Photo of Robert Somrak

Robert Somrak, Champion

  • 108 Posts
  • 28 Reply Likes
When you get your catalog to load have you tried selecting "All Photographs" and select all the photos and doing an "Export as Catalog". This may clean up some of the mess in the catalog and give you a working catalog.  Also, make sure you save all the metadata to XMP incase everything goes haywire in your catalogs. At least your edits except for virtual copies will be ok.  If I actually need a VC I have an export preset that creates another RAW on disk with the final VC settings and then get rid of the VC.  This "workaround" eliminates the problem of losing VC's in XMP.  Disk space is CHEAP.
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
"I have made a copy of it and compressed it using Finder and uploaded it to my Creative Cloud Assets area. "

Not sure there is a way to send a private message in this forum. You can send me the CC share link via a private message here: https://forums.adobe.com/people/trshaner
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Todd, I sent you a message with the link. Thanks again. Good luck.
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
@Robert
Thanks for the suggestion. As far as I know, the problem with exporting as catalogs are: no VCs (as you said, but you do have a workaround), no collections (not sure of this), and most importantly for me if there are no collections, no books. If I'm incorrect on these, I'd be happy to learn otherwise. But...

Even with those "problems", I gave it a go. Unfortunately, Lr "Quit due to an unexpected failure". I also tried to create a new catalog and then Import from Another Catalog.... It likewise quit after importing only 1505 of more than 57,000 photos. It "shutdown prematurely due to a problem reading the catalog."

Maybe that shouldn't be too surprising. Upon opening the 1505 image New Import catalog, I noticed that none of the keywords are present. Maybe that happens later in the process (after it imports all photos)? Also found it interesting that Importing did include virtual copies. Unfortunately, it didn't go to completion and it doesn't look like I can only import a portion of the "From" catalog.
(Edited)
Photo of Robert Somrak

Robert Somrak, Champion

  • 84 Posts
  • 15 Reply Likes
It was worth a try.  Considering the issues you are having I would be very hesitant to keep using these catalogs beings they don't export or import as catalogs.   

You could try this to create a new working catalog.

 Save  the metadata to ALL the files in your catalog
Import all the files to a NEW catalog using ADD
I realize collections and a lot of other stuff will be missing but the edits will be safe which is probably the biggest issue.

 The export/import catalog does have the VC's.  What I was talking about was saving the XMP to the files on disk so if the catalog does go haywire at least you have the edits.  When you save the XMP the VC's are not saved so my workaround.  After Todd was having all his problems with catalogs I quickly saved XMP for all my files and developed the workaround for VC's.  I wish the XMP would save VC data like they do snapshot data. 
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Gotcha. I always save to XMP. I go back and forth between having it done automatically and doing it manually, but it always gets done after an editing or keywording session, just in case. 

This is a very frustrating situation and really...(fill in the blank).

I wish Adobe or someone would create some way to save all the stuff in the catalog that can't be saved in XMP into something that isn't a catalog.
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
Dan, I did a check using SQLiteExpert Professional 5 with the below results. I then ran the SQLite ExpertProfessional 5 ’Repair’ utility, which was successful. It also passes SQLiteExpert Professional 5's integrity check, but fails LR 7.3.1's check. It also fails the LR repair process, but try it on your Mac system. I've forwarded the catalog file share links to Adobe Engineering (and you via PM) so perhaps they will have better luck.



Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Todd

Thanks for all the work you've done and time you've spent. I downloaded the repaired catalog and unzipped it. I double-clicked on the catalog and Lr 7.3.1 would not open it "due to an unexpected error."

I then Options-started Lr, selected the repaired catalog with an integrity test and it came back as corrupted. I clicked on Repair and Lr "encountered problems reading or writing from disk". The dialog box said it might work in future attempts, so I clicked on Try Again. Half a dozen tries did not result in anything new, so I gave up retrying. It sounds like this is familiar territory for you with this catalog.

Just for the heck of it, I tried double-clicking on the repaired catalog again, but to no avail. Thanks again for your help. I look forward to hearing from the Adobe Engineers.

Dan
Photo of SatishT

SatishT, Employee

  • 59 Posts
  • 19 Reply Likes
Hi Dan,

Thanks for sharing your catalog. The investigation for this issue is in progress but we have been able to recover your catalog.

I have sent you a private message with a link to the recovered catalog. Please check it out, it should get you unblocked.

-Satish
Photo of Todd Shaner

Todd Shaner, Champion

  • 954 Posts
  • 301 Reply Likes
Glad to help. Here are some more ideas on recovering the catalog data:

What can't be saved to metadata?

Pick/RejectFlags, Collections, Virtual Copies, Stacking, Sort Order, Develop History,Publish Services

Also Books, Slideshows, and Web Galleries cannot be saved external to the catalog file.


No metadata is saved in video files.


You can add Pick and Reject flags as keywords same as for collections. Virtual copies can be exported to TIFF or JPEG files.

For Books you could try selecting only the images used in the book and 'Export as Catalog.' Depending on the catalog corruption this may not work, but worth a try. This is also a good way in the future to 'Save' Books to a secondary backup. As you work on a book, at the end of each editing session, select all of the image files in the Book Collection, and 'Export as Catalog.' The catalog files shouldn't be very large and this gives you an incremental backup. I haven't tried it, but it should work.

Adobe Engineering has confirmed they are investigating your catalog file. I'll keep you posted. ;>)
(Edited)
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
All good ideas Todd. Thanks.
The biggest issue with books is the text. For the most part, for books that I send to Blurb to create actual hardcopy books, I also create PDF files. I make a lot of PDF books so I have the PDF files for those too. So not a total loss if the catalog would have been unrepairable. But I do have a rather large book (and associated collection) in progress that I would have lost (but wouldn't have been impossible to recreate (only slightly painful)) as I haven't yet written any words. I'll be using your idea about adding keywords for the collections. I'll also experiment with Exporting the book collection as a catalog to see if the text goes along with the images.

Dan
(Edited)
Photo of Todd Shaner

Todd Shaner, Champion

  • 954 Posts
  • 301 Reply Likes
Dan, I just tested a book 'Export as Catalog' and the file is ~330 KB per page using the below settings. it appears to save everything including text. Just select the Book Collection and Export. The export catalog only contains the image files in the Book Collection, but it's all there and can be edited further. LR books should be fully recoverable in the event of an unrecoverable catalog file. Just make sure your image file folder structure is backed up AND you have Develop edits saved to metadata. It appears to be a quick and easy secondary LR Book backup.

Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Yes, I had just tested it too around the same time you were. I tried a book that I just recently published that was over 100 pages and the catalog file came out to be 35 MB or so. Very nice. I'll have to remember this great tip.

Yep, I always have metadata saved to XMP sidecars.
Photo of Dan Roeder

Dan Roeder

  • 29 Posts
  • 4 Reply Likes
Adobe Engineers have recovered my catalog and it appears to be working fine (it also passed the integrity test upon opening the catalog). I am back up and running, with a copy of the catalog manually copied to two external hard drives.

My sincerest gratitude and thanks to Satish and any other Adobe Engineers who worked on this, to others who have replied to my post, and especially to Todd Shaner.
(Edited)
Photo of Todd Shaner

Todd Shaner, Champion

  • 989 Posts
  • 324 Reply Likes
Special thanks to SatishT and the Adobe Engineering Team for a speedy catalog recovery!  It looks like SatishT and I posted at about the same time. Anyhow the tips I provided for saving more catalog information to metadata should still be helpful.
Photo of Daniel Low

Daniel Low

  • 6 Posts
  • 0 Reply Likes
I am running the same OS X operating system version (10.13.4) and Lightroom version (7.3.1) as this person on two macs. I use a SAN to store my image files and Export as Catalog from one computer and Import from Another Catalog on my other mac. When attempting to import the catalog, the second mac gets the following error:

Lightroom cannot import from this catalog.  The catalog does not appear to be a valid Lightroom Catalog.  

I don't want to revert to 7.3, I don't even know how.  I just want the problem fixed so I can continue to move files from one catalog to another, maintaining my Lightroom edits. 

Please fix this problem. I feel this is a fundamental flaw in the current mac release and should be fixed ASAP.

Dan
Photo of Todd Shaner

Todd Shaner, Champion

  • 920 Posts
  • 280 Reply Likes
"I should be able to save as a catalog with one computer then import catalog with another computer, correct?  I have been transferring photos between computers this way for at least one year. "
If your copying the LR catalog to the SAN for the 'Import From Another Catalog' transfer function this is a know issue that started with LR Classic (7.0)

https://feedback.photoshop.com/photoshop_family/topics/lightroom-classic-cc-import-from-external-cat...
Photo of Daniel Low

Daniel Low

  • 6 Posts
  • 0 Reply Likes
i am not copying a catalog, I am selecting some photos and using the command "Export as Catalog".  This command copies the pictures and creates a new catalog file that can be imported into another Lightroom catalog with the command "Import from another catalog".  
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 3846 Posts
  • 1271 Reply Likes
Sorry Daniel, you're right, I meant to ask you to copy the CATALOG to the internal storage. I suspect the same issue Todd notes, and that's the easiest way to fix it.
Photo of Todd Shaner

Todd Shaner, Champion

  • 920 Posts
  • 280 Reply Likes
Daniel, this post concerns catalog corruption, which is not what you are experiencing. LR Classic no longer supports network based 'Import from another catalog' function. As Victoria mentions manually copying the catalog and associated picture folders to your local storage is currently the only way to use 'Import from another catalog.' You can add your comments, 'Me To' vote, and 'Follow' at the relevant problem report link I provided above. This will help get Adobe's attention. Thank you!
Photo of Daniel Low

Daniel Low

  • 6 Posts
  • 0 Reply Likes
I understand that the original post concerned corruption, but it was the closest I could find on the internet that related to both my version of OS X and version of Lightroom Classic CC.  I will try this with the catalog on the local driver versus a network drive.  

Thanks for the advice!

Dan