Lightroom: Cloud Catalog Syncing on 2 Machines (Catalog breakdown)

  • 1
  • Idea
  • Updated 5 years ago
  • (Edited)
I would like to suggest a way for the Adobe Photoshop Lightroom Catalog to be more incidental in its file structure. I'll explain. I currently have the cloud based service Copy. Through them I have 250GB of cloud storage. What I am trying to do is sync my catalog from my laptop to my desktop. I'm tired of editing on one machine and having to start over or convolutedly export and import each session at a time. The problem i have is that my catalog is 1.7GB. So if i open Lightroom and turn a single photo black and white and close lightroom, Copy is uploading the entire 1.7GB of catalog information. If somehow the Lightroom catalog could be broken up into folders, perhaps by the date accessed or the date of each corresponding file's "photographed" date, then every time i make a change only minor differential information will have changed and Copy doesn't need to upload 1.7GB of data every time. I realize the location of the stored files are of concern. People can either use external disks, or in my case i'm using the excessive storage i have through Copy. The catalog itself could also work for the cloud storage you already provide for Creative Cloud users. I would have gladly stored my catalog on the Creative Cloud but since i have 250GB with Copy I'm also importing the photos themselves onto Copy so the libraries can be the same too.
Photo of Pat Fenner

Pat Fenner

  • 5 Posts
  • 0 Reply Likes

Posted 5 years ago

  • 1
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2656 Posts
  • 341 Reply Likes
The cloud storage provider needs a way to do chunked/block updates so the entire files doesn't need to be copied each time. This method uses an intelligent agent on the computer that computes checksums of each block of the file and only transmits data for blocks that have changed. I would shop for a different online backup provider that specifically mentions bandwidth savings via partial file backup.

Others have requested this from Copy for smaller types of files so you might add your voice to theirs, and it's possible they already do this but don't mention it:
https://copy.zendesk.com/entries/2532...

Using Bridge/ACR is Adobe's answer to folder-centric photo adjustments.

You are free to break up your current catalog into as many smaller catalogs as you want. What you lose is the ability to filter and search across all those catalogs.

The issue with the approach you suggest, of splitting up the LR catalog into many smaller catalogs is that searching and filtering photos would become too slow. There is a bit of overhead in opening a catalog database file. Let's say it takes 2 seconds per database to open them. What happens if you want to search or filter for a specific keyword across dozens or hundreds of folders. It might take several minutes to open all the sub-catalogs. Besides this there would need to be a master catalog that keeps track of all the sub-catalogs.

You might imagine that searchable data could still be stored into the master catalog while non-searchable data, such as update history, could be stored in each folder where the files reside. Whether this makes sense would depend on if the amount of searchable data was much smaller than the amount of non-searchable data. I don't have a good answer for that, although Adobe would likely have some idea and may have nixed the idea of a multi-tier catalog structure based on their analysis.
--
As an aside, how many photos do you have where the photos fit into 250GB of online storage but the catalog is close to 2GB in size? I'd suspect you shoot JPGs rather than RAW because 250GB of RAWs wouldn't be that many photos so the catalog would be much smaller than 1.7GB. Presumably you optimize your catalog from time to time to compress out unused space.
Photo of Pat Fenner

Pat Fenner

  • 5 Posts
  • 0 Reply Likes
Hey Steve,
Thanks for the response. Everything you said makes sense, and i learned a few things! Do you know which cloud services operate like you mentioned? Adobes? Dropbox? I had already started a second catalog so that it would be much smaller and then in the future i can merge them if i need to. I would be great to be able to have them all in one but beggars can't be choosers.

I never intend to use all that storage but I do shoot a lot of raw. It's not near my whole archive, i just keep like 50gigs locally and the rest on redundant external hard drives.

Thanks Steve
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2656 Posts
  • 341 Reply Likes
Here is a large comparison of online backup services. I am not 100% sure but I think block-incremental is the capability you're looking for, which is listed next to various services down below the matrix:

http://en.wikipedia.org/wiki/Comparis...

It seems like your two needed capabilities are the ability to backup a large database that only has a few changes without transmitting the entire database file to and from the service each time, as well as wanting to synchronize a file or files between two computers smoothly.

I'd think many services are wanting to backup continuously but restore to the same computer only occasionally so may not have a smooth way to sync like you want.
--
Personally, I use a $80 USB3-speed portable 1TB hard-drive to hold my most recent photos and catalog, and carry it between home and work. I back it up nightly to another USB-attached portable hard-drive so if I accidentally lose it or destroy it I have the backup that I can use immediately. I actually back up the USB-drive to a secondary internal drive then backup that secondary internal-drive folder to a second USB-drive so I have two backups. Initially it takes some hours to copy everything to a brand new hard-drive, but after that it only takes a few minutes for a photoshoot's worth of work. I have the backup on a schedule so as long as the drive is plugged in at night it gets backed up. And if I forget or want to backup things at a particular time like just after I've offloaded photos before I start working on them, I have manual schedules that backup the current month's work so it doesn't waste time scanning the entire drive for files to backup when I do it on demand.

I do use an online cloud backup service for my computer's local pictures and documents folders (that do not include DSLR photos) in case of a drive failure but I don't plug that up with daily GBs worth of raw photos--that would take took long to sync up to the cloud.