Lightroom: More Space-Efficient Catalogue Storage

  • 2
  • Idea
  • Updated 7 years ago
Lightroom catalogues and their backups need a lot of hard drive space.

I suggest that they are stored in compressed form. As an experiment I compressed a LR catalogue with WinZip. A 56.3 MB catalogue was reduced to 5.86 MB. That's just a fraction over 10% of the original size.

I fully support the idea that files should be usable outside LR and may even be human readable (like as XMP files). However, if an open/common compression scheme were used, users could still perform the uncompression step themselves.

At least backups should be stored in compressed form. If there is no LR support for going back to a backup, the backup could be self-extracting, i.e., create an uncompressed version of the catalogue again.
Photo of TK

TK

  • 531 Posts
  • 110 Reply Likes
  • hopeful.

Posted 7 years ago

  • 2
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 135 Reply Likes
I dislike this idea for the on-line catalog, for two reasons:

- It adds an additional level of complexity and risk to the already-fragile catalogs
- It won't save an appreciable amount of space since the previews folder tends to be 100 times as large as the catalog anyway.
Photo of TK

TK

  • 531 Posts
  • 110 Reply Likes
I don't think proven compression techniques add any noteworthy risk. I wouldn't mind having less space hungry preview folders either, but that doesn't mean that catalogue size doesn't matter. In particular w.r.t. multiple backups.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
We could make the catalog much more efficient in its storage without using compression on it. Compression is surprisingly CPU intensive an ineffective at compressing the data except when applied to the whole catalog at once.

The wins for more efficient catalog storage are less disk I/O and more of the catalog fitting in memory in SQLite's page cache.

It could be argued as a valid disk space tradeoff for multiple backup catalogs, though at least for my part, I'd only back up very old catalogs and leave at least the most recent couple uncompressed.

But then, I know how to repair a damaged SQLite file, I know a lot less about repairing a damaged zip archive.
Photo of TK

TK

  • 531 Posts
  • 110 Reply Likes
I'm all for making catalogue contents more space efficient. I wasn't arguing for local -- within catalogue -- compression where the benefit / effort ratio wouldn't be good. But adding a global catalogue compression even on top of more efficient catalogue representations doesn't hurt (at least not for older backups).

I don't think damaged zip archives are an issue in practice. I've been using them for ages and never had an issue with them. Just to clarify, I'm definitely in favour of having catalogues and other LR files interrogable and manipulatable outside of LR. A user should still be able to unzip a catalogue and then feed it into SQLite, for example.
Photo of Sean Phillips

Sean Phillips

  • 159 Posts
  • 44 Reply Likes
I accept the risk of compressing my backup catalogs (using TPG LR Backup plugin) because they're huge and very expensive to store and backup across my local and cloud backup systems. I think that catalog backups should be compressed by default.

I backup my catalog weekly and it automatically gets distributed across my local and cloud storage devices. This takes a significant amount of time and bandwidth just to manage my image files, never mind my catalogs.

My 2 GB catalog backups compress down to under 200 MB, which makes it feasible to store them on multiple drives on my local network and into the cloud. If they were left uncompressed then simply backing them up would be far too time and bandwidth intensive and it would be difficult to keep up with everything else that needs to be backed up.
Photo of TK

TK

  • 531 Posts
  • 110 Reply Likes
Good point, Sean.
Photo of Dan Tull

Dan Tull, Employee

  • 172 Posts
  • 38 Reply Likes
Yeah, pushing across (relatively) slow networks is another good place to use compression on the catalog. Of course, after fixing the other inefficiencies, they won't actually compress nearly as well.