Lightroom: Regular expression option for text matching.

  • 2
  • Idea
  • Updated 4 years ago
Some of us power users could make good use of regular expressions for text matching in library filters and smart collections (and hopefully one day search & replace in metadata and/or folder+file names).

I'd settle for lua regex format only, but options for other popular formats would be good too :-)

I know this wouldn't be used by the majority, but that doesn't make it a bad idea.

Note: People with special needs who don't know how to write their own regular expressions could post requests on the forum (or they could even be shared via the exchange or the cloud...). Also, there's regex magic (translates human-friendly criteria into regex format) and regex buddy (to help build and test regular expressions before using).
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
  • not regular, but some of my expressions are.

Posted 7 years ago

  • 2
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
I'd add my vote for this, too, if the query language of the database engine that LR uses supports regular expressions, which it appears to do:
http://www.sqlite.org/lang_expr.html
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
Why not click the '+1' button then?
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
I was actually imagining the filtering being done in lua code as opposed to sql, but I guess there is more than one way to do it.
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
I can't imagine it would be very speedy, reading the entire database, matching one row at a time with a regex to see what photos are in a collection.

I wonder, now, if the "Searchable" fields in LR have full-text indexes built for them, and searching other ways, even with the database engine's help, would be prohibitively slow.

So I'm basically voting for adding regex to the current searching methods, whether LUA or SQL, but not going from fast SQL to slow LUA-filter loop on top of an entire database-scan.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
10,000 photo loop testing each full path for regex match takes 8 milliseconds.

Anyway, I think Adobe could pull it off either way...
Photo of Steve Sprengel

Steve Sprengel, Champion

  • 2586 Posts
  • 323 Reply Likes
I'm sure in your example, things are in memory already, rather than coming off the disk, so it's not really a database operation, anymore, it's a memory search operation. How fast is it doing a regex against "searchable metadata" or "searchable IPTC" or whatever the phrasing is in the Library filter panel, instead of a single field of filepath, and with a database that isn't already in memory, so perhaps on a USB drive w/o read-caching turned on if that's even possible.

I'm imagining your request for REGEX matching would be presented to the user as an added option to any search-operator dropdown that "Contains" is in, now, for various filtering or searching functions.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
Yes, memory search. Dunno how long it takes to load from database to memory. But, yes I think the dropdown would be a good place to add the option. Anyway, I suppose the implementation should be left to the experts...
Photo of John R. Ellis

John R. Ellis, Champion

  • 3384 Posts
  • 854 Reply Likes
I have an "almost finished" plugin (haven't worked on it for 5 months) that provides smart filtering across all metadata fields made available through the SDK, with an expanded set of operators including string equality testing, numeric comparisons against string fields, and regular-expression matching.

With a test catalog of 20K images, searches using the expanded metadata fields or operators take just a few seconds -- slower than builtin searches with standard fields and operators, but still pretty fast. The implementation is exactly what you'd expect -- enumerate all the images and apply the boolean expression to them one by one.

I think the reason that it's fast is that the data of almost all catalog databases end up being cached in memory by SQLite -- catalogs aren't that big. For example, one of my 20K-image catalogs is only 200MB, which is small change on an 8 GB machine.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
You rock, John! - Looking forward to that plugin if you ever finish it! I assume it will also support the custom metadata fields of other plugins too, right?
Photo of John R. Ellis

John R. Ellis, Champion

  • 3384 Posts
  • 854 Reply Likes
Yes, the intent is to support custom metadata fields (by reading the configs of other plugins). Not sure when I'll be able to finish this, though -- there's a lot of polish left and the demand for it is probably miniscule :-(
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
I'd be happy to alpha/beta test for ya :-)

It would be very valuable for users of DevMeta & ExifMeta, since all date & numeric metadata is in string form which drastically limits what can be done with it in smart collections.
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 372 Reply Likes
Update:

AnyFilter supports lua "regular expressions" (pattern matching), and is fairly fast.
SpaceUrchin also supports such stuff..

(plugins readily findable via internet search)

Rob