Lightroom: Please add face recognition to Lightroom (ability to specify region metadata)

  • 313
  • Idea
  • Updated 3 years ago
  • Implemented
  • (Edited)
Do you plan to implement a face recognition defined by keywords in Lightroom someday ?
Photo of ManuelL

ManuelL

  • 1 Post
  • 0 Reply Likes

Posted 7 years ago

  • 313
Photo of KyleT

KyleT

  • 3 Posts
  • 0 Reply Likes
Add face recognition algorithms similar to what Aperture has. Perhaps include something along the lines of people tagging so that people aren't just key words in a photo.

This reply was created from a merged topic originally titled
Face recognition & People Tagging..
Photo of John R. Ellis

John R. Ellis, Champion

  • 3724 Posts
  • 974 Reply Likes
If Adobe decides to add face recognition, I hope they do a better job than they did with Photoshop Elements, which had severe problems with its recognition.
Photo of Paeter Illman

Paeter Illman

  • 6 Posts
  • 1 Reply Like
well i hope they do something at all.

it ́s a shame that most apps have FR but lightroom not.

even apps you get for free have it.

but i guess like the MAP module it will be a medicore attemp at best.
freeware apps have better geotagging then lightroom.
Photo of BenD

BenD

  • 25 Posts
  • 10 Reply Likes
I'm all for this IF sufficient Dev resources are committed to to it's being implemented well. In other words, if it's going to be half-assed, don't bother. There's plenty of other things to tweak and improve closer to LR's core functions.
Photo of D-Two

D-Two

  • 2 Posts
  • 0 Reply Likes
@BenD: Well Put
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
First a face detection would be OK
-> I mean just detect faces with out saying who is there can help a lot.
Then add face recognition

But the most important thing is to be able to tag an image region (like a face for instance)
1. Start to make it general : ability to associate a region (saved in xmp) with a Keyword (normal kind of keyword we have already)
2. then these Keywords can be used for face taging
3. then in the database some of these keywords can be associated to face characteristic allowing an algorithm to make face recognition.

Finally the most important in face recognition is not how well the algorithm performs, but how well results are presented so that you can easily correct any errors or validate good guess.
Maybe we can group faces by similarities (with a given threshold) like Picasa seem to do.
A same person may then appear in different such group (because the person got older, or ...) and a same keyword (person keyword for instance) would then be associated with 2 or more face characteristic. this way we can eleminate the problems of photoshop element who get worse at guessing people faces when you have tagged too many faces of the same person (old faces pollute the ability to detect youger faces and vice versa)
regards
Eric
Photo of Brett Veenstra

Brett Veenstra

  • 3 Posts
  • 2 Reply Likes
Presentation is very important and something Picasa has got right (even over iPhoto). The annoying thing about Picasa Face recognition is that you can't tweak anything, or even tell if it's still running... until you suddenly get new results (try the Grouping Faces feature)

LR integration with Keywords is crucial, and possibly a way to integrate EXISTING Keywords to help with the detection would be great.

I typically take have a shooting spree where I have taken dozens of photos with the same people, so presenting that / building an algorithm should prefer that I would hope.
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
no they don't. At least in the version I use,I did not checked if it got updated.
By the way the anoying thing with PICASA and IPTC: person shown is that there is no hierarchy
-> whenyou have 1000 of names it becomes a real pain to find the one you need for tagging (though PICASA lets you type it and auto complete, which helps a lot)
Photo of john beardsworth

john beardsworth

  • 956 Posts
  • 196 Reply Likes
"By the way the anoying thing with PICASA and IPTC: person shown is that there is no hierarchy"

Lightroom could still add hierarchy without it being IPTC-supported (as it does with collections for instance). The trouble is, family structures are often messy, and can be fluid over time.
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
I would hate if the face information were not stored in the XMP data.
That is already one of the biggest issue with PICASA, and is why I do not use it anymore.
Note that Microsoft already created a XMP structure to store face information. (It is just a little bit too much microsoft centric since one of the record is related to a MSN account, but you do not need to fill it)
By the way it would be nice if lightroom could read these tags (and write them) as well.
PS: As for family structure I have no problem with them so far, keeping it simple, with just the family name.

But maybe we could invent a XMP tag that would allow for complex tree such as family tree ...
Photo of john beardsworth

john beardsworth

  • 956 Posts
  • 196 Reply Likes
Remember that XMP is only the language or syntax used. There's an IPTC standard field - Person Shown. Build out from there - and the X stands for (user) extensible.
Photo of Jim Isaacs

Jim Isaacs

  • 5 Posts
  • 0 Reply Likes
Please develop family tree face tagging
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
I added a related feature request:

ability to tag a picture region
Photo of John R. Ellis

John R. Ellis, Champion

  • 3724 Posts
  • 974 Reply Likes
The Metadata Working Group, of which Adobe and Microsoft are members, has already defined a standard method for representing regions and metadata associated with those regions. In particular, keywords, map locations, and persons (as already defined by the standards) can be associated with regions. This all gets stored in XMP metadata, of course.

See "Guidelines for Handling Image Metadata, Version 2", section 5.9 for more details.
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
that is exactly what we need!
I would like then LR to offer us the ability to tag region and link them with Keywords(whether it is a face region or not),( and Person In Photo IPTC/XMP tags when we specify it is a face region).
That would be the firrst step towards face recognition.
Second step would be to extract faces (just face recognition - no person identification) , so that it would improve manual face tagging.
Third step would add the face recognition.
When I speak in term of step I mean version of lightroom, as I understand that developping all that at once may take time and may not be free of bugs.
So we can take time to get there.
Photo of Steve McGinty

Steve McGinty

  • 7 Posts
  • 5 Reply Likes
Picassa's face recognition works really well..

I would hope though that if Adobe do add this feature (please do) that they do not get tempted to incorporate this into social networking sites... I do not like the idea that people can tag your picture on line without your permission. let alone be able to publish your photo online.....
Photo of Steve McGinty

Steve McGinty

  • 7 Posts
  • 5 Reply Likes
add into the product facial recognition

This reply was created from a merged topic originally titled
Lightroom: add face recognition.
Photo of Dan Zemke

Dan Zemke

  • 5 Posts
  • 11 Reply Likes
Google just acquired a facial recognition software company named Pittsburgh Pattern Recognition. See

http://blogs.wsj.com/digits/2011/07/2...

So Apple has it, and it looks like Google will be using it. It sure would be easier to keyword photos with face recognition, assuming it worked well. Instead of viewing it as a gimmick for the masses, as opposed to serious photographers, try viewing it as a workflow automation improvement that will become a requirement over time.

Fast forward a few years. Imagine LR creates an XML file, or mini DB of attributes attached to each photo, derived from things like image processing, GPS coordinates and date/time. Potential keywords could be presented for easy selection like: Dad, Julie, John Warnock, Christmas, Wedding, River, Mountains, Horse, Fog, Dawn, Yosemite, Motorcycle, Woman, Rain, Tornado, Fire, Red, Fruit ... You get the idea.

This kind of stuff will be prevalent for still and video captures in a few years, by all the computer-based applications (desktop and laptop and maybe tablet).

So Adobe - lead or follow?

Dan
Photo of Dan Zemke

Dan Zemke

  • 5 Posts
  • 11 Reply Likes
And not just Macs and PCs. What if Picasa and YouTube have auto keywording assistance and LR doesn't?

Will more LR users begin uploading their photos to a cloud for post processing that LR doesn't provide?
Photo of yr4uku

yr4uku

  • 4 Posts
  • 0 Reply Likes
add feature Face Detection with easy way to zoom, magnifying glass.
to test for large photo facial expressions
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 135 Reply Likes
I decided to take a second look at face recognition after my disastrous first test with PS Elements 8. I tried out Windows Live Photo Gallery's version.

I gave it a folder of about a thousand images, pretty much all of them of people (my kids, mostly). It took about 10 minutes, and came up with a bunch of faces for me to identify. That took a few minutes. Done right?

I then went through image-by-image and had a look. It missed about 75% of the people in the images, not identifying them as faces at all. This seemed to be caused by the fact that the full face wasn't in the image, either in profile, partially obstructed, wearing sun glasses, or something else.

I then proceeded to manually do what it hadn't done. It took a little over two hours.

Since I had to touch each image manually anyway, I estimate that the face recognition technology only saved me a few minutes over doing the whole thing myself manually.

I think this is a difficult problem and I don't see a way to overcome it.
Photo of Brett Veenstra

Brett Veenstra

  • 3 Posts
  • 2 Reply Likes
"I think this is a difficult problem and I don't see a way to overcome it." - this is the crux of innovation. Adobe needs to find/acquire/dedicate the resources to make this possible.

To Eric's comment: Picasa is a good alternative for now, but the workflow is a large burden. I used Picasa to index photos, recognize faces and then added standard Keywords for those images. In LR, I had to re-read metadata from this files and then assign the "Person Shown" in IPTC metadata. Ridiculous
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
Yes, and I gave up on this as well...Too much work to bring it back to lightroom.
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 135 Reply Likes
"From my experience : picasa is doing a much better job you Should try it. "

Okay, I tried Picasa on the same images.

It did much, much worse than Windows Live Photo Gallery, it was slower, harder to use, and very buggy. It missed a lot more really obvious faces it had no reason to miss plus it missed all the one WLPG missed too.

Took half an hour to scan the faces versus ten minutes, and would have taken at least 10 hours to manually enter them all versus 2.5 hours. I gave up and uninstalled it.
Photo of Lee Jay

Lee Jay

  • 990 Posts
  • 135 Reply Likes
Interesting. I just gave Elements 9 a try. It did slightly better than Windows Live Photo Gallery, missing fewer of the ones wearing sunglasses. It still missed a lot of the profile and partial face shots, but it also ran by-far the fastest at about 1.5 minutes for the same set of images, and it only took about half an hour to manually add the ones it missed. Interesting, especially given how badly Elements 8 did on a completely different set of images.
Photo of Derek Marsano

Derek Marsano

  • 1 Post
  • 0 Reply Likes
My PSE9 catalog contains over 99,000 images. A large number of these photos are of youth soccer players from a variety of teams, and many photos contain multiple faces (e.g. team photos). As many players are unknown to me, it is very helpful to tag a face with a player's name, and to find all photos (e.g. action photos) containing that player.

Yes, at first PSE9's face recognition appears to work substantially better than PSE8. However, my experience is that it simply stops working after some number of images are processed. If you re-run Find->Find People For Tagging, nothing happens. Worse, other features are impacted: you cannot use F11 for full screen (must exit and restart app), and you must re-load thumbnails for many images (right click->Update Thumbnail, may or may not work).
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
Note that part of the job could be done outside lightroom.
more and more camera have this feature, to adjust focus on the faces, even dslr (in live view mode). some of them store this info in metadata. So if lightroom could read that, this could help save computing time?
Photo of Vlad Plaiasu

Vlad Plaiasu

  • 9 Posts
  • 7 Reply Likes
It would be an excellent tool. At this point I use Picasa to detect and tag people but unfortunately the work around is not good for DNGs. I work events, set photography for films and publicity and face detection would save me lots of time. I even considered to migrate to Apple Aperture but the face detection in Picasa was better and the tools easier to use.

Please implement it.
Photo of Rory Hill

Rory Hill

  • 242 Posts
  • 35 Reply Likes
I have not used facial recognition tools and found the above conversation fascinating. My only reservations are that the technology does not appear to be reliable at this point and that LR still has basic stuff to get right, such as the keywording and performance rendering previews.

This seems to me to be an ideal function to be addressed by a third party addin.
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
What about LR4
It would be nice to have it in the final release
Photo of Michael Zacher

Michael Zacher

  • 2 Posts
  • 1 Reply Like
- It is 2012, everybody has it, including PSE -> it is doable (probably quite easily as adobe has the technology
- It makes life easier -> that what LR is all about!
- It is not dangerous, as people already use keywords for it -> perhaps even more secure, e.g. a small checkbox "do not export personal metadata" on export

-> Do it! I really miss it!
Photo of f_rele

f_rele

  • 7 Posts
  • 1 Reply Like
Please add this to LR. Even if it does not work properly it would be nice addition and start of a future development. This could be refined in each release like other features. Just the ablity to recognize a face and manually tag images would be great or search parameter "people in images".
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
Is it going to make it in LR4?
No idea but here is an interesting interview with Tom Hogarty on Dpreview

One feature that some users had hoped to see in Lightroom 4 is face recognition, in which the software identifies specific faces in images and embeds this information as metadata for easier image searches. Hogarty says that when allocating resources for a new release, highest priority is given to features that benefit the greatest number of users. 'Face recognition is very important to some', he says, 'but irrelevant to others, leading to [internal] debates about what solutions are tackled in a release cycle.' Perhaps even more important, he notes there are serious privacy concerns about, 'the ability of software solutions to collect person-specific information.' He says that the challenges in implementing a face recognition workflow in Lightroom involve: 'privacy controls, integration with third party solutions like Facebook, tolerance for false positives - and the effort required to correct them - as well as the time required [by the user] to teach recognition tools.'


So instead of having a full featured face recognition couldn't we have at least
-Region Tagging
-face detection (detect faces but not who does the face belong to)
This would at first greatly help the tagging process, with little recognition errors (less than trying to recognise who is who) and no privacy issue as the software does not automatically recognise people, just make it simpler for people to catalog photos with people on them (which photograph already do, today, anyway)
Regards
Eric
Photo of sean

sean

  • 256 Posts
  • 43 Reply Likes
EXACTLY!! Face-finding is very useful for image development shortcutting such as "quick face zooms" and to judge images based on people blinking or smiling or whatever. As a professional, I don't give a rip about who the hell the person actually is. I mean, it might be useful if LR could identify them as "Nameless ID 123" so that it could show me all of the images in this folder with that face, but it doesn't have to be "Jenny"
Photo of Michael Zacher

Michael Zacher

  • 2 Posts
  • 1 Reply Like
Well, interesting to hear this internal things. But also sad, as I really hoped to see that. As somebody else stated, the privacy issue is no issue to me: Now I do it manually, which is the same amount of information in the metadata but much much more work. Even a FR with say 90% accuracy would reduce my work by 90% (ok +- false positives and so on but hey, also 8% is nice).

best

Michael
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
I would even say that adding this function to LR will make privacy problem less important than today!
Why?
simply look at what they have done with geotagging

up to today I used to geotagg all of my photo outside lightroom, and when exporting them I had to option strip all of the metadata or keep all including private geotagged position. So my photo that were directly exported on the web had the position in them because I did not want to strip all of the metadata. (there are plugins that helps you do a better job - but not everybody knows about those)
Today LR4 let you define private position that will never get exported, or provide an option not to export any position.

Don't you think that the same thing will happen if face tagging is included?

Regards
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
We could add:
The thing is people always tagged (up to know using keyword only and not image region+keyword) people in photos and it won't stop. Auto tagging is just a tool.
It is like with knife
should we forbide knife because they can be used to kill
well no . This would be annoying for people that use them to cook, and people that may use them to kill will just use something else.
Should we avoid automatic face detection in LR
well I do not think so, this will keep "annoying" people that want to use it as a cataloging tool, this will not stop people that want to tag you on the web, as they will use other tools such as facebook, picasa, google+,...
regards
Photo of f_rele

f_rele

  • 7 Posts
  • 1 Reply Like
From Tom Hogarty's interview: "Lightroom involve: "...privacy controls, integration with third party solutions like Facebook..." Why on earth would we like to integrate our LR lib face detection data with FB. Why we can't just get a tagging tool with decent FR and leave it there. If someone decides to create FB plugin for integration that's an separate issue.
Photo of Timothy O'Callaghan

Timothy O'Callaghan

  • 12 Posts
  • 3 Reply Likes
seriously ... WE NEED face detection in LR4 with a NICE user interface like Picaso 3 ....The interface in PS10 Catalog is terrible.... I have 30,000 pictures and on each one where it says "Who am I", I have to type the entire name....when the suggestions aren't correct (most time) ....whereas in Picaso, if I start to type Bob, when I type the letter B, it gives me all the names starting with B and I click the name.
The option should be an option to deal with people concerned with privacy issues....they can turn it off.
Facial recognition is NOT a consumer only feature but something professionals need also !!!!
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Given that face recognition will probably not be in Lr4, I wonder if a 3rd-party plugin writer will step up. Its easy and fast to pull image data from the previews to analyze. If there's a command-line app to do the recognition, the plugin would be fairly easy to write... - The quality of results would be at the mercy of the command-line app of course...

Begs two questions:
- is there a good enough command-line app?
- would people use the plugin, or just continue to do without until native?

Rob
Photo of Babar_e

Babar_e

  • 138 Posts
  • 23 Reply Likes
Hi Rob
the problem with that approach is that you won't get to visualize the tagged region in LR interface which is a big draw back
Photo of Rob Cole

Rob Cole

  • 4831 Posts
  • 382 Reply Likes
Right - you'd have to view it in a plugin dialog box, or a window of an external sidekick app. - still not good enough, right?
Photo of John R. Ellis

John R. Ellis, Champion

  • 3691 Posts
  • 963 Reply Likes
I took a brief look at the implementation issues a while ago. I think a useful plugin needs the following components:

- A user interface that lets users view and modify the rectangular regions of where faces have been detected, view and modify the people tags associated with those regions, and confirm the faces that are automatically associated with tags. Windows Live Photo Gallery and Picasa are two examples of such non-trivial interfaces.

- Face detection: A module that detects the rectangular regions containing likely faces. Detecting faces reliably is well understood, and the OpenCV library has code that does it.

- Face recognition: A module that associates a face identified by face detection with the corresponding people tag. In a quick survey a while ago, I didn't find any well-established open-source library that does this well. The best libraries appear to be commercial or proprietary inside Google, Microsoft, and Apple. Anyone taking this on should be prepared to invest a lot in understanding the application of machine learning to get good results; as a counter-example, Photoshop Elements licensed a third-party face recognition library but botched its application (at least in PSE 8).

- Code for manipulating the XMP region metadata as defined by the Metadata Working Group (this is not supported by LR, or any other application at this point, as far as I know).