Lightroom: Imports duplicates

  • 2
  • Problem
  • Updated 1 year ago
  • (Edited)
What are the best practices to avoid dups? I shoot a few hundred images on a card and then import. I shoot a few hundred more images(cr2) on the same card and want to import just the new images.....LR many times imports duplicates of what it has already imported. And since it knows after the import that it just imported dups it marks them with a -2. Why did it import the dups if it knows that they are dups?
I have been using the method of selecting import, and then waiting a few minutes with the dialog box of the import open, and then cancel it. And then starting import a second time, many times this is enough for LR to figure out there are duplicates, however, it is hit or miss. How do I avoid duplicates?
Please improve the process of not importing duplicates and let me know how to avoid importing duplicates. Thanks.
Photo of William Virun

William Virun

  • 3 Posts
  • 0 Reply Likes
  • sad

Posted 3 years ago

  • 2
Photo of John Rhines

John Rhines

  • 1 Post
  • 0 Reply Likes
Lightroom developers have truly done miserable job when it comes to building a more accurate and dependable algorithm into their import routine. Lightroom has access to ALL relevant metadata and should be able to avoid importing a single duplicate. I'm more than happy to bark up any tree in order to get a resolution to what should be a no-brainer for Lightroom developers to solve. In fact this should have NEVER become a problem in the first place!
Photo of Becky Sowada

Becky Sowada, Adobe Employee

  • 508 Posts
  • 105 Reply Likes
Hi William,

First, make sure the "Don't import suspected duplicates" box is checked in Import. It's in the File Handling panel on the right side of the import screen.

Second, do you rename your photos on import? If so, Lightroom compares the name of the file you're about to import with files already imported, and does not find a match. It does check other metadata too, but when it isn't 100% sure the file is a duplicate, it will err on the side of caution and import the photo. If you want to rename your images, you can either rename them after import to avoid duplication, or you can format your card after each import and continue to rename as part of the import process.

Becky
Photo of William Virun

William Virun

  • 3 Posts
  • 0 Reply Likes
Becky,
Thank you for your response.

Is the Lightroom engineer team aware that they has a signification duplication problem when importing?

1st - I do have the "Don't import suspected duplicates" box checked.
2nd - I do rename files on the way in to put the time stamp into the image name....I do this because when I do have duplicates in the database, on export the image number is lost by Lightroom because of duplication. By renaming the file when importing with the time stamp YYYYMMDDHHMMSS this is a very unique name. Even with multiple shooters on the same project, I find very few images taken at the same exact second in time.

Based on what you have indicated, LR is unable to determine duplicate flies if the file is renamed. It seems that with the time stamp, camera serial, and other meta data that LR would be able determine duplicates. Please ask the development team to revisit this area to improve the import process, please make it better. This is an area that is clearly a weak spot in the product.

It is unclear how renaming the files after I import them is different from renaming on import. I believe LR would have the same issues not being able to detect duplicates.

Formatting the card after each import is a potential option, however, this does add risk to deleting images that may not have been imported, as the LR import process is untrustworthy. And removes the ability to import the card to the other editors and their systems who are on my team if the card is formatted prior to them importing the images. Many times the other editors on my team are not available to import the card immediately. LR does not really know what is has imported or has not. Losing client images is not an option for any professional.

I shoot many real time events, on location, where the time pressure to deliver images quickly is important. If you assume that 30% of the time LR can not figure out duplicates and creates duplicates and my typical event shoot is 25,000 images, this means I end up with an extra 7500 images. If each image is ~20MB, then this is an extra 146 GB of disk space and transfer time. This is for one of my editors on site. I typically have 3-5 editors on site. So, worse case 7500 extra images X 5 editors is 37,500 extra images due to the duplication limitation of lightroom. Shooting real time events with thousands of images, and having LR be unable to determine if an image is already in the database is a real issue for anyone using the product. I create a new LR catalog for each event shoot we shoot.

Whatever you can do to help the LR engineering team improve the intelligence of the import process, to avoid duplications while importing the new images and only the new images would be a welcome improvement. You assistance would be greatly appreciated!

Thank you. Bill.
Photo of Becky Sowada

Becky Sowada, Adobe Employee

  • 508 Posts
  • 105 Reply Likes
Hi William,

Thanks for sharing the detail about your workflow. This information really does help the engineers understand the exact problems you're facing, and that helps in crafting better solutions. I'll share this with the engineering team.

You said, "It is unclear how renaming the files after I import them is different from renaming on import. I believe LR would have the same issues not being able to detect duplicates."

When you rename on import, Lightroom sees the new name as the original filename in the database. When you rename after import, the filename from camera is saved as the original filename in the database (you can see it in the metadata panel if you rename a file after import). Thus, on later imports, Lightroom can compare with the original filename data as well as the existing filename.

Hope that helps clarify.

Becky
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4477 Posts
  • 1668 Reply Likes
Becky, do you remember when that changed? Because once upon a time (LR3 and later), the camera filename was used for duplicate detection (stored in the import hash?) regardless of whether the photos were being renamed during import.
Photo of Becky Sowada

Becky Sowada, Adobe Employee

  • 508 Posts
  • 105 Reply Likes
Victoria, you're right. My info was faulty. I had heard someone say recently to be careful about renaming on import because of potential for not recognizing duplicates, and it sounded very like William's description. But...I just tested renaming some files on import, and when I tried to re-import the originals, Lightroom recognized them as duplicates immediately. So, ignore that part of my post.

Becky
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4477 Posts
  • 1668 Reply Likes
Oh good, I thought it was an odd thing to change! Thanks for double checking that one.
Photo of William Virun

William Virun

  • 3 Posts
  • 0 Reply Likes
Becky,

Your feedback on rename after import is very interesting. I will update my workflow to rename after import rather than on import. And for your information, why do I rename. When we shoot shoot a real time event, knowing the exact time the image was taken is important. I rename the flies to be the time stamp of the image YYYYMMDDHHMMSS. And I check the exact time on all the camera bodies we shoot with each day. Thanks for your help.
Photo of Edward Bruce-Radcliffe

Edward Bruce-Radcliffe

  • 1 Post
  • 0 Reply Likes
This issue has never been resolved, I am shooting a real time event and rely on the rename on import. Now, I have about 200 hundred duplicates. Thanks ADOBE, way to listen to your customers!
Photo of Joan Ryder

Joan Ryder

  • 5 Posts
  • 0 Reply Likes
I rename the files on import and have always done so.  I end up with -2, -3, -4 and even -2-2, -2-3 on my duplicated files.  It is so frustrating and tiresome to have to keep checking and removing duplicates.  Please help!  Thank you.
(Edited)