Lightroom Classic CC device import misses photos

  • 1
  • Problem
  • Updated 2 months ago
I shoot RAW+JPG.  After a very long trip doing incremental imports from sdcard, I discovered that some of my images in Lightroom were missing the JPG sidecar, and others were JPG only.  Lightroom's catalog matched what was on internal disk after import.  But a check of the sdcard revealed that, in all cases, both RAW+JPG were present.  The number of affected photos was low, but it is my expectation that a DAM's import feature works reliably.

Since returning home, I have been able to consistently reproduce this issue.  Here is what I've discovered:
  • Affects Classic CC 7.3.x, 7.4, as well as CC 2015 6.0.
  • Always happens on first import after a reboot.
  • Always happens on alternating imports.  Imports 1,3,5 affected.  Imports 2,4,6 never affected.
  • Which file will not be imported is predictable.  It always starts the same after reboot, and counts up.
  • Is not related to "suspected duplicates".  Failure happens on fresh catalogs with and without this import setting checked.
  • Only happens when doing device-based import.
  • Happens regardless of if the device is ejected or not between imports.

I do not believe this to be an issue with this particular sdcard:
  • Command line copy of photos to internal disk always succeeds.  Integrity of copies validated through checksums.
  • A clone of this sdcard resulted in the same import behavior.

Here is an example of the behavior observed:
  1. Reboot
  2. New catalog, import via devices: P1840971.RW2 missing
  3. New catalog, import via devices: OK
  4. New catalog, import via devices: P1840972.JPG missing
  5. New catalog, import via devices: OK
  6. New catalog, import via devices: P1840972.RW2 missing
  7. New catalog, import via devices: OK
  8. New catalog, import via devices: P1840973.JPG missing
  9. New catalog, import via devices: OK
  10. New catalog, import via devices: P1840973.RW2 missing
  11. Reboot
  12. New catalog, import via devices: P1840971.RW2 missing

I've also tried incorporating file-based imports into the mix.  Interesting that the file-based imports succeed, yet still increment the file that will be missing when doing a device-based import later:
  1. Reboot
  2. New catalog, import via devices: P1840971.RW2 missing
  3. New catalog, import via devices: OK
  4. New catalog, import via files: OK
  5. New catalog, import via files: OK
  6. New catalog, import via files: OK
  7. New catalog, import via files: OK
  8. New catalog, import via devices: P1840973.JPG missing
  9. New catalog, import via devices: OK
  10. New catalog, import via devices: P1840973.RW2 missing
  11. New catalog, import via devices: OK

For reference, this is a Sandisk Extreme Pro 64GB UHS-1 card.  It contains a mix of photos and videos from a Canon S100 and a Panasonic LX100.  The file counts in the DCIM folder are:
    174_0501:      10
175_0517:      42
176_0530:      54
177_0703:      10
178_0711:      10
179_0813:       8
180_0814:       2
181_0108:      14
183_PANA:    1994
184_PANA:    1995
185_PANA:    1436
186_PANA:      48
CANONMSC:      11

During my import tests, I've also noticed that Lightroom is inconsistent on the number of photos it says are on the sdcard.  I raise this here as it is related to file scanning.  On the import dialog, the lower left number and "New Photos" number always match, but is a slightly different value each time.  But that value never matches the number shown under the destination folder structure (which is always consistent).

This is happening on OS X 10.12.6.
Photo of Nathan

Nathan

  • 4 Posts
  • 0 Reply Likes

Posted 2 months ago

  • 1
Photo of Polly

Polly

  • 2 Posts
  • 0 Reply Likes
Well presented detail Nathan.   I hope you get the answers you seek.

I'm coming over from Aperture and at the moment the two applications are like chalk and cheese.

I'm very new to LR and to LRCC and find similar errors to those you detailed, but I have not the skills, time and patience to detail them so well . . . .   I'm left wondering if it's me not understanding the system or is it the LR system.   From the details express as you have it's obviously the LRCC system. . . . shame.   Have I entered into LR at a too higher level ?  I wonder.

At this point I'm not even going to attempt using the Cloud between my devices. It's a shame that Adobe has rushed LRCC into the market . . . . they have done neither their name nor their customers any favours. 

Are they listening ?
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4261 Posts
  • 1544 Reply Likes
Hi Polly, welcome to the forum. Nathan's talking about the much older Lightroom Classic and its predecessors, which is different to Lightroom CC - and yes, the naming is really confusing. If you have questions, please feel free to post them on a new thread and we'll be happy to help figure out whether it's a bug or just a misunderstanding.
Photo of Roelof Moorlag

Roelof Moorlag

  • 113 Posts
  • 27 Reply Likes
Do you import with the camera connected or via an cardreader?
Lightroom is rather 'picky' about the connected gear so i would try to make some differences there and test that again. Switch all hardware when you can: Card, Cable, Reader/Camera.

The inconsistentcy on the number of photos ion the sdcard could due to fileformats that Ligthroom does not supports. You could check this with a new card with new (test) photo's.

I'm rather parainoid on my image management and i do always check the numbers. The only time i met problems was when i had hardware problems.
Photo of Nathan

Nathan

  • 4 Posts
  • 0 Reply Likes
I've been using the internal card reader on a MBP.  Ruling out hardware issues is always a good idea, so I dug out an old external sdcard reader.  Besides being painfully slow (1hr / 60GB), the exact same behavior was observed.

I don't think the inconsistency on the number of photos reported during import is due to unsupported file formats on the card.  If a format is unsupported, Lightroom should either always or never include that file in the count.  But the number shown in the lower left of the import dialog changes each time I launch Lightroom (e.g. 2812, 2819, 2818), even though the sdcard remains unchanged, and even if I don't import anything.

I feel that I've ruled out hardware problems.  Yes, I'm using the same physical card, but Finder, the command line, and Beyond Compare have no problem reading, copying, or verifying all files on this card.  Something about this particular photo set is throwing Lightroom for a loop.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4309 Posts
  • 1571 Reply Likes
Hi Nathan, that's a great bit of detective work. One more thing to check... if you use the OS X Image Capture app to transfer those photos, do they all transfer? The reason I ask is I saw a similar issue with Olympus a while back, and it turned out to be a OS X bug, so it would be useful to rule that out.
Photo of Nathan

Nathan

  • 4 Posts
  • 0 Reply Likes
That was a very good insight Victoria.  Image Capture does exhibit the same behavior.  And both applications use the same underlying Apple library to handle the imports.  So I do agree it is most likely an OSX bug.

But, Apple will most likely not backport any fix to Lightroom's minimum supported OS version, so it will be interesting to see how Adobe works around this bug.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4309 Posts
  • 1571 Reply Likes
If the card is showing in the Files section of the Source panel (further down) try selecting it there. That uses a different path, seeing it as a hard drive, so should work without error. 
Photo of Gary Rowe

Gary Rowe

  • 91 Posts
  • 33 Reply Likes
Great report - well done for documenting it so well. My import issues always seem random, I think I'd better take a leaf from your book and start looking for the pattern!
Photo of Sunil Bhaskaran

Sunil Bhaskaran, Official Rep

  • 321 Posts
  • 111 Reply Likes
Nathan,
Thanks a lot for the detailed report.
We have fixed a similar issue in Lightroom Classic 7.2 release.
Since you are on 7.3, looks like there can be some cases in which the fix is not effective.
We will try to reproduce it internally. 

Thanks,
Sunil
Photo of Nathan

Nathan

  • 4 Posts
  • 0 Reply Likes
I've spent my free time over the past week repeating the import tests, while monitoring all filesystem activity.  I used a combination of lsof and dtruss to watch every file read/write/list operation performed during import.  (Yes, I'm a software engineer, and this is going to get very technical very fast.  TL;DR?  Read the last paragraph.)

It appears that Lightroom is operating in conjunction with an Apple supplied MassStorageCamera application included in OSX to import from a device.  This process is started in the background as soon as the import dialog is launched.  During import, lsof shows that Lightroom is copying the RAW files, and MassStorageCamera is handling all the JPG files.
Adobe\x20 30824 nathan   28r      REG                1,9  15396352   411642 /Volumes/Untitled/DCIM/184_PANA/P1840974.RW2
Adobe\x20 30824 nathan   33w      REG                1,4  12914688 31319440 /Users/nathan/moo/TempParent/Temporary Folder C164CB01-AE26-4F25-BC1F-2D1B52F686FE/P1840974.RW2
----------
MassStora 30882 nathan    3u      REG                1,4   2097152 31319444 /Users/nathan/moo/TempParent/Temporary Folder C164CB01-AE26-4F25-BC1F-2D1B52F686FE/.tmp_P1840974.JPG_
MassStora 30882 nathan    4r      REG                1,9   5138944   411602 /Volumes/Untitled/DCIM/184_PANA/P1840974.JPG

The problematic photos are located in the 184_PANA folder on the sdcard.  Interestingly, Lightroom never directly asks the OS for a list of the files in this directory, yet knows the filenames to import.  MassStorageCamera does request the list of filenames from the OS.  Since these programs are operating in concert, I have to assume that Lightroom is receiving the list of filenames from MassStorageCamera.

In both successful and problematic imports, MassStorageCamera is always told by the OS that there are 1995 files (186*10 + 135) in the 184_PANA directory.  This is the correct value.  So the files that fail to import were known, at least by MassStorageCamera:
  684/0x1e45:  open_nocancel("/Volumes/Untitled/DCIM/184_PANA\0", 0x1100004, 0x1)                = 3 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 186 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 135 0
  684/0x1e45:  getattrlistbulk(0x3, 0x70000B5EE6C8, 0x7FD09F813A00)              = 0 0
  684/0x1e45:  close_nocancel(0x3)               = 0 0

I've determined that the success/failure of the import can be determined by the behavior of Lightroom during the initial scanning process.  This is useful because it isolates the problem to the scanning phase and not the actual import phase.  In particular, Lightroom asks the OS for the status of each file during the scanning process.  If the import will succeed, all file statuses are requested:
 1276/0x82cc:  11791928       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840970.JPG\0", 0x7FFF59A7DC50, 0x0)		 = 0 0
 1276/0x82cc:  11792111       6      4 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840970.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792177       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840971.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792366       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840971.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792427       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840972.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792609       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840972.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792671       5      2 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840973.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792853       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840973.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11792915       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840974.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  11793098       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840974.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0

But when the import will fail, Lightroom never requests information about the file that will be missed.  This trace is of the 5th import, so P1840972.RW2 is the file that will fail to import.  And as can be seen, Lightroom skips over it and never asks the OS about the file status.
 1276/0x82cc:  88659571       4      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840970.JPG\0", 0x7FFF59A7DC50, 0x0)		 = 0 0
 1276/0x82cc:  88659789       6      4 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840970.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88659859       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840971.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660043       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840971.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660109       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840972.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660326       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840973.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660510       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840973.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660594       5      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840974.JPG\0", 0x7FFF59A7DC50, 0x0) = 0 0
 1276/0x82cc:  88660777       6      3 lstat64("/Volumes/Untitled/DCIM/184_PANA/P1840974.RW2\0", 0x7FFF59A7DC50, 0x0) = 0 0

I've also discovered another way to reproduce this issue without restarting Lightroom:
  1. Reboot
  2. Launch Lightroom with new catalog
  3. Launch import dialog.  P1840971.RW2 is not lstat64'd.  Close import dialog.
  4. Kill MassStorageCamera process
  5. Launch import dialog.  All files are lstat64'd.  Close import dialog.
  6. Kill MassStorageCamera process
  7. Launch import dialog.  P1840972.JPG is not lstat64'd.  Close import dialog.
  8. Kill MassStorageCamera process
  9. Launch import dialog.  All files are lstat64'd.  Close import dialog.
  10. Kill MassStorageCamera process
  11. Launch import dialog.  P1840972.RW2 is not lstat64'd.  Import.  P1840972.RW2 will fail to import.

I can also cause Lightroom's first import after a reboot to be successful, by simply launching Apple's Image Capture program first.
  1. Reboot
  2. Launch ImageCapture.  Select the sdcard.  Close without importing.
  3. Launch Lightroom with a new catalog
  4. Launch import dialog.  All files are lstat64'd.  Import.  Success - all files imported.

Lastly, Image Capture is exhibiting the same problematic import behavior.
  1. Reboot
  2. Launch ImageCapture.  Shows 5622 total photos.  Import.  P1840971.RW2 not imported.
  3. Launch ImageCapture.  Shows 5623 total photos.  Import.  Success.
  4. Launch ImageCapture.  Shows 5622 total photos.  Close without importing.
  5. Launch ImageCapture.  Shows 5623 total photos.  Close without importing.
  6. Launch ImageCapture.  Shows 5622 total photos.  Import.  P1840972.RW2 not imported.
Under the hood, Image Capture is also using MassStorageCamera to read the sdcard.

Combined, these behaviors lead me to conclude that the problem is most likely in the MassStorageCamera application:
  1. The commonality is the reliance on MassStorageCamera.
  2. The ping/pong of import failure/success follows the number of times MassStorageCamera is used, irrespective of the invoker.  The failures thus manifest themselves in whatever application is using MassStorageCamera on the 1st, 3rd, 5th, ... invocation of MassStorageCamera after reboot.

Adobe Developers - It appears to me that the MassStorageCamera component you depend on to import from sdcards is buggy, and is hurting the reliability of Lightroom.  Since you know which APIs you are calling, it would be great if you could open an Apple Radar ticket so the root problem can be addressed going forward.  But your minimum supported OSX version is 10.11, and I doubt Apple will backport such a fix.  So I suspect you will need a workaround.  First, is there any reason to even use MassStorageCamera for an sdcard?  It is mounted as a volume, so importing can be done through filesystem operations alone.  Second, as a workaround, you could probably start/stop MassStorageCamera multiple times, using whichever session gives you the largest fileset.  Not pretty, but it could allow Lightroom to avoid the import failures and operate correctly.