Lightroom Classic: Multiple dng files from conversion

Hi Victoria,

I was simply following my regular workflow when dealing with files from this old camera.  I download the files into a directory created by a photography management program named Light Blue.  The folder structure is created and named automatically; in this case the folder structure is:

Mexico's Mayan & Aztec Illumination
Mexico's Mayan & Aztec Illumination\processed
Mexico's Mayan & Aztec Illumination\unprocessed

The images are copied with file verification using Path Finder 8 into the 'unprocessed' subfolder.  They are then renamed as 'Mexico's Mayan & Aztec Illumination - 'sub location' - 'original file number'.

In this case, the files were named:

Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.tif
Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.jpg

After copying and renaming, the files are imported into LR.

After import, depending upon the number of files and their size in the import, I might optimize the catalog.  I might keyword fist, and then optimize.  There's no hard and fast rule; I typically see how LR is responding.  If it is flying along, then I continue the workflow without optimization.  If it is dogging it, I will optimize.  If the optimization takes a long time, I will restart LR and then continue.

In this case, LR was humming along nicely, so I keyworded the images and then selected the tiffs for conversion to dng.

One (1) dng was produced for each tiff.  I optimized and restarted and viola!  Three (3) dngs in the collection 'Previous Import,' faithfully aligned with the tiff and the jpeg, all with the very same file number.  Except, the duplicates actually do NOT exist in any folder on any drive.  I verified this with Path Finder 8.

I've been using LR since version 3 and have never seen this behavior previously.  In fact, I'm pretty sure I have encountered more oddball behavior and bugs in LR 7.5 than in any previous version, and my observations have been borne out by reading the blogs since the release of LR 7.

That's why I delayed for months before upgrading, and I did that all because of a piece of hardware — Loupedeck+.

Ah well.  What do you think, Victoria?  Should I remove the dngs from the catalog and start again?  Should I move the real dngs where LR can't find them and then see of LR cannot find them and then move them back?

I can see several permutations one could apply here.

Update: After catalog optimization, the duplicate DNGs are still present.  However, when one right-clicks on the image in Library and selects 'Show in Finder', each DNG points right back to the single DNG file.

In essence, these duplicate DNGs exist only in Lightrooms imagination.

The question remains: "How to remove the extras?  Which ones should be selected, or does it make no difference?

All five (5) files in LR have exactly the same file name.  However, the two (2) extra dngs do not actually exist in any folder on any of my drives.  This is confirmed.

The tiff file is a raw file from an old camera which produces a tiff and a jpeg.  My Photoshop guru prefers working with dngs, hence the GOOD reason for creating one (1) from the tiff.    Only the tiff was selected when the dng creation was executed, and the fact that the others do not really exist except in LR is proof.

Your Photoshop guru is confused (or you are if indeed the original is a TIFF) and you really need to check up on what is rather silly “advice” IF so.
Verify THEN trust.
Ask him exactly what good it does you and report back. Raw, TIFF and DNG are all based on the same Adobe owned and controlled format.
A raw converted to DNG is not the same as a TIFF converted to DNG! One is rendered and the other is not.

My Photoshop guru is a she and Simone makes a very nice six-figure income in the UK providing Photoshop services to corporate clients.  They, and I, trust her implicitly, and they put their trust in her in the form of British Pounds.

I'm not completely aware of the history of the raw file, but back in the day there were several cameras which produced as their 'raw' file a tiff file.  However, I do know that raw files which are not tiffs are NOT an "Adobe owned and controlled format"—that is an idiotic premise.  The raw file format (e.g., .arw . CR2, etc.) is proprietary to the camera manufacturer.   I think Sony or Canon or Nikon would be surprised to know that they do not own their patented intellectual property.

You should also look up the definition of "rendered" as what you imply is grossly incorrect.

As to your claim about the basement of DNG and TIFF formats, once again you are incorrect.  The DNG format is based upon the international standard ISO 12234-2, titled "Electronic still-picture imaging – Removable memory – Part 2: TIFF/EP image data format".  This is different from the Tagged Image File Format, which is a standard administered by Adobe currently called "TIFF, Revision 6.0 Final – June 3, 1992."

In a nutshell, the major difference between DNG and the ISO 12234-2 standard is the increased capacity for the inclusion and handling of metadata.  That's why Simone—and many other Photoshop professionals working with images from professional photographers—prefers using the DNG format rather than a simple TIFF.

A TIFF is a rendered image. A raw isn’t. Both are based on TIFF/EP. If your camera produces an actual TIFF (rendered), there is absolutely NO reason to convert it do DNG. If it is actually a proprietary raw, there are indeed many reasons to convert to DNG:
http://digitaldog.net/files/ThePowero...

You stated you are converting rendered TIFFS to DNG but hopefully you are confused between proprietary raws and rendered TIFFs.

BTW, this is raw and I'd hope you would agree, it's nothing like a rendered TIFF:
http://www.digitaldog.net/files/raw.jpg

1) DNG file compression is better than TIFF ZIP compression (smaller file size)
2) DNG has a full-size Embedded Preview (speeds Import using Embedded & Sidecar Preview Building setting)
I'm not saying these are overwhelming reason to convert TIFFs to DNGs, but to some users it may be a benefit. Downside is DNG files don't support layers (layered TIFFs are flattened).

Ah but with ZIP or LZW compression? 
Why do you need a full sized embedded preview when the entire image is rendered and can be previewed? Or the embedded thumbnail we see in the Finder? 

Took a big layered image, converted to DNG, think the layers are gone. So that's a big problem! 

Took the same file, flatted it, saved as TIFF with LZW and DNG. DNG is smaller indeed. Bad news is, no preview or thumbnail showing up on DNG in Mac Finder (latest OS), but a preview from the TIFF. 

TIFF with LZW is 43MB
DNG is 28.5MB

Difference opening each and what one gets (using Photoshop). 

TIFF with LZW opens in 1 second Photoshop proper, ready to work on.
DNG opens ACR in about the same time (tad slower) but now I've got to process it through ACR.
No free lunch here. And no reason to convert TIFFs to DNG IMHO, at least a few reasons not to convert unless disk space is hugely important to you. And I really don't think anyone with layered TIFFs will wish to convert to DNG! Good-by layers it seems.

Andrew, I already mentioned DNGs do no not support layers and neither do raw files. This discussion concerns working with files inside LR so I would think results with PS, ACR, Finder are secondary.

"Why do you need a full sized embedded preview when the entire image is rendered and can be previewed?"

DNG files can be updated when develop edits have been applied using 'Update DNG Preview & Metadata.' This updates the embedded develop settings AND the embedded preview with those settings rendered. Using TIFF file format you can also embed the develop metadata, but the image data and resulting previews remain unchanged (not rendered). When importing DNG files the Library previews render very quickly since LR is simply reading the full-size embedded preview. When importing TIFF files with embedded settings LR must render Standard and 1:1 previews using the embedded develop settings, which takes considerably longer.

This may be a useful when multiple people are involved in the editing and reviewing processes. This benefit applies whether the original files are RGB rendered TIFFs or actual raw files. Regardless, converting TIFFs to DNG should not cause any issues such as creating phantom extra copies correct?

Yes Todd, one can update the DNG  preview and metadata but it's kind of pointless with a rendered TIFF, it's all updated every time too.
Multiple edits can be also be embedded by multiple users in the TIFF like DNG. 
 So I see no advantage in TIFF to DNG conversions here. Now the file size IS smaller and I didn't test TIFF with ZIP just due to possible compatibility issues. It may be as small as the DNG. But it's not enough where I can see any advantage and a few disadvantages (like the behavior of Photoshop on the docs) with a DNG from a rendered TIFF. I still see it as utterly pointless. DNG from a proprietary raw, now that's a useful workflow I practice and highly recommend. 

Nope.  Please see response above.  I think it is a catalog corruption too, and the catalog is currently being checked for integrity and then optimization.

Just re-sync the folder(s) or create a new catalog and import. Now what do you see? 

Didn't work super guru.

Of course, you didn't answer the question. 
Seems she's not so super after all. 

Troll alert!

dmeephd, I agree it's some kind of catalog corruption

And yet, when I asked him to try importing the DNG into a new catalog,  he reported no fix so I don't see how that's the issue. To copy and paste:

AR: Just re-sync the folder(s) or create a new catalog and import. Now what do you see? 

 dmeephd : Didn't work super guru.

Think he really did try with a new catalog?

Andrew, that's not what you were arguing about... and it's not the only reported case. You're just winding each other up and it's not helping anyone.

No it's not at all what we were arguing about. We were arguing about converting TIFFs to DNG. That's over with, dmeephd can't accept that his question about conversion just caused multiple people here to question what and why he was doing what he said he's doing. 

The question I asked and he answered was elsewhere from those debates. The problem appears to be simply what was asked first:

I converted the tiffs into Adobe dngs files.  After restasrting LR 7.5, I find that I have multiple dngs!?!   At least three (3)!

So the theory was, it's a catalog issue as the OP states there is only ONE DNG yet he see's three. OK, if it is the catalog, then why would creating a new, virgin catalog and importing that ONE DNG produce the same issue? Unless dmeephd didn't do as requested. 
IF the issue is the DNG (I suppose that's possible) he should upload it and allow others to attempt to import it into their catalogs to see what happens. But at this point, IF what dmeephd reported was accurate (?) in terms of a test I suggested with a new catalog, I can't understand how the catalog is the issue. 

I agree, Victoria, however some folks just can't seem to drop it, citing this as an open public forum and implying they can troll another for as long as they like.  I stopped responding last night but the posts just keep on coming, right up to 5:55 PDT this morning.  I will respond no further to this gentleman.

Back to the original issue at hand...

I found that each dng in the Library view pointed to the same dng file in Finder.  Not knowing which one to remove, I simply chose one and viola!, they all were removed (for a given image).  The actual dng file was, of course, untouched.

I repeated the process for all of the images with duplicate dngs.  Then I optimized the catalog and restarted LR.  Next I performed an import and the individual dngs in the folder were imported and added to the catalog, one for each image.  No duplication this time around.

I next imported another set of raw files in tiff format (as described by Olympus Medical themselves) along with their matching jpegs.  I then selected the tiis, used the 'Convert Photo to DNG' with the 'Only conver raw files' unchecked.  The dngs were created with no duplicates.

Subsequent imports of images have been without fault, albeit, slow compared to LR6.  Part of the problem I think lies with the fact that LR is performing preview building in parallel with the import process.  I observe that the center library pane remains blank until around 75% of the import has been accomplished, and then the images appear (whilst preview building—standard or 1:1; makes no difference) continues.

LR6 never attempted these operations in parallel.  Is this a new feature of LR 7, and if so, can it be disabled?

I have seen more problems with the catalog since upgrading and this parallel processing fo such computationally intensive operations might be contributory.  It certainly cannot be my hardware as my MacPro is heavily augmented with after market upgrades making it as powerful a MacPro as one can obtain.

Finally, as to sending you a copy of my catalog, I'm afraid it is far too large for that—just over 26GB (down from 37GB under LR6).  When it became corrupted last October (due to a new version of the mirroring program recursively writing the backups while LR was running) I had trouble sending it to the Adobe database guru you recommended (who was able to fix the db).

Regards,

David Martin, Ph.D.

Thank you David, that's useful information. I suspect it's a relatively rare issue but it is being investigated.