Lightroom Classic: Multiple dng files from conversion

  • 2
  • Problem
  • Updated 3 months ago
  • In Progress
  • (Edited)
After conversion of tiffs to dngs, I have multiple dngs!!!

Here's the details...

After importing a series of tiff images, I converted the tiffs into Adobe dngs files.  After restasrting LR 7.5, I find that I have multiple dngs!?!   At least three (3)!



Really?  Seriously?

My Finder does not show these duplicates as actually existing.  So where in the heck did they come from?
Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes

Posted 3 months ago

  • 2
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Oooooooh, you might have just found a clue in a bug I've been tracking. Can you take us through the steps you took leading up to that state please?
Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes
Hi Victoria,

I was simply following my regular workflow when dealing with files from this old camera.  I download the files into a directory created by a photography management program named Light Blue.  The folder structure is created and named automatically; in this case the folder structure is:

Mexico's Mayan & Aztec Illumination
Mexico's Mayan & Aztec Illumination\processed
Mexico's Mayan & Aztec Illumination\unprocessed

The images are copied with file verification using Path Finder 8 into the 'unprocessed' subfolder.  They are then renamed as 'Mexico's Mayan & Aztec Illumination - 'sub location' - 'original file number'.

In this case, the files were named:

Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.tif
Mexico's Mayan & Aztec Illumination - Ek' Balam - IMG_2019021.jpg

After copying and renaming, the files are imported into LR.

After import, depending upon the number of files and their size in the import, I might optimize the catalog.  I might keyword fist, and then optimize.  There's no hard and fast rule; I typically see how LR is responding.  If it is flying along, then I continue the workflow without optimization.  If it is dogging it, I will optimize.  If the optimization takes a long time, I will restart LR and then continue.

In this case, LR was humming along nicely, so I keyworded the images and then selected the tiffs for conversion to dng.

One (1) dng was produced for each tiff.  I optimized and restarted and viola!  Three (3) dngs in the collection 'Previous Import,' faithfully aligned with the tiff and the jpeg, all with the very same file number.  Except, the duplicates actually do NOT exist in any folder on any drive.  I verified this with Path Finder 8.

I've been using LR since version 3 and have never seen this behavior previously.  In fact, I'm pretty sure I have encountered more oddball behavior and bugs in LR 7.5 than in any previous version, and my observations have been borne out by reading the blogs since the release of LR 7.

That's why I delayed for months before upgrading, and I did that all because of a piece of hardware — Loupedeck+.

Ah well.  What do you think, Victoria?  Should I remove the dngs from the catalog and start again?  Should I move the real dngs where LR can't find them and then see of LR cannot find them and then move them back?

I can see several permutations one could apply here.

Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes
Update: After catalog optimization, the duplicate DNGs are still present.  However, when one right-clicks on the image in Library and selects 'Show in Finder', each DNG points right back to the single DNG file.

In essence, these duplicate DNGs exist only in Lightrooms imagination.

The question remains: "How to remove the extras?  Which ones should be selected, or does it make no difference?
Photo of Just Shot Me

Just Shot Me

  • 240 Posts
  • 71 Reply Likes
What is the Exact File Name of those 3 DNG files? Just because it ends with 2019021.dng doesn't mean they were created from the same TIF file. I see you have a JPG file that ends with that 2019021 group of characters. Could be one of the DNG files that were created was made from that JPG file.

Now the question I have to ask.
Why are you creating DNG files from TIF files? In my mind there is NO GOOD reason to do that.
TIFF is a multi platform/program recognized image format. The file can be read and displayed by many many different computer OS's and programs. DNG is not.
When I send an image over to Photoshop for further editing I save it as a TIF file. I would not then convert it back to a Digital NeGative (DNG) for any reason.
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 121 Reply Likes
“There are two ways to be fooled. One is to believe what isn't true; the other is to refuse to believe what is true.” -Søren Kierkegaard
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 121 Reply Likes
I am sure you are aware that many raw file formats, including IIQ (Phase One), 3FR (Hasselblad), DCR, K25, KDC (Kodak), CRW CR2 CR3 (Canon), ERF (Epson), MEF (Mamiya), MOS (Leaf), NEF (Nikon), ORF (Olympus), PEF (Pentax), RW2 (Panasonic) and ARW, SRF, SR2 (Sony), are based on the TIFF file format.  

CRW were not based on TIFF, they were based on CIFF.
http://xyrion.org/ciff/CIFFspecV1R04.pdf
https://en.wikipedia.org/wiki/Camera_Image_File_Format

Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes
You should get a life.  I think Trump adequately described you as the 400-pound fat man laying in his bed with his computer.

Your first reference clearly states "The first version of this RAW format was .CRW (see also here) and is used by Canon D30, D60, 10D, 300D, PowerShot Pro1, G1-G6, S30-S70. The EOS 1Ds is writing TIFF files."

I won't even bother to cite your second reference and how it also disproves your claim.  (BTW, in the interest of full disclosure, the paragraph you quoted from me came from Wikipedia.  So if you can cite them as your source, so too can anyone else.)

You are only proving my point that you are a classic internet troll.  LOL

Now, please go to bed—oops, my bad; you're already there!—and let the adults in the room continue the discussion.  (I can't believe that the ASMP, of which I have been a dues paying member for decades, allows you to posit yourself as a techincal expert.  I might have to resign in protest.)

Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 121 Reply Likes
The first reference From Canon does not state that with no reference to TIFF as they are NOT based on it as you posted.
Do resign in protest, please.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 121 Reply Likes
“I think Trump adequately described you as the 400-pound fat man laying in his bed with his computer.”

And I’m the Troll.

“Hypocrite: The man who murdered his parents, and then pleaded for mercy on the grounds that he was an orphan.”― Abraham Lincoln
Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
My question too; why convert a TIFF to what is basically another TIFF?
Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
Ah but with ZIP or LZW compression? 
Why do you need a full sized embedded preview when the entire image is rendered and can be previewed? Or the embedded thumbnail we see in the Finder? 
Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
Took a big layered image, converted to DNG, think the layers are gone. So that's a big problem! 

Took the same file, flatted it, saved as TIFF with LZW and DNG. DNG is smaller indeed. Bad news is, no preview or thumbnail showing up on DNG in Mac Finder (latest OS), but a preview from the TIFF. 

TIFF with LZW is 43MB
DNG is 28.5MB

Difference opening each and what one gets (using Photoshop). 

TIFF with LZW opens in 1 second Photoshop proper, ready to work on.
DNG opens ACR in about the same time (tad slower) but now I've got to process it through ACR.
No free lunch here. And no reason to convert TIFFs to DNG IMHO, at least a few reasons not to convert unless disk space is hugely important to you. And I really don't think anyone with layered TIFFs will wish to convert to DNG! Good-by layers it seems.
Photo of Todd Shaner

Todd Shaner, Champion

  • 1135 Posts
  • 366 Reply Likes
Andrew, I already mentioned DNGs do no not support layers and neither do raw files. This discussion concerns working with files inside LR so I would think results with PS, ACR, Finder are secondary.

"Why do you need a full sized embedded preview when the entire image is rendered and can be previewed?"

DNG files can be updated when develop edits have been applied using 'Update DNG Preview & Metadata.' This updates the embedded develop settings AND the embedded preview with those settings rendered. Using TIFF file format you can also embed the develop metadata, but the image data and resulting previews remain unchanged (not rendered). When importing DNG files the Library previews render very quickly since LR is simply reading the full-size embedded preview. When importing TIFF files with embedded settings LR must render Standard and 1:1 previews using the embedded develop settings, which takes considerably longer.

This may be a useful when multiple people are involved in the editing and reviewing processes. This benefit applies whether the original files are RGB rendered TIFFs or actual raw files. Regardless, converting TIFFs to DNG should not cause any issues such as creating phantom extra copies correct?

Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
Yes Todd, one can update the DNG  preview and metadata but it's kind of pointless with a rendered TIFF, it's all updated every time too.
Multiple edits can be also be embedded by multiple users in the TIFF like DNG. 
 So I see no advantage in TIFF to DNG conversions here. Now the file size IS smaller and I didn't test TIFF with ZIP just due to possible compatibility issues. It may be as small as the DNG. But it's not enough where I can see any advantage and a few disadvantages (like the behavior of Photoshop on the docs) with a DNG from a rendered TIFF. I still see it as utterly pointless. DNG from a proprietary raw, now that's a useful workflow I practice and highly recommend. 
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Guys, please keep the thread on topic. It's gone round the houses long enough. This isn't a debate about the DNG format. This is a bug report about ghost images.
Photo of Johan Elzenga

Johan Elzenga, Champion

  • 1606 Posts
  • 648 Reply Likes
I see a tiff and a jpeg with the same name too. It could well be that you also have a raw file with that name. If all three were selected when you converted to DNG (and they were not in the same folder) then you could end up with three DNG files with the same name (in three different folders, you can't get that in the same folder).

There is a problem though that has been reported a few times, where people have multiple copies of one and the same image showing in their catalog. It seems to be some kind of catalog corruption, but it is unclear what causes it. That is what Victoria is referring to.
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 122 Reply Likes
Just re-sync the folder(s) or create a new catalog and import. Now what do you see? 
Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes
Didn't work super guru.
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 122 Reply Likes
Of course, you didn't answer the question. 
Seems she's not so super after all. 
Photo of dmeephd

dmeephd

  • 238 Posts
  • 52 Reply Likes
Troll alert!
Photo of Andrew Rodney

Andrew Rodney

  • 651 Posts
  • 122 Reply Likes
"The highest form of ignorance is when you reject something you don't know anything about". -Wayne Dyer
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Guys, the arguing isn't helping. Drop it.

dmeephd, I agree it's some kind of catalog corruption. Here's a similar case which is the case used in bug LRD-4199245, currently in test.

If you're happy that all of the metadata you need would be retained by writing the metadata to the files, I'd run a backup and remove that whole folder and add it again. I suspect that may do the trick. 

Before you do though, can I get a copy of that catalog to add to the bug report please? You can zip it up and use www.wetransfer.com to send it to me at uploads@lightroomqueen.com and I can upload it to the bug.
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Thank you David, that's useful information. I suspect it's a relatively rare issue but it is being investigated.
Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
Finally, as to sending you a copy of my catalog, I'm afraid it is far too large for that—just over 26GB (down from 37GB under LR6).
Since you can use some education on this, you can Export as catalog 1 image only. The resulting catalog will be vastly smaller but most of the stuff we would need to evaluated the catalog would exist in this new version with a far, far smaller size. You could examine DropBox or Hightail too, if again, you really do want to share the catalog so Adobe or others can inspect it. 
IF you really needed help in uncovering possible catalog issues, you can upload one. I suspect that will not happen, like the request to upload a 'raw' TIFF your back produces. 

What also will probably fall on deaf ears is a clear answer if you indeed made a new catalog and imported the one DNG after which you did see more than one DNG in that new catalog as asked. There's a massive difference in what may be happening on your end if we know it's a single catalog issue or one that manifests itself with a new catalog. If so, the next test would be to try this on a different computer since by your own admission, you've had catalog problems and corruption. 

I've been using LR since before it went beta; I've never had a corrupted catalog. It's indeed possible! But it's also a factor based on your system/drive etc. Hence, if you really want to get to the bottom of this, knowing if the issue occurs in one new catalog or all would go a long ways to aiding people in figuring out the issue. Assuming that's your goal. 

Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Andrew, we all know how much experience you have, but with all due respect, YOU'RE NOT HELPING. Please, I'm begging you, let it drop. There's already a catalog attached to the bug, which is already being investigated, and if the engineers need a chunk of this one, they can ask for it. Stop putting David down - it's not fair.
(Edited)
Photo of Andrew Rodney

Andrew Rodney

  • 652 Posts
  • 122 Reply Likes
Victoria please explain how this would be a catalog issue if one imports the DNG suspect into a new catalog? I asked the Dr. to try this and he said it didn't make any difference.

Can you explain if this is so, why he'd even need to upload his entire original catalog which is doable and has been explained but maybe not at all necessary? 
Photo of Victoria Bampton - Lightroom Queen

Victoria Bampton - Lightroom Queen, Champion

  • 4528 Posts
  • 1718 Reply Likes
Andrew, you're clearly in an argumentative mood this weekend, both on this and other threads. I'm not quite sure why you're having a rough time or why you're so desperate to be proven right, but it's not fair that you keep hammering on other people. Let it go. It doesn't matter.
Photo of Rikk Flohr

Rikk Flohr, Official Rep

  • 4482 Posts
  • 889 Reply Likes
I have cross-referenced this thread against Victoria's bug referenced here and marked the thread as in-progress. When an update is available to fix this issue, we will post to this thread. Thank you. 

Let's keep the thread on-topic and take the peripheral discussions elsewhere. 
(Edited)