Details at http://www.BigTimberPhotoAdventures.com/

The Bocci web site is here

I’ve put together a two-part miniseries that describes the most important points of the Import process and how Lightroom helps you manage your files through the catalog. Click the links below to read the articles, published on Photofocus:

Part 1 – About Lightroom Catalogs

Part 2 – Importing and Organizing Your Photos

There’s lots more about Importing photos (and all other Lightroom topics) in my Lightroom 3 book.

In every class and workshop I teach, the subject of DNG inevitably comes up. There’s a lot of confusion and uncertainty about DNG. So when researching subject matter for my next book, I thought I’d polish up my knowledge of this essential image file format.

In doing so, I reached out to one of today’s leading imaging software developers, Eric Chan, Senior Computer Scientist at Adobe. Following is a [very minimally edited] transcript of our email conversation.

(NC – me, Nat Coalson EC – Eric Chan)

###

“NC: Can you give me a concise summary of the DNG conversion process? Specifically, what happens with the raw data, and what does the resulting DNG file contain? I’ve read the stuff on the official Adobe DNG pages, etc. But I saw recently where you stated that DNG is basically a TIF format and this inspired me to get a more useful/practical description from you.

EC: DNG is indeed very much like TIFF.  It’s actually a set of TIFF extensions, with tags to describe things like white balance, color profiles, and various calibration data needed for processing raw images (as opposed to already-rendered images like regular jpegs and tiffs).

A raw DNG typically stores 12-bit, 14-bit, or (rarely) 16-bit Bayer mosaic data, usually in lossless compressed format.  (Uncompressed is also supported, but rarely used because the files are so big!)  The lossless compression is part of the JPEG specification and is simply called Lossless JPEG.  It is a pretty common lossless compression scheme used by many vendors (i.e., both DNG as well as non-DNG raw files).

Aside from the image data itself, it’s important to have the metadata about the image, for image processing purposes.  This includes things like the so-called “as shot” white balance, the white point, the black point, etc.  Some of these tags are required, and others are optional.  The nice thing is that there isn’t much required to put together a perfectly valid DNG file, so a handful of vendors are now doing so in the camera (Casio, Leica, Pentax, Ricoh).

NC: Thanks so much, I think at this point the one biggest question I want to put to rest is this: When comparing the pure, raw, unmodified mosaic data in the original raw capture, is there any difference at all between that data and the image data contained in the DNG? A while back someone was trying to convince me they don’t like DNG because the raw data is being modified from the native capture. I’ve never believed this to be the case. Can you describe to me the nature of the image data in the DNG as it compares to the data in the original native file?

EC: This is a tough question to answer. It is a bit of a gray area. The shortest and most honest answer is: it depends!

Usually the reason for the difference has to do with calibration data. Many cameras perform internal calibration (eg, flat fielding, per-row equalization, etc) before writing out the raw file to the memory card, but others do not (eg, because of limited processing power in the camera). The ones that do not instead usually write a set of tags describing this calibration data to the private metadata of the raw file. It is then expected that the raw processing software (such as Lightroom) would apply the calibration data upon reading in the raw file. The image is practically useless without the calibration data, so it is always desirable to apply it. In other words, the camera vendor would have preferred to apply it directly in the camera, but couldn’t.

The reason I mention this is because when we build DNG images for such cameras, we will usually bake in the calibration data into the resulting DNG mosaic image. Thus, the resulting mosaic data is not identical to the original non-DNG mosaic data because the calibration data has already been applied. This has the advantage of making the image data much more usable to third-party DNG reading software (which usually doesn’t handle the vendor-specific calibration data), and it also makes reading the DNG faster (because the software can skip the calibration step).

There are some other cases where the converted DNG doesn’t have bit-identical pixel data. For example, usually the mosaic data contains extra pixels around the border to help with edge operations (like demosaicing), as well as black-level estimation.  In these cases, we will preserve this edge pixel data when producing the output DNG. However, in some cases the camera will write bogus values to these edge pixels (basically random values –uninitialized memory). These aren’t useful to preserve, and can confuse other DNG readers, so we usually omit these when building the output DNG.

So, we generally preserve the mosaic data exactly, but there are some exceptions.

NC: Thanks. So with all this in mind, is it still accurate to say that the image data in _all_ DNG files is _raw_ data?

EC: No, not necessarily. The reason is that the DNG file format can also store non-raw data.  For example, you can feed a JPEG to the DNG Converter and have it spit out a DNG.  However, that DNG isn’t raw, of course (you can’t take a rendered file like a JPEG and go “backwards” to make it raw again … it would be as miraculous as uncooking a steak!).

At its most essential level, DNG is just a container of image data and its associated metadata.  It can hold raw image data (a.k.a., scene-referred data), and it can also hold rendered image data (a.k.a., output-referred data).  For raw data, the data can be in the mosaic form, or it can be demosaiced (so-called “linear DNG”).  For example, Foveon cameras and BetterLight scanning backs store scene-referred linear data, but they aren’t in mosaic form.

NC: Great, got it. Just one final question, if you don’t mind: if I convert a raw file from a _typical_ DSLR (eg my Canon 5D Mark II) is the scene-referred data contained in that DNG any different than the original scene-referred data in the native raw file?

EC: No, they should be identical in the typical case (e.g., Nikon SLR, Canon SLR).  An exception in the case of Canon SLR is the S-RAW and M-RAW modes.  Those are not really mosaic files; they are stored in a quasi-JPEG like YCbCr space (luminance and color components), which DNG does not natively support.  As a result, if you convert a S-RAW or M-RAW Canon file to DNG, it will get mapped to a RGB color space before getting stored into the DNG container (and hence will not be the same as the original).

NC: Thanks a million for taking the time to explain all this to me. It’s a huge help! I’ve been a big fan of DNG for years and I’m glad my understanding wasn’t so far off that I got myself into trouble

I know that this kind of support isn’t technically part of your job and it means a lot to me and many of my colleagues that you are so willing to share your expertise.”

###

If the above all seems too technical, the main point you really need to understand is that if you shoot with a DSLR and convert your raw files to DNG you are not losing any data or image quality (assuming you’re capturing raw not sRaw). And the advantages of DNG are significant; see the links below for more about this.

As I mentioned in the introduction to this post, I believe in the practical, flexible, long term usefulness of DNG and will continue to use them as my raw file format. I welcome your comments and questions!

You can get more info about DNG at the links below:

Adobe DNG resource pages
Wikipedia

While exploring Warsaw last spring, I came upon this lovely old woman knitting on the sidewalk. She has the most amazing face! (click for larger image)

Buy a brand new calendar for 2012 featuring beautiful photography from around the world!

Choose from three themes: Italy, Desert Southwest and World of Water. All photography by professional photographer Nat Coalson.

Calendars are professionally printed and shipped by CafePress. Available now for only $34.99 plus shipping (and tax if applicable)

Click here for full product details and to place your order.