However, some of the conversations about responsive images end up going in circles because people are talking past each other. I believe this is happening because we don’t have a common framework to look at the problem.

I believe there are two separate, but related issues that need to be solved regarding the use of the img element in responsive designs. They are:

1. How do we enable authors so that they can display different images under different conditions based on art direction?

To understand this issue, it helps to look at a specific use case. Take for example the following photo of President Obama speaking at a Chrysler plant.¹

When the image is displayed at larger sizes, it makes sense for the image to show the automobile factory in the background. The background helps explain where the event took place and adds to the image. But look what happens when we scale the image down to fit a smaller screen.

At that size, you can barely recognize Obama. You can’t make out his face. Instead of simply resizing the image, it may make sense to crop the image to get rid of some of the background and focus in on him. The end result is an image that works better at the smaller size:

This is what I refer to as enabling art direction. Authors need to be able to provide different sources for images at different sizes not based on resolution or based on network speed, but based on the judgment of the designer for what is the best image at a particular breakpoint.

As an aside, showing photographs at different sizes to illustrate a point is more difficult when you’re dealing with flexible images in a responsive design. If those examples don’t make sense on your phone, I’m afraid you may have to look at it on a wider screen to see what I’m talking about! :-)

2. Enabling authors to provide different resolutions of images based on a variety of conditions.

When people talk about how to handle images for retina displays, they are talking about this second issue. The same is true of making decisions about the size of images based on bandwidth. All are dealing with how to deliver different resolutions based on conditions (pixel density, network speed, etc.).

Apple’s proposed addition to CSS4, image-set, is designed to solve this issue in CSS. Authors define images at different densities and the browser picks the best one based on some criteria that could be as simple as pixel density or as complex as a combination of observed network speed, pixel density, and user preference. What the criteria is remains to be defined.

Where the <picture> element fits

The proposed <picture> element attempts to solve issue #1. It focuses on how to give authors the ability to specify different images, but doesn’t do anything about pixel density or bandwidth.

I’ve seen a lot of feedback on the <picture> element that says we should have a new type of image format to replace JPEG, PNG and GIF that would be resolution independent. That would be awesome. And it would solve issue #2, but it wouldn’t help with the art direction outlined in issue #1.

When we discuss various solutions, it behooves us to figure out which issue we’re trying to solve. We can also debate whether or not the two issues I outlined are legitimate or if there are other issues that aren’t addressed by them.

But in order to have a fruitful discussion about how to solve these issues, we need to be clear about which issues we’re talking about or we’ll end up wasting more time. What I often see in comment threads and on Twitter is two people debating different solutions for responsive images, both looking at different issues, and neither realizing that the other isn’t looking at the same problem.

My hope is that by defining these issues, we can stop spinning our wheels and have more successful discussions.

Despite that fact, I’ll freely admit that I have a special affection for the Portland version of WebVisions. It was the first conference I attend and the first one I spoke at. There are talks that I saw at WebVisions that still influence my work to this day.

This is the 12th year of the conference. The organizers bring in fantastic speakers and do so at a price that is much less expensive than what it costs to attend most other conferences. At my previous job, I would often take our entire team to WebVisions.

This year is no different. The conference starts on May 16th. Lyza and I will be giving a workshop showing desktop web developers how to convert their skills to mobile. I’m also giving a talk called Casting Off Our Desktop Shackles. We’d love to see you at both!

If you’re in the Portland area, I encourage you to attend WebVisions. I love the community around the event. And I like playing host to all of the smart people who travel to our city to attend or speak at WebVisions. If you do attend, please say hello.

First, there is the weird brain-bending device pixel versus CSS pixel math we’re all trying to do in our head since we started seeing a lot of high-density digital displays on the market.

Then there’s the whole schizophrenia in the standards worlds about what a pixel even is. Ask the W3C currently, and you’ll get a wholly incomprehensible definition (kind of fun as a drinking game/party laugh) that in the end claims "absolute unit." Ask the Mozilla Developer Network, and you’ll get a brief answer of "relative."¹

The reference pixel is the visual angle of one pixel on a device with a pixel density of 96dpi and a distance from the reader of an arm’s length. For a nominal arm’s length of 28 inches, the visual angle is therefore about 0.0213 degrees. For reading at arm’s length, 1px thus corresponds to about 0.26 mm (1/96 inch). —Excerpt from the W3C definition of a reference pixel

And this all before we’ve even reached the lively world of viewports: layout viewports, visual viewports, viewport scales. And the different treatment by browsers of zooming—Webkit browsers don’t really scale pixel-based units after a zoom, Mozilla and Opera do. @ppk may understand pixels², but I think he may be superhuman.

So, pixels may be relative or absolute, big or small, device-specific or theoretical. Here’s the rub at this point: I don’t care.

What’s a pixel? I don’t care

When I wrote the post about The Ems Have it (em-based media queries) a few weeks ago, I was mostly just focusing on the surgical problem of getting Webkit browsers to behave, when you get down to it. I spoke of practicalities: em-based media queries don’t break when you zoom.

But whether or not pixels ultimately do scale in Webkit browsers is beside the (deeper) point. If we’re staying true to a "content first" maxim, it follows, logically, that our media queries—as well as our layouts overall—might benefit by being subordinate to the content itself.

From my perspective, ems are more true to content.

For me, thinking in pixels for media queries ties me into some bad habits. I start thinking as if there is some 320 x 480px inviolable "mobile smartphone" reality that we can rely on. On the other hand, 30em is a readable column width regardless of the surrounding pixel complexities.

We’re moving further into a world where content is king, and we know it needs to "flow like water." The containers we build to hold it can likewise be based on the proportions of the content itself. Ideal line lengths, space to breathe around text—those are things most naturally derived by using units tied to the text itself. Ems.

The fact that baseline ems tend to be loosely tied to pixels, i.e.: 1em ~= 16px ~= 14pt should help control freaks to unclench a bit. But, as I’ve been arguing for a while now, that control is illusory anyway. Your content has already broken free of its jail cell. You just might not know it yet.

In his talk at Breaking Development Orlando, Ethan Marcotte (the Great Father of Responsive Web Design) mentioned his support for em-based media queries going forward. While he didn’t delve into the specifics of his reasoning, my hunch is that his motives are generally in line with this notion: content first.

So, I’ve done some of the convolutions. I’ve tried to understand exactly what a pixel is in a zoomed viewport on a pixel-dense device vis-a-vis its CSS-to-device pixel ratio. I’m done with that for now. Long live ems.

Synposis

The mobile web is a complex place. There are lots of ways to build a mobile website—how do you choose the right tool for the job?

In this webcast presentation, we touch on a series of technologies and techniques including:

• Responsive Web Design (RWD)
• Mobile-first RWD
• Server-side device detection
• HTML5 and CSS3 on mobile
• Device idiosyncrasies
• Some of our favorite tools and tips for mobile development

Who is this webcast for?

If you already have web development chops but want to go mobile, this is the talk for you! Familiarity with HTML and CSS is a must; comfort with some JavaScript is helpful.

Register Now!

The webcast is free. Register now—we’ll see you there!

Resource Links

Here are some links to some of the topics and tools we talk about in our presentation.

• Ethan Marcotte’s original article about Responsive Web Design on A List Apart
• The Boston Globe’s new responsively-designed site
• A few articles about using the Fluid Formula and creating fluid layouts: from Jake Bresnehan and StudioPress
• You can also read Ethan’s brief and comprehensive book: Responsive Web Design
• Our blog post about em-based media queries
• Blaze.io
• Respond.js polyfill for media queries in IE
• WURFL device database, service by ScientiaMobile
• DeviceAtlas device detection service
• RESS (Responsive Design + Server-side Components) post by Luke Wroblewski
• Responsive.is for previewing responsive designs
• ResponsivePX for previewing responsive designs
• Chrome extension: Window Resizer
• Adobe Shadow testing and preview tool
• Future friendly
• Our book: Head First Mobile Web (O’Reilly)

Whereas my personal theme for the second half of 2011 was about letting content go, 2012 seems to be the year I obsess about content: what is content, what does it mean, and what does it look like? How do we go about untangling the CMS mess we’ve built over the last decade-and-a-half? How should content be marked up? Stored? Transformed? Delivered to devices and clients?

With the new Cloudfour.com site, we got a chance to Think Very Deeply about content and our mobile experience. We got to explore some of the nagging questions that keep us (well, me, at least) up at night. I’m not going to profess to have the answer, nor am I going to assert that the (long) thinking, designing and development process we endured was without convulsions or struggle¹. But I think we’ve been lucky in the opportunity to dig in.

Content is content is content is content…

At its core, www.cloudfour.com is content. Human-readable content written by regular people. Being bold and brave (maybe it’s hubris), we started with content before any design happened.

We’re not, by far, the first to shout, “Hey, Content First!” That notion’s been around the block a few times. But the practical challenge lies in unmooring oneself from 15 years of executing web development and design in particular ways and trying new habits and processes on for size. What sounds straightforward—have a point first and then make it pretty—actually goes against a staggering amount of what we’ve become used to doing with our web projects.

In our case, we opted for writing and storing our content in static text files. We selected markdown for this project, specifically, pandoc-extended markdown with Cloud Four-specific post processing extensions. Phew. More on that sometime other than now. Important piece being: human-readable text in a loose but meaningful format.

What a transformation!

Our happy, respected content sits there on the filesystem, grinning mysteriously, versioned, simple, ready to serve whatever purpose it needs to serve. In our case, obviously, we want a web site, so we can transform that core content into a set of web pages to make up a site that works in web browsers.

You’ll note that a lot of this sounds like, um, duh, yeah, we’re web devs, we make web sites. But in peeling back and thinking this hard about content, web pages as HTML isn’t necessarily a given for every context that we might use this content. But let’s talk about web pages for our purposes today.

Our content is transformed using a somewhat magical tool called pandoc², which processes it, plunks it in an HTML template and does some other miscellaneous things—out come HTML pages! These HTML pages are then pushed to the live server environment where they enjoy a performance-tuned, simple existence, just waiting for folks to look at them and enjoy their content.

Though the content changes from time to time—it’s versioned in a github repository—it isn’t actually dynamic, so serving it out of a database (here I am getting a bit pedantic again) provides more of a performance-maintenance drain than a boon. Or at least, that’s what I tell myself to feel awesome.

But what about the blog? Ain’t broke, don’t fix

As giddy as the simplicity of static content and transformations and a pure ultra magical shared philosophical language of core content makes me as a dev, I have to kneel to practicality and things that make sense sometimes.

WordPress is a blogging framework. It’s been around a long while, we’ve used it a long while, and it has the tools for, well, you know, blogging. We’re sticking with WordPress for our blog, though you may notice that our blog now lives a robustly independent existence as blog.cloudfour.com. Heck, maybe I’ll get the exciting opportunity to futz around with partially-static WordPress options with John!

There’s a lot more to talk about

The site is intensely responsive. It uses some edgy CSS selectors to get its job done semantically. The process of separating content from presentation can be deeply, deeply hard and we have stuff to say about that. We’ve spent a lot of time optimizing performance. We’ve battled specific bugs (many we knew about, some we didn’t) to get the site to look reasonably decent across a lot of devices. We are doing unprecedented things with :before pseudo elements. Our media queries are in ems.

We’ll continue to talk about the specific cool stuff we experimented with to get the new site working in upcoming posts. But I wanted to give a quick, high-level look at what we did before getting lost in the details.

Here is a nice picture

1. Static content pages and image content, managed in a github repository.
2. Pandoc and other transformation processing.
3. Layout, design and behavior resources shared between static web and our blog.
4. Blog content stored in WP database.

Thank you, Cloud Four Team

I owe a great amount of gratitude to the brilliant team at Cloud Four. Thinking the new site through and building it wasn’t always easy, fun or even sane. We had to (uncomfortably) learn new ways to do things that we were quite competent at before thankyouverymuch. There were confounding moments. It took John 327 hours or whatever to get Haskell compiled³ on our shared development server so we could run pandoc. And Aileen endured my weird experiments with SASS.

Here’s what you need to know.

Live in Europe or need a vacation? Attend Mobilism

You must attend Mobilism on May 10-11. We can’t speak highly enough of this conference.

• This is the only conference in Europe focused on mobile web technology.
• The speaker’s list is fantastic. Lyza went last year and vouches for how awesome the conference is.
• Lyza will be giving a talk entitled: Cutting through the crap: The essence of content on the future web
• I’m going to giving a talk called The Immobile Web which will focus on the upcoming explosion of smart TVs.

Seriously, register now. If you can’t attend, please pass on the word. This is a great conference, but they need your help making sure enough people attend so they can continue to put on this event..

Discount!!! EXPIRES SOON.

You can get 10% discount on the ticket price by going to a special registration page. This offer expires March 31st.

Attend Our Workshop

Lyza and I are also giving a workshop in Amsterdam. Here is a short description:

How to get from here to everywhere: Taking your web skills mobile

OK. You already know how to wield HTML and CSS, and maybe you can juggle JavaScript and server-side technologies, too. What you need to know is how to take your existing web skills mobile—quickly.

Lyza Gardner and Jason Grigsby, co-authors of “Head First Mobile Web” (O’Reilly), will show you the leading strategies for making web sites mobile friendly, whether you’re adapting existing sites or building new ones from scratch. There is no silver bullet. But in this hands-on workshop, we’ll show you the tools that make up the mobile web designer and developer arsenal—and how to choose which tactics to use for different projects.

This is a great opportunity for you or someone on your team to learn how to build things for the mobile web. It is a day-long immersion in the tools and techniques required to deal with device diversity. Seats are limited to please register soon.

In the States? Attend Breaking Development

Come join us at the Breaking Development conference in Orlando on April 16-18. I’ve written in the past about how much we love this conference. We love it so much that we sent the entire company to the last Breaking Development conference.

Here are the highlights of why you should join us:

• This is the only conference in the United States focused on mobile web technology.
• Like Mobilism, the speakers list is fantastic.
• Everyone at the conference is interested in mobile web. The conversations continue into the evening. You leave the conference buzzing with ideas.
• I’m presenting on Smart TVs again with my The Immobile Web talk.
• People are guaranteed to make fun of me. I’m sure this will happen at Mobilism as well, but the Breaking Development organizers have refined it to an art form. :-)

I don’t know what else to say. Register soon.

Discount!!! Limited to the first 10 people.

The first ten people to use discount code ‘ORGRI12′ will save $100 on their registration.

Attend our workshop

There is ONE and ONLY ONE seat left in the workshop that Lyza and I will be giving in Orlando. The workshop is called Zombie Apocalypse Preparedness 101. It teaches you the survival skills you need to survive the upcoming zombie apocalypse of devices. Register ASAP before the final seat is taken.

Support these conferences

If you care about the topics that we cover on this blog, then you need to support these two conferences. They are the only conferences focused exclusively on the challenges of building things for mobile devices.

Conference like this run on thin margins. I know the organizers of both conferences would like to make money, but are primarily motivated by helping our community get off the ground. If we want the conferences to continue, we have to support them.

Even if you cannot attend the conference, please pass the information on to your co-workers and friends. Tweet about it. Share on Facebook. Do whatever you can to help them out.

I would consider it a big favor if you would do so. Thanks in advance.

Bruce quotes from a survey of mobile web users and pulls out a few highlights including a huge demand by users for performance. Bruce writes:

This tells us that speed is more important than aesthetics. So perhaps some of the time and effort put into media queries, viewports, avoiding scrolling, line length would actually be better employed reducing HTTP requests and optimising so that websites are perceived to render faster.

Exactly.

This echoes something Brad Frost and I were talking about the last time I was in New York.

If you could only do one thing to prepare your desktop site for mobile and had to choose between employing media queries to make it look good on a mobile device or optimizing the site for performance, you would be better served by making the desktop site blazingly fast.

Most mobile browsers are pretty good about providing tools to help someone utilize a design meant for desktop on a small screen. People can double-tap or pinch and zoom to see and read the content.

But if your site is a bloated mess, there is nothing people can do about it. There is no magical gesture that people can invoke to make something load faster if the developer hasn’t built the site for speed in the first place.

The only gesture they are likely to use involves a single, upright finger as they ditch your site for one that responds to their requests in a timely fashion.

As you can imagine given my research on responsive images, I was keenly interested to see what approach Apple took.

What they’ve chose to do is load the regular images for the site and then if the device requesting the page is a new iPad with the retina display, they use javascript to replace the image with a high-res version of it.

The heavy lifting for the image replacement is being done by image_replacer.js. Jim Newberry prettified the code and placed it in a gist for easier reading.

The code works similarly to responsive images in that data attributes are added to the markup to indicate what images should be replaced with high-res versions:

1
2
3
4
5
6
7
8
9
10
11
12
13

<article id="billboard" class="selfclear" data-hires="true">
  <a id="hero" class="block" href="/ipad/">
    <hgroup>
      <h1>
        <img src="http://images.apple.com/home/images/ipad_title.png" alt="Resolutionary" width="471" height="93" class="center" />
      </h1>
      <h2>
        <img src="http://images.apple.com/home/images/ipad_subtitle.png" alt="The new iPad." width="471" height="54" class="center" />
      </h2>
     </hgroup>
     <img src="http://images.apple.com/home/images/ipad_hero.jpg" alt="" width="1454" height="605" class="hero-image" /> 
  </a>
</article>

Unlike most of the solutions I reviewed last summer, Apple is applying the data-hires attribute to the parent container instead of to the images to themselves. Also, the images borrow from native iOS development and have consistent sizes. So the high-res version of ‘ipad_title.png’ can be found at ‘ipad_title_2x.png’.

As far as I can tell, there is no attempt to prevent duplicate downloads of images. New iPad users are going to download both a full desktop size image and a retina version as well.

The price for both images is fairly steep. For example, the iPad hero image on the home page is 110.71K at standard resolution. The retina version is 351.74K. The new iPad will download both for a payload of 462.45K for the hero image alone.

The total size of the page goes from 502.90K to 2.13MB when the retina versions of images are downloaded.

Another interesting part of image_replacer.js is that it checks for the existence of 2x images before downloading them:

297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334

    requestHeaders: function (c) {
        var a = this;
        if (typeof a._headers === "undefined") {
            var b = new XMLHttpRequest();
            var a = this;
            src = this.hiResSrc().replace(/^http://.*.apple.com//, "/");
            b.open("HEAD", src, true);
            b.onreadystatechange = function () {
                if (b.readyState == 4) {
                    if (b.status === 200) {
                        a._exists = true;
                        var f = b.getAllResponseHeaders();
                        a._headers = {
                            src: src
                        };
                        var d, e;
                        f = f.split("r");
                        for (d = 0;
                        d < f.length; d++) {
                            e = f[d].split(": ");
                            if (e.length > 1) {
                                a._headers[e[0].replace("n", "")] = e[1]
                            }
                        }
                    } else {
                        a._exists = false;
                        a._headers = null
                    }
                    if (typeof c === "function") {
                        c(a._headers, a)
                    }
                }
            };
            b.send(null)
        } else {
            c(a._headers, a)
        }
    },

This is probably necessary as they move to providing two sets of assets in case someone forgets to provide the retina image. It prevents broken images. Unfortunately, it means that there are now three http requests for each assets: a GET request for the standard image, a HEAD request to verify the existence of the retina image, and a GET request to retrieve the retina image.

Another interesting bit of image_replacer.js is when they decide to go retrieve the double-size image:

26

if ((this.options.debug !== true) && ((typeof AC.Detector !== "undefined" && AC.Detector.isMobile()) || (AC.ImageReplacer.devicePixelRatio() <= 1))) {

Of particular interest is the test for AC.Detector.isMobile(). This is defined in a separate script called browserdetect.js (prettified gist version).

The browserdetect.js is full of user agent string parsing looking for things like operating systems and even versions of OS X. The isMobile() function does the following:

166
167
168
169

isMobile: function (c) {
  var d = c || this.getAgent();
  return this.isWebKit(d) && (d.match(/Mobile/i) && !this.isiPad(d))
},

Basically, is this a WebKit browser, does the user agent mention mobile, and let’s make sure it isn’t an iPad. Browsers not using WebKit need not apply.

Testing for yourself

AppleInsider’s instructions on how to test the retina version of Apple.com on your computer are very easy. Open Apple.com in Safari. Go to the console in the Web Inspector and type the following:

AC.ImageReplacer._devicePixelRatio = 2
new AC.ImageReplacer()

You’ll get back a klass object as shown below.

As an aside, notice SVG references in the console screenshot.

What can we learn from this?

Probably not a whole lot for a typical site because our goals will be different than Apple’s.

For Apple, it probably makes more sense to show off how wonderful the screen is regardless of the extra time and bandwidth required to deliver the high-resolution version. For everyone else, the balance between performance and resolution will be more pressing.

There are a few minor things that we can take away though:

• Planning ahead and knowing that you can depend on high-res images being available would be preferable to making extra HEAD requests to check to see if the images exist.
• Setting priority on which images to replace first is a good idea. This is something to look at and borrow from image_replacer.js.
• The retina version of Apple.com’s home page is four times the standard home page. Delivering retina images should be considered carefully.

Perhaps most importantly, Apple isn’t sitting on some secret technique to make retina images work well. Maybe they will provide better solutions in iOS 6. The way they handle images—downloading both sizes plus an additional HEAD request—may be the least efficient way to support retina displays. But for Apple, it likely doesn’t matter as much as it will for your site.

Hat tip to Jen Matson for pointing me to the original AppleInsider article.

Further reading

• ipad3′s retina display will wreak havoc on the web
• optimizing web experiences for high resolution screens
• 3.1 Million Pixels Are Heavy

If we take Steve Jobs at his word, then a question for web developers to ponder is whether or not this new TV will include a browser. Let’s take a look at some of the arguments for and against including a browser.

Reasons why a new Apple TV will not include a browser

• The current Apple TV does not have a browser.
• Browsers on TVs have not had the best user experience and adoption has been low.
• When announcing the latest Apple TV, Steve Jobs said, people “don’t want a computer on their TV. They have computers. They go to their wide-screen TVs for entertainment. Not to have another computer. This is a hard one for people in the computer industry to understand, but it’s really easy for consumers to understand. They get it.”

Reasons why a new Apple TV might include a browser

• Apple TV is built on top of iOS and has the WebKit rendering engine built in.
• People jailbreaking their Apple TV have been able to get access to the browser.
• There is a good chance that an App Store for Apple TV would be one of the major mechanisms for getting new content to the Apple TV.
• If Apple TV had an App Store, support for embedded webviews would be necessary as many apps these days combine native and web components including Apple’s own iTunes, Apple Store, and App Store apps.
• If embedded webviews and an App Store are available, someone will build a browser absent a specific policy from Apple against browsers. Such a policy seems unlikely as it would be difficult to enforce and cause an unnecessary backlash.
• If others are going to create browsers for the platform, it makes sense that Apple would want to define the default browsing experience by shipping Safari on the platform.
• Steve Jobs has a history of dismissing something until Apple has solved how it wants to implement it (see: multi-tasking in iOS).

Why worry about what Apple will do?

It wasn’t too long ago that people believed no one wanted to use the web on their mobile phones. The iPhone proved that to be wrong.

Now we hear similar refrains about how no one uses the web on their TV. If Apple releases a new TV and they include a browser, we may again see a shift in user behavior that would disprove those assertions.

If that happens, the challenges of building web pages will again increase as we all start to consider how to address a new form factor that we’ve long ignored.