Watching him work, it quickly became apparent that--while he knew what he was doing--he wasn’t efficient. Using the tools on his laptop was onerous; he fought the system, taking the long way each time to achieve the desired results, but at the cost of lost time for both of us. As I was watching him, I wished he were more lazy.

Wait. Lazy? Well, sure.

Laziness is the first great virtue of a programmer:

“The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don’t have to answer so many questions about it. Hence, the first great virtue of a programmer.”

—  Larry Wall

Fortunately, laziness isn’t just for programmers; it’s for all who work and take pride in their accomplishements, and with computers, it’s possible to be lazy (and efficient!) with just a little effort.

And, look at all the ways you can be lazy!

App launchers

Moving hands away from the keyboard, or away from their default position if you use a secondary input device consistently, inherently creates inefficiencies. Use an app launcher that you can customize. Four or five keystrokes is all I need to open just about any app on my system: two to open the launcher, one or two to select the app or file, and enter to open it. You can also use app launchers to set up scripts and run workflows.

On OSX, options include Alfred, Launchbar, and Quicksilver.

On Linux, options include Gnome Launch Box, Gnome Do, and Launchy.

On Windows, options include Skylight and Launchy.

Launchy also lets you use Python to extend its functionality, so you can automate common work patterns.

Or, if you’re on OS X, use the highly-underrated Automator workflows for tasks on your system. You can have your workflows triggered manually or automatically. An example could be instead of the default action to import all photos into iPhoto whenever you mount an SD card, you could have all images post to a private folder on OpenPhoto and tag them with the date.

Befriend the command line

Not all workflows, however, need be triggered from an app launcher or automatically. Using the command line, you can trigger most apps to run just the way you want. While some developers may roll their eyes, not everyone uses the command line. Befriending the command line is the best form of laziness.

If you’re on a Unix platform — yes, that includes OS X — these twelve commands will get you going: cd, cp, less, which, ls, rm, pwd, cat, find, grep, !!, and !$.

Want to execute the last command again?

!!

Want to repeat a command you typed in a while ago?

!{command}

Want to go to the beginning of a line?

^a

Want to go to the end to edit the last argument?

^e

Want to switch back to your previous directory?

cd -

Want to look through your history to find the command you ran a while ago?

^r

My personal favorites are pushd and popd, with dirs to tell me where I am.

pushd with an argument will add the current directory (pwd) onto the directories stack (listed with dirs) and change the working directory to the one specified by the argument:

% pwd
/Users/kitt
% pushd ~/work/projects/web-advent
~/work/projects/web-advent ~
% pushd ~/work
~/work ~/work/projects/web-advent ~
% dirs -v
0       ~/work
1       ~/work/projects/web-advent
2       ~

pushd without arguments will swap the top two directories on the stack — the same behavior as cd -.

% pushd
~/work/projects/web-advent ~/work ~
% dirs -v
0       ~/work/projects/web-advent
1       ~/work
2       ~

The advantage here is when you have a number of places you need to be working:

% dirs -v
0       /usr/sites/example.com/8929/website
1       /var/log/apache2/example.com/
2       /etc/apache2/extra
3       /home/kitt/work/reports/example.com

Using the +N notation moves you among the directories, either popping the Nth directory to the top of the stack, or rotating the stack around N positions:

% pushd +2
/etc/apache2/extra /usr/sites/example.com/8929/website /var/log/apache2/example.com /home/kitt/work/reports/example.com
% d
0       /etc/apache2/extra
1       /usr/sites/example.com/8929/website
2       /var/log/apache2/example.com/
3       /home/kitt/work/reports/example.com

I’m lazy even when using pushd, popd, and dirs -v. I have them aliased to pu, po and d. I alias my commands heavily, setting up aliases for project directories, common commands I run frequently, and uncommon commands (I hate trying to remember the full syntax).

If you run an even semi-complicated command more than once, write an alias. No one wants to type in this command more than once:

alias gen_ctags='/usr/local/bin/ctags \
  --langmap=php:.engine.inc.module.theme.php.install.test.profile \
  --php-kinds=cdfi --languages=php --recurse --exclude="\.git" \
  --exclude="\.svn" --exclude="\.hg" --exclude="\.bzr" \
  --exclude="CVS" --totals=yes --tag-relative=yes \
  --regex-PHP="/abstract\s+class\s+([^ ]+)/\1/c/" \
  --regex-PHP="/interface\s+([^ ]+)/\1/c/" \
  --regex-PHP="/(public\s+|static\s+|abstract\s+|protected\s+|private\s+)function\s+\&?\s*([^ (]+)/\2/f/"'

Practice and repetition are the best ways to learn shortcuts. If you’re looking for a concentrated resource, ShortcutFoo for the command line is a great one.

ShortcutFoo also has drills for learning how to use the shortcuts for a large number of programs, including a number of editors.

If you’re a developer, the best thing you can do for efficiency and, well, laziness, is to become proficient in your editor, learning its shortcuts and learning how to write editor macros. If your chosen editor doesn’t have macros, pick a different editor; no need to handicap yourself with a tool that doesn’t actively help you be lazy.

Emmet

When looking at editors or writing macros, anyone who writes any significant HTML should look at the Emmet (formerly Zen Coding) plugins.

The easiest example of how powerful Emmet can be is the abbreviation expansion, which takes a abbreviated line like this:

div#banner>div.logo+ul#navigation>li*4>a

And, expands it into this:

<div id="banner">
 <div class="logo"></div>
 <ul id="navigation">
   <li><a href=""></a></li>
   <li><a href=""></a></li>
   <li><a href=""></a></li>
   <li><a href=""></a></li>
 </ul>
</div>

Emmet also provides editing shortcuts, tag manipulation (matching, removing, navigating to), and other generators (vendor prefixes, gradients).

There are, of course, thousands of other ways to be more efficient and lazy in a common workday. Being aware of repetition, asking co-workers how they accomplished a task faster than you, and recognizing that laziness can be a virtue are ways to increase that efficiency and leave room for more important things in life.

Like relaxing. Or, doing nothing.

How I started

I’ll not pretend I was kicking around back when the Web was formed. I was too busy trying to work out where I fit in the high school oligarchy, and how to talk to girls. I spent every other lunch time in the school computer rooms, but I was mostly in IRC chatrooms (talking to girls might be easier if you can’t see them), not mucking about with HTML. In 2001, five years after high school graduation, I made my first production site. It was the year Netscape, Opera, and Internet Explorer all released 6.x versions, and neither Safari nor Firefox existed. I was part of a small team developing a business around a content management system to help golf professionals run their own web sites. These days, we’d call it a Software-as-a-Service startup. I was the programmer (at first), and the only technical person, so I did everything.

The business owner, a golf pro himself, came to me with a feature request.

“We need pros to be able to change the layout themselves from the admin. Put the logo on the right instead of the left. Change the background and text colours. Switch the size of the columns. That sort of thing.”

“Uhhh… Yeah, so that’s going to be pretty difficult.”

“What? I don’t care. Can you do it or not?”

“Give me two weeks, and I’ll know.”

This was my first lesson in client-developer relations. He didn’t know or care what tables were and how they worked. He just wanted the feature he wanted. So, I went looking for answers. That’s when I discovered CSS-P.

CSS-P, yeah, you know me

The P stood for positioning. Until then, we’d only been using a style element in the head of each document to change the fonts and link colors. I thought I was clever for using a Server Side Include, so I only had one file to change. If you search for CSS-P these days, you’ll find that most of the old articles and tutorials are gone, but for whatever reason, two remain, unchanged: MaKo 4 CSS and CSS-P 3box.

And I remember these. I’ve never known who MaKo is, but she or he is responsible for the basis of my entire career.

This almost perfectly valid page from 2001 looks and works exactly the same now (December 2012) as it did in when it was made. We probably no longer need to worry about the Netscape 4 hacks or the browsers that don’t support CSS. (View Source for fascinating tidbits like that.) This is one of the things the Web gives us.

Longevity

Jeremy Keith has written far more, and far more eloquently, than I ever will on the importance of preserving digital artifacts and the horror that is linkrot. And, if you haven’t read Anil Dash’s retrospective The Web We Lost, I strongly recommend it.

Inherent in the nature of the Web is a sense of preservation. The possibility of archive. The Web is pure information, and we have the capacity to store it all. So many of us produce content for our friends, family, colleagues, and peers to share. Beautiful, interesting, informative, dull, hilarious, ugly, fascinating content. Like MaKo’s article, or the others I’ve linked in this article, I’d love to know it will still be available in 10 years or 20. Perhaps not in quite the same format, but still readable, still findable. Still offering benefits and insights years later.

This is what the Web offers us. We need to find ways to hold up our end of the bargain.

It must be a small world after all. Every day I am in communication, over the phone, text message, email, Facebook, Twitter — you name it — with someone that is in a different city, county, state, country, continent, but somehow not a different planet. I guess it is a small world, but in a huge, sparse universe.

Before the Web, and all these social apps, there was a time when you dictated the number for your phone to dial by diligently typing in every digit, rather than asking your smart phone what to do next. During the time when your parents — or maybe grandparents — were your age, were they in near constant communication with someone living in another country? The answer to that probably depended on whether they were one of the few to have a friend from grade school through high school, or a more likely reason for knowing someone abroad was participation in the service.

Today, I am posting this article that will likely be read by people from all around the globe. I might collect more Twitter followers, and I will likely learn about new happenings in the web development community from those whom I follow. I will write at least one email to someone not residing in the same country as I do, and I will compose, and reply to, several more e-mails to people I will not talk to in person for at least a month, if ever. The odds of me commenting on or liking a photo or post of someone I have not seen since my tenth high school reunion is not improbable, either. It must be that technology and the Web has shrunk the world, brought us closer together… but has it really?

Many of us participate in a communication graph encompassing a large geographical region, but has this made us any more worldly?

This past year, I spent a decent amount of time traveling to places I had never been to. My travel was mostly to web development conferences. The developers were excited to meet more people in the community; the conversations were informative, and sometimes became a heated debate. It was difficult to be disinterested. How could anyone electing to attend a web development conference not be excited to participate in a lecture, discussion, or a debate about web development?

The world is not as small as it seems. The Web has just made it easier to create small, niche communities, based on shared interests that cover a large global expanse, as opposed to the past where small, niche communities often times were more or less dictated by geography. Participating in a community spread thin across the globe, but tightly knit by its passion surrounding common interests can create a false sense of worldliness. Within such a group, the world is as small and narrow as the role the common interest plays in the world population.

I am not suggesting that the Web is insignificant. Most would say that the majority of the world’s population is impacted in some way by the Web. The common interests and shared topics of the web development community, as an example, tend to be far more specific. Is it rarely about the significance of the web? Usually, discussions about which technology to use, which practices to follow, which programming language to use, or something else that is more personal, teach individual members day-to-day. Beyond that, I am talking more generally about all types of self-selecting groups. I am warning against being lulled by the false sense of reality that self-selection bias can breed.

Still using the web development community as an example, the truth is that most people (also known by the dehumanizing term: user) do not care what language your web site is implemented in, which database you used, whether or not you host the site from the cloud or your own bare metal. Most people do not care if you use continuous deployment or how good your test coverage is. Read this last bit to someone who is not a web developer, and you are more likely to discover that he doesn’t even know what these things are. Some of you reading this may think, “we should teach everyone.” That’s not my point.

My point is that everyone — no matter who they are — needs to actively put themselves in an uncomfortable position every now and then. It is comforting to surround yourself with people just like you. Next time you are at the company party, rather than spending most of the time talking just your co-workers, take the opportunity to listen to your co-workers’ party guests. It is a great opportunity to ask them about their professions; find out what their interests are. It is an opportunity to be educated, a chance to view the world through the eyes of someone unlike you. Now this is just a starting point because these party guests obviously have something in common with your co-workers who likely have something in common with you, but you can use this as a start.

Another good start is to change the topic in your common interest group, every now and then, to something off-topic. While I was traveling this past year, conference to conference, I decided to ask most people that I met, “What did you learn in history class?” It did not matter whether the person grew up in Poland and now resided in Great Britain, or if the person simply lived in the south as opposed to my north-east roots. The question was enough to remove the person from the normal biased answers to the ever reworded common questions. It was also an opportunity for me to glean some notion of how a person’s view point of the world might be different from the view point I was raised with.

All of this ties into common things we all eventually tackle during our web development experience, like learning that internationalization is more than handling different character sets and translating verbatim to another language. That it is more about a quest for localization, realizing that different cultures exist. Sometimes it is even recognizing that the dividing lines could be education, occupation, or some other basis besides culture. It’s also like recognizing not to trivialize the world through Big Data. When attempting to find the commonality amongst your user base, remember that those are also individuals. Do not take for granted the opportunity to actually sit down with someone, listen to them, observe them, just because you have data.

For me the goal of all this is to encourage you to get a little uncomfortable, break out of whatever small worlds you have joined, and see the world as vast, huge, uncertain, and as ever-unlimited as a wide-eyed child does. Otherwise, your endeavors to take the world by storm with your web development will be as limited as your outlook.

First things first

Before we go wild with “creativity,” we need a simple HTML structure to work with.

<body class="is-anchored">
    <div id="navigation">
        <p>My awesome app</p>
    </div>
    <div id="panel-main" class="panel">
        <h1>Click me to slide panel</h1>
    </div>
    <div id="panel-sidebar" class="panel">
        <h1>Click me to slide panel</h1>
    </div>
</body>

Next, we need to bind our panels (.panel) so they toggle the class name .is-anchored from the body when clicked. For that we are going to use the following jQuery snippet:

$(document).ready(function() {

    // this should be your wrapping element,
    // in this case I use the `body`
    var $body = $(document.body);
    var $panels = $('.panel');
    var dom_classname = 'is-anchored';

    $panels.on('click', function() {
        $body.toggleClass(dom_classname);
    });

});

Now for the fun part

Here is where we get crazy with our CSS. First, let’s add the rules that will be applied to .panel elements when .is-anchored is present. I’m omitting some styles in order to focus on the sliding panel layout, but I’ve supplied the complete style sheet.

body {
    /* prevents horizontal scrollbar */
    overflow-x: hidden;
}

.panel {
    /* allows each panel to have its own scrollbar, if needed */
    overflow: auto;
    /* makes the browser window the containing block */
    position: fixed;
    top: 0;
    width: 100%;
    height: 100%;
}

.is-anchored #panel-main {
    left: 0; /* moves this panels into view */
}

.is-anchored #panel-sidebar {
    left: 100%; /* moves this panel off the screen to the right */
}

Next, we add the styles for positioning the panels when .is-anchored is not present. Note that left is the only value which varies based on the presence of .is-anchored. This single property change allows us to create a smooth sliding effect using CSS transitions.

#panel-main {
    left: -100%; /* moves this panel off the screen to the left */
}

#panel-sidebar {
    left: 0; /* moves this panels into view */
}

At this point, clicking anywhere inside .panel should move the hidden div into view. It is time to add some sliding to our sliding panels with a dash of CSS transitions.

.panel {
    /* applies CSS transition to `left` property */
    -webkit-transition: left .5s ease;
    -moz-transition: left .5s ease;
    -o-transition: left .5s ease;
    transition: left .5s ease;

    overflow: auto;
    position: fixed;
    top: 0;
    width: 100%;
    height: 100%;
}

Slow clap

Here is the final product. I have used this technique for displaying individual assets on Gimme Bar, a magic little app that relies on infinite scroll for its pagination. Using sliding panels helped me preserve a user’s browser window position when scrolling, all without using a complex JS solution. Simple!

We are living in what I perceive to be an information Big Bang. Our lives are disrupted exponentially more by information and technology than was the case a decade ago. I have to squelch the questions asked by my own brain just to get through the day. Thoughts cross my mind like: which blog should I add to my reader? Who should I follow? Am I missing out on anything important? Is there new technology that I should learn? Are my skills still relevant? Will they be relevant a week from now? What are people saying on Twitter right now? Ahh! What to do‽ Unfortunately, I don’t have the answers for How to Stop Worrying and Love Information Technology Overload.

If you can manage to filter out all the worrying and get to the most important part, self-education, then we can begin to make some progress. It’s what got me here. I’m apparently writing this article because I have convinced somebody that I might know something, likely due to something I’ve taught myself. As web professionals, we have to learn all sorts of skills. Each skill had to be given some portion of your attention, and likely that study made you a better person. As the years go by, people come to know you by your accomplishments, and you get paid to do those things. But what about the things on your self-improvement list? Do you ignore them because you ran out of time?

Every time I approach a new task, my own little mini Big Bang of ideas goes off in my head. The task could be something simple like “add a notification bubble next to the label whenever something funny happens.” Unfortunately, I’ve already thought of 100 associated things that I would like to do to add my personal touch, as well as technologies to help me achieve my goals. Obviously, I have deadlines to meet, and I can’t just add new tickets at will without discussing it with my team, but I can choose one of them that is important to me, that will help me add value to the project and learn something at the same time.

Lately, I’ve been interested in automated web application testing. That shit is hard. Real hard. There are so many daunting words just to get started: unit tests, functional tests, integrated tests, acceptance tests, test targets, test-driven development, behavior-driven development… holy shit! What gives? What approach and technology should I use? Someone might recommend: PHPUnit, Selenium, Cucumber, Behat, Mink, CasperJS, and Jasmine. That’s a lot of things just to make sure my application works. Dude, I’m a developer. I’m always trying to ensure that shit is working with my eyes, and now you’re throwing all this crap at me? Now I’m not so sure my shit is working… aw man…

Relax. Time out. Yes, there is surely a lot of information out there. There are infinite things to learn. It’s going to take some time, but you can do it, I keep telling myself. First I tried PHPUnit, then Selenium, then Behat, &c. Did I get to my destination in the 15 minutes I wanted to? No, I did not. Am I learning? Yes, I am. When did I find the time to do this? I made time. I knew I had to work 8 hours today and for 1 of those 8, I changed gears and learned something to further my long term goals.

PHPUnit is for testing small blocks of code. Sure, I can write a test to make sure my function works, but how do I know it works in the larger context of the application? How do I know that when I click on a button, it loads the Ajax request, and then ultimately gets to that small block of code that already has a test. Well, then I would need to use something with a larger test target. In fact, I would love to write tests without using a programming language at all. Selenium IDE is built exactly for this reason. Now I have to learn two technologies just to make sure that I’m being a good developer and my feature is working. So last week, I set aside some time to learn both of these things, and I know more than I did a week ago. Testing is an incredibly deep topic. It’s going to take me a long time to learn how to do it right, but I had to start somewhere.

This is the only way to self-educate. You have to take the time to make the time. Knowledge is power, and in the technology field, relevant knowledge is evolving at an extremely fast rate. If you don’t take the time to keep your skills sharp, you will ultimately be left behind. Devote a small portion of your day for learning, and know that even a little is enough.

Let’s take a brief look at each of them.

Insufficient server-side Transport Layer Security (Peerjacking)

The easiest way to fathom the mysteries of Transport Layer Security is to remember that your web app sometimes behaves like a modern browser. It can make GET and POST requests like a browser, over HTTPS. It can act as an intermediary, shuffling a user’s personal data to and fro between your server and the server of any third party (including within your own network). It can consume masses of information, store it, and display it back to users, over HTTPS.

Browsers and SSL/TLS have a simple relationship. Browsers implement it correctly, strictly, and monitor their implementation and the CAs they trust very closely. Any failure on their part would be incredibly embarrassing and harmful to their share of the browser market.

Unfortunately, it is far from unusual to find PHP libraries and applications where SSL/TLS protections have been accidentally or even deliberately disabled. Since SSL/TLS is designed to prevent data interception, request manipulation, request replays, and other attacks that are designed to do harm to users, disabling SSL/TLS or being unaware of how to configure it correctly is not an acceptable behavior in PHP. Yet, it remains ludicrously common. As programmers, we also need to be familiar with the prevailing data privacy legislation, privacy ethics, and corporate guidelines which may apply to user data in the jurisdiction or corporate setting we operate within.

Here are two examples of inappropriate HTTPS usage in PHP followed by their correctly configured variants. There is one each for PHP Streams and the cURL extension. The common factor between them is actually very simple — they disable two essential checks in SSL/TLS which we can call peer verification and domain matching. Peer verification guarantees the validity of the SSL certificate offered by the contacted server. Domain matching ensures that the offered SSL certificate is for the host or domain name we connected to. If we fail to verify the peer’s SSL certificate, then we’d never notice if was a self-signed fake, or if it was signed by an untrusted Certificate Authority. It might also have expired. If we fail to perform domain matching, then the attacker could use any valid SSL certificate for any domain or host (whether one they purchased or stole the private key for), so long as the certificate used is capable of passing Peer Verification. So, you need both Peer Verification and Domain Matching enabled in order to be completely secure.

PHP streams (the wrong way)

$url = 'https://api.twitter.com/1/statuses/public_timeline.json';
$result = file_get_contents($url);

PHP streams (the right way)

$url = 'https://api.twitter.com/1/statuses/public_timeline.json';
$contextOptions = array(
    'ssl' => array(
        'verify_peer'   => TRUE,
        'cafile'        => __DIR__ . '/cacert.pem',
        'verify_depth'  => 5,
        'CN_match'      => 'api.twitter.com'
    )
);
$sslContext = stream_context_create($contextOptions);
$result = file_get_contents($url, NULL, $sslContext);

cURL (the wrong way)

$url = 'https://api.twitter.com/1/statuses/public_timeline.json';
$req = curl_init($url);
curl_setopt($req, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($req, CURLOPT_SSL_VERIFYPEER, FALSE); // Disable Peer Verification
curl_setopt($req, CURLOPT_SSL_VERIFYHOST, 0); // Disable Host Matching
/** OR **/
curl_setopt($req, CURLOPT_SSL_VERIFYHOST, TRUE); // TRUE = 1 when it should be set to 2!
$result = curl_exec($req);

cURL (the right way)

$url = 'https://api.twitter.com/1/statuses/public_timeline.json';
$req = curl_init($url);
curl_setopt($req, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($req);

$error = curl_errno($req);
if ($error == CURLE_SSL_PEER_CERTIFICATE || $error == CURLE_SSL_CACERT
|| $error == 77) {
    curl_setopt($req, CURLOPT_CAINFO, __DIR__ . '/cert-bundle.crt');
    $result = curl_exec($req);
}

While cURL on a platform like Ubuntu will be configured with access to a bundle of trusted Certificate Authority certs (e.g., the manually-added cert-bundle or cacert files from above), we should assume that this is probably not the default for many servers (and certainly not when using PHP’s HTTP stream wrapper). Make sure to configure a path to such a file as necessary and ensure libraries you use are doing the same!

As an exercise in awareness, use some of the configuration names above to run a search on libraries you use (or search on GitHub if feeling brave). The same exercise can apply to the following sections, too — these are not only common vulnerabilities, but also easy to locate with grep — their existence revealed by the detection, or lack thereof, of the key configuration constants and function names needed to enable or disable them.

For further reading, see Insufficient Transport Layer Security (HTTPS, TLS and SSL).

XML injection (XMLi)

This class of attacks revolve around PHP’s reliance on the libxml2 extension used by DOM, SimpleXML, and XmlReader. While it is typical to view XML as a simple text format, we should remember that XML can drive an interpreter (via PHP) to perform a number of unexpected actions. One of these is resolving external entity references.

A simple entity like & expands to a lone ampersand when you use DOM to extract the text it’s included in. Similarly, you can also define your own custom entities in XML. A simple entity defined as a very long string which is then repeated an awful lot of times in an XML document can turn any document with a modest filesize into a rapidly expanding RAM eating monster when the custom entity is resolved into ever larger chunks of text, i.e., a Quadratic Blowup Attack. For example:

<?xml version="1.0"?>
<!DOCTYPE results [<!ENTITY long "SOME_SUPER_LONG_STRING">]>
<results>
    <result>Now include &long; lots of times to expand
    the in-memory size of this XML structure</result>
    <result>&long;&long;&long;&long;&long;&long;&long;
    &long;&long;&long;&long;&long;&long;&long;&long;
    &long;&long;&long;&long;&long;&long;&long;&long;
    &long;&long;&long;&long;&long;&long;&long;&long;
    Keep it going...
    &long;&long;&long;&long;&long;&long;&long;...</result>
</results>

This raises a serious risk — we can get XML from third-party services, from users via the browser (e.g., Ajax), and even from our own local filesystem in configuration files both written by us and distributed with third-party libraries and applications. That’s a lot of exposure to XML — a lot of potential targets for an attacker.

Custom entities can also refer to external XML resources, i.e., an external entity. Such external resources can be retrieved by using a PHP stream reference like a file path, a URL, or a PHP filter wrapper URI (to base64 encode non-XML content retrieved). This feature, which is enabled in PHP by default, means that any interpreted XML can potentially access such resources, have them inserted into the textual context of any XML element, and then potentially displayed back to the attacker. This therefore leaves local files readable by PHP and local URLs subject to localhost style access controls subject to Information Disclosure. It may also facilitate making the victim an unwilling participant in a DDoS attack on themselves or a third party.

<?xml version="1.0"?>
<!DOCTYPE results [
    <!ENTITY harmless SYSTEM
    "php://filter/read=convert.base64-encode/resource=/etc/secretfile"
    >
]>
<results>
    <result>I am &harmless; - honest!</result>
</results>

The defense to XML Injection is simply to disable custom entities wherever possible. Luckily, libxml2 has an innate defense against other exponential attacks such as the Billion Laughs attack where deeply nested entities (all referring to each other) can be packed into an even smaller XML file.

libxml_disable_entity_loader(true);

And, optionally, as a sanity check before accepting any XML for processing:

$dom = new DOMDocument;
$dom->loadXML($xml);
foreach ($dom->childNodes as $child) {
    if ($child->nodeType === XML_DOCUMENT_TYPE_NODE) {
        throw new \InvalidArgumentException(
            'Invalid XML: Detected use of illegal DOCTYPE'
        );
    }
}

For further reading, see XML Injection Attacks.

Cross Site Scripting (XSS)

XSS is never too far from the minds of a PHP developer. One area that still needs improvement is weaning PHP developers away from the concept of htmlspecialchars() being the sole defense required. In reality, XSS can target not only HTML, but also JavaScript, URIs, and CSS. Outside of the HTML context, our obsession with htmlspecialchars() is utterly useless and borders on misinformation, since articles and books on the topic frequently omit anything beyond HTML escaping. We need other escaping methods suited to these other contexts. This is referred to as context-based escaping.

Aside from context-based escaping, there is the continual need to be wary of allowing users to submit textual data as HTML. The two usual methods for allowing users to use HTML securely is to employ something like HTML Purifier (the only solution of this type I recommend) or a simpler intermediary syntax such as BBCode or Markdown. Markdown has gained popularity among programmers — it’s the primary format used by GitHub. The problem with intermediary languages is that they are converted into HTML, so you still need to use HTMLPurifier on their output! It’s a subtle misunderstanding, but intermediary languages are for the benefit of the users not the security of your application. Markdown actually includes all of HTML (including script tags) as valid Markdown syntax.

In terms of context-based escaping, your best choice is to use an established solution that can be peer reviewed and rapidly tested. Zend Framework 2 offers the reusable \Zend\Escaper class as a starting point. If you use Symfony 2, the Twig library includes \Zend\Escaper compatible escaping methods for HTML, HTML attributes, JavaScript, and CSS.

Outside of this approach, you should be wary of custom JavaScript escaping. A recent recommendation emerging online is to use the json_encode() function for escaping JavaScript string literals and integers. These recommendations usually neglect to mention that JSON encoding and JavaScript encoding are not the same thing. If using json_encode(), you should be aware that it only supports UTF-8, and you must use many of its allowed flags from PHP 5.3. Avoid it altogether for PHP 5.2 — it doesn’t even escape the ampersand, which can be used to construct HTML entities interpretable in XML-serialised HTML5 or any other HTML where a DOCTYPE has not been defined for the document (i.e., what you expect to be CDATA data may end up being PCDATA in some circumstances).

For further reading, see XSS (Cross Site Scripting) Prevention Cheat Sheet and Escaping RFC for PHP Core.

In business, you’re often able to test ideas quickly and without major costs, which means that an experimental approach to development is common. In software development, you aren’t able to test your ideas as quickly, because you need things to be stable and that costs time and money. This means that software development depends on well-specified needs that business people have a hard time articulating, because they are not used to doing that.

There are many ways to go about improving this. One option is prototyping, and another option is for the business people to acquire a better understanding of how software development works, thereby improving the foundation for good communication.

I think learning Node.js with Express.js can be a great way for business people to gain an understanding of how software development works that lets them improve the relationship in two different ways: by building prototypes that can help them explain what they want, and by being better at speaking the language of developers.

Node.js

Node.js is JavaScript, and the very expressive nature of the language makes the code relatively easy to understand. This lets you learn as you go with a very limited understanding of the language. That might not be the optimal way to go about it, but it builds motivation that keeps you going.

It has a big and active community that builds awesome frameworks and modules, so when obstacles arise, chances are good that you can find a module or a fellow developer that can help you out. This keeps you moving forward.

Node can be used to a lot of other things too, like building a web or chat server, and that encourages you to try and understand how it all works together, and it makes you feel very empowered by just learning one language.

Express.js

Building web apps with Node lets you iterate and change the structure of your app very easily, because it is JavaScript on both the server side and the client side. Express.js is one of the most popular web app frameworks for Node, and it takes care of a lot of things while remaining light and flexible. This fits an experimental and iterative approach very well.

It’s super easy to install, and it’s amazing how quickly you can get an app running. The only steps you need to go through are1:

• Install Express.js: npm install express
• Generate the app: express myapp
• Install dependencies: cd myapp && npm install
• Run the app: node app
• Access the app at: localhost:3000

Express defaults to using Jade, which is a template engine (in other words, a different way to write HTML and insert data into it) for Node. But, if you prefer using HTML and Handlebars.js, you can do that. First, install a few packages:

npm install html
npm install hbs

Next, change some settings. In the app.js file, change app.set('view engine', 'jade') to app.set('view engine', 'html').

Then, configure the app to use hbs by adding this line:

app.engine('html', require('hbs').__express);

Now, rename index.jade to index.html and have it print the title with HTML and Handlebars.js:

<h1>{{title}}</h1>

Relaunch2 the app, go to localhost:3000, and it should display the title that is set in the route.

Development

The framework doesn’t make any assumptions in terms of how you want to structure your code, so you’re free to do what makes sense to you for the current project. That can be a bit confusing at first, so let’s look at an example of how you’d begin the development of an app.

Let’s say that you want to have a page that shows some numbers about your business. To handle that, you’ll need a new route and a view, and you’ll have to tell the app where you want the page to be.

1. Add a numbers.js file to the routes directory, and paste in the code from the index route. Configure it to render the numbers view instead of the index view.
2. Add a numbers.html file to the views directory.
3. Add this line to your app.js file: app.get('/numbers', routes.numbers);

By adding this line of code, the numbers route will be called when /numbers is requested, and the route will then grab the data and render the view.

You can now start adding logic to the route that gets the data you need, but you’ll soon need a model that can take care of pulling in data from other services such as a database. Express doesn’t suggest a place to keep the models, but as long as you only have a few, just put them in the root directory. As you app grows, and you get more routes, views, and models, you can adjust the structure to your liking, and by that time, you’ll know exactly what that is.

Conclusion

In summary, Node.js with Express.js is great for business people, because it’s easy to learn, it can be used to do a lot of different things, and it encourages you to understand how the Web works. It empowers you to build things quickly and iterate until you’re happy, which is a great way to prototype and demonstrate your ideas.

So, if you’re a business person (we all are), and you have ideas that you’d love to see brought to life (we all do), consider Node.js with Express.js. It’s awesome.

Footnotes

1. Assuming that you’re on a Mac and have Node.js and NPM installed. If you don’t already have Node.js installed, head over to nodejs.org. Return to footnote 1 source.↑

2. Nodemon is a great tool that automatically relaunches the app when you save. So, if you don’t like to constantly relaunch it, this will be your friend. Return to footnote 2 source.↑

Pixel density? Retina ready? Say what?

Many new mobile devices are built using special display panels that are capable of displaying more pixels per inch than the standard display you’re probably used to. For Apple products (iPhones, iPads, &c.), the density is twice as many pixels as usual. Android device pixel density can range from 1.3 to 2 times as many. When you view an image on a high pixel density device, it is doubled in width and height to compensate for the smaller pixels, meaning your image only has 1 pixel to fill for every 2 (or so) on the screen. This is why “1x” images look blocky on high pixel density displays. One of the fastest and easiest ways to improve the look of a website on retina displays is to offer a “2x” version of every image, so the pixels fill up natively and without up-sampling.

Here’s an example showing a 1x image (left) and the 2x version (right):

How do I make my web site retina ready?

Compass, a framework for the CSS pre-processor Sass, has built-in image functions that will build 1x and 2x assets for your web site with just a couple lines of code. By specifying the location of a folder full of images, Compass can stitch them together into a single sprite and provide you with all the CSS classes, background positions, and pixel dimensions you need to start using your sprite right away.

The first step is getting all the individual images. To be “retina ready”, you’ll need the regular versions of all your images as well as one twice as big as the original (the 2x version). The method works by displaying the 2x sprite only on retina devices and CSS-resizing it back down to its 1x dimensions. That way your image fits into your design the same way it did originally, but all the extra pixels are there for your device to use to display a sharp picture. A tool like Slicy will make exporting multiple images from Photoshop a snap, and even automatically generate 2x versions of vectors and layer styles for you.

Once you’ve got all your images, just split them into two folders, one for the standard 1x images and a second for the 2x images:

images/
    icons/
        star.png
        heart.png
    icons-2x/
        star.png
        heart.png

Next, you’re ready to start coding. Just a few lines of Sass will get you everything you need:

$icons-layout: smart;
$icons-2x-layout: smart;

$icons-sprite-dimensions: true;

@import "icons/*.png";
@import "icons-2x/*.png";

@include all-icons-sprites;

Let’s break this code down:

1. Lines 1 and 2 tell Compass how to lay out the images in both the 1x (icons) and 2x (icons-2x) sprite. “Smart” mode arranges the images in the least amount of space possible. We want both sprites to use the same layout mode.
2. Line 3 line tells Compass that we are going to want the generated CSS to include the dimensions of each image.
3. Lines 4 and 5 tell Compass to build a sprite using the PNG files in the imported directory.
4. Line 6 tells Compass to generate the CSS for the 1x sprite.

Once you compile your Sass file, you should see two new images appear (your sprite files) and one new CSS file containing all the class names (based off the image file names), background positions, and dimensions of each image in the sprite. Your sprite is 100% ready to go; just apply one of the class names to an element in your markup, and your image will appear. How easy was that‽

Here are the 1x and 2x sprites:

If you’re wondering why we didn’t also generate CSS for the 2x sprite, think back to how we said we would CSS resize it down to fit into its original 1x dimensions: because both sprites are laid out identically, we can let the CSS-downsized 2x sprite rely on the background-positions and dimensions of the 1x sprite. The last step is to simply add CSS to swap out the file path and CSS downsize the sprite when on a retina display:

@media (min--moz-device-pixel-ratio: 1.3),
       (-o-min-device-pixel-ratio: 2.6/2),
       (-webkit-min-device-pixel-ratio: 1.3),
       (min-device-pixel-ratio: 1.3) {
    .icon {
        background-image: sprite-path($icons-2x-sprites);
        background-size: image-width(sprite-path($icons-sprites)) image-height(sprite-path($icons-sprites));
    }
}

That’s it! The browser will now serve up the 2x sprite in place of the 1x on high pixel density devices, and your web site will look sharper than ever when you view it on your new iPad 4 (or iPhone 5, or Galaxy Note II, or Nexus 10, or…) this holiday season. Happy browsing!

There has always been theater of one sort or another. As long as there’ve been stories to tell, there’ve been people sitting in a group enjoying them. Theater troupes were formed to bring a more professional kind of presentation to crowds in the area, traveling around and sharing their stories with the masses. If you have attended any kind of play in recent times, though, you know that the audience is always in the front, facing the stage and enjoying a one-sided look into the lives of the characters on stage. They laugh, cry, and suspend their disbelief for a while, losing themselves in the story unfolding on stage.

It wasn’t always like this. Sure, you can do some great things when the audience can’t see certain parts of the stage — amazing sets, quick costume changes, and special effects that wow audiences. There’s another kind of theater that wows the crowd in a completely different way, though. The theater in the round format, where the audience surrounds the actors and sees them from all angles, brings a certain openness and engagement to the experience. Audience members are no longer just spectators; they feel involved in the play. The actors have to work harder to bring audience members into their world — no elaborate sets, no hidden tools for special effects, just actors doing what they do best.

The key to the theater in the round format is that the audience can see all sides at once. Nothing is hidden, and the actors have to try even harder to make things work.

On the security stage, we’re the actors, and it’s essential that no matter how we go about implementing the security of our apps, we consider more than just the code.

A “security in the round” view, much like the audience’s perspective at a performance, allows for visibility into all parts of the environment. Depending on the company (and the environment), this can either be as easy as opening a document and checking out a schema, or it can be as difficult as herding cats into getting their knowledge out of their heads and down on paper. Fortunately, most development groups have a pretty good overall understanding of where their app lives, what kinds of systems they connect to, and what technologies they use.

“But, I’m just a developer,” you exclaim! “Other than making sure there are no exploits in my code, what else can I do?” This is a dangerous mindset to get into and can lead to some big issues down the line. Much like the actors in the theater in the round, you can’t ignore the other parts of your environment. You have to take the whole app into account, making sure to view it from the perspective of everyone in the audience. Without a good overall view of the app, you can’t possibly create an effective security plan to keep your users and their data safe (sometimes even from themselves).

A holistic view of your app’s environment lets you not only look at the possible issues that could come up as a result of some of the most common exploits (like cross-site scripting and SQL injection), but also lets you assess other threats like data corruption or code injected from other malicious sources.

Here are a few questions you can use to map out this overview:

1. What kind of storage does my app use? Where is it?
2. Are there any firewall or network restrictions in place?
3. Do I have anything extra installed on my server(s) that I don’t need?
4. Is my deployment method secure? Could bad code get into the live site without me knowing?
5. Is my data protected if my app has an exploit that allows access?

These five are just a start to the long list of questions you might need to answer. (Be sure to check the list twice.) The answers can be very revealing, especially if you’re dealing with a dreaded legacy app.

So, get your head out of the code and take a look around at the entire environment surrounding it. Take out a piece of paper (or something more high tech if you prefer), and diagram how things are set up; figure out what points will connect to your app. This gives you a much better picture of what parts you’ll need to protect and how you can best keep your users safe as they use your service. Keep this diagram close are you progress through the life of the app. Map out the new sections and features similarly, and figure out where their connections are and what risks might be associated with them.

With this in hand you can then create a security plan tailored to your app, helping you and your users have even more silent nights knowing your data is safe.

Humankind has always felt a burning desire to record information and transmit it into the future. What that dot meant, no one really knows. Perhaps it was the first expression of binary?

Perhaps it was just a dot.

7,000 years later, and our ancestors were expressing themselves in the form of horses, panthers, cave bears, mammoths, and much more inside different caves, this time in the south of France. It’s the first example of recorded history, the first transmission into the future.

It took many thousands more years until we discovered writing. The Sumerians started scraping sigils into clay tablets and baking them — an act which preserved them almost forever, and because of it we, know so much about their society and the way it worked.

Now, of course, we have the Web, and the world has been transformed. So much information is available that our biggest problems are sorting, classifying, filtering, and absorbing, and the rate at which information is growing is staggering. In 2010, we broke the zettabyte barrier, and in 2011, nearly 1.8 zettabytes of information was created, stored, and replicated. Numbers aren’t available for 2012 yet, but we can safely assume that it will have been greater, and that 2013 will be greater still.

The scale is phenomenal. If a penny represented every gigabyte of storage that was consumed, you could use nothing but pennies to construct a 1:1 replica of the Empire State Building in New York and still have some change left over for a cup of coffee when you were done with your labors.

As we are creating all this information, the way in which we are realizing we can use it is changing. We’re structuring the unstructured, and turning the unknowable into knowledge. We’re making tools for navigating the data more effectively, and we’re learning that sometimes raw information is knowledge — you just have to know to ask the right questions.

Improvements in the browser technology available is allowing web developers to push the boundaries of what was thought possible. Our interfaces are richer, and the interactivity is far greater than it ever has been before.

It’s not just the browser — far from it. As our processing spills out of our homes and offices and onto our smart phones and smart devices, the Web has ceased to be just about the browser. The Web has become a message bus. A conduit by which information is ferried back and forth, not just from server to client, but from client to client as well. I can share my thoughts and feelings with friends on the other side of the world instantly, and to the extent that sometimes it doesn’t feel as though there is any distance at all.

Each development spurs a rush of new apps and new ways of looking at older data, which creates new data that pushes us forward again. It’s not that our time to market is shrinking, it’s that our time to market has become instantaneous, and our tools and techniques are adapting to cope with that.

However, as the end of the year approaches, I get reflective, and I find myself looking at the Web and worrying about one thing. 40,000 years from now, when our descendants cast their gaze back on the Web, will they understand what we were saying?

Have we preserved the knowledge we’re creating for future generations, or are we just doing the best we can to keep up with the rate that we’re creating it? Can we be sure that we’re doing the equivalent of writing on clay tablets, baked in the sun and made to last forever?

And, once we are sure of that, will our children’s children’s children still be able to make sense of what we have to say, or have we become so myopic that we no longer care about transmissions further into the future than our own lifetimes? Will our apps still serve them as they have served us? Are we the architects of a library, or of Babylon?

I’d certainly like to think we’re creating more than just a dot.

Screen is the solution to this problem. Screen allows you to start terminal sessions that you can disconnect from and resume at any time.

I personally use screen a lot with Node.js web servers, so I can kick off the process, and resume my terminal session to check logs or errors, or restart the process if it died.

This article is an introduction to Screen for the average developer, including a few tips and tricks.

Installing

Screen is a Unix tool, so with Ubuntu, you can use apt-get to install it:

sudo apt-get install screen

It’s also possible it’s pre-installed; just type screen to find out.

Instant win with Screen

When I remote in to a server, I run screen, and it starts a new session (and a window, but more about these later). There’s a blurb about what Screen is, I hit return, and the session is ready.

From here, I might start my process. Let’s say I’m doing a large database export.

Now, I need to detach, so I can cleanly close the remote session or do something else while the export is happening. To do this, I type Ctrl+A, followed by D.

This will leave the session running and detach from Screen, so you’re back to your original terminal prompt.

To resume your Screen session:

screen -r

But, what if my connection drops and closes while I’m inside Screen? When I resume my SSH session, the screen could still be attached, so when I type screen -r, it won’t resume. This is simple to get around; you can detach from outside of the Screen session (and, in our case, immediately resume using -r):

screen -dr

Now that you have a basic understanding of Screen, how about using some of its features? Multiple Screen sessions, multiple windows, naming Screen sessions & windows, setting defaults like scrollback, &c.

Screen commands

When you run Screen more than once, you’ll have multiple active Screen sessions. To list the available sessions:

screen -ls

If you have more than one, you’ll need to indicate the session you want to resume:

$ screen -ls
There are screens on:
    17566.ttys001.remys-mba (Detached)
    18778.ttys001.remys-mba (Detached)
    19014.ttys001.remys-mba (Detached)

$ screen -r 18778

This will resume the second session (identified by its process ID). You can also resume the last session using screen -RR. But, resuming using PIDs is ugly, so let’s name the sessions as we create them using the -S argument:

$ screen -S database-dump
[Ctrl+A D]
$ screen -ls
There are screens on:
    17566.ttys001.remys-mba (Detached)
    18778.ttys001.remys-mba (Detached)
    19014.ttys001.remys-mba (Detached)
    18898.database-dump (Detached)

$ screen -r database-dump

Now that you know how to have multiple (named) Screen sessions, let’s look at multiple windows per session.

Windows

Screen supports having multiple windows inside a Screen session, so you can have one screen and multiple windows, each dealing with specific jobs.

To create a new window inside a Screen session, type Ctrl+A, followed by C.

The initial window is 0, the second is 1, and so on. There are a number of ways to switch windows:

Ctrl+A [n]     // where [n] is the window number
Ctrl+A Ctrl+A  // switch to the last used window
Ctrl+A "       // show a list of all the windows
Ctrl+A A       // change the title of this window

There are lots more key bindings you can use to navigate windows, but they’re beyond the scope of this article.

Tricks

Even if you use Screen in its simplest form (as I usually do), I wanted to share a few tricks I’ve discovered.

Multi-user sessions

To connect two (or more) users to an existing screen session:

screen -rx

Instead of detaching any users attached to the session (which is what would happen if we just used -r), this lets me join the session, and anything I type in my window is echoed to any other users connected to the screen.

Maybe this could be useful for remote support, or perhaps training. I’ve not used this in the wild yet, but it’s certainly a fun idea!

Keeping history

One problem with Screen is the scrollback. If you try to scroll up inside a session, instead of doing what you’d expect, the session itself just scrolls away. To scroll back inside a session, you have to use `Ctrl+a [. Now, you can use your keyboard to navigate up and down (this is actually in copy mode). You can hit escape to return back to the prompt.

What might be more useful than a scrollback is a log of the Screen session, which can be toggled on using Ctrl+A H. This can be enabled by default with a .screenrc.

Screen Defaults

Finally, I’ll leave you with my own .screenrc. Create this file in your home directory to set your own defaults for Screen:

# Always show a status line in the window footer.
hardstatus on
hardstatus alwayslastline
hardstatus string "%{.bW}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H %{..Y} %m/%d %C%a "
            
# Auto-detach session on hangup instead of terminating Screen.
autodetach on 
    
# Turn off the splash intro.
startup_message off

# Enable log on all windows.
deflog on

Now that I’ve got that out of the way, I’ll give you the common definition, which is that you ship changes on a commit-by-commit basis, or alternatively, as frequently as you desire. What most people think of is push-per-change, which is often not actually the case.

In practice, most CD setups are automated, not automatic. What this means is that although every step from code commmit to ready-to-push may be automated, at the end of the day there’s some kind of Big Red Button or one-line script or IRC bot that you activate in order to deploy.

Why would I want to do CD?

Hipster cred. Seriously, though, the goal is to minimize the amount of work needed between finishing an idea and making it real. That’s a worthy goal.

A side benefit is that building the tools, processes, and skills needed for CD will make your working life better.

Good practices for CD

I put up a list of practices in a talk and called them “requirements.” Laura Beth Denker called me on it. She said these things are not true requirements for doing CD, because people do it with none of these requirements satisfied. She’s right. Let me suggest instead that it would be a good idea to do some, if not all, of these things before adopting CD. Most of these practices are a good idea in any case.

The practices in the following sections are not all-encompassing, and I’m sure there are great practices of which I am totally unaware.

Continuous integration

This is the practice of having developer changes land on a shared version control system as they are completed. (The original version, in Extreme Programming, says “several times a day” but velocity is irrelevant in my view.) CI presumes you are using a VCS. You are, right? Since GitHub made version control a commodity, there is no longer any excuse to not have a source repopository.

Build-on-commit

“But, I’m writing code in PHP/Python/Ruby, and I don’t need to build anything. That’s for compiled languages.” It may be true that, in your environment, the build part of the process is a NOOP, however, it may involve steps like:

• Minification of CSS and JS
• Running a localization script to combine templates with string files
• Converting your dynamic code to some kind of intermediate format

The net result of a build process like this is to spit out at the other end a build artifact which will be deployed.

Test-on-commit

This is where you take that build and run your tests on it. Tests. Yep, I said it. I sincerely hope this part of your process isn’t a NOOP. There are great tools for this now. My team uses Jenkins, and lots of people like Travis CI.

Recently, we’ve been automatically running tests on pull requests using Leeroy. Travis CI does this, too. It’s so nice to avoid a code review, because I can see that your pull request doesn’t pass tests!

Good test coverage

Having 100% unit test coverage is nice, but it can make developers over-confident. Full test coverage doesn’t cover integration testing, or performance testing, and in an ideal world, your automation would do both of those things. The most important thing is to know what your tests cover and have a realistic assessment of what the holes are, and the level of risk.

A staging environment that reflects production

The developer’s own machine is a horrible staging environment. Get a real one. Make it as close to production as possible. The biggest issue I see in staging environments is what I call the “single box of fail.” In staging, you have one machine. In prod you have, say, ten web heads, one DB master, three DB slaves, some Redis or Memcache machines, and a queue, all running on different servers. In this environment, you will have failures.

I have been bitten by this a hundred times, and it still bites me. It bit me last Wednesday, in fact, when it turned out our PostgreSQL puppet manifests were ever-so-slightly different in staging and production. Don’t be me. Make your own novel mistakes.

Managed configuration

Speaking of Puppet, you should use something to manage configuration of your machines. Puppet, Chef, CFEngine — pick something.

Single-button deployment

You should have scripted, single-button deployment to all of the machines in your infrastructure. (You should also be able to deploy to individuals or groups.) Again, there are a ton of tools that will do this for you. Pick one, or roll your own.

Failure plan

You need a plan for what you’re going to do if things go horribly wrong. In practice, there are two basic approaches:

Rollback plan

This can be as simple as being able to deploy a previous build. Remember, you need reverse data migrations.

Feature flagging

Build your code so that you can turn features off and on, or turn them on for sub-groups of users (beta testers, or employees, or a random 10% of the user population). This requires more forethought, but it’s like writing tests — an investment.

Track, trend, monitor

Measurement is key to process improvement. From page load times, to conversions, to TCP sockets, every measurement gives you insight into something. And, remember, if you don’t have good visualization tools for your information, it’s only data.

Excellent source code management

Know thy tools, craftsman. At least one person on your team should know your SCM tool inside and out.

High levels of trust

If you have separate development and operations teams, or even if you have an awesome devops team, you won’t get far without trust. It doesn’t matter when things are going well; it’s when everything turns to crap and you’re firefighting when the finger pointing starts. So, start from the premise that people are trustworthy. Trust them to do their jobs. (And, if they can’t, coach them or sack them. It’s that simple.) It doesn’t matter what your job title is. You’re all on the same team. You all want the same thing.

Realistic risk assessment and tolerance

Know what your risks are, and have an appropriate level of tolerance. If you are trying to build something with five nines of uptime, you will want more assurance before deployment, and it may be that CD is not for you. Know what parts of your site are acceptable to break, and what parts need to be up, and act accordingly. Availability requirements are not the same across all components of a system. Managing this is key to resilient systems.

Excellent code review

Code review has its place; on many projects I work on, every line of code is reviewed. On things that are less mission critical, or have very experienced teams, this is less the case. Code review is useful as both developmental (“I know a better way to do that”) and copy editing (“You made a critical typo here”).

Finally

If there are only two things you remember after you read this article, I would like them to be these:

1. You should build the capability for continuous deployment, even if you never intend to do continuous deployment.
2. As with everything else in life, the only true way to get good at deployment is to deploy a lot.

I’d be interested to hear about your thoughts and practices for CD — get in touch!

While we’ve mostly done away with Flash, the trade-off is that static web sites are still the majority. Pages on the Web are full of text, images, and links. Sometimes a video, too, although it’s almost always stuck in a rectangular box, slapped in the middle of a page, waiting for a click of the play button to come alive.

Shake the humbugs

Things don’t have to be this way. The Web is naturally interactive, with users clicking around from page to page and site to site, discovering all sorts of things along the way. The Web is flexible; it flows from one device to another, regardless of screen sizes or browser functionality. Pages and elements have states, positions, and layout that can be manipulated through code and through input from users.

With all of its potential, why does the Web feel so stiff? We should be taking advantage of its interactive nature and flexibility. Design details shouldn’t end in a Photoshop mockup; they should continue on to the code. Interaction design for the Web extends beyond hover states and animated GIFs.

CSS is more powerful than ever. Designers and developers alike should take advantage of these advancements in code and browser support. Good design and good code only take the Web so far. It’s the layer between, the one that makes your design come alive, that can take the us to the next level.

Add some tinsel to the (DOM) tree

One of the least attractive things you can offer on your web site is a form. If I can paraphrase Luke Wroblewski for a moment, “No one wants to use your form; they just want what’s on the other side of it.” That said, forms are a necessary evil in some cases, so why not make them more interactive and less painful?

Have a look at 45royale’s project planner. Not only does it clearly communicate the number of steps involved and provide helpful tooltips for each input, but those same tooltips are highlighted along with the text input as you move through the form. Placeholder text appears in each field to improve the quality of the input, and each step appears without a new page load, thanks to JavaScript.

Not all enhancements need to be so functional, though. Canvas’s client list, for example, features your typical grid of company logos that the agency has worked with. But, move the slider from “Day” to “Night”, and watch as the heading, icons, and logos themselves light up in a bright neon as the background fades to black.

And, it’s not just about buttons, inputs, and links. Other elements of the page can be given just a little touch of interactivity. Scroll down Mosaic’s single-page site and watch as the calendar rocks into place. Transparent clouds move past the blimp on Helium’s site. A looping video element fills the background of Sevenly’s team page.

Keep Delivering Gifts

Remember, the Web isn’t static; your design shouldn’t be, either. And, as browser support for CSS improves, so too does your ability to enhance the user experience of the Web. Use design and markup to deliver your best content to users, then delight them with the joy of interaction.

The definition of confidence.

But as I thought about it more, I realized how important confidence is for new and veteran developers, alike. The following are the values and practices that I use to keep up my confidence. I hope that they can be encouraging to anyone just starting out, or considering starting a journey into the Web, as well as good reminders for those up to their elbows in code.

You will never know everything about everything

The world of the Web is huge, and it’s changing all the time. If you start out wanting to learn it all right away, you will feel overwhelmed pretty quickly.

I wrote my first “Hello, world!” in early 2009. When I was first learning the fundamentals and came across something new, I immediately wanted to know everything there was to know about it, and would get frustrated that I didn’t. Allowing myself access to this kind of impatience and frustration hindered my ability to think clearly and logically, and made it hard to let things sink in.

Even though it’s been almost four years since the beginning, I still consider myself relatively new to development. I’ve learned a lot of things along the way, and it’s fun to look back even just a year or two and realize what I didn’t know. Yet, I still experience many of the frustrations that come along with being a new developer.

One of the things I’ve learned is that there is a huge difference between learning how to write code and learning how to implement code. When it comes to implementing code, there is so much to learn, from security to debugging to caching and optimization, and they all involve understanding how to architect your code and applications. They go way past the fundamentals, and even now I feel like I’m only just beginning to really feel confident in these areas.

Secondly, I’m still getting the hang of how to pace myself as I learn. It’s easy to fall into my old way of thinking and to expect myself to know everything right away. The best approach is not to take on too many challenging things at once, but also not to shy away from things that are just out of reach. It takes a lot of dedication, time and practice to become fluent in all of the different areas of the Web. Don’t expect to gain all this knowledge in the first few years.

Stay out of your comfort zone, and learn to love the pain

Don’t overwhelm yourself, but it’s critical to also stay out of your comfort zone. It’s really easy to keep doing what you know, because it makes you feel great about your ability. While I agree that this is a good confidence-building approach, it prevents you from learning new things. “Stay out of your comfort zone” implies being in a state of discomfort, and sometimes it’s true! This pain is a path toward gaining new knowledge and confidence.

Practice, practice, practice

The best way to get better at writing code is to write it — lots of it! Find examples online, ask your colleagues if they need help with their projects, even try to recreate your favorite web sites. Take a look at some of your old code, because you’ll most likely see errors and will be able to refactor it based on new knowledge. Practice is progress, and every bit of progress counts.

Some days are good, and some days are bad

Every developer has a story about an embarrassingly awful and impactful error they’ve made.

Mine was a few years ago, when I was rushing to implement a new feature, and I forgot a really important bit of code to check whether or not a save was successful, which essentially broke validation. Oopsies.

Me, after I realized what I had done.

Luckily, my teammates and I noticed right away, and it only created 80 botched records. But it could have been much, much worse. The it-was-me pill was a hard one to swallow — I had to admit to 80 users, never mind my managers, that I had made a mistake. The only thing I could do was own up to it, fix it, and move on. As a developer at any skill level, you are going to make mistakes, and sometimes you might make really bad ones. This is OK.

It’s easy to remember the bad days and easy to forget the good ones. When you feel a sense of accomplishment, take a moment to pat yourself on the back. You’ve earned it. These feelings are the ones that will keep you going when you make those mistakes.

Know when to ask for help

My mentor used to tell me to “stop staring at the code waiting for Jesus.” If I ran across a problem I couldn’t figure out, I would stare at the code, thinking that if I were a developer worth my salt, I’d be able to discover the solution. It took me a long time to accept that sometimes I needed help, and that admitting that was not admitting defeat, but instead was the smartest, most efficient way to solve the problem.

When it finally was time to ask for help, I always felt a bit embarrassed to show people my code. What if I’d made a silly mistake? What if I’m doing it all wrong? I had a hard time accepting the fact that I was new, and other people were not. This self-consciousness inhibits progress. Time spent worrying about someone judging you or your code leads to missed opportunities to learn.

Teach other people

You don’t need to be an expert on the Web to teach someone else about it. Teaching someone what you know, no matter how little it is, will reinforce those skills in your own mind. Your student, mentee, whoever it may be, will most likely ask you lots of questions, and you’ll learn new things in the process of answering them.

Science cat wants to teach you things.

Heather Payne of Ladies Learning Code says that sharing your knowledge, “means a lot to the people you help.” I can say this is true in my own experience as a new developer and as an instructor. The inspiration that comes from teaching goes both ways. Students feel confident and excited, and teachers feel great about helping others to feel that excitement. This pay-it-forward approach is also one of the main reasons why the tech industry is so healthy and strong. If you learned through mentors, user groups, and friends who spent the time to teach you, you should give back!

At the end of the day…

Remember that learning how to be a web developer, above all else, should be fun. Don’t do it for the money, don’t do it so you can impress people with your mad programming skills, and don’t do it because you can’t think of anything else to do. Do it because you love it and it makes you happy. I love being a web developer because I know that the possibilities are endless. I can make any web site, application, or game I want, and there are countless ways to create them. Who wouldn’t love that?

What follows are some principles I try to keep in mind.

Learn languages, not frameworks

I like PHP, Python, and JavaScript, and I like making things in PHP, Python, and JavaScript. I’m not a Symfony developer, or a Django developer, or a jQuery developer.

I think this is an important distinction. It’s entirely possible to be a jQuery developer, but not a JavaScript developer. It’s possible to be a Django developer, but not a Python developer. Those are all certainly valuable and useful tools, but if I only know how to use one framework, my options for using the right tool for the job get pretty limited, and in my experience, large, full-stack frameworks are often not the right tool, particularly if flexibilty and performance are major concerns.

My experience has been that I become a better, more versatile developer when I focus on learning a language first. Diving head-first into a full-stack, complex framework when I’m starting out has allowed me to build finished products more quickly, but it also becomes a detriment when I need solutions outside of the framework’s scope. I often end up with a “plug and pray” approach to development, where I find a library or plugin that sounds like it will meet my needs, cross my fingers, and shove it in. That might get my app launched faster, but it makes stuff a lot harder down the road.

Additionally, learning a full-stack framework can be as complex as learning a new programming language. They often have complex architectures and nomenclatures — parts that don’t carry over to other frameworks and tools. I’d rather spend my time learning more about the language itself, knowing the skills that I learn will apply to anything I build in the language, no matter which libraries I use.

Build small things

Small units of code are good. The smaller it is, the easier it is to understand. It’s harder to screw it up — and I screw stuff up a lot, so limiting that is really important.

So, I strive to build small modules of code with a single purpose, or at most a few closely-related purposes. They should be self-contained pieces that solve individual problems. These pieces then work together to solve larger, more complex problems.

With simpler, modular code, fixing bugs is easier, because I can look at an individual piece and see clearly what it’s doing. And, if the modules are self-contained, testing my code is much easier.

Less code is better than more

To paraphrase Biggie Smalls, “more code, more problems.”

I want to manage less code. Larger codebases get harder to manage. Searching across the codebase takes longer. Navigating through complex file structures takes longer. Tracing execution through dozens of files is hard. Keeping it all straight in my brain gets challenging.

Bigger libraries and longer code seem to overflow my brain buffer. I have trouble keeping track of code flow when the source gets too long, or when execution is jumping between several source files, and visual noise really affects me, too. This is why I love syntax coloring so much, and consistent whitespace really helps. I use the code folding feature in my editor a lot, just to hide things, so I can focus better on the task at hand.

I also want to support less code. I’m responsible for all of the code that my app uses, not just the stuff I wrote — every single line of it. This means that bugs and security holes are my responsibility. How often do we see a WordPress or Drupal plugin get abandoned after a year or so? Am I sure I want to use a piece of code I don’t really understand, when a year down the road I may have to fix an exploit in it?

This is not to say that I would never use code that I didn’t write — I’d be hard-pressed to get much done that way, and frankly there are better programmers out there than me — but every line of code in my app matters, so I need to justify each one.

Create and use simple, readable code

I want code that is easy to understand. Understanding things quickly means getting stuff done faster. My time to productivity is shortened. My time to fix bugs is shortened.

I also want code that is easy to verify. My code should be testable, whether or not I get around to actually writing those tests, and I’ve consistently found that simpler, more modular code is more testable.

Readability should be a feature. Code should be simple and terse, but clear. Value clarity over cleverness. When I’m writing code, I try to consider how quickly another developer will be able to understand it at first glance. Alternately, will I be able to understand it when I come back to it in two months? The less time I waste trying to figure out how things work, the more there will be for getting things done.

I’d be lying if I said I always follow these principles. Sometimes, I just get lazy. Sometimes, time constraints mean I write hacky, complex code as fast as possible to get something working, or I pull in a library without reviewing it carefully, and hope it works. In the short term, writing simple, clear code is harder — it requires more discipline and frequent reassessment of technique. Particularly with time-sensitive projects, it’s hard to ever feel like I ever am entirely happy with the results.

But, when I make the time and put the effort in, it always pays off down the road — not just for myself, but for the other members of my team and people who use the code I’ve written, and that’s a really good feeling.

Email clients disable embedded images to protect you from spam. Spammers will use an image as a beacon; when downloaded, it signals that you have an active and valid address. Spam protection is a good thing, but automatically-disabled images make designing a compelling marketing email tricky. Fortunately, there are a few things you can do.

Style the image’s alt text

Many email designers are hip to adding an alt attribute to all important images, ensuring that — at the very least — text will show up when the image does not. Instead of just a straight description of the image, a savvy email designer uses the alt attribute to place an offer or call-to-action. An even savvier email designer takes it one step further and styles the alt text.

One very simple way we do this at Etsy is with our logo. Instead of showing a box with alt text that just says “Etsy”, we style the alt text to look as much like our wordmark as possible. The result is that when viewing the email with images off, you see something very close to what our logo looks like.

The code looks like this:

<img src="etsy_logo.png" width="47" height="24" alt="Etsy"
  style="color:#d15600; font-family:georgia,times,serif;
    font-weight:normal;text-decoration:none; border:0" />

Not every logo lends itself so easily to this technique, but there are plenty of other ways to use it. For example, if your email makes use of image-based buttons, you can style the alt text to come up with a pretty convincingly clickable fallback.

Of course, like any feature of HTML email, alt text doesn’t display consistently across all clients. Campaign Monitor has a handy breakdown that shows how alt text displays across different clients.

Make your image into pixel art

Take that would-be empty image block one step further and make a lo-fi fallback version of your image using tables and background colors. Pizza Express uses this technique in a variety of ways.

Hand-coding a custom pixel art table for each email is a time-consuming task, but there are a few apps on the scene that are making this easier.

Style Campaign created a free app that pixelates your image while keeping email-friendly compression in mind. Here’s how it works:

“The application [...] can be run with multiple parameters to allow HTML output in a variety of formats, from standard HTML table layout to full CSS. The application utilizes a simple run length style compression system, using table cell colspan to create rows of contiguous color. This saves significantly on file size and client processing time. A background color can also be passed as a parameter, so that cells containing the specified color do not have to have their color defined in the table cell, further saving file size.”

They readily admit that compression for large images continues to be an issue, and they recommend that the tool be used for smaller items like logos and icons.

Email on Acid offers a very consumer-friendly app called Mozify that does the same thing. Among other features, Mozify lets you control how pixelated your image will be be, letting you experiment with larger feature images while saving on file size. Watch the Mozify intro video to see it in action.

Designing for the inbox is rife with challenges. There’s a dizzying variety of clients, devices, and browsers to contend with, along with more emails being sent than ever before. Inboxes are a special mix of a personal and practical space. Once you get them to open the email, it’s worth taking the time to greet them with more than a blank screen.

It’s hard for those of us in this field to consider our daily chores of pixels, markup, and databases to be anything more than a job. A great job, but a job nonetheless. We are multi-lingual experts skillfully constructing experiences that are responsibly responsive. We bring skeuomorphic interfaces to life with little more than photo editing software that we then plug into a machine that responds to our touches and gestures. I ask it a question, it tells me an answer. In some places throughout the world, it would probably seem like witchcraft. In fact, more often than not, we find ourselves bemoaning one feature or another that doesn’t live up to our astronomical standards. I think it was Louis C.K. who said “It’s going to space! Can you give it a second to get back from space‽”

My friend Joshua Blankenship put it this way: we are wizards. I agree. We have the ability to do things that most folks find too complex to bother with. We create the worlds they engage in on their computers and their phones. To us it’s no big deal, but to the users, it’s straight-up magic.

Building for the Web is the future, of this I am convinced. Software and hardware do more with greater efficiency than ever before. What we do with smart phones today once took a semi-truck-sized pile of hardware. The future is here and we are the architects.

So with that said, I have to ask — why do we spend so much of our time, effort and money on building useless junk?

Don’t get me wrong, your disruptive-social-mashup-startup sounds awesome but is it solving any problems? Yes, yes, it may be solving the age old dilemma of how to take a 30 second video of me eating breakfast while staring at my shoes and share it with “friends” (friends I wouldn’t recognize if we crossed paths on the street), sans audio. But what real problems are you solving? Researchers of chronic diseases who could benefit from a worldwide sample of patients, educators unable to accommodate the various learning styles of children in their overloaded classrooms, or unemployed people in desperate need of training for a chance at an interview — what are we doing to help them solve their problems?

Some of us are ideators, some of us are doers, others are a bit of both and, while the solution may not come from you, the ability to execute a viable solution can. Write an algorithm to detect patterns in patients, customize an online course to teach lessons based on a student’s previous answers, or build a simple tool to allow others to share their knowledge with those willing to learn. We do this stuff every day — maybe not in the context of solving real problems, but we do it nevertheless.

You don’t have to work at a non-profit to solve these problems. You don’t even have to quit your job. Our tools of creation have improved so much that we’re able to build in hours what used to take days. Frameworks, auto-magically-scaling-distributed servers, one click this and that… it’s getting faster and easier to do what we do. A few hours in the evenings or on the weekends is all it may take to breathe life into a new idea.

When you dream up your next big idea, or you find yourself looking for a side project to take on, consider your magical powers and wield them for something good, something that makes a difference and solves real problems.

However, the movement has been going too far — expanding to cover every kind of design contest and groups where design and art are created for fun, not just profit. Anti-spec is detrimental to community and social good projects, and it hurts more designers and artists than it saves by going too far.

Anti-spec is hurting pro bono design

At Brooklyn Beta 2011, Todd Park of the US Department of Health and Human Services spoke passionately about the need for designers to work for the public good, to spend their free time building and experimenting on current public issues. He called for designers to take the massive amount of health data released by the government and use it to build public solutions, and he specifically mentioned that, yes, the government could pay one individual or studio a massive amount of money to find one solution, but that the most amount of innovation and creativity comes from the community.

Recently, The Designer Fund and the White House presented the Health Design Challenge dedicated to redesigning the electronic medical record. On the TechCrunch coverage, The Designer Fund co-director says, “There are so many meaningful problems in the world. Healthcare, clean energy, environmental issues, city design. We feel like we’d be able to make a much bigger dent in these problems if designers went at them.” Of course, there were anti-spec responses such as this:

The Designer Fund and the White House could pay a consulting firm to create a solution for the electronic medical record, but like Todd Park mentioned, the greatest amount of innovation and creatively comes from a massive amount of designers working together — more innovative, cost-effective, and interesting solutions will always come from a community project such as this rather than one paid consulting firm.

Anti-spec is hurting open source, design collaboration, and education

Being a designer jumping into development, I would have been completely lost if it wasn’t for sites like Stack Overflow, where people can post problems they’re having with development and get answers and help for free. The free advice and help from Stack Overflow made it easy to jump into development. So many people are looking to jump into design, but huge communities for design help like Stack Overflow are lacking — Stack Overflow for design isn’t big enough and Dribbble exists more for sharing small details. Large communities don’t exist due to the stigma against asking for design help for free.

For developers, open source development has lead to innovation on the Internet and online products. I was able to build my startup using a programming language called Python and a framework called Django, both of which are available online for free, even though they represent thousands of hours of work. The developers worked for free, in their spare time, to build something to help others. It’s worth noting that it would be odd to have anything but free programming frameworks and plugins — developers in the open source community embrace working for free in order to build free resources for the benefit of others.

What if designers were more open with their free time? What if we embraced opening up to others, embracing community? A huge issue in the open source community is the lack of design help — poor UX and UI design on open source projects which hinders adoption and understanding by non-developers. Designers should work with developers to further promote and aid adoption of great open source projects, which, in turn, helps nearly everything we use on the Internet today.

Anti-spec is hurting community

Moleskine, the popular retailer of artist notebooks and sketch pads, decided to hold a design contest for their online blog, the Moleskinerie. The Moleskinerie is dedicated to sharing art and stories around the Moleskine brand, so it probably seemed natural to use their current community to help design the logo for the Moleskinerie. The anti-spec work comments poured in:

“I am shocked and dismayed that Moleskine is running an unethical contest like this. You are expecting thousands of artists to work for free.”

“Like a lot of professional creatives, I am very disappointed by the fact that Moleskine actually helps devalue our profession and potentially tries to get rid of a lucrative part of its user demographic, instead of making a fist against crowd-sourcing or any initiative involving speculative work for that matter. I'm quite sure that I'm not the only one who won't buy Moleskine products anymore.”

Moleskine had already built a community through design and was just further developing that community by having the headline piece of the community (the logo) built by the community. Moleskine wasn’t redesigning their brand or any piece of their company that is distinctly for-profit. We shouldn’t discourage designers using their talents for fun to help and build upon something they love.

The anti-spec movement needs to embrace pro bono pursuits

The anti-spec movement can be productive, since bad spec work still exists — there are still employers making potential employees work for free with the possibility of a job (bad), and there are still companies running spec contests for logos and branding (very bad — I’m looking at you, 99Designs and Crowdspring).

However, we should never discourage designers and artists from using their art and design skills for fun and public good. These pursuits don’t deny designers jobs, nor do they lower our wages. As designers, we should contribute to open source projects, donate our skills to important public projects such as the Health Design Challenge, and embrace art for fun — not just profit.

PhantomJS is a rather new kid on the block. It’s a headless WebKit-based browser. It’s statically built against QT, so it works across platforms (Windows, Linux, and Mac OS X downloads available), and it has minimal dependencies. The great part about it being headless is that you don’t need X installed to use it, so it’s suitable to install on your servers. In fact, I’ve already installed it on 80 of our servers powering Where’s It Fast.

Once you’ve got everything installed, it’s time to give the examples a whirl. Load speed is easy and interesting: ./bin/phantomjs ./examples/loadspeed.js http://webadvent.org/ yields:

SyntaxError: Parse error

Page title is Web Advent 2012
Loading time 610 msec

It takes 610 msec for PhantomJS to load the page inside its headless browser. Note that this isn’t a basic request with cURL to simply grab the index document; it’s “rendering” the HTML, grabbing the style sheets, images, &c. The “Parse Error” we’re seeing is the output of the JavaScript engine erroring out as it grabs a misconfigured resource. Whether you prefer the JavaScript or CoffeeScript source, the code involved is almost disappointingly trivial. Record the current time, attempt to open the page, on success subtract then from now and print. If there’s an error, print the appropriate message. That so few lines of code were required to return such a useful monitoring metric really pushed me to look harder into PhantomJS.

For a more in-depth look at the resources it’s grabbing, the netsniff.js example is perfect. It similarly retrieves the requested page, but it also returns detailed information on each resource that it requests in its log from requesting webadvent.org.

Here, we get some basic app info, followed by the details on the page, including the pageTimings variable — it took only 393ms for the browser to hit onLoad (the Wi-Fi must have improved). What follows is an entry for each resource requested, including both request and response, as well as various timing details. This far more granular view of the page allows us to not only ensure that it loads quickly, but also determine which resources took the longest. Using startedDateTime as well as the timing information, it’s possible to replicate the detailed page load waterfall you’d receive from Firebug or similar tools.

I find these figures tremendously interesting and exciting. An incredibly large pool of research has shown that the speed at which your page loads and becomes usable is tremendously important. Faster pages equals more sales. Being able to automate monitoring of these full page statistics provides an additional layer of assurance that pages are working the way their provider expects. These are often values that are easily hidden if it’s your static JavaScript server slowing things down, rather than your main dynamic server, or your database.

But wait, there’s more!

PhantomJS is also capable of rasterizing your page, and spitting out a PDF for your viewing pleasure. This site uses a second (and appropriate) CSS file for printing, so I’ll use my own site for comparison here. The PDF that PhantomJS generates (with /bin/phantomjs ./examples/rasterize.js http://wondernetwork.com/ wondernetwork.pdf letter) can be compared directly with wondernetwork.com. Apart from needing to do some work to better support narrow browsers (my bad) I’m incredibly impressed with how well that worked. Generating thumbnails of pages was cutting-edge stuff very recently (and let’s be honest, generating PDFs programmatically is almost always a pain).

Further reading

Beyond these examples (and the clear, though unspoken, implication that you can edit the rather clear code to do more), there’s also been a few things built on top of PhantomJS to further utilize this headless web browser. In particular, I’d like to call your attention to the friendly CasperJS, which eases scripting and testing with PhantomJS. Their site documents a simple way to iterate over some Google search results, possibly to ensure your continued ranking, but I’m more interested in the testing aspect. I’ve explored using Node.js for testing my JavaScript in the past, but was never really happy with how it required me to change how I was handling things (since I wasn’t deploying with Node.js). With CasperJS, I’m able to test my pages the way I’ve written them, without any modifications or other chicanery.

All in all, I think PhantomJS is a fantastic tool to add to your toolkit. Monitoring, imaging, and testing, all in one package.

To SSH to a remote server, simply take a terminal program (Windows users should grab PuTTY) and type:

ssh username@server

You’ll be prompted for your password (or passphrase for your key; more on those in a tick), and then — presto! — you’re in. If you don’t supply the username, you’ll be prompted for that first, and the server can take either a domain name like lornajane.net or an IP address.

SSH keys

While it’s fine to use a username and password for SSH access, it’s quicker and more secure to access using an SSH key. SSH keys are made of two parts: a public key and a private key. You generate the keys with the command ssh-keygen (Windows users, download PuTTYGen).

You will be prompted to answer some questions when creating the key, and, in general, the defaults are fine. One of the questions will ask you to create a passphrase. The passphrase is a password that must be used with the key when accessing a server. In general, it is recommended to have passphrases, but you might choose to omit them if, for example, this key will be used by an unsupervised automated user account.

There’s an excellent GitHub resource on generating SSH keys that gives you more information, and it also provides a clue to some of the other things we can do with SSH!

Sharing SSH keys

Once you have your keys, you will need to send the public half (the filename will end with .pub) to anyone who needs to give you access to their server or other app. It is safe to email public keys and to share them, but keep your private keys safe, protected, and backed up. These are the real keys to the kingdom!

GitHub uses SSH keys to grant access to your GitHub account. You paste the contents of your public key into their site (look under “SSH Keys” on your account page), and then you can access your repositories using the key.

Many keys

Some people only have one key, but I find it more helpful to have many keys — perhaps one for work use, and one for personal use, or one created for a specific project or system. This approach means that I can revoke an individual key if I need to, without needing to change all of my keys.

To SSH with a specific key is very simple. You can supply the information on which key to use with the -i switch to the ssh command. For example, I have a separate SSH key to use with GitHub:

ssh -i ~/.ssh./github_rsa git@github.com

Using multiple keys in this way is great for accessing servers, but many of the programs which rely on SSH, such as scp and git don’t have an option to specify which key to use. For this situation (and many others), you can use the SSH config file.

SSH configuration file

The config file for SSH is the source of many very useful little tricks! (On Unix systems, it lives in ~/.ssh/config.) For example, you might set up an alias so that you can refer to a server by a shorter name:

Host lyra
  User=lorna
  Hostname=lyra.lornajane.net

Now, I can simply SSH to the alias, and it will pick up the hostname and username accordingly:

$ ssh lyra
lorna@lyra.lornajane.net's password:

Using the config file, you can also tell SSH which key to use when accessing a different server. Here’s the entry I use for accessing GitHub:

Host GitHub
  Hostname=github.com
  IdentityFile=~/.ssh/github_rsa

This does mean that I edit all the URLs I use to be GitHub, where normally there would be github.com. It reminds me that there’s some magic somewhere!

SSH tunnels

Earlier, I mentioned that it’s possible to use SSH connections as a transport for other protocols. I use this mostly to get around the issue that some clients’ tools are locked down to my home static IP address. There’s a server in the house that does a bunch of handy things, but also accepts SSH connections, so I can tunnel through it to make my traffic look like it came from there.

To achieve this, I use the -L switch to bind a specific port on my local machine (such as 8080) to a specfic port on the server that I actually want to access. This will only work if the connection is made through my static IP address, so the house becomes the target of the SSH command. It looks something like this:

ssh -L 8080:client-tool.example.com:80 home-server.lornajane.net

I am prompted for my SSH credentials for home-server.lornajane.net, and the connection is created. Anything I send to 8080 on my local machine will be forwarded over an SSH tunnel via the house to port 80 at client-tool.example.com. So, I open http://localhost:8080/, and by SSH magic, I have access to the client tool that’s only available from the whitelisted IP address!

This is an invaluable trick and one that I use when I’m on the road most of all. It can be used to send all kinds of traffic, not just HTTP. I’ve used it for rsync, telnet, other file transfers, and who knows what else. I’ve also used it to route my web traffic through a proxy as a way of working around restrictive firewalls.

The power of SSH

SSH is such a key skill in working with remote commands, but it’s one of those awkward things that isn’t usually taught; you don’t learn SSH when you perfect your technical craft, yet you need it to be able to share what you create. Hopefully this post has given you a useful snippet or two to make sharing a little bit easier.

But, because the browsers are new, we can make things a lot better.

Validation

Since you’re doing server-side input validation anyway (right?), client-side validation is just to make the user experience better. But, loading a bunch of JavaScript to do this is going to slow down the site.

Fortunately, you can bake a lot of the validation into the form itself and let the browser do it for you.

There are a ton of new input types in HTML5, and for common types, there is good support on mobile browsers.

First, let’s look at the pattern attribute. The value should be a regular expression that matches allowed values. This works on text, search, tel, url, and email fields. For example:

<input type="text" pattern="\d+ \w+">

When a user enters text that doesn’t match the pattern, the browser will apply the :invalid pseudoclass to the element, which you can style with CSS however you’d like. Here, I set the background to pink.

There’s also a :valid pseudoclass that you can use.

Browsers apply the pseudoclass immediately, so if you want to let the user finish entering the text before calling them out, you can use a selector like this: input:invalid:not(:focus). (This worked for me on iOS but not on FirefoxOS.)

Or, use specificity to reset the change:

input:invalid {
    /* Set something here. */
}

input:focus {
    /* Undo the something from above. */
}

There’s one particularly special value for patterns, which is matching numbers only:

\d*
[0-9]*

iOS, in particular, will react to this pattern.

Entering data

Now you can skip the big JS download to validate your forms. Do it in HTML and CSS and let the browser handle it.

The other part of forms that is hard on mobile is entering data. Keyboards are small, and typing a lot is a pain. Here’s where the input types really become valuable.

Mobile browsers will change their keyboard in response to certain input types. Here’s how it looks with a regular <input type="text" />:

Here’s how it looks with <input type="email" />:

The difference is subtle, but both keyboards have made the @ symbol easier to get to. Now here’s <input type="url" />:

Gone are the space bars, now we have ., /, and .com readily available.

<input type="number" /> is an interesting case. We need to add the digits-only pattern (pattern="\d*" or pattern="[0-9]*") for iOS to recognize it, but then we get:

A nice, big, numeric keypad.

Some browsers support even more. Here are <input type="range" /> (obviously this one needs some additional visual feedback) and <input type="datetime-local" />:

Use it today

This all works today, right now, in the browser on my real phone.

If a browser doesn’t know what to do with a type value, it defaults to type="text", so it still works fine — so you may want to load a script in that case to do client-side validation.

If you’re already using all the tools available, get your friends and coworkers to do it, too. Peer pressure is a good thing. I have seen some of this in the wild, but it’s not everywhere yet.

The most surprising thing was how much I enjoyed myself. I loved figuring out every piece of that first project. It was frustrating at times, of course. But, most days, I could spend hours and hours researching things and making tweaks to the site. Soon, I started thinking about my next one, and about what I should learn next.

Fast forward a few years. Now, I get to watch beginners experience the joy of web making or programming all the time. My personal experience with learning how to code, plus a healthy dose of inspiration from PyLadies in Los Angeles, lead to the founding of Ladies Learning Code, a not-for-profit organization that runs workshops for women (and men) who want to learn beginner-friendly computer programming and other technical skills in a social and collaborative way. We now have chapters in Vancouver and Ottawa, a permanent 1,100+ square foot workshop space in downtown Toronto, and a thriving girls’ program, called Girls Learning Code. Over 2,500 women, men, and girls have participated in a Ladies Learning Code or Girls Learning Code workshop since we started in August 2011. In Toronto alone, over 400 individual developers and designers have volunteered their time to help beginners at Ladies Learning Code workshops. That number might be the most staggering of all. We’re constantly amazed by the support we’ve received from the tech communities in Toronto, Vancouver, and Ottawa, as well as in other cities where we don’t have chapters yet. The fact that so many amazing developers and designers are willing to volunteer their time to help beginners get started with developing technical skills still blows me away.

What I’ve learned through Ladies Learning Code is that there is a huge group of people in our society who are ready to become creators — not just consumers — of technology and the Web. They want to build web sites, they want to prototype app ideas, they want to design and print things in 3D, they want to understand how computers and the Web work, and they want to be able to better use technology to improve their personal and professional lives. Not all of these people are comfortable venturing into the space on their own, though. It’s intimidating, and when you start, you don’t know what you don’t know. If we can get this group of people to the point where they have the knowledge and confidence to begin exploring the world of making — well, it would be a big deal. Because they want that. They want to know what all of you know.

So, enjoy the holidays and take some well-deserved time off. But, when you get back to work in January, I want to encourage all of you — every single person who has some sort of technical knowledge — to share it with someone else. It means a lot to the people you help (believe me!), and if you’re anything like the group of developers and designers I work with in Toronto, you’ll have a lot of fun, too.

Here are some ideas for those of you who are interested in sharing what you know:

• Do a quick search for a group like Ladies Learning Code or Girls Learning Code in your area. If you can’t find one, email me at heather [at] ladieslearningcode.com. I might be able to point you in the right direction. Ladies Learning Code will also be expanding to new cities in 2013, so if you think your community would be a good fit, let me know!
• Check out CoderDojo and Dojo Directory to see if they have a group in your area. If they don’t, why not start one?
• Reach out to your local elementary or high school and see if they have a computer-related after-school program that you can help out with. If they don’t have one, maybe you can partner with a teacher to make it reality.
• Boy Scout and Girl Guide Leaders are always looking for fun activities and experiences for their groups to try. Maybe you could meet at the local library and take them through an introductory HTML and CSS lesson. Need slides? Try Jon Buckley and Kate Hudson’s slides from an event we ran together in September.
• If you’d like to start small, try hosting a “Kitchen Table” event (a term coined by Mozilla earlier this year). They’ve published a great how-to.
• Speaking of Mozilla, have you tested out Thimble or Popcorn yet? These are great tools for teaching kids about the Web.

In addition to a low barrier to entry, a somewhat unfortunate side-effect of the current ecosystem is that many people (especially those new to web development) are at a loss when it comes to scaling their app to accommodate an influx of traffic and users. Often, these problems are relegated to people with fancy titles such as Systems Architect or Senior Architect, and with good reason; this stuff is difficult. Very difficult. It takes quite a bit of blood, sweat, and sleepless nights to understand just how hard of a problem this can be.

Nonetheless, a superficial understanding of the challenges in scaling a web app can be useful to developers who wish to progress further in their career and to better understand the trade-offs that are made during the lifetime of an app.

Let’s look at a broad and general overview of how a typical app might grow, what problems might arise, and some general-purpose solutions to these problems.

Money does solve problems

You might think that there’s something inherently cool or fun about maintaining all of the disparate servers and services that your app requires — and you might actually enjoy it, at least for a little while. After the initial honeymoon phase wears off, however, the hard facts start to set in. Every minute that you spend fiddling and maintaining your architecture is a minute that could be spent improving some facet of your app that could benefit your users in a much more direct way.

As is often the case, the simplest (and sometimes most cost-effective) solution is to simply throw money at the problem. With the recent affluence of general platforms as a service (PaaS) such as Heroku, Engine Yard, and even language-specific ones like Gondor.io, the need for a full-time operations person on staff for a small- to mid-sized web app has decreased dramatically.

Take it from someone that has been paged at two o’clock in the morning in a foreign country when servers go down due to network issues, and your Internet connection amounts to a hotel using an AOL CD they found in the trash; if you can pay someone else to have that problem, do it.

Every problem you pay someone else to have is time that you can use to focus on what makes your app special.

Starting Small

If you’re like 99% of the web sites in the wild, you most likely started with a single server for running your app. Or, you might even have started on some form of shared hosting. There’s nothing wrong with that; we’ve all been there.

There comes a time when that single host isn’t able to keep up with the demands of your app — your database grows too big, you become memory or IO bound (depending on your app and your service stack), or you’re not able to attain whatever response time or throughput goals that you’ve set for yourself. This might be stressful, but remember, problems of scale are generally good problems to have. Most apps struggle to hold on to a few hundred users, let alone have the privilege of needing architectural changes to scale upwards.

Your go-to solution at this point should be one that minimizes the changes in your procedures, development time, and complexity of the system and app architecture. For most situations, this means getting a more powerful machine. If you’re on some sort of virtualized hardware like Linode or Amazon Web Services, then this is simply a matter of a few clicks in their respsective web consoles, and a few minutes of downtime. While the latter might not be desirable, generally, it is tolerable for your users if handled correctly.

Considering the various memory, processor, and disk configurations available from virtualized or bare-metal providers, this very simplistic approach of making your servers more powerful (also known as vertically scaling) can get you quite far.

Necessary complexity

If your user base continues to grow, vertical scaling will only get you so far. The next step is to separate out your system architecture by broad components and services.

The obvious choice for most is to separate the web server from the database. In doing so, an additional layer of complexity and an inherent network overhead is introduced. However, you suddenly gain the ability to independently scale both major components with respect to their own performance characteristics and requirements for your app as a whole.

Once the database and web server have been separated, the next phase is to add some high-availability to the mix. It’s great that you can now serve more users, but if your web server node goes down for whatever reason, no one cares about your mean response time or throughput. What matters is that your app remains up and functioning for as many minutes during the year as is possible.

Begin by performing an analysis of the SPOFs in the current architecture. A simple heuristic is this: if any particular server were to become unavailable, would the app continue to function? If the answer is no, then you’ve found a SPOF.

In our current architecture, we have two very obvious (and separate) SPOFs: the database server and the web server. If either is disconnected, our app is no longer able to operate normally.

At this point, anyone not familiar with the concept of a shared-nothing architecture has some reading to do. Go on. I’ll wait.

Now, the easiest SPOF to address in this situation is the web server. As long as you aren’t storing any stateful information pertinent to a particular user on the web server node (file-based sessions being a typical offender), then adding additional web servers is feasible.

Now you run into yet another SPOF: your DNS A record, which ties your host (example.com) to an IP address.

(I assume you aren’t managing your own DNS for the sake of simplicity. If this is not the case, then you probably don’t need this overview.)

One solution is an HTTP-capable load balancer such as HAProxy, nginx, the Amazon Web Services Elastic Load Balancer, or one of many other similar products. A properly configured load balancer will let you distribute incoming requests according to a set of rules or heuristics, thus eliminating a SPOF. You can then add or remove web servers from this pool at will; you need only ensure that there are enough active servers in said pool to ensure that your app is able to process requests in a timely manner.

Now you’ve managed to remove the web server as a SPOF. If one or more web servers go offline, our app will continue to respond and serve requests as long as the number of remaining web servers is enough to satisfy our normal performance requirements.

The next obvious SPOF, your database, is a tricky one. If you’re smart and can afford it, this is something that you might want to make someone else’s problem. Products such as Amazon’s Relational Database Service can take a huge chunk of pain out of managing a relational database yourself. If you’re more adventurous and willing to try new things, there are also a plethora of non-relational technologies such as MongoDB, Riak and CouchDB that make various trade-offs in an attempt to better tolerate network partitions and provide better availability.

The little things that make a big difference

Open up your favorite web app in your browser. Now open the network traffic inspector panel (or whatever the equivalent is for your preferred browser). Chances are good that there are several dozen (if not hundreds) of requests for external stylesheets, Javascript, images, and fonts, not to mention any asynchronous XHR requests to load dynamic content after the page has loaded.

Assuming all of these files have been properly minified and compressed, and you use all the best HTTP headers for caching, your web server still needs to respond to every request. A very simple and effective way to reduce the load on your web server is to move these assets somewhere else, either to a dedicated pool of static content servers, or to an external service such as Amazon S3. There are a few caveats to this, but it’s generally a good idea and can save you quite a few CPU cycles that could be put to use elsewhere.

Going even further

As with most things worth doing, your job is far from done. You might have a solid, mostly-available foundation for your web app, but there’s always more work. Here are some additional things to consider:

1. If your app lets users search through content on the site, moving search-related querying and storage to a separate tier is highly advisable. Products such as Solr and Elasticsearch are well worth your time to investigate.
2. Caching content that is expensive to generate is sometimes a necessary evil, and it’s always a double-edged sword. Familiarize yourself with common read and write cache policies before adding something like Memcached or Redis.
3. Often, there are expensive requests that you are unable to cache, such as requests to third parties like Twitter or Facebook for lists of friends or other volatile and personal information. In these cases, an asynchronous job queue is your friend. Anything that does not need to be immediately computed, calculated, or queried to serve a request to a user should (at some point) be moved to an asynchronous task. There are good solutions, including Celery, an abstraction for several adapter-based brokers and result stores ranging from Amazon Simple Queue Service to RabbitMQ and even Redis and MongoDB.

Not quite there yet

This article didn’t begin to scratch the surface of what’s possible for a modern web app. Many things were omitted, caveats were left out, and naive simplifications were made. Fear not! One of the great advantages of web apps over desktop apps is that it is possible (and advisable) to make incremental changes to both functionality and performance while remaining relatively transparent to the end user. All of the above strategies do not have to be implemented all at once. Thankfully.

Many skills and intuition are learned through trial and error, and can take years before becoming part of your natural tech vocabulary. For all you new and aspiring developers and devops out there, don’t give up! Systems and app architecture are not only challenging topics, but also incredibly rewarding.

Many developers use empirical approaches, systematically addressing each possible problem branch. If A, then B. If B, then C. If C, then D. This analytical approach works well and is easy to teach. The problem I find, though, is that many focus on identifying solutions before fully grokking the problem. This leads to approaches that begin with D and work their way backwards to A, resulting in wild goose chases and wasted time.

That’s not usually how I approach debugging. For me, an approach based on intuition is key. But intuition is a tricky thing, isn’t it? After all, having a gut feeling about something seems mystical and a little paranormal. I don’t think it is, though. Wiktionary defines intuition as “immediate cognition without the use of conscious rational processes.” Here’s how I think it works. Your brain is constantly making computations, much like a Frank Herbert mentat. It has the data it needs, makes the calculation faster than you can blink your eyes, draws conclusions, and gives you the response in the form of a short-but-sweet gut instinct. That’s immediate cognition, and you weren’t conscious of any rational process being used.

This constant computation is something everyone does, but in Western civilization, we have suppressed it in favor of the rational approach. We can learn to develop it, listen to it, and, along with analytical thinking, make it a part of our standard thought processes. Here are some ways to make intuition a part of your debugging habits.

Slow down

As developers, when we encounter problems that require intense debugging, we’re often in a rush, especially if it’s something that affects a production environment. This rushed mindset creates the worst possible conditions for allowing you to effectively target the problem and identify a solution. When rushed, you don’t think clearly about where to begin, much less how to solve the problem. To think clearly, you must slow down.

When you find yourself rushing to fix a bug, it’s best to stop what you’re doing, close your eyes, take a deep breath, and let it out slowly. I know it sounds hokey, but this isn’t transcendental meditation. It’s perspective. It’s focus. When you take the time to slow down and focus, you can tap into your intuition, using it to discover where to begin and how to solve the problem.

In his biography of Steve Jobs, Walter Isaacson quotes Jobs as saying this about intuition and slowing down your mind:

“If you just sit and observe, you will see how restless your mind is. If you try to calm it, it only makes it worse, but over time it does calm, and when it does, there’s room to hear more subtle things — that’s when your intution starts to blossom and you start to see things more clearly and be in the present more. Your mind just slows down, and you see a tremendous expanse in the moment. You see so much more than you could see before. It’s a discipline; you have to practice it.”

Stop saying “I don’t know”

As developers, we like to focus on constraints. Software construction is all about finding the constraints of a problem domain and programming a solution to fit within those constraints. It’s no wonder our lives and jobs revolve around constraints, and we like it that way. But constraints can hold us back and squelch our intuition. When we assume we don’t know the answer to a problem, we impose a constraint upon ourselves that makes it more difficult for us to find the answer.

In her book, Grow Your Intuition: 6 Simple Steps, Suzan Bond says that we shut off our intuition when we answer questions with, “I don’t know.” She goes on to say, “When you stop using this phrase as much, you will open up your intuition. You’ll be able to see more possibilities, rather than limitations.”

It’s not wrong to admit lack of knowledge in an area, and I’m not advocating that you make up answers, but the truth is that you may already know the solution. Your brain already has the data it needs. Slow down. Take a breath. Ponder on it. Let your intuition speak to you. If you have trouble finding a solution, ask yourself what may be blocking you from seeing it. Don’t accept that you don’t know the answer. Accept that you must discover the answer.

The problem is in your code

Here’s a humbling but crucial notion. It’s never fun to accept the blame for something, so it’s easiest to point fingers at someone or something else. This third-party library or framework is terrible! John wrote this spaghetti code three years ago, and it’s awful! This programming language is a nightmare! Let’s face it, developers are proud folks and masters of hyperbole. I have learned, though, that the problem is most often found right in your own code, so it’s best to begin with that assumption.

Very early in my career, I worked with a developer — we’ll call him John — who always blamed someone else’s code when he found a bug. He loved doing this so much that he would leap from his desk, giggling, and run into my office to boast about finding a bug in some third-party library or even in the programming language we were using. John loved being able to say that the language was faulty and that he was responsible for finding it out. On more than one occasion, he had the programming language’s bug report form open and filled out before I was able to catch him and force him to readdress the issue by carefully looking at his own code. Every time, his code was at fault.

When you start with the assumption that it’s someone else’s code that has the problem, then you’re hurting yourself in several ways. First of all, you’re impeding your own ability to find the real problem because you start by looking in the wrong place. This costs you time. Secondly, you’re setting yourself up to look like an buffoon when it turns out your code was the problem all along, and you wasted time, energy, and resources trying to blame something else. Lastly, you hurt your reputation as a problem solver by passing the buck.

When you start with the assumption that the problem is in your code, then you display humility and leadership by accepting the responsibility to solve the problem. If it turns out the problem was, in fact, in someone else’s code, then you have instilled in others a great deal of trust in your ability to actively pursue solutions to problems.

Focus on the problem, not a solution

When you focus on identifying a solution before fully grokking the problem, then you’ve put the cart before the horse. Start at the beginning; start with the problem. I think this is difficult for many developers, because we like to jump to conclusions. I see both A and B, so it must be C, therefore D! Finding and implementing solutions is thrilling and satisfying for us. Finding problems? Not so much.

Keep in mind that the symptoms may not indicate the real problem. We know what they are, so we fix the problem by testing for and addressing the symptoms. Once the symptoms no longer appear, the problem is solved, right? Nope, now you’ve buried the problem deeper, and it’ll be harder to find next time.

A specific story from my career comes to mind where multiple, identical records were added to a database table for a specific action that should have added only one record each time. This resulted in the customer generating reports that contained multiple, identical records. That was the symptom. In an effort to appease the customer, a solution was devised to combine the records at the point in which the report was generated. This made the customer happy, but it didn’t solve the problem. In fact, now that the problem was buried, it was more difficult to see, and the database continued to fill up with bogus records.

Always get to the heart of the problem. Don’t just look at the surface symptoms and assume they are what needs resolving. Focusing on the solution won’t get you there. You need to focus on the problem. I find once I understand the problem — once I fully grok it — my intuition kicks in with a handy solution.