Engineered Web

Drupal 7.14 API Compatibility Breaking Change

2012-05-15T20:30:00-04:00

Minor Drupal versions are usually for bug fixes, security updates, and the occasional new feature that doesn't break backwards compatibility. Compatibility changes are reserved for major Drupal releases. There are exceptions such as Drupal 6.2. It was such a big deal there is an update documentation page just for this release. When Drupal 7.14 made an API breaking change without providing documentation or notification to module developers I was quite surprised. The lack of detail made it difficult to track down the changes when I had a broken codebase. Here are the details so others can, hopefully, have an easier time if they run into this problem.

What Is An API Breaking Change?

I consider a change to be API breaking if for the same input a function or method has a change to the output. For example, the data structure for the response from a function is different from before. This is what happened in Drupal 7.14.

The Change in Drupal 7.14

Prior to Drupal 7.14 the database connections were configured, by default, to fetch all field and column names as lowercase. As of Drupal 7.14 this has changed and they come back as the database sent them.

For many modules who have defined their schemas to use lowercase table names and fields this change won't cause an issue. If however a module makes a SQL query like show table status (using a function like db_query()) the results will no longer be lowercase. Say you get the results back as an associative array the key names will be different (e.g., Name instead of name). This has caused some modules to break.

For the record, I like this change in theory. Bringing it into the middle of the Drupal 7 life cycle without documenting the change or notifying the develops was a bad idea.

How To Work Around It

The following work around will work for all versions of Drupal 7 by generating lowercase field and table names. This should only be used for the places it's absolutely needed. Instead of using db_query use the following function:

function _custom_db_query($query, array $args = array(), array $options = array()) {
  if (empty($options['target'])) {
    $options['target'] = 'default';
  }

  $connection = Database::getConnection($options['target']);

  // Backup and restore PDO::ATTR_CASE. Prior to Drupal 7.14 this is
  // PDO::CASE_LOWER. Afterwards it is PDO::CASE_NATURAL.
  $orig = $connection->getAttribute(PDO::ATTR_CASE);
  $connection->setAttribute(PDO::ATTR_CASE, PDO::CASE_LOWER);

  // Execute the query statement.
  $result = $connection->queryRange($query, $from, $count, $args, $options);

  // Restore the attribute.
  $connection->setAttribute(PDO::ATTR_CASE, $orig);

  return $result;
}

Ethics, Software Engineering, and The Web

2012-05-15T08:20:00-04:00

One of the required classed for my engineering degree was an engineering ethics course. They said the reason this class was required was due to all the ethical failures engineers had been making. At the time I thought of people working on children's toys or military applications. An ethics course could be good to help them work through or realize any dilemmas they had going on. At that time I would never have imagined I would see more ethical failures working on the web than I did working for a military sub-contractor. Unfortunately, this has been the case.

Ethics

Ethics a the branch of philosophy that deals with right and wrong behavior. For example, putting sharp small objects inside a toy for a one year old is considered wrong. A lot of wrong behavior can get you in trouble or hurt those around you. This is why large companies tend to have ethics training for their employees. Well, that and CYA.

Fail, Fail, Fail

I've seen numberous developers and web companies fail at ethical issues. My gut reaction is to think they don't realize what they're doing. I want to give them the benefit of the doubt. Many of these developers are focused on themselves and their product well more than 40 hours a week. Being so me focused leaves little time and energy to focus on the implications to others. And, where I had a university education that covered ethics many of those building the web today didn't. Ethics may not be on their mind.

A Nonthreatening Example

I first saw this one a few years ago. A startup that was doing quite well had done it. This startup web application had taken a JavaScript library and in minifying it stripped out the license and attribution. It was part of their web interface and the JavaScript was distributed from their servers to a users browser. This was a violation of the license and the developer contacted the startup over the issue. The license he chose had been violated and he felt wronged. The startup didn't respond to him.

This isn't a one time failure or a singled out case. I've seen this disregard for licenses (even open source ones) happen numerous times. I've talked with angry developers who've been wronged.

It Doesn't Stop There

There are a lot of cases of more problematic ethical failures. They happen all around us and they are hurting people. My professors were right to think an ethics course was a good idea.

If you've never taken an ethics course and don't know much about it consider taking a professional development course, watching some of the free iTunesU videos, or picking up a good book (anyone have a good recommendation?). I'm regularly asking myself, when we build the future of the web are we building an ethical place? I'm not so sure the answer is yes.

Speedy 1.0 Released

2012-05-07T15:30:00-04:00

Minifying the JavaScript distributed to a browser is important for performance. The speedy module came into being in an effort to make this easy for Drupal sites to implement everywhere. Today the 1.0 release of the module comes out providing minified JavaScript for Drupal 7 core.

Path Forward

While there are many performance problems we can solve over time we want to get those that work out there as quickly as possible. In the speedy issue there are more features we are hoping to add over time. The initial release is just one step to better performance.

Default JavaScript to Footer

Unless you have a reason to your JavaScript files should be included in the footer. By default Drupal places them in the header which is a much safer approach. When included in the footer there are times this won't work well. We are working to make the footer the default except for cases it shouldn't be.

A Pluggable System

While I use UglifyJS to minify my JavaScript there are other tools. Some you may want to automate into build scripts or some other method. For example, using an external service to generate the minified files. There are a number of issues in the speedy issue queue targeted at making this work.

Minified Contrib JavaScript

One of the hard problems to solve is to legally minify JavaScript libraries included in contrib. If a library is include in a module with license information we have to be sure not to remove that. We want this module to be safe for anyone to install without worrying if they are violating any license. This is one of those non-trivial problems to solve but we are working on it.

If you would like to help please join the issue queues, create some patches, start some git branches with changes, write some documentation, or even just spread the word. JohnAlbin and arthurf already have.

Understanding Other Developers - Users Of Our Development Products

2012-04-23T15:00:00-04:00

Other developers are not like me. This is something I have to tell myself regularly. I'm reminded of this by the simple numbers for Wordpress usage. 16.3% of websites are powered by Wordpress and it accounts for 54% or the CMS market. While we can argue how these numbers are measured we can't argue that A LOT of websites are built on technology often railed on in tech blogs.

I find this even more interesting when it comes to creating development tools, documenting methods, and advocating use of the new hotness. All too often this happens in the bubble of people who think like us and do thinkings in a similar manner. We wonder why others struggle with what we are suggesting. We wonder why something isn't catching on. Sometimes we even write off these other developers as incompetent because they suggest using some piece of technology we've grown to despise. None the less, they keep trucking along, thinkings change slowly, some camps move further apart, and may good ideas go stale.

Part of the equation we often miss is that other developers are different and how those differences matter. They have different needs. Their customers have different needs. The list of tasks they have to do for their job is different. The priorities they have are different. Understanding these differences can help us make better all around products and move the Internet in a better direction in a healthier manner.

Differences of Note

After talking to a lot of developers in many different circles I've found a few differences that are worth noting and understanding. They provide us with some keys into others making it easier to explain things or even know what to explain.

The specialist vs. the generalist - A simple case is someone who only works on JavaScript compared to someone who writes JavaScript along with PHP, dealing with deployments, working with clients, managing others, or any number of other tasks. The JavaScript developer is going to dig into the nuance of patterns, play with different techniques, quote Crockford, and be really into it. The generalist will likely just want to touch enough JavaScript to get the task in front of them done.
The 501-ish developer vs. the I live for this developer - Some developers just want to get their tasks done quickly, easily, and meeting the needs in front of them. At the end of the day they disconnect from developing and go do something else. There are other developers who don't disconnect. Their hobby is what they do for work. In their off hours they are doing other development practices. It's not just a job, it's a passion.
I love change vs. I avoid change. Some developers love change. Sometimes they even like to stir the pot to inspire change. Others what to stay with what's been tried and true for them. There are a lot of reasons that cause this. For example, someone may want to avoid change because they don't have time to learn all the new stuff while meeting the needs or their clients of project. Someone else may love change because they are easily bored and change keeps them interested.

Just looking at these three dimensions we can see we have developers that can range from generalists who don't develop outside working hours and don't like to change their development practices all the way through specialists who live for this stuff and are always changing what they are doing. That's quite a range.

Where Things Can Go Terribly Wrong

I've watched stuff go wrong time and time again. Frustrations start to show up. And people wonder why it didn't work. Unfortunately, technical problems are usually the easy ones to solve.

Let me give you an example of something gone terribly wrong. Imagine a JavaScript developer on a team of non-JavaScript developers. The JavaScript developer does a lot of the front end development while the others do back end development, APIs, deployments, and the other parts of the stack. Some of these developers do some front end work. To get better overall JavaScript would it be better to teach them all the intricacies of front end development or to make it easier for them to know very little about front end development but output good end products through the use of documentation and helpers.

I've seen and been told that the rest of the team should learn the latest patterns. That it's in their best interests. Trying to make non-JavaScript developers into JavaScript developers in addition to what ever else they have to do can prove to be fairly in-effective at best and piss people off at worst.

Picking Your Goal

With such a wide range of developers it can be important to pick your goal. Looking at the this case is the goal to make the rest of the team write better JavaScript or to make the output JavaScript be more performant, better to manage, or something else? This is important because you need to take the goal and mix it in with a good dose of the difference in a group of people to come up with a productive path forward.

In the open source world this can be harder to do. Continuing this example, imagine a handful of JavaScript developers get together and want to make a change to a product. They think like JavaScript developers and make the change they want to see. It gets released and then the complaining begins from all the people not like that first group that envisioned and built it.

Making It Easier On Yourself

Failure you can learn from has lead me to a number of tips and tricks that help me with these occasions come up.

Learn about the others affected by what you're doing. I prefer having a beer or coffee with some others in the affected group and talking frankly about the subject. Learn what they think and why. How it impacts them.
Make it easier on others. Once you understand their needs, questions, and problem write documentation they would want to read. Present on it. Do a video demo. Alter the concept to make it easier for them. Provide a helper library. Or, if the idea isn't going to be helpful consider scrapping it and sharing why. Or, making it pluggable so you and they can both co-exist.
Plan for the hard stuff. Making technology changes is often the easy part. The hard part is working with others, explaining yourself, answering support questions, and all the not-as-fun-as-technology stuff. Plan for it. That may mean getting others to do the dirty work or setting different expectations.
Be kind to others. Being a dick does not help others accept what your doing or help them want to follow you in the future. Treating others with respect and kindness goes a long way.

The next time you want to make or support something that's a change consider the implications to those actions. They may be more than meets the eye and there may be some things you can do to set yourself up for success.

Note: If you think any example here is about you it's not. Many examples have influenced me but none of those are directly reflected in an example.

Why It Is Hard To Minify On The Fly

2012-04-17T09:55:00-04:00

One of the tips to speed up websites is it minify the JavaScript. It's simple, effective and the major players already do it (except Drupal). In the discussions for Drupal 8 and how we can speed up Drupal 7 we've been talking about how we go about providing minified JavaScript files for browsers and there are two camps. One camp wants to ship minified files and the other wants to minify on the fly. While minifying on the fly would be nice it turns out that this is really hard for Drupal and other CMS systems.

Not A Technical Problem

When it comes to problems the technical ones can mostly be overcome. It's not hard to tie in with UglifyJS or another tool to minify JavaScript. PHP libraries like Assetic already do this and there are even web services we could use.

The License Problems

Note: I am not a lawyer and this is not legal advice. If you want legal advice go see a lawyer.

Licenses are something virtually every software project uses. Some will even argue that you need to use license X instead of license Y. When we sign up to use a license we need to respect what we signed up for and follow them. Many of us work for companies with lawyers that will make us respect them/follow the law. This ends up creating a complication because many of the JavaScript files we use contain license and copyright information in comments. When a file is minified comments are stripped out. When we manage how they are minified by hand we can make sure licenses are still properly handled where on the fly processing can't do this.

Removing comments may seem simple and safe because they aren't needed for the application to technically run. But, when we ship files from our web servers to execute in a browser the way JavaScript does we are distributing a file. This brings the license sections on distribution into play.

We can see examples in the GPLv2 where is says "keep intact all the notices that refer to this License" and even the MIT License where it says "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software." At the very least removing notices pointing the MIT license violates the spirit.

Does It Matter?

I look at it in two ways. Does it matter legally and does it matter to the developers who put the license and copyright statements in the code? I think we need to look at both to be a legal and kind community.

On the matter of legality I would recommend you go see a lawyer. If you aren't going to get one look at what companies with lawyers are doing and saying. From a legal standpoint this was all first brought to my attention by a company I wasn't working for whom I suggested they minify their JavaScript. Figuring out how they could handle the license issues was what was stopping them.

Being kind is entirely different. Several years ago I watched as a big name company used a JavaScript library and strip out the copyright and license information. The author was hurt because he felt violated. He contacted the company and got no response. He felt violated. On the flip side it did hurt him. Anyone else technically sound who looked at how this functionality worked would not see the authors name. He wouldn't get credit nor would it cause anyone to seek him out to congratulate him, or try to hire him.

Seeing the legal impacts and personal ones I do think it matters.

What Can We Do?

I see three things we can do about this.

Use the Speedy module to minify Drupal core JavaScript. No license notice exists so there is nothing to strip.
Help get other Drupal contributed modules to ship with minified JavaScript files that properly handle licensing.
Help us try to find a way to minify on the fly and preserve licenses.

Using Bitbucket In Addition To Github

2012-04-13T11:40:00-04:00

Github is awesome. I've used it for open source projects and with companies I've worked for. The pull requests, comments, gists, and tools are fantastic. So much so that I'd even go so far as to recommend large enterprises checkout Github enterprise. Despite all of this I find myself happily using Bitbucket for personal projects that aren't open source. There are cases where using Bitbucket just makes sense for me.

Some Background

I first learned about Bitbucket when it was used solely for Mercurial hosting and Nathan Smith had moved his 960 grid over there. Git won the distributed VCS wars and 960 moved back to Github. But, I was introduced to Bitbucket.

Since that time Bitbucket added support for Git, improved the UI (though it's not as good as Github), and was purchased by Atlassian. Atlassian purchased SourceTree and made it free. SourceTree is a Git/Mercurial/SVN Mac GUI that integrates with Github and Bitbucket nicely. Bitbucket pricing has also been made really friendly.

Why I Use Bitbucket

Not every project I do is open source, for work, or something I want to share with the world. I used to run my own git server for these projects. Sometimes I would want to collaborate with others on these projects or share links to changes with others. Yet, they still weren't meant to be public. To make my own setups have the features I wanted was more work than I was willing to put in.

Github plans are nice for some cases. For example, a team working on a product where they just need a handful of repos. Or, for an individual collaborating with a few others on a project. But, when you get into a company doing client work and they have a lot of clients you need to have a plan for repos you aren't actively working on. This may be an issue if you have a lot of clients you are supporting who aren't in active development. Or, you can pay quite a bit per month. What if you are an individual who want's to collaborate on a hobby project with 3 or 4 others on a private project? Or, you have 5 or 6 or 7 personal sites plus backup scripts and you want them in VCS.

If there were no other options I would probably recommend paying Github. But, there is competition in this space so I'm embracing it. Of the different alternatives I've found myself using Bitbucket and have been happy with it. For my personal private projects it's turned out to be the alternative I'm the most happy with. Atlassian is a successful stable parent company. If you are looking for an alternative for some cases consider Bitbucket alongside Github.

Why I Continue To Choose PHP

2012-04-12T10:35:00-04:00

While I was responding to comments on a recent post I had the realization that the motivations I have for choosing the development tools I use are quite different from those others have. I find a little self evaluation to be healthy so I thought I would share what I found when I looked at why I continue to choose PHP as my language of choice.

A Little Background

Context is everything when it comes to making personal decisions. Understanding how someone got to where they are can show insights in the decisions they made (or failed to make). So, there are a couple pieces of context I should share.

I've always liked to help others. I've volunteered with non-profits. I've taken my own time to try and help others solve problems. I try to learn how to do things and share it with others so they can learn from my experiences.
I like solving practical problems with practical solutions.

Low Barrier To Entry

PHP has a low barrier to entry. Some would argue the barrier to entry is too low.

This low barrier isn't just about helping someone learn to program because they want or need to learn. It's also helpful when you want to help enable someone whose skilled in another discipline do something on the web. Maybe a web designer who targets a niche audience and needs to know just enough coding to be dangerous.

Sure, you could say they might write some ugly code or there are security issues. But, I've seen ugly code out of seasoned "professionals" and just look what happened with the mass-assignment rails vulnerability recently. I consider these all education points not reasons a low barrier to entry isn't good.

Host It Virtually Anywhere

Hosting can be complicated. Just last night I had a light discussion on rails hosting. I regularly talk about cloud hosting. So many of these solutions aren't for the mom and pop stores or small local non-profits that can benefit form a basic web presence. Cool hip hosting solutions are great. They just aren't great for everyone in the long tail of the web.

PHP can be hosted all over the place. There are so many hosting options (some that even drive me a little nuts). Some even provide free options for non-profits. Sure, it's not powerful but many of these organizations don't need it.

At Scale

PHP can scale. And it's not hard to make it do so. So, if a small hobby project, non-profit, or something else takes off it's not hard to scale the project. If a project remains small and steady it can do that well. This is well documented, well shared, and people know how to do it.

It Makes Me Replaceable

I'm easily distracted and I have a great day job. So, when I help others get going I want them to have other easy to get to resources because that means they don't need me anymore. I don't really have the time to help everyone I'd like over the long haul.

There are a lot of PHP developers. I'm happily replaceable when I become to busy or distracted by the next thing.

What This Means

I've found a couple nuggets of usefulness in these.

When I generally work on and share useful PHP stuff, even if not targeted at others, it can help others. Sometimes the long tail of web people who I want to see be successful.
Our motivations can often shape our choices. If I just wanted a good programming language for me and I didn't care about helping others I might choose a different language or tool set. What motivations do you have for your language of choice?

Whether you agree or disagree with me hopefully you understand why a little bit more.

4 Reasons To Choose PHP For Your Next New Project

2012-04-11T08:20:00-04:00

With all the new development toys why would someone want to choose PHP for their next new web project? Why would someone choose PHP over node.js or Rails? It's easy to say something is a legacy application in PHP and it's not cost effective to migrate away from it. It's entirely different to choose something for an entirely new project all together. So, here are 4 reasons to consider PHP for a new web project.

There Are Bodies To Hire

What happens if your project takes off and you need to hire people? Or, you hired someone to develop your site and they are not longer available and you need them? If a project has a need for people choosing a language where there are fresh warm bodies is important. There are a lot of developers with PHP experience. Compare this with Ruby (and Rails) or node.js. While these projects may be younger or hotter how deep is the available talent pool of available developers? I'll give you a hint, it's not deep enough. PHP has a deep pool of developers.

Hosting Is Everywhere

People who know me might say I point this out too much. But, if you are going to create the next product for the long tail of sites or want to try and create the next big CMS you'll want to start with the most fertile ground for hosting. There is a reason PHP is currently used for 77.5% of sites. You can host PHP stuff almost anywhere.

There Are Good Tools Available

This is where I can imagine getting push back. Good tools? Really? It's not like I'm suggesting you build a Wordpress site (16.2% of all sites are wordpress). I'm talking about frameworks like Symfony, dependency managers like Composer, editors like Sublime Text 2, and documentation tools like phpDocumentor 2 or Doxygen. The tools to write good quality code exist in PHP. I'm not joking. Really.

Lots and Lots and Lots of Documentation and Knowledge

Floating around the PHP community and the web there is a lot of documentation on PHP. Has your project grown and it's ready to move off shared hosting or a single VM? The documentation and knowledge for scaling is all over the place.

This even applies to searches. When I search for a programming topic in PHP I tend to get more results (sometimes significantly more results) than the other server side languages I've searched for.

In Conclusion

Using PHP might not get you fame and accolades from computer science purists. Or even from cutting edge hackers. But, that hasn't stopped Wikipedia, Facebook, The White House, Etsy, Wordpress, HP, and so many others from being successful using it.

Why I Love PHP

2012-04-10T12:00:00-04:00

While I was just reading an article titled PHP: a fractal of bad design I felt the need to defend PHP. Over the years PHP has gotten a bad reputation and is regularly railed on. Yet, I've recently grown to like using PHP again. If you're curious why someone would like PHP (or insight into my possibly twisted world) here you go.

Disclaimer

I'm an engineer. I like to build things that work and take into account the ilities. If you want to discuss the nuances of language design talk to a language designer. If you want to talk about designing the perfect language you might consider talking to a philosopher. Or, if you look at a lot of well engineered failures consider looking at bridges). When they fail it's a big deal.

Some Context Surrounding PHP

It's important to understand some elements of PHP and how it's used. Making judgments about something without understand the context surrounding it can be short sighted and cause problems. Understanding context and history can help us understand why decisions were made and even evaluate if something was a healthy decision or not.

Target Audience

Not all programming languages are created alike. PHP is built for the web. On php.net it puts it this way:

What is PHP? PHP is a widely-used general-purpose scripting language that is especially suited for Web development and can be embedded into HTML.

Many languages are general purpose languages or have their own targets. For example, python.org states:

Python is a programming language that lets you work more quickly and integrate your systems more effectively.

PHP is targeted at the web while Python isn't. That means decisions are based on different criteria. So, when it comes to PHP a mass majority of the discussion centers around the web. For other languages this may not be the case. For example, I'm writing this post in an editor written in Python. Can you imagine a desktop GUI application in PHP?!?!

Long History Of Popularity and Usage

PHP is used by 77.5% of sites where the language can be detected. Let that sink in for a moment. PHP is popular and has been for a long time. Ten years ago PHP was really popular. While we often talk about the amount of time a language has been around there is a difference between a popular language and one that's a lot less popular.

To put popularity into context we need to look at the web. The web is a lot more than Facebook, Google, the other huge sites, and the hot new startups. The long tail of the Internet is really long. That includes the blogs of people living with cancer, the stores in suburban villages, the websites sharing pictures of cats with the world, and so much more. Most of these sites don't have engineers or IT on staff. When you make language decisions and have this audience you are attempting to be benevolent with it's not as simple as making decisions on a whim or even those to clean up language design.

Now that I think about it, handling this long tail well is one of the reasons I love PHP.

Ever Language Has It's Ugly Spots

I've written terrible code in over a dozen languages. I've seen the ugly spots in many languages. For example, Python doesn't have interfaces. Some people may like it and others don't. To some, this is an ugly spot. While it's popular to rail on PHP for it's ugly spots, every programming language has ugly spots for some developers.

Why I Love PHP

Let me count the ways...

The language can be hosted almost anywhere. I've worked on applications for the web intended to be installed and used by others. To help the long tail. The wide availability of hosting is something amazing. While this isn't great for people who are writing their own application in an environment they intimately control, for some of the use cases I work in this is great.
PHP has some nice underutilized features. For example, PHP Stream Wrappers. If you want to store files in a remote storage (e.g., S3) you might use a library. In Ruby you might use Fog or Java you might use jclouds. To use these you have to write you application around them. With PHP you can have your code use the same underlying file handling code for a file system and implement a stream wrapper to make the remote location work. No rewriting your codebase to work with remote apis. You just work with files like you always have. This was a feature we took advantage of with some recent HP Cloud Object Storage work. This is just one example among many.
Since PHP doesn't force us into a coding style there are now some nice frameworks being developed and well written articles being shared. In PHP you can write maintainable, well written, code that takes into account the illities.
PHP is well documented and contributing to the documentation is something anyone can get involved in. I love documentation so this hits close to home.
Developing with PHP is becoming a wide open field again. For some time PHP felt out dated and crufty. There was a lot of ugly code and ugly processes. When I looked at PHP I thought of applications like Wordpress and Pear. PHP has been reinvigorated with projects like Symfony, Composer, and Assetic. There is a movement of good quality code and it's fun to watch or contribute to.

I could go on and on but, I have to go write some PHP code. If you have reasons to love PHP or maybe just like it please share. I'd love to read what others think as well.

An Hour A Week To Better Documentation

2012-04-10T07:50:00-04:00

If you're like me, a lot of your projects aren't very well documented. If you look over Github or Drupal.org a significant number of projects are poorly documented. I'm only aware of a few people who regularly document their projects well. In an effort to improve the documentation on the projects I maintain (or try to maintain) I've started to schedule an hour per week to just write better documentation.

An important element to note is that this is scheduled time. I've added it to my calendar and scheduled it into my week. To keep this as a useful part of my time I've scheduled it for the second hour of my work week. This gives me time to get through my email and any communication I need to get done (or any slow starts). Then, I start writing documentation before the week gets in the way.

If you'd like to start writing better documentation I invite you to join me in spending an hour a week improving the documentation in our projects.

Faster Mobile Sites

2012-04-05T12:00:00-04:00

87% of the time it takes to load a web page happens in the front end. This it taken from actual measurements of the top 50k websites. When we want to look for ways to speed up our sites we should look to where most of the time is occurring. At DrupalCon Denver I presented an introduction to front end performance in the session Faster Mobile Sites. This introduction talked about why front end performance is important, covered some details about what's technically happening, offered some simple solutions to get going, and talked about some useful tools.

The Slides

The slides cover most of what I talked about. They were more than an aide to the presentation since many people don't watch the presentation videos.

If you have trouble viewing the slides they are also available on Speaker Deck. A mobile friendly version is available through slideshare.

The Video

If you have trouble with the video it is up on Blip.

Drupal 8 and Front End Performance

2012-04-03T00:00:00-04:00

At DrupalCon Denver I spoke on Front End Performance improvements we can make to Drupal, why they are important, and provided some information about what's happening technically so we can spot more cases to improve. This presentation was targeted at low barrier to entry changes we can make in Drupal 8 with the short amount of time we have until the feature freeze. If you're interested in front end performance, making Drupal really perform on mobile, or just want to heckle me checkout these slides and video.

The Slides

Most of the relevant information is in the slides. There is a portion at the end of the video where some questions were discussed and some other topics came up.

If you have trouble viewing the slides they are also available on Speaker Deck. A mobile friendly version is available through slideshare.

The Video

If you have trouble with the video it is up on Blip.

Why Minify JavaScript?

2012-03-29T00:00:00-04:00

When I was at DrupalCon Denver I suggested that all production JavaScript should be minified. This seemed like something simple that I wouldn't get any push back on. After all, the major JavaScript packages do it and the performance recommendation tools all recommend it. I was wrong. Over the past couple weeks I've received push back from numerous people who've told that me if you use gzip (or deflate) compression there's no need to minify JavaScript. So, here is a case for minifying JavaScript.

A good place to start is with size examples for different files. The table below has a breakdown for discussion. The three files used in the example here were chosen because they are used on ever page in Drupal where JavaScript is used.

JavaScript files and their sizes when minified and gzipped. File sizes are in bytes.
File	Original	Minified	Gzip	Min + Gzip
jquery.js	252881	94840	73775	33656
drupal.js	13852	3338	4708	1457
jquery.once.js	2974	568	1081	384
Total	269707	98746	79564	35497

Why Are drupal.js and jquery.once.js Smaller When Minified?

One thing that surprised me right off the bat was that drupal.js and jquery.once.js are smaller minified than compressed with gzip. These two files have lots of fantastic documentation in them. When a file is minified this is stripped out as it's not needed for functionality. When the file is compressed with gzip all of this text is compressed and shipped. Well documented files like these can really benefit from being minified.

Total File Savings

The savings between gzip compression and gzip plus being minified is 44067 bytes. If you figure you have 10,000 downloads of this bundle of files per month that's a savings of 420 Megs of bandwidth savings in a month. No small amount. Even though drupal.js and jquery.once.js are small we have over a 3 Meg savings in bandwidth for just these two files at 10,000 downloads a month.

Round Trips Saved

The real fun starts to appear when we look round trip requests. To understand why this matters I would first suggest reading TCP and the Lower Bound of Web Performance by Steve Souders where he talks about connections, TCP slow-start, and how it matters. To put it way to simply, when a browser fetches a file from a server it does it in multiple round trip requests. Each of these round trips has a delay as communications travel between a browser and a server. On mobile networks this delay can be several times that of a wired connection for each of the round trip requests. So, to speed up performance for an individual user it can be helpful to remove unneeded requests.

With JavaScript files in the head of a page this can be extra important. JavaScript files can block the rest of the assets in a page from being downloaded. Drupal puts these assets in the head of a page. Saving time on these files means the rest of the assets in a page start downloading sooner.

To download the original uncompressed set of files would take 10 round trip requests. To downloaded the gzipped version would take 6 round trip requests. By minifying the JavaScript in addition to gzip compression we save 2 more round trip requests to get the same functionality.

Extending This To All The JavaScript

Here I've only touched on the base JavaScript included in Drupal. Now lets extend this to ajax.js, form.js, jQuery UI, and all the other JavaScript in Drupal. By minifying files we can reduce the bandwidth and save on round trip requests giving our users a performance improvement without sacrificing functionality.

To do one last test I took an aggregated from from drupal.org. This file already includes a minified version of jquery.js so the improvements we will see are from JavaScript files bundled with Drupal and contributed modules. The sizes were:

Original: 156297 bytes
Minified: 97984 bytes
Gzipped: 53810 bytes
Min + Gzip: 37698 bytes

This produces a savings of:

15.7 Kb per download.
1 round trip per user.

Minifying files can have a real world performance difference. There is a reason jQuery calls the minified version the production version and the original source the development version. If you want to start using minified Drupal core JavaScript files now checkout the Speedy module.

Update: I wanted to note that I used UglifyJS to minify the files. This is the tool jQuery uses.

Introducting the Speedy Module

2012-03-21T00:00:00-04:00

Website front end performance is important. Better than 85% of the time it takes for a page to load occurs in the front end. One of the low barrier to entry ways to speed up a site is to make sure the JavaScript sent to browsers is minified. This is considered a standard practice with other common platforms, like Sharepoint and Wordpress, already doing this. The Speedy Module brings minified JavaScript files to Drupal core.

What Do Minified Files Give Us?

An example can help us see where the gains are. A minified version of drupal.js is 24% the size of the original. That means you have 76% less to send to users.

This happens because a minified JavaScript has most comments remove, whitespace removed, and can even transform functions to make them smaller. One of the simplest function alterations is to rename safe variables to single letter names.

What About GZip Compression?

I've been told that minified files aren't necessary if you use gzip compression. This isn't really the case. When you compress with gzip (or Deflate) everything is compressed. That includes the comments in the code and the spaces used for indentation. Then in the browser (even mobile browser) all if this is ungzipped. Minified JavaScript should be used along with gzip compression to get the best of both worlds.

How The Speedy Module Works

Some of the JavaScript libraries used in Drupal core are already minified. But, most of the common Drupal specific JavaScript is not minified. The Speed module contains minified versions of these core files and swaps out the core one for the minified version. This files are handled per Drupal version so alterations to core JavaScript are taken into account.

If you want a simple way to speed up your websites consider installing Speedy.

Drupal, HP Cloud, and PHP

2012-03-20T00:00:00-04:00

Cloud computing is something a lot of Drupal developers are interested in these days. As someone who works on the cloud, works for a cloud provider, and does a lot of Drupal work I want to see Drupal and HP Cloud (my cloud of choice) easily work together. So, we are releasing PHP Bindings and a Drupal module that uses them to bring the HP Cloud to Drupal.

Language Bindings

At the core of enabling Drupal to work with HP Cloud is a PHP library to do the heavy lifting. In the initial release of the library the functionality for identity services (authentication), object storage, and CDN for object storage are available. More features will come to this library over time.

For the technical minded this library:

Is PHP 5.3+ based.
Uses PSR-0 Naming.
Provides a low level API and stream wrappers.
Works with Composer.
Has a ton of documentation.

Drupal Support

Imagine mounting public or private files from Object Storage right into Drupal. With all the public files being served out of a CDN. Or, automating backups to Object Storage. The HPCloud Drupal module enables this. This module ties HP Cloud services right into the core of Drupal.

If you're a Backup and Migrate user you can use the module to backup right to Object Storage.

Kudos to Matt Butcher

Matt Butcher wrote the code powering both the language bindings and Drupal module. The code is a great example of well documented, tested, and well written code. Kudos to Matt.

Full Discosure: I work for HP Cloud in the developer experience department.

Learning From Verizon Wireless Experience Failures

2012-03-13T00:00:00-04:00

Providing good customer service is important. Providing secure well thought out technical solutions when you deal with someone finances is important. So, when I experienced repeated serious failures using Verizon Wireless I decided to take it as a learning and actionable experience. If you're curious what I learned keep reading.

Lesson 1: Provide An Upgrade Path To Changes

The serious of problems I had started because my username stopped working. When I set it up I was allowed to have special characters in the username. At some point they stopped allowing special characters but didn't provide an upgrade path for users with special characters in their usernames. They simply stopped working.

The lesson is to provide an upgrade path when you change your data structure.

Lesson 2: Prepare Your Customer Service Agents

When I had the problem I connected with an online customer service person. This was a chat window right on the website. So, when I was asked to reset my username through the site by this agent I was surprised. This isn't a feature of the site. Shouldn't the customer service agent have known how they website works?

Lesson 3: Don't Use Social Security Numbers

When I connected with a phone representative I was asked about an account password. This is different from the online password and one I setup years ago so I couldn't recall. It's not like I call in often. I've made all my changes via the web for years. When I didn't remember this password the representative wanted the last 4 digits of my social security number. This is a problem.

To put some background to this, I used to work in a call center. I know a number of others who have worked in call centers. I'm aware of situations where account information was stolen and used for bad things. Because of this history I don't like reps having easy access to any part of my social security number. Verizon uses this as a fall back when you don't have your password.

A lesson is to have other information on the account you can use to verify a user. This could be payment dates (a bank I have uses deposit dates and amounts) or numerous other pieces of information they collect.

Lesson 4: Make Customers Happy

By the time I had gotten through all of this I was not really happy. Unfortunately, without my password or part of my social security number I wasn't allowed to talk to a manager. To get this all taken care of I might have given the last 4 of my soc to a manager who listened to my complaint. Instead I was told I couldn't talk to a manager without it.

Managers or supervisors should be equipped to make happy customers and deal with problems. Sometimes they just need someone to listen. If unhappy customers aren't able to connect with people who have training like this you end up with blog posts like this floating around.

A simple lesson, enable representatives to make customers happy.

Lesson 5: Use Secure Passwords

When I was finally helped I was told I needed to have my account deleted and to create a new one. There was no way to change the username even if they could somehow get me logged in (this could be a failure but I'll keep going). While registering for a new account I was really limited on my password options. No special characters and no more than 20 characters. This makes me really suspect they are storing the password rather than a salted hash.

The lesson here is to be secure with passwords. I should be able to have a 64 character random password if I wanted. I'd even appreciate a password strength meter.

Lesson 6: Don't Have WTF Moments

When this whole process started I was logging in with my phone number. For security reasons they wanted me to input my username. You are allowed to login with a username or phone number. The username was failing for special characters. After I created a new account I logged in with my phone number. This time there was nothing asking for my username. WTF.

When we build websites and web applications we need to think about users, their experiences, security, and caring for them. No matter how big or small an organization. On the bright side this was a reminder for me.

Understanding The Hard Stuff

2012-03-12T00:00:00-04:00

Building websites is easy right? You install Rails, Drupal, Symfony, or some other tool and just start coding (or site building). If you want to code from the ground up you can grab a book or one of the millions of tutorials online and off you go. But, what happens when a customer or boss asks about security, performance, or something else highly technical?

This topic came to the forefront of my mind with the recently published security issue in Drupal. In the time between the initial report and a response from the Drupal security team I investigated the issue, read the code in question, and even ran some tests. In my discussions with others I was finding a bit of FUD and people taking sides behind someones opinion rather than understanding (or discussing) the problem space.

This topic doesn't just relate to security issues. For instance, Google, Amazon, and others have found the time it takes for a page to display to a user matters. The faster it displays the more likely a user is to do more interactions (that includes shopping). Yet, how many web developers know the ins and outs of making pages faster? Where are the low hanging fruit in this discussion? How does this all work?

Lately, I've started to wonder if there is a distinction between the easy stuff getting sites and apps up and going versus the hard stuff in the form of security, performance, scalability, and so on. It can be easy to get into web development and start building sites. And, with everything else we have to do in our lives and jobs it can be easy to stop there. Instead, let me challenge you to dig into the hard stuff because it's not as hard as it may seem and there are payoffs.

Switching From Bash to Zsh

2012-03-05T00:00:00-05:00

As a long time shell user I've spent a lot of time using bash as my default shell. In most of the hosting environments I regularly use bash is the default shell. Because of this bash is what I've been using on my local development machine until I recently switched to Zsh and oh-my-zsh. This post outlines why I prefer Zsh and how I made the switch.

Why I Prefer Zsh

There are a few reasons that Zsh has really popped for me.

In the image above you can see my current Zsh config in action. This theme allows me to see details about my system and the current working directory. oh-my-zsh ships with a bunch of themes and you can create your own.
I love plugins. My CMS of choice has them. My text editor has them. Why shouldn't my terminal. Zsh has them and oh-my-zsh ships with a bunch of useful ones like git integration, github, osx, and so much more.
Because of it's compatibility with bash commands I only had to learn the new stuff such as how to use the plugins.

iTerm2

My development systems are Macs. The default terminal shipped with the operating system is nothing special. Instead of using the default terminal I use iTerm2. iTerm2 has tabs, multiple panes, and you can even configure it to operate like visor.

Installing oh-my-zsh and Customizing

Zsh is already installed on most modern operating systems and is already on OS X. Installing oh-my-zsh and setting zsh as the default shell is a simple one line install script in the readme documentationn.

Out of the box it won't look like the image of the shell above. I've switched themes from the default one.

The first step I did was to migrate my current bash customizations from my .bashrc and .bash_profile files to the new .zshrc file in my home directory.

The next step is to customize to your liking. First, there are a number of plugins, some of which are documented on the wiki. Plugins are enabled in the .zshrc.

# Which plugins would you like to load? (plugins can be found in ~/.oh-my-zsh/plugins/*)
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
plugins=(git git-flow osx github)

To make the display look the way you want you can choose or create a theme. A number of the themes that come with oh-my-zsh are pictured on the theme wiki. The one pictured above is the dstufft theme.

# Set name of the theme to load.
# Look in ~/.oh-my-zsh/themes/
# Optionally, if you set this to "random", it'll load a random theme each
# time that oh-my-zsh is loaded.
ZSH_THEME="dstufft"

Jekyll Deploy Scripts

2012-02-20T00:00:00-05:00

One of the goals I had when I converted my blog to Jekyll was to speed up front end performance. To do this I wanted to minify the CSS and HTML files I deployed (I'm already manually handling the minification of the JS files). Luckly, grenadesandwich.com by Steven Merrill was already doing this and provided a good outline. I'm now able to use the command make deploy to generate the site, minify everything, and deploy it to the server.

To do all of this I take advantage of yuicompressor, htmlcompressor, make, and rsync. While make and rsync are installed on the system the Java jar files for yuicompressor and htmlcompressor are in a _build directory within my Jekyll site.

At the root of the site I have a Makefile containing the code snippet below (with some modifications for paths).

server:
  @@jekyll --server

render:
  @@echo "Building the site..."
  @@jekyll --no-future
  
minify:
  @@echo "Minifying the CSS..."
  @@java -jar _build/yuicompressor.jar --verbose --type css -o _site/path/to/style.css _site/path/to/style.css
  @@echo "Minifying the HTML..."
  @@java -jar _build/htmlcompressor.jar -r --type html -o _site _site

build: render minify

deploy: build
  @@echo 'Deploying site.'
  @@rsync -avq --delete-after _site/ sshAlias:/path/to/site/html

.PHONY: server render build minify deploy

You can get this as a gist.

The path to the css file and the location where rsync places the files are customized for my site and you should update those as well.

LossLess Theme Image Optimization with ImageOptim

2012-02-13T00:00:00-05:00

I used to think that exporting images from Photoshop for web and devices produced optimized images ready to use in my production sites. Sure, there are optimization techniques you can use for PNGs and JPEGs that I would need to incorporate. But, since Photoshop is considered one of the best imagine editing programs on the market I expected it would do a fantastic job exporting images.

I've since learned there are a number of techniques for producing smaller file sizes without compromising quality that Photoshop doesn't use. For example, 8-bit PNGs with alpha transparency. On the advice of Nathan Smith I've started using ImageOptim to losslessly compress my theme images.

Why Smaller Images Matter

If you aren't sure why using smaller images is better here are a couple reasons to consider.

Smaller files are less to transfer for the sender and receiver. This means using less bandwidth to serve the site and the receiving browser receives the site faster to load it faster.
Smaller images take up less room in memory. With the rise in use of mobile devices with different memory constraints, using memory more efficiently is a good thing.

ImageOptim

Using ImageOptim is simple. You just drag and drop images onto ImageOptim or it's icon on the dock and it does the rest. Under the tools menu option you can select which tools are used to do the optimizations.

If You're Not On A Mac

ImageOptim is a mac program which is what I use for development. If you are on Windows of Linux and are looking for a tool consider the web based tool Smush.it. Other than this I don't really have a tool I can recommend. If you know of one please offer it up in the comments.