Navigating a remote file system in order to find the file you want to edit can be cumbersome. Many text editors and IDEs have features that attempt to mitigate this pain, but I have never found one that works reliably and is comparable to the speed of local development. Saving a file over a network can introduce delay that adds up to a significant amount of lost productivity over time. File navigation aside, nothing beats the speed of working locally. In order to work on your local machine, you have to install all of the packages that the team is using, and each member has to do so as well. This approach introduces inconsistency. As developers, we seek consistency, speed, and, as much as we love the Internet, we would like to be able to work without it sometimes. So, what do we do? We build a machine within a machine that works on all platforms and ensures homogeneity across development environments.

Thanks to advances in virtualization technology and the open source community, a number of free tools exist that one can leverage to facilitate providing consistent virtual machines to your team.

VirtualBox, Vagrant, and Puppet

There are a few apps that are required in order to get started. VirtualBox is hardware virtualization software that allows you to run another operating system on your local machine. Vagrant is a set of scripts that allow you to easily manipulate those virtual machines from the command line. Puppet allows you to define what configuration changes should be made to the machine to have it operate how you would like.

Install VirtualBox

Download and install VirtualBox. You are now ready to install any operating system and run it locally.

Install Vagrant

Vagrant is a command line tool for building and distributing virtualized development environments with VirtualBox. Executing only a couple of commands, you can download and provision a virtual machine which matches that of your team.

Ruby and its package manager RubyGems are required to get going with Vagrant. There are detailed instructions by platform to help.

Once Ruby and RubyGems are installed, execute the following command on your local machine to ensure that your packages are up-to-date.

gem update --system

To install Vagrant, issue the following command on your local machine:

gem install vagrant

Configure Vagrant

Assuming that everything has gone well, configuring Vagrant should be easy. I have created a sample Vagrantfile and base Puppet configuration in order to provision a machine with Apache, MySQL, PHP, MySQL, Xdebug, PEAR, PHP CodeSniffer, and PHPUnit. You can clone my repository.

Once you have cloned the sample repository, it is time to install the Vagrant base image. The following command will download the box via the Internet and cache it. It is a 300 MB download, but you should only have to do it once.

The next step is to modify your Vagrantfile. Its configuration will forward ports from your local machine to the VM, mount a local folder of your choice within your VM, and initialize Puppet as your default provisioner. I have already done this for you.

There are a couple important configuration directives in the Vagrantfile. Let’s take a look.

# This enables Puppet as the default provisioner
config.vm.provision :puppet

# For verbose output, and you wish to do a dry run, substitute this line
config.vm.provision :puppet, :options => "--verbose --debug"

Once provisioning is complete, you will access the LAMP environment from your browser via http://localhost:8080/. The following line is the magic that accomplishes this:

Being that this entire exercise exists in order to work with local files that are served by the VM, we will need to share a local directory to the machine. The following line mounts the folder named www in your home directory to Apache’s DocumentRoot/var/www) on the virtual machine.

config.vm.share_folder "www", "/var/www", "~/www"

Whether you’ve elected to leave this configuration as-is or modify it, you can boot the virtual machine and begin provisioning it with the following command on your local machine. Make sure you are in the directory containing the Vagrantfile.

vagrant up

(If you get a stack dump and a permission error, an explanation is available.)

Vagrant boots the base image with VirtualBox, then executes Puppet with the configuration located in manifests/base.pp. I have commented the lines in the base.pp file for you to better understand their function. Also, I have kept the configuration deliberately simple. More advanced configuration will require that you create modules that can be inherited, included, parameterized, &c.

For reference, here is more documentation on Puppet directives.

Many developers have also written and shared Puppet scripts with the world. Here is a more complicated LAMP setup.

Once the machine is up and running, if you wish to SSH to the machine, you can do so like this:

vagrant ssh

In order to SSH to the machine without using Vagrant (for example, when creating a tunnel to use a GUI MySQL client), use the longer syntax:

ssh vagrant@localhost -p 2222 (password: vagrant)

There are very few reasons that you should have to configure the virtual machine from within itself. Connecting to the machine in this way should be limited to running environment-dependent command line tools such as PHPUnit and MySQL.

Distributing and Provisioning with Puppet

Puppet is the true hero here. Once a virtual machine is distributed, Puppet makes it easy to keep the configuration of machines synchronized. Think of your virtual machine as disposable. You should be able to destroy and recreate it without losing functionality. Any configuration changes that need to be made to the machine should be propagated to the Puppet configuration file and distributed via a Git repository or a Puppet server (puppetmaster). Remember, the key is for everyone to have the same environment, and to do that, no custom modifications should be made to the virtual machine without sharing your changes in configuration via the Puppet file.

If you have modified base.pp, you can easily reprovision the machine as follows:

vagrant reload

You can also completely destroy the machine and begin anew:

vagrant destroy
vagrant up

If you just want to suspend your virtual machine for later:

vagrant suspend
vagrant resume

Once you have configured a box, you can also package it up and distribute a binary version using Vagrant’s package command. I prefer creating a Git repository with many folders containing different configurations. For example, you could have a stable LAMP environment in one folder, and another containing a script to install the new PHP 5.4 release candidate. Yet another one could contain a Ruby or Python environment. Switching between them is as simple as suspending one machine and booting up another.

Developing with a Consistent, Local Environment

Now you should be able to easily modify files within your www directory on your local machine, reload your browser at http://localhost:8080/, and see the results.

This configuration serves as a great development environment with which to begin. Each app will have its own set of dependencies. You can even distribute the configuration of the virtual machine within the app’s repository to serve as an explicit definition of the dependencies that it has. With this, you could configure your continuous integration server to deploy a machine, provision it, install the app, run its tests, and report back to you.

There is an ultimate level of configurability and flexibility with a setup like this. Above all, though, your team will now be able to work faster and more efficiently, and you won’t end up in a situation where something works differently in one environment from the next. I hope you find a way to use these tools for your own sanity.

Developer Gift

I can think of no better gift to give a developer than the Phantom II RC helicopter. When your codes get you down, and you just need to let off some steam, you can dive bomb your housemates or family members with this easily-manueverable craft. It’s reasonably durable and cheaper than other models — you can get two of them for $25.

The BrowserID Solution

You will be surprised at how easy it is to implement BrowserID. Are you ready? Check out our instructions. Go on, I’ll wait right here.

Not bad, right? It’s super easy to use, too! Check out the user flow. Did I mention that it works on Firefox, Chrome, Safari, Opera, and IE — even on iOS and Android?

You may have noticed that the flow is centered around choosing an email address to sign in with. That’s no accident. Email as a login identitier already makes sense to users and sites alike, which is no easy feat. Because email is inherently distributed, BrowserID can enable any email provider to act as an identity provider. For now, though, browserid.org verifies your email by sending you a message (a common pattern on the Web).

Of course, you want more than just an email, and in early 2012, the BrowserID team will start to add experimental features to get at least a name and an image. That’s just the tip of the iceberg. The goal is to enable rich APIs to the user’s identity.

We Need You!

The BrowserID team is very eager to get your feedback. Try it out on your own sites! Instead of coding it yourself, you can trivially add support via plugins on WordPress or Drupal. You have no excuse; give it a whirl and let us know what you think on our mailing list.

We’re listening for your feedback. BrowserID is easy to implement directly, and it’s even easier to add support via plugins on WordPress or Drupal. Improve your users’ sign-in experience; try out BrowserID, keep up with our blog, and let us know what you think.

The Web needs this. Let’s make it happen.

Developer Gift

I really like my SousVide Supreme Demi. Sous vide cooking is really fun, and there’s a lot to try — highly recommended for anyone into food hacking.

I had a more DIY sous vide setup before, but I switched to the SousVide Supreme Demi recently, and I like it better. It’s an all-in-one unit that looks good on the counter, so I can leave it out permanently without it looking like I’m running a science experiment.

You haven’t had scrambled eggs until you’ve had sous vide scrambled eggs. :-)

On the flipside, our product Perch is a small content management system that aims to make it easy for non-developers to deploy a site. In this situation, we have to try and stay out of our customers’ markup. In doing so, we’ve learned a lot about how to keep out of the front-end code while still enabling powerful functionality.

Don’t Assume a Flavor of HTML

Currently, people are using one of several flavors of markup. You might come across front-end developers using HTML4 (which allows unclosed tags, unquoted attributes, and upper or lowercase tags), XHTML (which requires a well-formed document, quoted attributes, closed tags, and must be lowercase), and, increasingly, HTML5 (which can be written HTML or XML style).

This variety of markup means that if your PHP is outputting any markup, it may well invalidate the documents created by the front-end developers. If they are good front-end developers, this is likely to irritate them. It will also make their job harder, as each time they validate a document, they will see the errors for the incorrect markup style, making it hard to see errors they may have introduced.

Quoted attributes are optional in HTML4 and HTML5 when written in an HTML manner. However, they are valid in HTML4, XHTML, and HTML5 (whichever way you code it). So, ensuring that you quote any attributes you must output (for example when displaying form elements) will mean they are valid for all DOCTYPES. Similarly, XHTML and XML require that elements are correctly nested and that tags are lowercase, and while these are not requirements for HTML, ensuring things are correctly nested, tags are lowercase, and that the document is “well formed” will be valid across all flavors.

That really just leaves the thorny issue of self-closing elements. Elements that have a separate closing tag — for example paragraphs or list items — are optionally closed in HTML DOCTYPES, but closing them is valid across all flavors. However, self-closing elements (for example, images <img> and line breaks <br>) are invalid in HTML if they have the closing forward slash, and invalid in XHTML without. If you need to output these elements, which include input elements for forms as well as images, you should ideally allow the front-end developers to set their preference as to which they use.

In Perch, we have a number of options that can be set by the designer implementing the CMS. These include whether tags should be closed and using HTML5 so we can utilize the new HTML5 form elements if that is the case. We can then safely output form elements using the correct syntax from our templating engine.

Output Single Elements Only

Even when using a templating engine, there are times when you might have to output some markup. If this is the case, a good rule to follow is that you should only output a single element at a time. This lets the front-end developers wrap that markup in some other element if needed. For example, if you are generating a list and need to output the li elements, avoid also outputting the list itself. Find a way to let the front-end developer decide whether this should be an ordered or unordered list.

Avoid Inline Styles

Try to avoid the temptation to output any inline CSS. Due to the rules of the cascade, inline styles take precedence over those set in the stylesheet, therefore your styles will overwrite those created by the front-end developer. This will displease them greatly, and it also means they need to contact you to get those inline styles changed should the need arise. If you need to add something to show that a particular item is distinct from others, it would be preferable that you added a class to the element and tell the front-end developer what that is and in which circumstance it will appear. That way the front-end developer keeps control over how the element looks.

Allow HTML Class Attributes to Be Passed

If you need to generate markup such as an image element, let the front-end developer pass a string that will be added to the element as an HTML class attribute. This helps them add their required CSS without needing to add additional markup around the element and putting a class on that.

Use Familiar Syntax for Templating

Many web designers and front-end developers are unfamiliar with PHP, so when implementing any kind of templating that will be used by non-PHP developers, it can make all the difference to use a syntax that will seem friendly and familiar to them.

We do this in Perch by using XML tags within our templates. For our designer customers — many who have never installed a CMS before — these tags and attributes look like something they understand. To see how use of familiar syntax can promote use of a system you need only look at jQuery, which has become the JavaScript library of choice for so many designers due to its use of CSS selectors to target elements in the DOM. A familiar syntax goes a long way toward helping front-end developers and designers feel confident that they can work with the back end without breaking anything.

Let Front-End Developers Control Their Area of Expertise

Good front-end developers will be as concerned about the performance of their code as you are about the back end. For performance reasons, it is seen as good practice to include JavaScript at the bottom of the page rather than at the top. Ensuring that the developers have control over this sort of thing means they can place them there, if appropriate.

By handing control of the front-end development to front-end developers, you allow them to do their job to the best of their ability, unhampered by things appearing in their markup that they were not expecting. You also shift responsibility for that area of the site to them. If you generate reams of markup via PHP, then any time it needs to be changed, or if there is a problem with validation or any CSS issues, the request will come back to you.

Developer Gift

I don’t wear a lot of jewelry. However, I would definitely make an exception for the iNecklace:

“Sophisticated. Elegant. Open Source. The iNecklace is a gorgeously machined aluminum pendant with a subtle pulsating LED.”

There are also iCufflinks, if you are not the necklace wearing type.

It wasn’t so long ago that a closed development process was the norm. You offered up a finished product to the world, and people either liked it or moved on to the next. Users might send in some feature requests, but whether they’d end up in the next release was anybody’s guess. Projects were developed by a certain group, and it was up to them if they wanted to listen to the feedback from their users.

Then came social coding. Developers were given a new road to follow, one that opened up a whole new world of contribution and commmunity. Suddenly, apps became living, breathing things, begging to be shared. They reached out into the world, an extension of the developer, and asked to be accepted and improved. GitHub and its fellow code sharing sites made this revolution possible. They made sharing a few simple commands away and revolutionized the way development works.

I propose that 2011 be known as the Year of Social Development, a point in time where it has become so widely accepted to share code that there’s no excuse not to.

I won’t bore you with yet another introduction to GitHub, though; there are plenty of sites out there for that. Instead, I want to share just a few of the ways that you can use these same social development sites to give back to the projects you love.

First on the list is code contributions. This is, by far, the most powerful way you can give back to a project. GitHub and Bitbucket both call these pull requests — code improvements and commits created by another developer in an effort to make the project better. These can range from simple documentation changes to brand new features that have been on the TODO list for a long time. I highly encourage you to find a project that you love to use, see if they’ve opened up their source, and contribute back if you can. It’s a satisfying feeling seeing an acceptance message and seeing your commits included in future releases.

Next on the list is something that can make or break a project, it’s docmentation. Just now, I mentioned contributing to the documentation via a pull request. What I’m talking about now is a different sort of documentation, the kind found in manuals, example code, tutorials, and wikis. Find a piece of code that you don’t see a wiki page for and make it! If you found it useful enough to track down, chances are others will want to know about it, too.

Finally, if you don’t know enough about the project to contribute code, or there’s no way for you to update the documentation, there’s still a great way to give back — submit bugs. Too many developers find things that aren’t quite right and work around them. Next time you come across something like this, don’t code the other way; let the developers know there’s a problem. Unit testing and general usage will only reveal some bugs. Issues really show through in specific situations. Be sure to include as much detail as you can in the report; those make for happier debugging for the devs.

Even if you can’t contribute to a project via the code, documentation, or bug reporting, there’s always one more way you can offer support — send the developers a message letting them know how much you appreciate their hard work. Nothing feels better than someone halfway across the world telling you what you’ve made helped them.

Social development is here to stay. Get involved, contribute where you can, and maybe even start a project of your own!

Developer Gift

My developer gift suggestion is more for the up and coming geeks in your lives. I’m a huge fan of the stuff ThinkGeek sells and they have some great stuff for the kiddos. If you haven’t picked a gift for this holiday season may I suggest one of these bibs, a t-shirt like this or maybe even a ship of their own.

One of the great joys of creating software is solving problems. Not just academically solving problems, but really fixing things for people. Making other people’s lives better by enabling them to achieve more with less. The satisfaction gained by having another human being pleased to be using what you’ve created is what makes it all worthwhile. If that feedback loop is broken, you end up releasing code into a void like a machine churning out functionality to spec.

One thing you quickly find when you’ve reduced your role simply to one of coding to spec is that your default position for any request is “no.” To keep the project on time, on budget, and to prevent treading water, you have to stand firm against what could be seen as unnecessary changes and stupid requests from users who don’t really know what they’re talking about. It’s tiring saying “no,” and resentment toward those asking the questions quickly builds up.

Before you know it, work is not fun like it used to be.

The solution is easy. Stop coding for the spec, and start coding for the users again.

Functionality Is Not Enough

I think we’ve probably all been in a situation where users ask for something that already exists. With our small CMS, Perch, users often asked for the ability to reorder content items.

Now, of course, it was already perfectly possible to reorder items — there were options to set the content to order on any field, and in either direction. If the user needed an arbitrary order, they could add a new field, add a number to their items, and order by that. On paper, it was quite a powerful and flexible system.

Yet, users were still asking for a way to order content. The functionality was there, but users were either not finding it, not understanding it, or both. The spec was met, but the functionality was failing the users.

Remember, the spec is a bare minimum requirement. To do a really great job, it’s sometimes necessary to go beyond what the spec says and make something that really works for the people who have to use it.

Accept When Users Are Having Trouble

A big part of this is learning to accept when working code is broken. Sometimes the wet problems are the most difficult to solve, and as with all wet problems, the first step to solving them is admitting you have a problem. As a developer, it’s a really tough lesson to learn that a perfectly functioning, bug-free feature is broken if users don’t understand it.

What you’ve developed may meet the spec’s requirements enough to get sign-off, but we’re not working for the spec anymore. How do we go about getting user sign-off?

Sometimes the issue can be resolved with better user messaging, or by rethinking the interface, but other times you just have to accept that users are having trouble, and you need to go back to the drawing board — and that’s okay. If what’s there doesn’t work for users, then it doesn’t work.

My number one recommendation in this situation is to simply listen to your users. Ask them what they’re trying to achieve. Much like Henry Ford’s debated “faster horses” quote, your users may come up with the wrong solutions, but their core requirement can often been distilled from those solutions. So ask them, and then listen carefully to the answer.

Really Great Developers Solve Problems

I went back to the drawing board with my content ordering and implemented a JavaScript drag and drop reordering interface. The core of what my users were asking for was a quick way to set an arbitrary content order. I’m not one to throw masses of JavaScript at every problem, but in this case, it seemed that drag and drop was a good way to achieve that goal.

Now, users can pick up a bit of content and easily drop it where they want it. Is the end result any different to the functionality we provided before? Absolutely not, but the process of getting there is more closely aligned to how users want to use the software.

Not only does this mean they’re not bugging me with requests for features that exist, it means that they’re able to achieve more with the software.

FAQs Are Bugs You Haven’t Fixed Yet

We have a policy of having no frequently asked questions. If users are asking about something with regularity, then it means we’re either not explaining things clearly, or the product can be improved.

With Perch, users needs to add their site’s domain and corresponding license key before they can log in to a new installation. If they missed all the instructions and failed to do so, they were presented with the following message:

“Sorry, your license key isn’t valid for this domain.”

While accurate, this message very rarely resulted in the user taking the correct course of action to resolve the problem. What they’d actually do is raise a support ticket saying that they couldn’t log in.

One way of addressing this would be to add an FAQ to our site that explains what to do when encountering the message. That would be a bandage on the wound — it would stem the bleeding but not alleviate the pain. The real solution was much simpler. We changed the error message:

“Sorry, your license key isn’t valid for this domain. Log into your Perch account and add the following as your live or testing domain: example.com”

With example.com being the domain the installation is running at. Following this simple change, nobody asks us this question anymore. I’m sure plenty of people still miss the setup instructions and see the error, but it’s no longer a problem for them — they just make the change and carry on without incident.

The very worst thing you can do with frequently asked questions is make a list of them on your web site. Frequently asked questions are bugs, and the only satisfactory answer to them is to fix the problem so the questions don’t get asked again.

Being a developer isn’t about providing tools; it’s about solving problems. If we want to enjoy what we do, make awesome things, and not feel burned out, then we need to find ways to solve problems that work really well for users. If you make great software, users will appreciate your work and you’ll feel proud about what you’ve achieved. That sounds like a good recipe for being happy in your work to me.

Developer Gift

Did you ever program a robotic turtle back in the 80s? Remember the fun of having code on the screen translate to movements of a physical device on the floor? Drawing lines on a big sheet of paper only has limited appeal, but there’s a grown-up equivalent with an open source electronics kit from Arduino. Give the gift of hardware hacking!

This is not a post about cloud computing and its benefits, but rather a refresher about understanding some basic rules of scalability and how to get started with building scalable apps while keeping in mind that most people and companies have limited budgets.

The concept of shared-nothing architectures is well known in the scalability community — if such a thing even exists — but it seems to be frequently forgotten when it comes to building web apps.

Scalability

Before we dive into scalability theory, a reminder is in order; fast apps and infrastructures don’t necessarily scale.

Although scalability is a complex and arduous concept to explain, the generally-accepted meaning for load scalability is the ability for a system to handle a growing amount of work in a capable manner. In other words, it’s the ability to handle more load by adding more computing power. Will you be able to handle more users if you add more servers to your cluster?

Load scalability is about the ability to adjust and adapt. As Darwin once said (or meant to say):

“It is not the fastest web app that survives peaks and popularity; it is the one that is most adaptable to change.”

Share Nothing

For years, the PHP world (most of it) was deploying to a single server that contained the web server, the databases, the uploaded files, the session files, &c.

This outdated setup looks like this:

This was fine until web sites started going down because of a mention on Digg or Slashdot (for those of you who remember Slashdot). Apps were fast, but they couldn’t surpass a certain threshold of users.

This is about the time the concept of shared-nothing architectures began to take hold. Infrastructures are now decoupled, and every component can be easily replaced. This improved setup looks like this:

By sharing nothing, every server that powers your app becomes egomaniacal and does not care about the rest of the infrastructure. Like modular object-oriented code, parts of your infrastructure become independent. For instance, your web server should no longer save sessions in local files, because at any given point in time, the number of web servers can change.

A system that is tightly linked to its filesystem for file uploads, databases, sessions, &c. is not scalable. Luckily, in the PHP world, we have quite a few tools to help us attain a high degree of selfishness.

Sessions

Storing sessions on a single filesystem means that when the system scales and adds or removes servers, some sessions will be lost. There are a few solutions to the problem.

Memcached

Memcached has been used for many years in the PHP world, and it is a great way to store objects and sessions across a cluster of servers. For an even more scalable Memcached infrastructure, I recommend Membase, which is open source and provides elasticity as well as persistence for Memcached.

Once you set up your Memcached infrastructure (and the PHP extension), you simply modify your session handler to point to your Memcached server (or pool) as follows:

<?php
    ini_set('session.save_handler', 'memcached');
    ini_set('session.save_path', '1.2.3.4:11211');

Many people dislike using Memcached for session storage, because the data is ephemeral, so if your server dies, all of your sessions are going to disappear, and your users will be logged out.

Membase avoids this problem and is fully compliant with the Memcached protocol.

Redis

An alternative to distributing sessions persistently across clusters of computers is to use Redis and the Redis PHP extension.

Redis is an advanced key-value store (data structure server) that can contain strings, hashes, lists, sets, and sorted sets. Redis can be naturally replicated, and, despite being an in-memory system, it can also fall back to storing data on disk.

After installing the aforementioned PHP extension, you have modify your session handler to use Redis:

<?php
ini_set('session.save_handler', 'redis');
ini_set('session.save_path', '1.2.3.4:6379');

In the event where you want to use a cluster of Redis instances, you can specify multiple servers:

<?php
ini_set('session.save_handler', 'redis');
ini_set('session.save_path', 'tcp://1.2.3.4:6379?weight=1, tcp://4.3.2.1:6379?weight=2');

SessionHandler

PHP 5.4 has a new class called SessionHandler. This new class lets you write a custom object-oriented session handler.

Saving your sessions to Memcached using SessionHandler would look something like this:

<?php
class MemSession extends SessionHandler {

    public function read($key) {
        return parent::read(sha1($key));
    }

    public function write($key, $value) {
        return parent::write(sha1($key), $value);
    }
}

ini_set('session.save_handler', 'memcached');
ini_set('session.save_path', '1.2.3.4:11211');
session_set_save_handler(new MemSession);

Of course, sessions can be, and often are, saved to a database. Using SessionHandler, it is now easier than ever to do so.

Distributed, Redundant File Storage

Another limitation of most traditional web apps is saving files to a local filesystem.

If an app scales by adding more servers, saving files to the filesystem has the same limitations as saving sessions to the filesystem. The presence of files will be inconsistent across nodes, and your users will be frustrated.

A solution to this problem is to use systems that are built to store, distribute, and replicate files across multiple regions. A good example is Amazon S3 which can be easily supported with Zend_Service_Amazon_S3 or PEAR::Services_Amazon_S3.

For Symfony2, I use a bundle I made named symfony2-s3streambundle. It lets you easily write logs and similar files directly to Amazon S3 using Monolog.

Once the bundle is installed, and your Amazon S3 bucket is created, all you have to do is modify the app/config/config_prod.yml file to add the following:

monolog:
    handlers:
        nested:
            type:  stream
            path:  s3://your-bucket/%kernel.environment%.log
            level: debug

There are other ways to distribute files, such as building your own filesystem cluster using GlusterFS, NFS, or any other variant of a distributed and eventually consistent file-storage system. I usually recommend using Amazon S3, because it is fast, reliable, highly available, and widely supported.

Conclusion

With cloud computing becoming more standardized, and platforms such as Orchestra providing scalable architectures at the click of a mouse, application architecture is becoming more important, while infrastructure architecture is becoming simpler and more reliable.

There are more things to ponder when building a truly scalable architecture, such as message queuing, database replication, automated backups, self-healing, elasticity, and locality. PHP has many extensions and tools to help your app scale.

On a related note, if you are unaware of the Gearman and ZeroMQ projects, I recommend learning more about them and considering whether they might fit into your current or next project.

Developer Gift

Einstein once said, “God does not play dice with the world.” Unfortunately for him, he may have been a few years too early for the very cute mathematician’s dice, and they might have changed his mind. These are a great gift for adults and children alike. Happy Christmas!

The early days of commerce on the Web were difficult, to say the least. If you wanted a comprehensive solution for your business, you needed a merchant account specifically for online transactions, and you needed to process transactions through a gateway that could connect to your merchant account. On top of that, you’d have to estimate sales tax and shipping rates, and handle fulfillment and inventory on your own.

Solutions are easier now for small businesses and the developers who work with them. Simple APIs exist for use at affordable rates to handle transaction processing, sales tax calculation, and fulfillment.

Transact

In the early days of the Web, if you rolled your own cart, you likely used services like Authorize.Net and VeriSign PayFlow Pro (now PayPal). For developers, these weren’t easy to integrate with. For small businesses, the online merchant account was a headache, and SSL certificates were expensive, so many opted to redirect their carts to a third party, where the transaction would take place and then redirect back to the business’s site. This did not provide a very good user experience.

Stripe is a new service aimed at developers and focused on simplifying transactions and integrations, allowing you to focus on user experience. They have a simple API and do not require a merchant account or gateway. In a sense, they are the gateway. They provide all the features you’d expect, including storing credit card data, offering the ability to set up subscription payments, and paying out directly to your bank account, but there’s no hassle. You simply accept the data from your own forms and send it to them over SSL. (You still need your own SSL certificate, but those are much cheaper these days.)

Calculate

Many businesses choose to avoid the hassle of calculating sales tax on the front end, and bake it into the cost of the good or service, but this means there’s hassle on the accounting side at the end of the year. Your accountant must then determine where you have nexus (a fancy term for regions where you are required to collect sales tax), examine the shipping address of everyone who bought from you to see if they need to calculate sales tax, and, if needed, calculate the sales tax for that jurisdiction — or guess at it and (hopefully) overpay.

SpeedTax (recently acquired by CCH) provides a service with an easy-to-use API to reduce and eliminate this hassle. You can set up your SKUs within the SpeedTax system and categorize them so that SpeedTax will know how to calculate the sales tax according to each jurisdiction’s laws. Then, you specify where you have nexus. When you send a request to the SpeedTax API, you include the SKU and the shipping address of the customer. SpeedTax takes care of the rest and gives you back the appropriate sales tax that you should collect for that jurisdiction.

Fulfill

Fulfillment includes three different processes, none of which are easy for a small business to perform alone. The first is calculating shipping fees. This can be done online, but you have to make decisions about which provider to use (USPS, UPS, FedEx, DHL, &c.) and integrate with their API (APIs if you choose multiple providers). The second is determining whether you have the inventory required to fulfill the order. The third is actually picking, packing, and shipping the product. Most small businesses do this on their own at first, but if you want to scale your business, you’ll need to outsource this, so you can focus on running your business instead of shipping your product.

Shipwire provides all three of these services. Their APIs calculate shipping rates for a variety of vendors (which you can provide to your users as options), and you can store inventory within their global warehouses, so you can query their APIs to determine whether you can actually fulfill an order, based on inventory available. Furthermore, since you store your product in Shipwire’s warehouses, they will pick, pack, and ship your product based on API calls you make to their system.

Seize the Opportunity

When you or your clients are out and about, either at trade shows and conventions or just through the course of normal, everyday life, accepting payments whenever and wherever the opportunity arises is important.

Square provides a credit card reader and app for iOS and Android devices that allows you to swipe and accept credit cards virtually anywhere. They don’t provide any APIs, so they’re not an application integration tool, but if you have clients who need to accept payments online, then they likely to need to accept payments offline, so suggest this solution to improve their business. Square is similar to Stripe in that they don’t require a merchant account, and they deposit directly to your bank account. What’s best is that the card reader and app are free!

Simplify

I’d be amiss if I didn’t mention that your mileage may vary. Depending upon your transaction volume and customer base, you may need to consider PCI compliance or TRUSTe certification, among other compliance and regulatory considerations.

These are also not the only options out there. There are many, many products and solutions; I have only picked one from each category I’ve identified as important. I think these are good for small businesses; they’re easy to integrate with, and they’re affordable. They will simplify things for you or your clients. You may want to consider other options, though. For example, PayPal, Amazon, and Google provide excellent transaction alternatives that many of your customers will trust and appreciate.

Finally, I don’t want to close without promoting a simple solution provided by one of my Nashville friends, Luke Stokes. FoxyCart aims to simplify the checkout process for small businesses by providing a secure (PCI and Safe Harbor compliant) and fully-customizable (CSS and HTML) cart experience, while eliminating the need to host or develop your own cart or maintain your own SSL certificates. They provide features such as coupon codes and shipping rates.

Developer Gift

It’s no secret that I love beer. The other night, Luke Stokes asked me what my favorite beer was. It’s no small task for me to answer this. I like malty, I like hoppy, and I like sour. I really love Yazoo Fortuitous, but it’s hard to find. So, for those who like malty beers, I recommend the gift of Clutch from New Belgium, and for those who like hoppy beers, I recommend Le Terroir, also from New Belgium. Both are sour. Both are delicious.

At Etsy, we have a variety of command line utilities that are all written in PHP. Having our web site and command line utilities in the same language means easier mental context switches between the two environments. It also lets us reuse web code from the command line, particularly for data access or asynchronous processing.

Arguments

Just like web pages, command line scripts usually do more interesting things when they take arguments. The command line arguments can be accessed with the $argc and $argv globals. $argc contains the number of arguments, and $argv holds an array that contains each argument; $argv[0] is always the filename of the PHP script being executed.

For simple options, using $argv directly is easy enough. Consider using getopt() or this Args class if you want to allow more complex options and flags like: ./script -n -g --near 4 --orange rectangle

Command line scripts often have different memory requirements than web requests. If you find your script running out of memory, don’t hesitate to explicitly increase the memory limit: ini_set('memory_limit', '256M');

`Backticks`

The first handy scripting tool is the execution operator known as backticks (``). As with shell scripting, backticks will execute a given command and return whatever was written to standard output.

One place where we make extensive use of backticks is in our builda script, which we use to compile CSS and JavaScript as part of our deployment process. Rather than using PHP functions to navigate files and directories, we get everything with one shell command, as in the following example.

$files = explode("\n", trim(`find $js_dir -type f`));

Note that the backticks expand variables like $js_dir just as with double quoted strings. Remember that any input from strangers should be escaped with escapeshellarg() or escapeshellcmd() before sending it to the shell.

Builda was originally written in “not-PHP” but stagnated, because not everyone at Etsy was comfortable enough with that language to jump in and scratch their own itch. Since being rewritten six months ago in PHP, which all engineers know from working with the web code, seven different engineers have contributed to making the tool better.

Standard Error

print and echo are easy ways to write to stdout. But, sometimes, command line programs want to write to standard error. For quick writes to stderr, try the following:

file_put_contents('php://stderr', "error message\n");

Alternatively, you can open stderr like a file and write to it with the standard file functions.

$err = fopen('php://stderr', 'w');
fwrite($err, "another error message\n");

Pipes

In cases where you require more interactivity with child processes, proc_open() is often the right tool. It launches a process and opens pipes so that the two processes can communicate with each other.

As an example for how you might use proc_open, consider the case of loading data into sharded databases. Originally, we had only three shards, and it was easy to manually run one script for each shard. As the number of shards grew, this became more time consuming, so we wrote another script for loading data to all the shards at once.

The new script is actually two scripts. First, the child process, shard_load, is very simple. It opens a connection to a single shard, reads input from stdin and inserts rows into the database. By breaking the job into two scripts, each task is kept simple and can be tested independently.

The main script, load_all_shards, is equally simple. It just reads a stream of input from stdin, opens a child process for each shard, and pipes rows of input to the appropriate child process. The pipes opened by proc_open can be read from and written to with functions like fgets() and fwrite(), just like any other file handle.

do {
    $line = $fgets($stdin);
    list($shard_id,$row) = split("\t", $line, 2);
    fwrite($shards[$shard_id]->stdin, $row);
    $line = fgets($handle);
} while ($line);

All that needs to be done is to connect them with proc_open:

function openShard($shard, $tbl, $fields) {
    $desc = array(
            0 => array('pipe', 'r'),
            1 => array('pipe', 'w'),
        );
    $pipes = array();
    $cmd = "./shard_load $shard $tbl $fields";
    $proc = proc_open($cmd, $desc, $pipes);
    return new ShardProc($shard, $proc, $pipes);
}

PHP Ubiquity

With easy access to the shell and interprocess communication with pipes, there’s no reason not to take advantage of existing engineering experience (and web code) and use PHP for utility and offline scripts as well.

Developer Gift

Have you ever looked at the inside of a hard drive? It’s round and very shiny and perfect for a clock face! Any of these clocks would make a great gift for your friendly neighborhood software developer.

In the Beginning, There Was Apache and CGI

And there was much rejoicing.

In 1994, Rasmus Lerdorf created the “Personal Home Page Tools,” a set of CGI binaries written in C. These tools looked little-to-nothing like the PHP we know today. Embedded in HTML comments, and using a syntax bearing no resemblance to C, they still contribute one critical principle to modern PHP; it just worked.

PHP is a language of convenience. It was built with the idea that anyone could toss together a few lines of code and have a working CGI script, without having to worry about the server interface, the cryptic syntax of Perl, or the pitfalls of C. It’s a great idea, in theory, and for the most part, it’s worked very well in practice.

Unfortunately, nothing’s perfect — PHP included. Over time, PHP has suffered everything from security failures to bad design decisions. Some of these problems were avoidable, but others weren’t.

Backward Compatibility Is a Female Dog

Backward Compatibility (or BC) is the bane of every library and app writer in existence. It stifles improvements, holds back innovation, promotes unsafe practices, frustrates users, and slows development. PHP, being a language intended for beginners, suffers from it even more than most.

When BC is broken, apps break. Operating system vendors who shipped the new and improved version of PHP tend to come under fire for having a broken system, even though they did nothing wrong. More often, the writers of the apps are vilified for not providing working software, despite having done nothing but follow the manual.

Sometimes, the source of the problem is correctly identified, and the PHP developers are berated for trying to make a better language. No matter who takes the blame, though, one thing remains constant; users rarely understand anything other than “it’s broken!” They don’t care whether the new version is better. The old one worked. They want it to keep working. It’s a reasonable expectation.

Unfortunately, with a programming language, it’s often impossible to meet that expectation without sacrificing features, safety, or speed — usually more than one of these.

PHP 4.4 was released to fix a bug that caused memory corruption when references were misued. The fix changed the internal API, forcing every extension module to be rebuilt. Unfortunately, rebuilding extensions can be an arduous process in some environments. Some extensions don’t come with source code. Others are ancient, and code that managed to struggle along finally stops compiling cleanly. Vendors (such as those who provide various flavors of Linux) who ship packages have to rebuild and test not only PHP itself, but also every extension they ship before pushing to their repositories.

The results of all this are twofold. First, almost everyone holds off on updating to the new version to avoid the work and cost involved in fixing the problems that arise, leaving them all running a version with publically disclosed memory corruption bugs. Second, PHP itself is discouraged from making similar changes in the future, lest adoption of compatible fixes be slowed. It only gets worse when the change breaks source compatibility, forcing app writers to change their code even after the vendors catch up.

All of this is bad enough when the change is essential for whatever reason. It’s far worse when the compatibility break is the result of bad choices or poor planning.

Innovations Are the Devil’s Playthings

There are several examples in PHP’s history of changes that were made by well-meaning, forward-thinking developers who were trying to make PHP better, only to be shouted down because of the trouble it would cause to implement them, or, worse, to actually make the change and suffer the chaos that ensued, because they didn’t realize how far the effects would reach.

A recent example is a change in the behavior of the is_a() function between versions 5.3.6 and 5.3.7. is_a() used to allow a string parameter as its first argument; it was changed so that it would call the autoloader when passed a string referring to a class that doesn’t exist. The new behavior was technically correct and consistent with the is_subclass_of() function, but calling into the autoloader when it previously hadn’t caused a great deal of working code to break. Several PEAR packages started throwing exceptions from the autoloader due to supposedly missing classes. Fixing these packages required adding an extra is_object() check to every place is_a() was used.

Unfortunately, by the time the scope of the problem was realized, and a solution was agreed upon, a further version in the 5.3 series, 5.3.8, had been released, with the new behavior intact. To revert the behavior at that point would have created a whole new BC break and necessitated a second round of code changes. By the time the situation was settled with a patch to the 5.4 tree and a reversion in 5.3.9, a CVE report had been entered for the new behavior, and several mailing list threads regarding poor testing coverage and lack of procedures for BC breakage had reached impressive length.

All of this could have been avoided if a unit test had caught the break, or a procedure regarding BC breaks had been in place to prevent the change. The original author of the change can’t be held responsible for fixing a bug and correcting inconsistent behavior, but in an environment where fixes and corrections can have such far-reaching consequences, blame tends to get assigned (that fortunately didn’t happen in this case), and people become less willing to fix things.

On a rather less excessive note, for a long time there have been complaints about the inconsistent naming of array functions. It would be quite nice if, for example, the [uka][r]sort() family of functions was instead named array_[uka][r]sort(), but while the new function names could be added, the old ones couldn’t be removed for a very long time — at least two major PHP versions. With that limitation in mind, it seems pointless to add the new names. So, the old ones just stay — the result of a design decision far in PHP’s past, reasonable at the time (shorter function names were easier to understand, remember, and use).

When the Porpoises Ask the Few Survivors What Went Wrong

If, in the past, someone had asked me what was wrong with PHP, I probably would have said something like “developer apathy.” In early 2009, I took the lead in moving PHP’s source code from the increasingly creaky CVS repo to a shiny new SVN repo. I found that individuals with specific knowledge, such as whether a particular module needed to be saved or not, were in plentiful supply, but it seemed to me as if there was a lack of people to help with the overall process.

Let history be my judge in that regard. It may have been that offers of help were made that I misunderstood or chose to ignore. If so, I apologize to all who made the effort and were rebuffed. I do, however, stand by my assertion of apathy as a problem; PHP 6’s failure is an equally good example.

Even if we accept that perception as valid, it’s certainly not so any longer. A great deal more is happening with PHP now at the end of 2011 than was going on in early 2009. If asked what the problem is today, I would say, “no design and no plan.”

PHP has always been an evolving, almost-organic language. It has been rewritten from the bottom up at least four times, with massive internal changes to the engine at least twice more. Through all these mutations, however, its external interface — the language itself — has remained quite similar for a long time. Nearly everything that can be pointed to as different between PHP 3 and PHP 5.4 is an addition or extension to the language, not a change in existing behavior. There are exceptions, such as the new object model, but by and large, a PHP coder looking at PHP 5 code will be able to make complete sense of PHP 3, and vice versa. All of these versions share one flaw: there is no single specification of the language!

External tokenizers have to be implemented by reading the Zend Engine’s re2c and Bison input files. Reimplementations of PHP have to refer constantly to Zend’s implementation to understand the quirks of the engine. The behavior of the language is inconsistent, and it often feels clunky. In particular, expressions do not reduce recursively to values as one might expect. (new SomeClass)->methodReturningClosureReturningArray()()[5] causes a parse error, for example, though a similar construct in Objective-C, [[[SomeClass alloc] init] methodReturningBlockReturningArray]()[5] works fine.

There are a variety of reasons why this behavior is part of the PHP language, but they boil down to two major points. First, there is no specification that says how the language should work; there’s nothing to compare against and say, “this is wrong” or “this is wrong.” Second, fixing issues like these in a complete and lasting form would necessitate a parser rewrite, and that means reimplementing the entire language differently. “BC break” doesn’t even begin to cover it.

I Can See Clearly Now

I’ve gone on at quite some length about PHP’s problems. I’ve mentioned some solutions to those problems, but I haven’t said much about what’s actually happening. So, here’s the situation, and it’s not nearly so bad as I may have made it sound.

BC breaks

A new release process was adopted in June of 2011, clarifying the timeline for releases, including the proper times for changes which break BC. This has also put PHP on a track for more regular releases in general, which is a significant help for vendors who bundle PHP.

Communication problems

PHP has historically had trouble with no one outside the core team knowing what was happening. In the last several months, there has been considerably more communication with OS vendors and others affected by changes and the release timeline. Some of them came to us, but in other cases we went to them.

Lack of specification and standardization

The lack of a language specification remains a significant issue, but at the very least, awareness of the problem has increased. An initiative to document the language behavior in a format such as EBNF has been suggested.

Unit tests ignored and broken

This particular problem, brought to the public eye by a major security bug in the 5.3.7 release which was caught — and ignored — by a unit test, was dealt with shortly thereafter. Since that time, a huge number of failing tests were fixed, and the release managers now pay a great deal more attention to the test suite.

Apathy

Developer apathy in the PHP community has largely disappeared. Participation and discussion are considerably improved from where they once were. Development is now active on the 5.4 branch, the development line that grew out of the PHP 6 effort, and it’s already in RC status as of this writing.

Undocumented, confusing engine API

Unfortunately, the cruft of the Zend Engine’s API remains a sticking point. The API is complicated, mostly undocumented, and completely unintuitive. zval reference management comes to mind. Efforts made to document the API have stalled time and again. The news isn’t all grim, though; an RFC is in discussion to completely separate the internal API and create a new, clean external API.

Bit rot (old code and features holding back new things)

The new release process eases this quite a bit; it’s now safe to say there will be a new version at some point which isn’t afraid to break BC. This follows from a simple and obvious fact: the people affected by BC breaks will now have warning that it will happen. Deprecation in point releases is a bad way to handle future changes. Giving the expectation of major changes at a defined point in the future is a good way, and that’s where PHP is headed.

No Unicode

The lack of Unicode support in PHP remains a serious problem, but at least we’re no longer nursing a dying animal (PHP 6) in the vain hopes of making it work when the entire surrounding situation has changed. This opens the door for new ideas on how to fix the issue. For more on PHP 6’s history, see this excellent set of slides by Andrei Zmievski.

A Little Nonsense Now and Then Is Relished by the Wisest Men

It’s safe to say PHP still has a long way to go to be the shining beacon of light we’d all like to see in a language, but it has withstood the test of time better than any other language of its kind, and it sees daily use across millions of servers. Nothing that’s wrong with PHP or its development is unsolvable, and tremendous progress has been made in the last year. No matter how dark I may have sounded during some parts of this article, I’m proud to be a part of PHP, and I hope I’ll continue in the future.

Developer Gift

A gift I recommend for any developer is a copy of any comprehensive book on hardware architecture and operating system design, with Inside the Machine and Operating Systems Design and Implementation being two excellent examples. I have long held the belief that any developer’s skills — whether they’re writing raw machine code or PHP or anything in between — can be improved by a clear and comprehensive understanding of the machines themselves.

Since the Ajax API is different across browsers, and most developers use a JavaScript toolkit, examples within this post will use the MooTools JavaScript framework. CORS will work with any JavaScript framework, since the core philosophy and code is configured on the server, not the client side. Due to its popularity, Apache will be used in server-side configuration examples.

Basic Ajax Requests

Each of the JavaScript frameworks abstracts XMLHttpRequest objects (or in the case of Internet Explorer, ActiveXObject or Microsoft.XMLHttp) to make Ajax requests. These requests feature a URL and may contain extra headers, data, different request types (GET, POST, PUT, or DELETE), and much more. A basic Ajax request would look something like this:

// Create a new Ajax request
var request = new Request.JSON({
	// The URL to get content from
	url: "/countries.json",
	// The success callback
	onSuccess: function(countries) {
		// Log out the content
		console.log("The countries are: ", countries);
	}
}).send(); // Send the request

The URL within the request above is local; countries.json is located within the same origin. What if we try to get tweets from Twitter, though?

// Create a new Ajax request
var request = new Request.JSON({
	// The URL to get content from
	url: "http://twitter.com/statuses/user_timeline/davidwalshblog.json",
	// The success callback
	onSuccess: function(tweets) {
		// Log out the content
		console.log("The tweets are: ", content);
	}
}).send(); // Send the request

The request to Twitter will fail because the request destination, twitter.com, is not the same as the origin. Each browser provides its own error message; Chrome will warn you with: XMLHttpRequest cannot load http://twitter.com/statuses/user_timeline/davidwalshblog.json. Origin http://davidwalsh.name is not allowed by Access-Control-Allow-Origin.

In the case of Twitter, we could use a JSONP request instead, but the problem with JSONP is that the destination server must support it. Even if the destination supports JSONP, you cannot POST to the URL or send specified request headers. How do we fix this conundrum? CORS, of course.

Implementing CORS

CORS allows for cross-origin requests with little fuss. Since the destination server is the entity in control and “at risk,” it must be configured with the proper headers and security settings. A few of the key headers include:

Access-Control-Allow-Origin

A specific URI or * which identifies what domain(s) may make cross-origin requests to this (destination) server. (* is an undesirable configuration value, since it allows any and all origins to make requests to your server.)

Access-Control-Allow-Methods

A comma-separated list of allowed request methods.

Access-Control-Allow-Headers

A comma-separated list of allowed request headers.

Assuming that a web site is hosted on an Apache server, the virtual host could be configured as follows:

<VirtualHost *:80>
	DocumentRoot "/path/to/website/root"
	ServerName domain.tld
	Header set Access-Control-Allow-Origin http://example.com
	Header set Access-Control-Allow-Methods POST,GET
	Header set Access-Control-Allow-Headers X-Authorization,X-Requested-With
</VirtualHost>

The configuration above only allows remote requests from http://example.com, the request type may only be POST or GET, and allowed headers are X-Authorization and X-Requested-With.

If you want to allow anyone to make Ajax requests to your domain and with any request type, you could opt for this configuration:

<VirtualHost *:80>
	DocumentRoot "/path/to/website/root"
	ServerName domain.tld
	Header set Access-Control-Allow-Origin *
	Header set Access-Control-Allow-Methods POST,GET,DELETE,PUT,OPTIONS
	Header set Access-Control-Allow-Headers X-Authorization,X-Requested-With
</VirtualHost>

With the destination server configured, we can send cross-origin requests from wherever we want:

// Create a new Ajax request
var request = new Request({
	// The URL to get content from
	url: "http://example.org/content.php",
	// The success callback
	onSuccess: function(content) {
		// Log out the content
		console.log("The content is: ", content); // Works!
	}
}).send(); // Send the request

The big question is, “which browsers have implemented the CORS spec?” Most of them, actually. Firefox 3.5+, Safari 4+, Chrome 3+, and IE9+. IE8 doesn’t support standard CORS, but it does have an XDomainRequest object, and Opera has not yet implemented CORS. That current level of support does, however, make CORS a viable option for many web sites. If you manage a web service or simply want to tinker with Ajax strategies, keep CORS in mind; I have a suspicion that CORS will play a large role in the future of web development.

Developer Gift

I’m terrible when it comes to figuring out what other people want as a gift. One gift that never misses is a good bottle of wine or a hard-to-find beer. Sometimes, a smooth wine or hoppy beer fuels a great, late-night coding session. Other times, it helps me get away from the text editor and relax. One added benefit of choosing wine is that your spouse might more quickly forgive your all-night hacking sins. My favorite wines come from Kendall Jackson.

Things were a little dicey those first few months. I’d seen really bad code before (I’ve written some truly awful code myself), but I’d never seen anything quite like what I was dealing with.

Rather than run away screaming, I decided I’d try to wrangle this code into shape, one tiny piece at a time. Here are a few of the things I’ve learned over the past year about dealing with legacy code.

There’s a Difference Between Internal and External Quality

I once heard Stefan Priebsch comment on the fact that there’s a difference between internal and external quality where web apps are concerned. That was an “Aha!” moment for me, and I’ve never forgotten it. As bad as the code at my new gig was (and in many cases still is), the apps worked, and most of the users were very happy with them. Where I wanted to rewrite every bit of every app (including all of the front end), I had to remember that most everyone at the company was pleased with the way things were. Sweeping changes would cause more problems than they would solve.

Version Control

One of my favorite stories to tell about my first week on the job was when I asked, “What version control system are we using?” I naively expected we were using Subversion or Git. To my shock and horror, the developer replied, “There’s only one developer; we don’t need version control.” As soon as I recovered from that, I installed Subversion and versioned every bit of code and configuration I could find.

Organize

Not only was the code itself a mess, its location and organization was such that it was nearly impossible to find the code I was supposed to be working on in the first place. The situation with the server config files was nearly as bad. I did two things that helped immensely:

1. I placed all of our sites in /sites and named each site’s directory after the site’s URL.
2. I reorganized the existing Apache config files, breaking out the numerous VirtualHost directives into their own files.

Do. Not. Rewrite. Seriously, Don’t Do It.

Rewriting should never be the first option. It shouldn’t be the second option, nor even the third. Previous developers have domain knowledge you don’t; they’ve solved bugs and addressed user issues you don’t know anything about. Unless the external quality of the app has degraded to the point a complete rewrite is necessary, the best thing to do is take it slow.

Take It Slow

Refactor the code one little bit at a time. When I say one little bit, I’m not joking. Tiny changes. Small things like instituting a coding standard and reformatting the code you’re working on can work wonders for readability. You might change variable names to be more descriptive, or move functions closer to where they’re being called. At that point, and before you begin refactoring, it’s time to write tests.

Tests Are Your Best Friend

The code I found seemed to be untestable, but over time, I’ve discovered that there’s very little code that’s truly untestable, even if you have to go through contortions to test it. The orthodoxy of automated unit and integration testing is excellent in theory, and amazing when you can make it happen (it should be a number one goal), but sometimes the reality of your situation won’t allow for that. I frequently use the following technique when I begin testing disastrous code:

1. Make a small change.
2. Refresh the browser.
3. Rinse. Repeat.

Don’t Forget to Have Fun

Derek Siver’s post, Because It’s Fun!, really resonated with me. I started programming for many different reasons, but one of them was that programming is fun. There’s nothing more exciting — nothing that makes me happier — than solving a really difficult programming problem. What bigger problem is there than a disastrous legacy codebase? When I stopped crying over how bad I had it and started laughing at what I found, I started enjoying my work again. Now, I find I enjoy coming to work and making something beautiful out of something bad. I hope you can, too.

Developer Gift

I’m a huge fan of gift certificates — huge — and I can’t think of a better place for a developer shopping spree than ThinkGeek.

SimpleTest is very easy to install. Just download the latest version, upload it to your server, then unzip it. That’s it! Now you can get started writing tests.

SimpleTest can handle all of the unit testing for your normal behind-the-scenes code, but it also has an internal web browser, so it can also test the front end, too. To get started, create a PHP file for your tests. Include autorun.php from SimpleTest. For tests that will use the internal web browser, you also need to include web_tester.php. Then, create a class for your tests that extends the WebTestCase class.

require_once 'simpleTestDir/autorun.php';
require_once 'simpleTestDir/web_tester.php';

class contactForm extends WebTestCase {

}

It’s now time to get down to the actual tests. SimpleTest offers a lot of options for ways to test. You can check the page title, whether specific text is present on a certain page, whether a certain cookie exists, and so much more. See the SimpleTest documentation for a complete list of options.

One of the things that I like to test is to make sure that a specific error message displays when bad data is sent in a form. With SimpleTest, this is easy. Load a form using $this->get() and provide the URL of the page you want to test. Enter data into a field using $this->setField() and provide the name of the field as the first argument and the value of the field as the second argument. Commit an error, such as leaving one of the required fields empty, then submit the form with $this->clickSubmit() and provide the title of the submit button. Finally, you can check that the error message is displaying on the resulting page by running $this->assertText() and providing the text of the error message. The code would look something like this:

require_once 'simpleTestDir/autorun.php';
require_once 'simpleTestDir/web_tester.php';

class contactForm extends WebTestCase {
   function testUsernameField() {
      $this->assertTrue($this->get('http://myWebsite.com/myForm/'));
      $this->setField('username', 'bad username');
      $this->clickSubmit('Send');
      $this->assertText('Please enter a username with no spaces.');
   }
}

Now, when you view your test file in a browser, you will get a report telling you your test case is complete and that — hopefully — you have two passes, one for your assertTrue and one for your assertText. You did it; you are testing your form! SimpleTest also integrates with many of the continuous integration platforms, so you can incorporate front-end testing into the rest of your testing toolbox.

Developer Gift

I have two gift suggestions.

What do you get for the developer who has everything? How about a motorcycle repair shop in Honduras? Kiva lets you become a microfinancer, lending money to small, locally-owned businesses around the world. Kiva handles everything for you and sends you updates on the status of your loans. It’s a great way to take a small amount of money and make a big difference in someone’s life.

Love to play guitar, but don’t want to deal with the bulk of carrying one around all the time? Get the Electronic Rock Guitar Shirt! You can keep coding, but your guitar is ready for rocking at a moment’s notice. (And yes, it really plays!)

Let’s explore how to securely store data on the client side, and how to detect unwanted modifications.

Ingredients

Nearly every web app needs some kind of authentication, and most use cookies to help with identity. Cookies are a very convenient, portable, and scalable method to identify users after they have been authenticated. Unfortunately, you have to store those cookies on the client. The phrase “all user input is evil” is often used when you have to deal with data from the client, so it is important to realize that cookies are under the client’s control.

The first thing to remember is that cookies can provide more information to the user that you might think. For example, if your cookie contains a PHPSESSID key, an attacker doesn’t need fancy attack tools to immediately recognize that your app probably uses PHP. Every bit of information is important for an attacker, even if it doesn’t seem important.

An example from the Microsoft world is if you see ISAWPLB in a cookie, chances are that the target app uses the ISA Server as a reverse proxy or load balancer. Just look at the cookies of apps that you use regularly, and Google for some of the keys. Attackers can use this information in ways you would never consider. If you don’t believe me, take a look at the presentation entitled How I Met Your Girlfriend, by Samy Kamkar, which was presented at DEFCON 18. He shows you how to reduce the entropy of PHP sessions from 160 bits to 20 bits, which is much easier to brute force.

Attackers may also use cookies to bypass your authentication mechanisms. For example, if you implement a “remember me” feature by storing the user ID in the cookie (and no additional checks are performed on the server side), then the attacker may randomly change the ID in the cookie until he finds a valid one. It’s not very hard to guess that admin users have low IDs, right? If you want to learn more about attack vectors like this, I recommend you to start with the OWASP Top Ten Project where you can read about the top ten security risks in web apps, and how you can prevent them.

Does all of this mean that you should avoid cookies altogether? Of course not. Let’s take a look at two ways to encrypt your cookies and detect possible changes made to them. Note that the following code snippets are taken from the Lithium core. (Make sure to check it out if you haven’t already!) Additionally, I’ve modified the code snippets a bit to make them easier to understand. Lithium handles the following techniques transparently, so in practice, you only add and read data and don’t need to care about the implementation details.

Directions

A secure way to ensure that no one has tampered with your cookies is called HMAC (Hash-based Message Authentication Code). In a nutshell, your cookie gets signed with a hash that is generated from your data and a secret password. As a result, if the data changes, the hash will change, too.

PHP provides a handy hash_hmac() function that does the actual work for you. Here’s the code snippet from Lithium:

public static function signature($data, $secret = null) {
	unset($data['__signature']);
	$secret = ($secret) ?: static::$_secret;
	return hash_hmac('sha1', serialize($data), $secret);
}

It all boils down to calling hash_hmac() with an algorithm (in this case sha1), the actual data, and a secret password. Because you can only hash a string, you may need to serialize a non-scalar payload first. The signature will be appended to the payload, so you need to unset it first, so that the hash actually represents just the payload. It looks similar to this (let’s pretend that the static signature method is wrapped in a Cookie class and is public):

// password and payload
$secret = 'phpadvent';
$data = array('christmas' => 'fun');

// create the signature
$signature = Cookie::signature($data, $secret);

// store the cookie
$data += array('__signature' => $signature);
setcookie('mycookie', serialize($data));

To verify the cookie, remove the payload and generate the hash again. If the two hashes don’t match, Lithium raises a RuntimeException. This isn’t strictly necessary, but it makes sense in certain environments (you can then write exception handlers that log attack events, send emails, or block further requests).

public function read($currentData, array $options = array()) {
// ....

$currentSignature = $currentData['__signature'];
$signature = static::signature($currentData);

if ($signature !== $currentSignature) {
	$message = "Possible data tampering: HMAC signature does not match
data.";
	throw new RuntimeException($message);
}
return $data;
}

I recommend you read both the hash_hmac() documentation and the actual implementation in Lithium.

Message digests are a good way to detect data tampering, but in a lot of cases, it is an even better idea to encrypt all of your data. In order to do this, make sure to have the mcrypt extension installed. I recommend you use the AES algorithm in CBC mode (don’t use ECB, as it is considered far less secure).

The code for encryption is a bit more complicated, but it’s definitely worth the trouble.

protected static $_vector = null;

public function __construct($secret) {
	$this->_config = array(
		'secret' => $secret,
		'cipher' => MCRYPT_RIJNDAEL_256,
		'mode' => MCRYPT_MODE_CBC,
		'vector' => static::_vector(MCRYPT_RIJNDAEL_256, MCRYPT_MODE_CBC)
	);
}

protected static function _vector($cipher, $mode) {
	if (static::$_vector) {
		return static::$_vector;
	}

	$size = static::_vectorSize($cipher, $mode);
	return static::$_vector = mcrypt_create_iv($size, MCRYPT_DEV_URANDOM);
}

protected static function _vectorSize($cipher, $mode) {
	return mcrypt_get_iv_size($cipher, $mode);
}

public function encrypt($decrypted = array()) {
	$cipher = $this->_config['cipher'];
	$secret = $this->_config['secret'];
	$mode   = $this->_config['mode'];
	$vector = $this->_config['vector'];

	$encrypted = mcrypt_encrypt($cipher, $secret, serialize($decrypted),
$mode, $vector);
	$data = base64_encode($encrypted) . base64_encode($vector);

	return $data;
}

public function decrypt($encrypted) {
	$cipher = $this->_config['cipher'];
	$secret = $this->_config['secret'];
	$mode   = $this->_config['mode'];
	$vector = $this->_config['vector'];

	$vectorSize = strlen(base64_encode(str_repeat(" ",
static::_vectorSize($cipher, $mode))));
	$vector = base64_decode(substr($encrypted, -$vectorSize));
	$data = base64_decode(substr($encrypted, 0, -$vectorSize));

	$decrypted = mcrypt_decrypt($cipher, $secret, $data, $mode, $vector);
	$data = unserialize(trim($decrypted));

	return $data;
}

The actual work is done by the mcrypt_* methods. (Lithium provides internal wrappers.) The constructor (not shown here) creates a vector (by calling _vector()) and sets the cipher (MCRYPT_RIJNDAEL_256), the mode (MCRYPT_MODE_CBC), and a custom secret. When encrypt() is called, the unencrypted payload is serialized and encrypted with mcrypt_encrypt(). Because the encrypted data may contain incompatible characters for cookies, we need to base64_encode() it. Decrypting works in exactly the opposite way (decoding, decrypting, and unserializing). Here’s an example:

// password and payload
$secret = 'phpadvent';
$data = array('christmas' => 'fun');

// encrypting
$cookie = new Cookie($secret);
$encrypted = $cookie->encrypt($data);

/* Possible output:
* string(88)
"aAH84W30XJTRC1aFdvleJdv3H0Dzj/TPLVSYKyWe2yo=LtoWXG8ewGK6btLnO2OLGsOfTc6T97TLwwogmMXORHI="
*/
var_dump($encrypted);

// of course, we can decrypt it again
$decrypted = $cookie->decrypt($encrypted);

/* Output:
* array(1) { ["christmas"]=> string(3) "fun" }
*/
var_dump($decrypted);

If you are very careful, you can first add an HMAC signature to your payload and then encrypt it all.

Tasting

Now that we know the ingredients, lets bake secure cookies with Lithium:

use lithium\storage\Session;

Session::config(array('default' => array(
	'adapter' => 'Cookie',
	'strategies' => array('Encrypt' => array('secret' => 'p$hp#4dv3nTT'))
)));

Session::write('mykey', 'myvalue');

That was easy, right? Lithium provides you with a transparent abstraction to both session adapters (cookies and native PHP), and security strategies (HMAC and encryption), so you can mix them as you like. You can even write your own adapters (like a custom database adapter) and still profit from encryption strategies out of the box.

The key message I want you to take away is to take care of your cookies! Don’t store a single bit more than necessary on the client side, encrypt it, and make sure it was not modified outside of your control. PHP and the mcrypt extension provide you with everything you need, and good frameworks provide you with handy abstractions, so use them!

Developer Gift

Here’s a recipe for original viennese Vanillekipferl. You may need to convert the measurements into your own.

• 300 grams of flour
• 250 grams of butter (at room temperature)
• 150 grams of ground hazelnuts
• 100 grams of icing sugar
• 1 teaspoon of vanilla sugar

Roast the hazelnuts in a pan without fat, and then let them cool. Mix all ingredients together in a bowl until they contain no chunks of flour and the dough is smooth. Afterwards, wrap the dough into wrapping film, and put it into the refrigerator for one hour.

Preheat the oven to 160°C. Chop the dough into small parts and form small Kipferl (U-Form, like seen in the Wikipedia article). Put them into the oven for about 15 minutes — they should only be slightly brown (not too dark!). While they are hot, turn them over into a mix of icing sugar and vanilla sugar so they are fully covered (like in the Wikipedia picture).

Let them rest for a few days in a box before serving!

One of the first interview questions I was asked was to describe, in my own words, the difference between an inner join and an outer join. It is a question I have adopted, and I now ask it to every candidate. Over time, I’ve been interviewing more and more senior developers, most of whom know the correct answer, so this question has become a way to gaze into the developer soul instead, providing a brief glimpse of the developer’s professional experience. Typically, answers take one of the following paths:

• A dry, textbook answer. Bonus points for using table EMPLOYEES in the example.
• A MySQL-specific answer, illustrated with a set of tables to support a blogging app.
• Data relationships (and therefore joins) explained as a set of properties of objects. Most answers here include User objects and Friend relationships.

I see the last answer more frequently these days, which makes sense. NoSQL is making huge inroads in the world of software engineering, and frameworks are popping up left and right divorcing developers from the need to interact with databases (sorry, “data stores”) directly. The combination of Moore’s Law and developers’ access to as much computing power as they need — or want — creates a situation where developers can get away with abandoning a schema-first approach to app design. This is great for those who have never learned SQL properly, or those who don’t want to feel shackled by a relatively inflexible schema and prefer to focus on business priorities instead.

Not designing the schema first is cool, but data still has to live somewhere and continue to be managed and monitored. We have not skipped over databases; we’ve just changed, quite drastically, how we think about them. Over time, relational databases battled each other, pushing features and vertical designs (I’m looking at you here, Oracle) that promised end-to-end data management. In the meantime, NoSQL data stores thought about separation of function and specialization. Instead of trusting relational databases to solve the problems behind the CAP theorem, they went all postmodern on data-driven software and deconstructed how data is stored, retrieved, synchronized, and persisted within an app.

Gone are the days where a web app was a database with some PHP in front of it. Web apps are now layers and layers — a lot of them independent, many of them using completely different technologies, and some interacting with each other via HTTP-based APIs. The art of scaling web apps is the art of shifting bottlenecks around, and in data-first design, the bottleneck was frequently the monolith database. These days, you might encounter some of the following within your app:

• A caching layer from which most stored data is retrieved, usually implemented using Memcached or Redis.
• Decoupling the volume of user activity from the expectations of app responsiveness by delaying the computation of some tasks. This is sometimes implemented by using a queue-specialized data store — either by using a great key/value store like Redis or an off-the-shelf product like Gearman.
• Data that is separated horizontally across many databases — perhaps a few MySQL instances with sharded/replicated data.
• Data that is separated based on natural parameters, such as user data in a graph database like Neo4j for easy relationship digging, and articles the users write in a MongoDB instance for quick reads and writes.
• A standalone search index of the content of your app in a search-specialized data store, such as Solr.
• A layer — invisible to the end use — that siphons samples data from designated data store locations, then crunches it until it’s transformed into information and, later on, knowledge.

The bottlenecks still exist, but they are isolated explicitly in a way that can be either easily refactored or replaced with a new technology when it is invented. Your search might be slow, but in a few months, Solr might get replaced with X. Your articles might require more error-tolerant write mechanism, but in a few weeks, your MongoDB instances might get replaced with Y.

From the standpoint of an SQL aficionado, I enjoy seeing how relational databases continue to be used but are forced to evolve to fit the needs of the new world. Your MySQL is no longer responsible for searching or managing user relationships, but it may be used as a queueing mechanism, or perhaps an interim caching layer. Relational databases are no longer used as the end-all solution for data management, but they are not discarded and can still be used in a lot of the new ways developers build web apps. Perhaps, in a way, the slight distrust developers still have of NoSQL (e.g., how certain products are still “immature”) means that relational databases are here to stay purely out of inertia, but this also gives developers a chance to be creative with the existing technology.

Here are a couple of examples of using MySQL in interesting (and it’s up to you whether unwise) ways:

• MySQL as a graph database, like Twitter’s FlockDB.
• MySQL as document store, like FriendFeed’s extremely custom schema design.
• MySQL as a key/value store. This lets you play with NoSQL concepts using MySQL.

In the last decade, we have changed how we interact with relational databases and what we expect from them. NoSQL has not (yet) offered us a flying car in the form of a be-all and end-all data store. In the meantime, we must make do we the tools that we already have, while hoping for and working on new inventions!

Until next time! (Starts her jetpack and flies away.)

Developer Gift

Gadgets and software are always nice, but think about making your developer gal or guy’s workspace a bit nicer. Nothing makes your desk feels cozier, saner, and healthier than an awesome plant. My favorite is the easy-to-grow monstera deliciosa, also known as the Swiss cheese plant. If you suspect your gal or guy to be terrible at plant management, I suggest a set of nicely potted succulent plants. They’re easy to grow and nigh impossible to kill. Perhaps a terrarium of succulent plants from this Etsy shop?

But the moment you introduce concepts like objects as return types, people suddenly can’t figure out how the darn thing works. It’s remarkably similar to when people forget how to drive the instant it starts raining. Most PHP software has trained us to expect native types as responses, but if you return something like a `SimpleXMLElement` object or some other sort of custom object, confusion erupts!

Stopping the Insanity with Collections

In an effort to find a better solution to the array versus object debacle, I started investigating the possibility of object-oriented arrays in PHP. This way, you can return an array-like object which behaves just like you’d expect an array to, but has all of the power of an object.

PHP has two classes called ArrayObject and ArrayIterator that are more like stubs than anything useful. What they provide, however, is a reasonably solid foundation for building a useful object-oriented array class on top of. These classes also implement a set of interfaces that give us a significant amount of functionality relatively for free.

Since the word array already has a specific meaning in PHP, I’ll call our new object-oriented array class the Collection class. The goal of this class is to behave identically to an array, but support a variety of methods like the kind you’d find in everything-is-an-object languages like Ruby and JavaScript.

Constructor

Let’s begin with our class definition and constructor:

class Collection implements IteratorAggregate, ArrayAccess, Countable, Serializable
{
	private $collection;
	private $iterator;

	public function __construct($value = array())
	{
		$this->collection = new ArrayObject($value, ArrayObject::ARRAY_AS_PROPS);
		$this->iterator = $this->collection->getIterator();
	}
}

Note: The IteratorAggregate interface extends the Traversable interface, which enables foreaching through the collection.

Interface Methods

Next, we need to implement all of the methods that are defined by the interfaces. These include:

• current, key, next, rewind, seek, and valid from the Traversable interface.
• getIterator from the IteratorAggregate interface.
• offsetExists, offsetGet, offsetSet, and offsetUnset from the ArrayAccess interface.
• count from the Countable interface.
• serialize and unserialize from the Serializable interface.

For the most part, these can be implemented by simply calling out to the already-existing methods on the $this->collection and $this->iterator properties.

If you want your collections to be able to be serialized and unserialized, you’ll need to custom implement these methods. The trickiest thing to watch out for is handling cases where your collection contains a SimpleXMLElement element. Since SimpleXMLElement can’t be serialized, you’ll need to manually convert the object to an XML string on serialization, and re-parse it on unserialization.

Converting Types

So far, you’ll already be able to do things like this:

$data = new Collection(array(1, 2, 3, 4, 5, 'a', 'b', 'c'));
$numbers = new Collection();
$letters = new Collection();

foreach ($data as $entry)
{
	if (is_int($entry))
	{
		$numbers[] = $entry;
	}
	elseif (is_string($entry))
	{
		$letters[] = $entry;
	}
}

However, since array functions in PHP require the inputs to be real arrays, we’ll need to be able to easily export the collection to an old-school array.

public function to_array()
{
	return $this->collection->getArrayCopy();
}

Maybe you even want to add functionality to implement things like to_json(), to_yaml(), or to_object(). This is really easy to do with off-the-shelf tools found in PHP itself and popular third-party packages (such as Symfony YAML).

Magic Methods

What would an awesome collection class be without some magic?

// Support looking up nodes using $obj->key
// Also, call methods without parenthesis: $obj->method
public function __get($name)
{
	if (method_exists($this, $name))
	{
		return $this->$name();
	}
	elseif ($this->exists($name))
	{
		return $this[$name];
	}

	return null;
}

// Support setting values with $obj->key = $value
public function __set($name, $value)
{
	if (method_exists($this, $name))
	{
		return $this->$name($value);
	}

	$this[$name] = $value;
	return $this;
}

// Support cloning this object
public function __clone()
{
	$this->collection = clone $this->collection;
	$this->iterator = clone $this->iterator;
}

// Support echo $obj
public function __toString()
{
	print_r($this->collection->getArrayCopy());
}

Now What?

So, you’ve put together this class. Now what? Well, now you have an object that behaves like an array, can do all of the things that an array is designed to do, and has all of the advantages of being an object. Let’s look at what works and what doesn’t. (Pretend we’ve also taken the time to implement methods like first and last.)

$array = array(
	'key1' => 'value1',
	'key2' => 'value2',
	'key3' => 'value3',
	'key4' => 'value4',
	'key5' => 'value5'
);

$array[0]       // Error!
$array['key1']  // "value1"
$array->key1    // Error!
rewind($array)  // "value1"
end($array)     // "value5"
$array->first() // Error!
$array->first   // Error!
$array->last()  // Error!
$array->last    // Error!

array_values($array); // ["value1", "value2", "value3", "value4", "value5"]

Versus…

$collection = new Collection(array(
	'key1' => 'value1',
	'key2' => 'value2',
	'key3' => 'value3',
	'key4' => 'value4',
	'key5' => 'value5'
));

$collection[0]       // Error
$collection['key1']  // "value1"
$collection->key1    // "value1"
rewind($collection)  // "value1"
end($collection)     // "value5"
$collection->first() // "value1"
$collection->first   // "value1"
$collection->last()  //" value5"
$collection->last    // "value5"

array_values($collection->to_array); // ["value1", "value2", "value3", "value4", "value5"]

You can now access collection keys as indexes (like arrays) or properties (like objects). The advantage here is that you can extend your class even further by implementing methods such as push, pop, first, last, flatten, slice, sort_by, reduce, each, map, min, max, every, any, grep, ungrep, and more, all by manipulating the value of $this->collection. You can’t do that with an old-school array!

Homework

To make a really robust collection class, I would recommend borrowing as many ideas as possible from everything-is-an-object languages like Ruby and JavaScript. Once you have your powerful collection class fully implemented, you’ll never have to worry about needle-haystack issues ever again, and your developer-customers will have far less confusion functionality when using your code.

At some point, maybe we can do the same thing with object-oriented strings. :-)

Developer Gift

I have two ideas.

When I was working on SimplePie, the developers would post our Amazon wishlists online in case anybody wanted to get us something in appreciation for the software. I would randomly get cool presents left on my doorstep throughout the course of the year. It was kind of like a Christmas present that would sneak up and pounce when I wasn’t looking. Much like Hobbes the tiger. It was awesome.

If the developer in your life doesn’t have any kind of wishlist, I would say the single greatest tool I use in a normal programming-intensive workday is a really good set of noise-canceling headphones. There are few things better than being able to fall into the coding zone with great tunes, and very legitimately not be disturbed by someone walking over to your desk and interrupting you. You won’t be able to hear them! :-)

PHP has multiple categories of errors and exceptions, each with its own handling semantics within the PHP interpreter. For our purposes, we can classify these into the following categories:

• Fatal errors, like out-of-memory and syntax errors, E_ERROR, E_CORE_ERROR, and E_PARSE.
• User errors, including E_USER_ERROR, E_WARNING, E_PARSE, and all other non-fatal errors.
• Exceptions (all subclasses of the Exception class).

Fatal errors are triggered from within the PHP interpreter and cannot be triggered by app code written in PHP. (It is possible, however, that some PHP extensions written in C may trigger fatal errors.)

User errors are largely deprecated in favor of exceptions, although they still get triggered by the PHP interpreter and some third-party libraries.

Exceptions behave the same way exceptions do in many other languages like Java, Python, and Ruby.

In an ideal world, all these categories would be condensed into one category, so you could apply the same rules and syntax to all of them. However, as you might have guessed, we do not live in an ideal world. Nonetheless, we will try to approximate it.

Fatal errors

There is a lot of literature out there about how to handle fatal errors using shutdown functions. Basically, set_error_handler() will not work for fatal errors. Instead, you can use register_shutdown_function() and pass it a function that performs some custom actions when fatal errors occur.

The problem is that, by the time the shutdown function is called, the call stack is emptied, and you have no useful backtrace to work with, except for where in the file the error has occurred. If you try to call debug_backtrace() in your shutdown function, it will return an empty array.

One way to work around this is to log a global variable in the shutdown function that can help debug the problem. This global variable could be the session identifier, user identifier, or anything that can help correlate log lines produced by the shutdown function with other log lines produced within the normal run of the app.

Furthermore, following some examples, calling error_get_last() means if any additional errors or warnings occur before the shutdown function is called, these will overwrite the error that actually caused the code to halt. Sadly, there is no way to avoid this, except to make sure these errors do not happen in the first place. These usually happen if PHP is incorrectly configured, or if you use poorly implemented third-party modules or C extensions that throw errors on shutdown.

Unifying user errors and exceptions

Much literature also exists about handling user errors, including in the PHP manual. However, most of the error handling solutions around suffer from two drawbacks:

• You have to maintain two separate handlers for errors and exceptions.
• You still cannot handle PHP errors like exceptions locally in the code. Instead, you have to rely on a global error handler.

The first drawback can be worked around by having both handlers simply call a central error handling facility instead of duplicating code. The following trick to wrap PHP errors as exceptions can be used to ameliorate the second drawback:

<?php

// These should usually be in your php.ini, but I'm putting it here to
// make sure this demonstration is portable across different configurations
ini_set('display_errors', 'stdout');
error_reporting(E_ALL);

function handle_error($errno, $errstr, $errfile, $errline) {
    // Note that you could also check for a larger set of PHP errors like:
    // $is_halting_error = $errno & (E_USER_ERROR | E_WARNING | E_USER_WARNING);
    $is_user_error = $errno & E_USER_ERROR;
    if ($is_user_error) { 
        throw new ErrorException($errstr, 0, $errno, $errfile, $errline);
    } else {
        // do some logging, custom actions, or even ignore the error
    }
}

set_error_handler('handle_error');

function test_fake_error() {
    trigger_error('Test error', E_USER_ERROR);
}

function test_caught_error() {
    try {
        test_fake_error();
    } catch (Exception $e) {
        print "I caught the exception with message: " . $e->getMessage() . "\n";
    }
}

function test_uncaught_error() {
    test_fake_error();
}

test_caught_error();
test_uncaught_error();

If you run the program above with a PHP 5.3 interpreter, then something like this should be on the standard output:

I caught the exception with message: Test error

Fatal error: Uncaught exception 'ErrorException' with message 'Test error' in test.php:22
Stack trace:
#0 [internal function]: handle_error(256, 'Test error', 'test.php', 22, Array)
#1 test.php(22): trigger_error('Test error', 256)
#2 test.php(34): test_fake_error()
#3 test.php(38): test_uncaught_error()
#4 {main}
  thrown in test.php on line 22

As you can see from this example, you can take advantage of both localized try/catch facilities as well as a full stack trace that comes for free from wrapping the error as an exception!

As an aside, you may notice something strange in the stack trace above; it looks as if the error handling function was called from inside the function that triggered the error. This is because, behind the scenes, what trigger_error() is really doing is simply calling error handlers with information about the user-defined error. If there is a user-defined error handler, then that is called first before calling the default error handler. (For details, see trigger_error() and zend_error().) The call from trigger_error() to the user-defined error handler is done using the call_user_function_ex() C function (which happens to be the same C function used in PHP’s call_user_func()), which records the call in the stack trace and leads to the backtrace seen above.

Developer Gift

Look for those unsung heroes in your workplace, the ones who keep things humming along without anyone ever noticing, simply because they are so good at what they do. Send them an email, ask to accompany them for a day or two, and put yourself in their shoes. It will most likely be an enlightening experience for you — a chance for you to explore what you have come to take for granted. Your interest in what they do will be a rare acknowledgement — a gift! — of how critical they are.

The TDD thesis is that we write tests (known as unit tests) that consume the functions, classes, and interfaces of the code that we intend to write before we write it, and that doing this improves the quality of our code due to increased foresight and better ability to catch a specific type of bug known as a regression, which is the technical term your boss uses for something that worked fine before you added that last bit of code.

The gist of TDD is:

• Each test should test one unit of code.
• Tests should not cross process boundaries in a program.
• Tests should not establish network connections.

If you’re writing tests that don’t follow these rules, tsk tsk, you’re doing unit testing wrong! The mantra is that instead of doing this, you need to tease apart the code so that you can introduce mocks and fake access to those other components.

While I’m on board with the idea that we should be writing code with testing in mind, it is all too easy to get swept away with the purist TDD approach, and — before you know it — you’re not actually writing code anymore; you’re thinking about how to change your real code so that you can write fake code to test your real code. (As a humorous semi-diversion, getting lost in JUnit-derived test frameworks just makes me think of building a spice rack.)

The problem I see with this approach, especially if you’re retrofitting test facilities to a project, is that the cost of implementing all the mocks and fakes can be prohibitive, and it may not even be an effective use of time! Are your users going to run your fake code? What if there’s a bug in your fake code that doesn’t match up to the real world? Will your tests save you from that problem?

Integration Testing

There’s another type of test known as an integration test that describes testing the ability of your program to talk to other modules, databases, and services. The idea is that your test suite starts up instances of its own services (web server, database, &c.) for the test programs to execute against.

Running its own services guarantees that everyone who runs the tests are running with a consistent environment; this is how you eliminate the “works for me” class of problem.

I’ve recently started to build up a test suite for Mtrack (yes, it is long overdue) that uses this approach. Mtrack has a couple of key interfaces:

• Web UI (PHP, with a bit of Backbone.js for the frontend)
• REST API
• SSH access to repositories

It also relies on a couple of services, depending on how it has been configured:

• Zend Framework Lucene Full Text search index
• Solr Full Text search index
• PostgreSQL database
• SQLite database

A driver script called runtests sets up an Apache web server, initializes a SQLite database, starts Solr, starts an SSH server, and launches the Selenium server. It then runs the Perl prove tool to execute the test programs.

I tend to use a Test::More-derived test framework in my integration tests (because we work with Perl and other systems a lot); you can certainly use PHPUnit if you’re more comfortable with that. I like the Test::More style, because it feels very lightweight.

With the services up, we can then write tests that exercise our code via the REST API; these tests can use PHP’s streams or cURL functions to talk to the web server. The test driver for Mtrack exports the web server port number to an environment variable, so the test program knows which server it should talk to.

The Mtrack REST API tests look something like this (simplified for the sake of this article):

<?php
require 'Test/init.php' // Include test functions.
plan(10);               // Plan to run 10 tests.

// This makes a POST to the /ticket API, returning the
// HTTP status code, the headers, and the content. It
// automatically converts to/from JSON and gets the
// host and port from the environment variables set by
// the runtests script.
list($st, $h, $body) = rest_api_func('POST', '/ticket',
    null, null, array(
      'summary' => 'my test ticket',
      'description' => 'the body of the ticket'
    ));
if (!is($st, 200, 'created ticket')) {
  // Failed; this is not something we expect, so dump
  // out the body.
  diag($body);
}

If the script doesn’t run 10 tests, the prove utility will fail the test run, because something unexpected happened. If the is() function returns false (indicating that something didn’t work), then that will cause the test to fail and indicate where and why. We augment that by outputting some diagnostics via the diag() function.

We use Selenium (albeit very lightly right now) to drive through some workflows like creating a new ticket via the web UI (and this has already caught a couple of corner cases for fresh installs).

The SSH server isn’t currently used by tests, but it is there to validate that the repository creation and permissions work correctly when folks use their SCM tools (hg, git, svn) over SSH.

I use a couple of command line options to toggle Selenium on and off (it takes more than a few seconds to launch, making the tests take longer to run), and another to switch between using the ZF Lucene code and the Solr search server.

Each time the runtests script is run, it wipes out the Solr, SQLite, and other state that was initialized from the last run so that we have a consistent environment on each run.

Summing Up

Automated testing is definitely something that I would advocate to anyone writing any kind of serious code. Designing your code for testing from the outset is also strongly desirable. Unit testing is important, but don’t get swept up in the purist approach; mocks and fakes have their uses, but if you’re investing a significant amount of time to build them up, you may want to consider spending that time on automating your integration tests instead.

Developer Gift

It can be tricky to think of good gifts; so much of what software developers do is intangible. I’ve come to appreciate things that help me relax and unwind, and these days, this tends to be assisted by a nice pour of a single malt whiskey. Depending on your budget and your insight into the tastes of your prospective gift recipient you might consider:

• Whiskey stones. “on the rocks” is not something you hear too often from a single malt drinker, because the ice melts and dilutes the complex flavors. Whiskey stones are a non-ice alternative.
• A nice set of tumblers. (Here’s another option.) I have a couple of nice, weighty glasses that I particularly enjoy handling. If your giftee is really into tasting and nosing, there are special whiskey snifters to be had.
• A bottle of whiskey. There’s an element of taste to factor in here; some have very strong flavors that might be fine for an occasional glass, but otherwise unappreciated. If you have no idea, try to find out if there are favorites, and go for something similar. If you really have no idea, find someone else that does, and try to avoid the cheaper blends; it’s better to go for a smaller bottle of something nice than a larger bottle of something not.

An emergency line for the kids

For $2 per month, I set up an emergency toll-free phone number for my kids. They can call this number from any phone, anywhere in the US, for free. The toll-free number will ring my phone, then their dad’s phone, then their grandparents’ phone, and if all else fails, then a neighbor’s phone. It will try each phone number until it finds one that answers. Basically, the kids can call this one number when they urgently need to reach one of us, and they won’t have to remember a bunch of phone numbers. They also won’t need any money. $2 per month is not a lot to pay for my own peace of mind. Setting this up was easy; here’s how I did it.

First, make sure you have a Twilio account and somewhere to host your files. I use Engine Yard’s Orchestra PHP platform and GitHub, but you can use whatever works for you.

To make the app, you’ll first need the Twilio PHP library.

If you want to copy and paste some PHP, you can either use the Find Me Twilio Twimlet or the Call Screening guide in their list of HowTos, depending upon what you want to do. You can also avoid writing your own PHP entirely by using the hosted Find Me Twimlet, but you have a little less control.

If you want to make everything from scratch, feel free. There are numerous Twilio Quickstart Guides.

Make sure you modify the PHP to include your own phone numbers and Twilio account information ($id and $token), and upload it to where it’s being hosted.

Next, get a phone number from Twilio, and provide the URL of where to go when that phone number is called (where you just put your code). The number is now ready to be used!

I think this idea is not only good for young kids, but also for teenagers and even college kids. And, what about aging parents or family and friends traveling around? Knowing they can always call a toll-free number and get help if they need it is something that should make everybody feel better.

Personal messages from afar

My Dad turns 70 this January. I thought it would be kind of awesome to get quotes from some family and friends that he hasn’t seen in a while. The only problem is that he has friends all over the world, and a lot of them aren’t too tech savvy. They can sometimes use email, but they don’t know about Facebook, Twitter, or even Microsoft Word.

I came up with the idea that I could get them to record a small message for him to wish him a happy birthday, burn all the messages onto a CD, and give it to him as a gift. Hearing someone’s voice is much more personal than an email or card, and this will be one gift that he will always remember. Because many of them don’t know how to record something, I needed to make it easy for them to record a message for him and send it to me.

Twilio makes this super easy!

1. Get a Twilio account. (You don’t need anything else, unless you want to make this more robust and do more things.)
2. Check out the Voicemail Twimlet. Give it the parameters you want, then grab the URL that is generated.
3. Get a phone number from Twilio and in hook it up to the URL the Twimlet just returned.
4. Distribute the number with an explanation of what it’s for. When someone calls, they will be prompted to leave a message, and you will get an email with a link to an MP3 of that message.
5. As emails come in, you can follow the link and grab a copy of the message they left, download it, and then burn them all to a CD.

I think this also would be cool for “get well” messages for a hospital-bound friend, or a loved one that is stationed overseas. Responses could be sent to them individually, or to someone else who could compile them first.

Basically, there is no limit to what Twilio can do; the only limit is your own imagination. Go check out their Quickstart Guides, their HowTos, and their Twimlets, and make your life or someone else’s better today!

My first thought for a developer gift were these awesome pancake cushions, but nobody is going to spend that much, even for something that cool. Instead, how about these leather cases for iPods, iPads, and MacBooks that resemble old books? They’re called BookBooks, and they are made by Twelve South. I think these are awesomely unique and help keep your devices separate from the sea of others.

Also, I have personally asked for some Arduino goodies, so I hope Santa comes through!

The truth is, dates and times are hard! There are many different rules that make dealing with dates and times non-trivial. Often, people make mistakes, causing loss of information and general confusion.

Deciding on a format

One of the observations I’ve made is that nearly every system out there lacks a standard for passing around date and time values.

The most common formats are MySQL (YYYY-MM-DD HH:MM:SS), timestamps (what time() produces), and DateTime.

There are benefits to all of these, but it’s important that you need to decide on one. Make sure any date/time that enters your app through any means is converted immediately to this standard. This includes, for example, the result of a MySQL query, data from $_POST, or the response from a web service.

Make sure this enters your coding standards document.

Using DateTime

Deciding on a standard date format and following it religiously is much more important than which standard you choose. If you have a choice, though, I recommend the DateTime object.

The standard inclusion of the DateTime object in PHP is one of the best things that has happened to PHP in recent times. It replaces all the functionality of the existing date functions, and there’s really no need to use anything else.

One of the benefits you’ll get is that you can type hint it like every other class, which means it will be easier to spot bugs, and it’s self-documenting. Also, you can pretty much treat it like a timestamp:

$dt1 = new DateTime('yesterday');
$dt2 = new DateTime('next wednesday');

if ($dt1 > $dt2) {
    echo "Hell froze over!\n"
};

As you can see, because DateTime is actually a PHP extension, it gets the privilege of overloading operators such as < and >. Plus, it handles timezones.

DateTime also makes it easy to do relative calculations. If you want to increase a date by exactly one month, how many seconds you need to add the start date depends on the month of the year, wether it’s a leap year or not, which timezone you’re in, and whether a leap second has been scheduled that year. It’s easier let DateTime do it:

$dt = new DateTime('December 25th, 2011');
$dt->modify('+1 month');

Using time zones

I’ve noticed that a lot of people simply use their local timezone. This is a bad idea, because it’s not always immediately clear, it’s ambiguous, and it can cause obscure bugs. It will also not always be possible to convert to different timezones if your app ends up going international.

The reason it’s ambiguous is simple. Every year, in most European countries, the end of Daylight savings time is the last Sunday in October. On that day, every second between 02:00 and 03:00 occurs twice. Thus, one ambiguity when using time zones is that these seconds that occur twice are not unique, so you can no longer determine if a time is referring to before the switchover or after. This can cause bugs when doing date comparisons. Adding a second could mean the same thing as subtracting 59 minutes and 59 seconds.

The start of DST is similar, which is why you might see gaps in log files and whatnot.

So, make sure that when you deploy your app, the server as well as PHP are configured to use UTC or GMT. UTC time is time zone independent, so every second of the year is, well, unique.

You’ll have to get into the habit of converting times to your local time zone in your presentation layer. On the plus side, it will be easy to support other time zones in the future.

You can always accurately convert from UTC to your local time zone, but you cannot always accurately convert from your local time zone to UTC. (It is a lossy conversion.)

// Changing time zones
$dt = new DateTime('2011-10-07 12:00:00');
$dt->setTimeZone(new DateTimeZone('Europe/Lisbon'));

echo $dt->format(DateTime::RFC2822);

Storing dates and times in MySQL

There are three possible data types you can use to store dates and times in MySQL. You can use DATETIME, TIMESTAMP, or INT (for storing a timestamp).

DATETIME is a popular choice, and in most cases, it’s fine. Just remember that DATETIME is 8 bytes, whereas a TIMESTAMP or INT is just 4 bytes. The biggest benefit of DATETIME is that it’s easy to do calculations in queries, whereas this may is harder with a TIMESTAMP and INT.

I personally dislike TIMESTAMP, because it does automatic time zone conversion. This is fine if you keep everything set to UTC, but if this ever changes, it can cause unexpected results.

I tend to store dates and times as timestamps (using INT). The biggest drawback is that you’re missing some of the date handling features MySQL would otherwise provide, and it’s not clear what date and time you’re looking at when checking out the raw data in a MySQL console or other app. The benefit is that it’s never ambiguous. Timestamps are always based on UTC, so the chance of making mistakes is lower.

Storing time zones

There are a bunch of cases where you might want to store time zones. One example is that your users may have their own preferred time zones.

If you’re in need of this, never store the offset (e.g., -0500). Instead, store the local time zone name, such as Europe/Amsterdam, Canada/Eastern, &c.

The offset alone is ambiguous. For countries that observe DST, the offset changes twice each year, and not necessarily on the same day. Many countries don’t observe it at all.

The US and Canada recently (in 2007) shifted the day DST starts by a few weeks. In Australia, some states observe it, and other states don’t. Newfoundland in Canada has experimented with a 2-hour shift, and the time changes at 00:01. In Brazil, it’s completely crazy; they tend to change the rules quite frequently. (The cost of everyone having to deal with this is huge.)

By storing Europe/Amsterdam, if the rules change again (and they do almost every year, somewhere in the world), you will be prepared.

Summary

• Dates and times are difficult to work with, but very interesting!
• Use PHP’s DateTime.
• Always store dates and times in UTC.
• Make sure your operating system, PHP, and MySQL are all set to UTC.
• If you need to store time zones, use the region, but store the actual date and time in UTC.

I hope this was helpful. Many programmers hate having to deal with dates and times and time zones, but if you stick to these simple rules, it will be much easier.

Developer Gift

I’m embarrassed to say that, until recently, I hadn’t read a good book in several years. That all changed when I read what is hands-down my favorite book this year, Hyperion by Dan Simmons.

Among sci-fi geeks, it’s considered a classic. It’s a relatively easy read, and it’s not very expensive. If you do read it (or have read it), let me know what you thought!

It is easiest to describe the N+1 problem by example. Take a look at the following bit of PHP in a hypothetical blog app:

<?php
/**
* @var Solar_Sql_Adapter|Zend_Db_Adapter
* $sql An SQL database connection object.
*/

// 1 query to get 10 blog posts.
$stmt = 'SELECT * FROM posts LIMIT 10';
$posts = $sql->fetchAll($stmt);

// 10 queries (1 per blog post) to get
// the comments for each post
foreach ($posts as &$post) {

   $stmt = 'SELECT *
            FROM comments
            WHERE post_id = :post_id';

   $bind = array(
       'post_id' => $post['id'],
   );

   $post['comments'] = $sql->fetchAll(
       $stmt,
       $bind
   );
}

We have a master set of rows (the blog posts), but we need to additionally fetch many rows of comments for each blog post. To do that, we loop through the master set of rows and issue one query for each. This gets us the many rows of comments for that blog post.

This is the N+1 problem. In order to build up a list of 10 records, we issued 11 queries: 1 for the master set of 10 records, and then 10 more (N) queries to fetch the comments. If we also needed to get multiple tags for each post, it would be 20 more, 10 queries for the comments, and an additional 10 queries for the tags, for a total of 21 queries to build 10 records.

This can quickly become a performance problem even for small sets of data with many relationships. It is especially bad when you need to fetch, for example, 5 additional sets of related data for a master set of 40,000 rows. There will 1 query to get the master set of 40,000 rows, plus 5 queries per row for the related data, for 200,001 queries in total.

The Solution

There are at least two ways to solve the N+1 problem and reduce the number of queries dramatically.

One way is to write a query that joins the comments to the posts in a single massive result set. We will see repetition of the same blog posts over and over, so we would need to loop through the result set and pick out the distinct posts and retain the multiple comments for each one. There are several drawbacks:

1. It’s hard to do LIMIT/OFFSET on the master set of rows that we want to retrieve.

2. We get back an oversized result from the database with lots of repeated information. To extract the related data, we need to loop through the result set in PHP and discard the repeated parts of rows, keeping track of which ones were repeated and which ones are new.

3. If we have two or more “to-many” relationships, it becomes difficult to construct the query and then pick apart the result set into your domain objects. The repetition problem from point 2 becomes exponentially more troublesome.

Instead, I suggest a way to solve the N+1 problem that is easier to implement and more scalable across several to-many relationships. After we query for the master set of rows, we issue a single additional query to get all of the related rows in one shot, and then stitch them into the master set in a PHP loop. The following is a rewritten version of the earlier example using the query-and-stitch technique:

<?php
/**
* @var Solar_Sql_Adapter|Zend_Db_Adapter
* $sql An SQL database connection object.
*/

// 1 query to get 10 blog posts.
$stmt = 'SELECT * FROM posts LIMIT 10';
$rows = $sql->fetchAll($stmt);

// Find the ID of each post and key a 
// $posts array on them.
$posts = array();
foreach ($rows as $post) {
   $id = $post['id'];
   $posts[$id] = $post;
}

// 1 additional query to get all
// comments for all the posts.
$stmt = 'SELECT *
        FROM comments
        WHERE post_id IN (:post_ids)';

$bind = array(
   'post_ids' => array_keys($posts),
);

$rows = $sql->fetchAll($stmt, $bind);

// Stitch the comment rows into the posts.
// It is easy to find the right post for
// the comment, because we have keyed the
// posts on their ID.
foreach ($rows as $comment) {
   $id = $comment['post_id'];
   $posts[$id]['comments'][] = $comment;
}

We now have one additional loop in the PHP code compared to the original solution. The tradeoff is that we have saved ourselves 9 additional queries overall; we have gone from 11 (1 for the posts and 10 for the comments) to 2 (1 for the posts and 1 for the comments).

Extending The Solution

We can add rows from as many relationships as we like using the same technique. For example, if we also needed to fetch multiple tags in a many-to-many relationship for each post, we would do something like this:

<?php
/**
* @var Solar_Sql_Adapter|Zend_Db_Adapter
* $sql An SQL database connection object.
* 
* @var array $posts The master array of
* posts, keyed on their IDs.
*/

// 1 query to get all tags through the
// association mapping table.
$stmt = 'SELECT
           posts_tags.post_id AS post_id,
           tags.*
       FROM tags
       JOIN posts_tags
           ON posts_tags.tag_id = tags.id
           AND posts_tags.post_id IN (:post_ids)';

$bind = array(
   'post_ids' => array_keys($posts),
);

$rows = $sql->fetchAll($stmt, $bind);

// Stitch the tags into the posts.
foreach ($rows as $tag) {
   $id = $tag['post_id'];
   $posts[$id]['tags'][] = $tag;
}

In the original solution, we would have had a single loop in the PHP code, but a total of 21 queries. With query-and-stitch, we have 3 loops, but only 3 queries. The performance of the multiple PHP loops is practically guaranteed to be better than the performance of 21 queries.

Automating the Solution

The query-and-stitch technique is applicable in any context, whether a new codebase or a legacy one, using a framework or not. However, at some point we are going to want to generalize the technique so we can wrap it in a class for reuse.

In fact, query-and-stitch is what many ORMs do under the hood when you ask to include a relation in an eager-fetch against the master set of rows. Using an ORM is one way to automate away the N+1 problem.

Lots of PHP developers don’t like ORMs, though. After having developed a robust ORM system with Jeff Moore, I am fully acquainted with the tradeoffs involved in using ORMs. They can be opaque, ineffective, or unpredictable in edge cases, and almost always require you to learn a special way of constructing your retrieval requests. (That way can be either through a SQL-lookalike language or through a particular set of methods and informational properties.) Even then, you may not be fully insulated from the N+1 problem, especially if the ORM lazy-loads records in a loop where you did not eager-fetch the necessary data.

In contemplating the N+1 problem combined with the tradeoffs of using ORMs, it occurred to me that the core of the N+1 problem is not the SQL being used. The problem is the marshaling of the results into your domain model objects. With that in mind, I have put together an initial version of a data-marshaling package called Aura.Marshal that uses the stitching technique above.

The package is very limited in its scope. All it does it take result sets and marshal them into domain model objects. It is completely database access layer agnostic; you can use the PHP MySQL functions, PDO, Solar SQL, Zend DB, or anything else. When you define your domain relationships in a type manager and then feed the types with your own query results, Aura.Marshal will link up the objects for you using the stitching described above. This helps to provide an automated way of avoiding N+1 while giving us complete control over the queries being used to retrieve the data we need.

Conclusion

I hope the query-and-stitch technique helps you to solve your own N+1 problems. If you have comments, questions, or criticism, please feel free to leave a comment on my companion blog post.

Developer Gift

I have one gift idea, in three parts. Make it an ATF Christmas! If the nerd in your life likes alcohol, tobacco, and firearms, he or she is sure to appreciate a pack of good cigars, a bottle of fine scotch, or perhaps some appropriately-sized ammunition. The prices on these things are only going up, so get them now while you can still afford them.

Let’s start with content types and formats. There are precisely two formats that you should use to respond to a request that I send to your service: the format I indicate in the Accept header or the format you tell me in your documentation. What I don’t want to see is what commonly happens on two different web services that we integrate with; we send it XML messages, and in an error situation, it occasionally responds with an HTML message! This is very disconcerting, and SimpleXML doesn’t appreciate being asked to handle malformed HTML, either!

Related to this, if you are going to serve in a given format, please ensure that you provide well-formed XML/JSON/whatever. We integrate with what we affectionately call a “nearly-XML” web service. This web service returns XML that nearly parses, except for a number of wrinkles. Specifically, we have to remove some binary characters and fix some remarkably poor entity encoding. We also have to fix the character encoding of the message to UTF-8 to match the declaration. Issues like this are sloppy and do not instill confidence in your users.

If you are supporting a common format, then please do not provide additional restraints on the format that are not generally known. My particular pet peeve is when an XML based service expects the elements within a container to be in a particular order. If you insist on requiring a specific order of elements, please document this requirement! I’ve not yet seen this problem with JSON-based services, so I assume it’s related to particular XML parser used in some enterprise middleware components.

Speaking of documentation, errors need to be really well documented — ideally within the returned error. Your users have no idea why your service doesn’t accept the request, and they want a clear reason. They do not want a message that says: Unable to store entity [CLIENT] - Status -6. This is especially useless if you don’t tell us what -6 means. Nothing slows productivity more than wasting time trying to get a request to work when you don’t know what you’ve done wrong, so throw us poor client developers a bone and provide good error messages. A better error would be: Unable to store entity [CLIENT] - Status -6: Missing NAME field.

All of the issues that I’ve talked about in this article are really easy to get right, and they all point to a root cause, the developers of the web service have never consumed their own service. If there’s one thing I want you to take away from this article, it’s this. Please write an app that consumes your web service. Also, consider writing a test suite for your web service and adding a new test case for every single support ticket you have to deal with. Create a polished web service, and everyone will be talking about how awesome you are!

Of course, if you are feeling all smug because your web service doesn’t make any of these mistakes, I thank you. You are making the Web a better place for us all.

Developer Gift

As all good English people will attest, the best way to get through a long coding session is tea. Lots of tea. In that vein, may I suggest this excellent mug as the ideal gift for your developer friends. Of course, if you haven’t yet embraced good tea, I expect that you could even use this mug for coffee.

How can we get there? I wanted to share my thoughts on things you can do to reduce the deployment friction, even if you can’t get to continuous delivery any time soon.

Identical Environments

Failure to have a development environment that matches your production environment as closely as possible is the number one cause of the oft-used “unable to duplicate” response to a bug. The rise of easy-to-use virtual machine tools like VMware and VirtualBox means that it is easier than ever to duplicate your production environment on your desktop or laptop.

It also allows you to experiment with new features and work on code that can potentially alter or destroy data. Accidentally erased an entire table? No problem, just grab another copy of the virtual machine you were using and try again.

Environmentally-Aware Apps

If you are using third-party APIs in your app, you can often use a sandbox for testing purposes, and the real deal in production. You might also want to use separate databases for your staging and production servers. Either way, your app needs to know where it is actually running so you can control what it communicates with.

Web server environment variables are good for this when coupled with a registry that contains information unique to each environment where your application runs.

Automated Tests

Yes, I know, I know. Testing is a holy war unto itself — even more vicious than tabs versus spaces. Sometimes, your app is incredibly difficult to test in an automated fashion. I understand, but you do have options.

Your best option is to start using PHPUnit for your testing needs, as it can be run from the command line, which also means that running your tests could be triggered by a commit to your version control repository.

If you can’t do that, consider a tool like Selenium, which can run automated tasks in a browser for you and report back its findings. With a little work, you can get Selenium to also ensure certain values are present on specific pages.

Push-Button Deployment

Manual deployments are full of potential problems if they are beyond issuing one command. Database changes, asset compression, minifcation of JavaScript, restarting of services — the list of tasks that need to be performed as part of a “normal deployment” grows longer with every passing day. Every step is an opportunity to make a mistake that could have huge implications.

Tools like Phing, Capistrano, or even Bash can quickly and easily reduce your deployment process to the execution of one command.

Keep Fighting the Fight

Creating frictionless deployment workflows are hard, but they are worth the effort, because once you’ve reached that point, you’re able to concentrate on solving difficult problems instead of worrying about whether or not your new code is going to work this time.

Developer Gift

I can’t think of any programmer who wouldn’t benefit from having a copy of The Pragmatic Programmer, by Andrew Hunt and David Thomas. Mine is well-worn from its use as a reminder that while we are sometimes solving difficult problems, there are often simple solutions to them.

Coding standards can be controversial on many levels. Many have argued over which existing standard to use. Some have entered into arguments over which aspects to pull from multiple coding standards into a new coding standard. Others have argued over conjuring a a new coding standard which often leads to mundane arguments over where to have whitespace and how many characters per line. On the other side, there have been several that have argued that all of this arguing over what to have in a coding standard is simply wasting so much time that there could not possibly be a way that implementing and enforcing a coding standard could ever help. Assuming you can settle on a coding standard, enforcing the standard on a legacy code base can be yet another hurdle.

Many have traveled the path of choosing a coding standard and then attempted to enforce it on a legacy code base with PHP CodeSniffer with the very typical result of abandoning the whole notion of a coding standard due to the seemingly insurmountable number of errors.

The skeptics have won! There can’t be any possible benefit for spending time fixing all of these code standard violations! After all, it’s just style!

The general argument for having a coding standard is that having a common coding style will allow the programmer to focus on the problem and less on the formatting of the code.

Coding standards typically look like just style, but style can be helpful beyond reducing distractions while implementing new features. For example, disallowing superfluous whitespace can help keep your file history cleaner, as you will not have commits that only contain accidental changes to superfluous whitespace.

Coding standards are commonly referred to as style guides, but PHP CodeSniffer also has several sniffs that can save you from parse errors, like checking for duplicate property, parameter, and class names. There are other sniffs that can save you from potential runtime errors, like the sniff that detects jumbled incrementers in for loops.

There are also sniffs that can help you find where your code is complex and likely very buggy, like the nesting level and cyclomatic complexity sniffs.

Haven’t upgraded to PHP 5.3 yet? Someone has written a coding standard for that!

Not a fan of a particular language construct? You could write a sniff for that, too!

Coding standards enforced with PHP CodeSniffer can help with so much more than style, so how can we move beyond the insurmountable number of errors?

Start by picking some sniffs you would like to include in your coding standard, and see how many violations you have. If you have a small number of violations for a particular sniff, clean that up and enforce that sniff. If you find a sniffs with a large, but not too scary, number of violations, then consider distributing those across your team to fix by either by using a pre-commit hook or organizing a “fix-it” day.

Some sniffs could result in a ridiculous number of violations. This is where you will need to stop and think. Is this something worth enforcing? Did we “vote with our feet” to be doing something slightly different from this sniff? It is not a terrible thing to write your own sniff, as it is quite possible that there is some aspect of a coding standard that is already more dominate in your code than what you chose when picking out the pieces of your standard. It might be better to keep that aspect of your existing style intact rather than to modify. Also, some of the existing sniffs contain hidden subtleties that might not be for you.

From here, you should watch the number of violations for the currently unenforced sniffs go down, and eventually just fix the remainder and add those sniffs to your enforced coding standard. It may take awhile, but rest assured that your code base is becoming much better.

Developer Gift

I have two gift ideas.

I love to knit. It gives me something to do while watching my husband play video games on a frigid weekend morning, and it makes my subway commute more productive. I tried many times to knit with straight needles on my morning commute, but too often I would drop one of my needles and scramble to get it back. Then I discovered circular needles. I could now knit a hat a day, and really, who couldn’t afford to have an extra ski cap? Now, I have the Options Interchangeable Zephyr Acrylic Circular Knitting Needle Set. The acrylic tips are so light, nimble, and sharp. I am no longer dropping a needle on the train, as they are attached. An interchangeable knitting needle set is also the most cost and space efficient way to have the needle combinations needed to tackle any knitting project. You may not be a knitter (yet), but chances are you know someone who is, and it would be mighty nice of you to get them (or yourself) a set (and then ask them to make you something).

Knowing the difference between baking soda and baking powder is important, which is why I am glad there is a chapter for each in Cooking for Geeks: Real Science, Great Hacks, and Good Food. Besides, cooking is more fun when you have the know-how to make it your own, rather than programmatically executing a recipe verbatim.

I learned how to download Linux and install it myself for free — then, FreeBSD and OpenBSD. Then, I was building my own servers. Now, the whole web server/Internet thing was less of a mystery. This was, again, fascinating and empowering.

Ten years ago, I was the founder/owner/CEO of a music distribution company that grew from just me to 85 employees.

As the company grew, I enjoyed building all of the office PCs and web servers myself. I enjoyed setting up the same FreeBSD distribution for everyone, and writing shell scripts to automate the installation of everything. It was a fun challenge, and I’d often stay until midnight setting these things up, totally “in the zone” and loving it.

Ten years later, I mentioned this to a friend. He said, sarcastically, “Pffft. Yeah, that’s a good use of the CEO’s time!”

In the moment, I laughed, but later I thought, “Well… yeah! It was my company, and I was the CEO. So, I could do whatever I wanted, and that’s what I wanted to do! What else should I be doing? Something less fun? Sitting in meetings? Why?”

Just this week, an investor friend asked what I’m working on now. I spun my laptop to show him a screen filled with PHP. He said, “Why are you doing this? You can hire someone to do it!”

I said, “This is the only thing I love doing. Programming is pure creativity. Why would I hand off the fun stuff to someone else?”

It was clear that both friends were just assuming that what I should have been doing is whatever is the most profitable thing, as if that’s the only measure that matters.

Even now, I’m trying a geeky project; I’m setting up a JSON-only REST API and requiring even my own site to use only that. When I tell this to other programmers, a few have asked, “Why go to that trouble? You don’t need it!”

My answer is the same for everyone. “Because it’s fun!”

No other reason is needed.

Never forget that all of these things we do — making money, building a company, whatever — are all leading to the eventual goal of having enough money to do whatever we want. So, take a shortcut to that goal, and do whatever you want in the meantime. Never feel embarrassed or defensive about doing something just because it’s fun.

Developer Gift

I recommend any loud, ticking timer to use with the Pomodoro Technique.

Don’t use an app. Get the actual physical ticking thing — it really makes a psychological difference.