Here's how:

• I decided to upgrade to Vagrant 1.1, which broke bento: the current bento master is incompatible with Vagrant 1.1. But selecting the right rbenv env and installing the latest available Vagrant gem (inside the rbenv environment) fixed it.

• My base box build, but for some reason, the guest addition setup broke and while it worked on Mac OSX, it broke the image completely on Ubuntu. Don't ask me why.

Especially the last bit reminded me to share another small tip, or work-around.

Fixing up base boxes

So every once in a while something doesn't work as expected. In my case, the guest additions were installed but not loaded and also failed to load when I started a VM. This in itself wasn't so bad, but it broke the vboxfs shares and while there may be cases where you don't need that (e.g. for a database VM), a VM without your code mounted into it is pretty useless.

The fix wasn't too hard:

• I created a blank Vagrantfile (no recipes, just a simple box definition).
• vagrant up and wait for the box to fail.
• vagrant ssh to enter the box, or start with the GUI option and login through it.
• execute sudo /etc/init.d/vboxadd setup

Then, exit the VM and execute the following: vagrant package vm_name --output box_name.box.

Import the box again with vagrant box add etc., update your Vagrantfile and test — then distribute.

Thoughs

Let me use this to reiterate on how critical a fixed set of versions are.

VirtualBox 4.2.x and bento (or veewee) seem to be incompatible somewhere and even though an error doesn't surface while the box is build or validated, it's still happening. It's paramount that there's always a rollback of some kind before you end up breaking too many things and stall your team for long.

Fin

That would be all for today.

Good enough of a reason to write down or re-cap some things I've learned with or about Vagrant over the last two years.

Base boxes

There are lots of base boxes available, but don't be tempted to rely on them (e.g. via config.vm.box_url).

1. Vagrantbox.es doesn't actually mirror images and that is a huge pain.
2. Available base boxes tend to be outdated. (Think kernel, packages, etc.) Running updates each time you provision is painful.
3. Available base boxes use U.S. mirrors only/mostly/always — because we all live in the U.S. of A..
4. Your VirtualBox guest additions may not match with your system and this may create random issues.

Your best bet is to create your own base box and establish a workflow e.g. using veewee or bento.

At EasyBib, we use bento and we created a definition which replaces the sources with Ubuntu's nifty mirror syntax (since we're pretty distributed at times, everyone appreciates this) and upgrades the base system. Either one of these tools introduce more Ruby into your organization and you may think "WTF — why do I need this?!", but the clear advantage is that no one has to write down a lot of steps how to recreate these boxes and anyone can do it.

In bento's case, the requirement is Ruby 1.9.1+ (getting this installed is IMO the hardest) and bundler. bundle install in your bento-clone gets you everything needed and then the three commands require build, validate and export a box which is ready to use. Ensure to put whatever you need into the definition — for example in the update.sh. Avoid too many manual steps before you export because the next person will have to know and repeat them. Bento serves as documentation as well.

I version our boxes with like easybib-something-10.04.4_vbox-4.1.8_0.1.box and upload them to an S3 bucket. The first number is the Ubuntu release and the second is the version of the VirtualBox guest additions. Simple. The third version is our internal iteration — typically a base box isn't perfect from the beginning while e.g. the Ubuntu and VirtualBox part are settled, there might be other improvements. With an extra version you avoid conflicts and extra work like vagrant box delete etc. and ensure the latest box is always used.

Also — in case the software stack is very different across your projects, it also helps to to create different boxes which come with different software pre-installed.

Standardize on versions

Vagrant and VirtualBox have frequent releases. I suggest to standardize on one so members of your team don't have random issues at hand and fires to fight.

Even for a small team of up to ten developers this makes a lot of sense. Because people tend to add a lot of randomness anyway — different hardware, operating systems and so on. Fight only the battles you want to fight, and deploy otherwise.

Vagrant also recently went from being a rubygem to providing installers. I haven't had the time to roll this out yet, but I expect this to help as well since at least as far as ruby is concerned all the dependencies are bundled.

This of course still implies that testing is required so you and your team don't walk into a stupid little regression and waste away the day trying to figure out what went wrong. And of course even if Vagrant is smoother, it still leaves you with VirtualBox and tools like bento and plenty of potential breakage.

Chef Versions

On a side-note — Chef 10 and 11 may also introduce a lot of breakage in recipes. It helps to roll your base box with a specific version as well. With bento the work-around was pretty straight forward: I replaced the chef-client.sh and installed Chef 10 (instead of 11 — or whatever the latest is).

VirtualBox and guest additions

In theory, it's alright to run with different guest additions in a box than the version of VirtualBox you have installed on the host. It should at least match the main release — for example: 4.1.8 guest additions and 4.1.12 VirtualBox should do fine. That's a big should though, because it also may cause random issues like crashes and hangs.

If you don't want or cannot rebuild the base box for some reason, you can also use vbguest which is a Vagrant plugin to update the guest additions when you start the virtual machine. Keep in mind that this adds a couple minutes to the bootstrapping.

Learn some Ruby

There are little things where it helps to know a little Ruby. And by Ruby, I don't mean Rails. A Vagrantfile itself is Ruby code — this implies that it is fully customizable.

An example of something we as a team couldn't agree on is the location of where projects (and essentially cookbooks) are located on your local disk. Every other team member has a different preference:

case ENV['USER']
when 'till'
  local_cookbook_dir = "~/Documents/workspaces/easybib-cookbooks"
when 'someonelse'
  local_cookbook_dir = "~/dev/till/easybib-cookbooks"
else
  local_cookbook_dir = "~/Sites/easybib/cookbooks"
end

if not File.directory?(File.expand_path(local_cookbook_dir))
  raise "You need to checkout your cookbooks into #{local_cookbook_dir}"
end

# ...

web_config.vm.provision :chef_solo do |chef|
  chef.cookbooks_path = local_cookbook_dir
  chef.add_recipe "ohai"
  # ...
end

It's as simple as that.

Another example — setting VirtualBox options for everyone but a certain user:

web_config.vm.boot_mode = :gui unless ENV['USER'] == 'mr_I_dont_run_X'

Bonus tip: Once you made changes, make sure to at least re-provision. Commit and push after!

Learn Chef or Puppet

I often see projects where developers end up writing a lot of shell script to bootstrap VMs, but learning Chef or Puppet is not really that hard.

I find it harder to validate exit codes (again and again and again) in bash than using a DSL (which is what Chef and Puppet essentially are). The code in your cookbooks (Chef) or manifests (Puppet) is certainly not faster than a shell script but a lot easier to read and more maintainable in the end.

Bash-scripting is not hard either, but in order to produce a set of scripts which can be ran again and again (not just to bootstrap a fresh VM but e.g. also to run updates on one that is running), defensive coding is paramount. And while that is certainly not impossible, it's often a waste of time when frameworks like Chef or Puppet have that covered.

But let's skip on the benefits of using identical tools to bootstrap Vagrant, staging and production because I find them more than obvious.

Learn some Linux

Every once in a while you will run into weird issues with the VMs. These may include one of your VMs losing connectivity (sudo restart networking to the rescue) or weird behavior like assets not refreshing (sendfile off; in nginx). Take it as an opportunity to learn some about the system that is run in production.

In the end all required configuration changes will go back into your provisioning and make sure to share your experience with at least the people on your team.

Backup everything

Whatever you find and use — make a copy of it and put it on Amazon S3 or the local network. With larger teams even a local Ubuntu mirror (or whatever you use) can come in handy.

This includes base boxes, packages, etc.. Nothing is more annoying than waking up and not being able to bootstrap your VMs because someone decided to remove something in order to force you to upgrade.

Don't dumb it down!

Typically, PHP applications are developed on a single host — Apache, PHP and MySQL on localhost. With Vagrant it becomes surprisingly easy to mimic production.

Not to say that I have to run 20 virtual machines to copy my cluster of application servers, but it's perfectly acceptable to set up an environment with four VMs where one is a loadbalancer, two are application servers and then a database server.

Networking and port forwards

Unless you regulary let others use your VMs, don't add port forwards — or at least install a firewall.

For networking, I suggest you either use static IPs (and keep track of them in a sheet) or DHCP. I prefer static IPs though since that makes configuration (e.g. of an application to connect to the database) easier.

It also doesn't hurt to assign names, so you know which VM you're dealing with when GUI is enabled:

    db_config.vm.customize [
      "modifyvm", :id,
      "--name", "DB",
    ]

Hardware

It doesn't hurt to have lots of CPU and RAM, but also configure the VMs accordingly. I run up to four virtual machines on a Macbook Air — usually configured with 256 to 512 MB. I imagine this would go smoother with VMWare Fusion, but since our team contains Mac and Linux as well, we haven't moved on this.

Here's an example how to give 512 MB RAM to a virtual machine:

    db_config.vm.customize [
      "modifyvm", :id,
      "--memory", "512"
    ]

Fin

That's all I can think of right now. Happy development!

For some reason, sometimes it didn't work right away and I had to re-run git svn clone a couple of times to get it right.

git svn clone

Here's a snippet to make it work for you:

$ git svn clone --authors-file=./authors.txt --no-metadata \
--prefix=svn/company_repositoryname/ \
--tags=tags \       
--trunk=trunk \
--branches=branches \
http://company.unfuddle.com/svn/company_repositoryname \
./new-git-repo

Make sure to replace the company and the company_repositoryname part in above scripts.

For some reason, I was never able to get prefix and all that completely right from the start. That is, until now — needless to say the above works. And for the sake of documenting and not re-learning each time (e.g. today, as a migrate another Subversion repository to git (Github :-)), here's my documentary blog post.

Tags and branches

Tags and branches are slightly different concepts in Subversion and git.

In Subversion, we usually ran pre-processing on tags before we deployed them (because doing this in a branch was a huge pita due to size of the repository and the overall joy of merging commits in Subversion. So in the end, a tag we created in Subversion, is not a tag in git because we modified the tag — which makes it a branch.

So as a follow up to my prior snippet to convert a repostitory, I've used this script to convert the branches it created to proper tags in git:

#!/bin/sh

branches=(`git branch -r`)

for branch in "${branches[@]}"
do
    case $branch in
    *tag*)
        tag=${branch//svn\/company_repositoryname\/tags\//}
        remote="remotes/${branch}"
        echo "$tag from $branch, remote: $remote"
        git checkout -b "tag-$tag" $remote && git tag -a $tag -m "SVN tag: $tag"
        ;;
    *)
        echo "Skipping: $branch"
    esac
done

Again, you will have to adjust company_repositoryname in this piece. :-)

Once the script completes, I verify the tags with git tag -l and delete the branches with git branch -D foo.

If all looks ok and the tests confirm this, I add an origin, push branches and also git push --tags.

Fin

That's all — happy migrating. Just in case: the code is BSD licensed, which means, you can do whatever you want with it.

Another obstacle is test setup and environments: When I write and run tests, there is sometimes only so much I can do to mock and avoid actual calls to my storage backend. While I prefer to run my database tests against a SQLite in memory database, there are these edge cases, where I work with multiple database or I write a direct query (and by-pass the ORM-magic).

In these cases I need to have that database server available in my test environment!

The following blog posts explains how to solve these things with Travis-CI. I will walk you through the setup on Travis-CI's business service. But most of this applies to their open source offering as well.

Step by step

I'll try to break it up into small steps.

Travis-CI

The first step is to login at http://travis-ci.com and add the repository tests should be run for. To be able to add repositories of an organization, you have to be the owner of the organization. The easiest way to get access to the service right now is donating to these guys or in case you have done that already: email them. ;-)

The second step is setting up a .travis.yml file.

Mine looks like this:

language: php
php:
  - 5.3
  - 5.4
before_script:
  - ./composer.phar -v install --dev
  - psql -c 'create database testdatabase;' -U postgres

Run-down:

• run the tests against PHP 5.3 and 5.4
• before_script defines your test setup (outside PHP)
• I omitted the script stanza because the default (phpunit) works for me

Composer

I am using composer to manage my dependencies and you should too. I don't want to go into details here, but a short example of my composer.json is the following:

{
    "name": "mycompany/project",
    "description": "A super cool project.",
    "require": {
        "doctrine/orm": "2.2.2"
    },
    "autoload": {
        "psr-0": {
            "MyCompany\\Project\\Test": "tests/",
            "MyCompany\\Project": "src/"
        }
    }
}

Side-note: We also currently commit a composer.phar into each repository for two reasons:

1. To ensure a change in composer won't break our setup.
2. Downtime of (or connectivity issues to) their website don't break our deployments and test runs.

Test framework setup

There is not a whole lot to setup since Travis-CI installs phpunit already. Just make sure you have a phpunit.xml in the root of your repository and you are good to go.

Database schema

The next step would be to generate your schema and check in some .sql, right? I'm not a huge fan of this, because I hate running through a lot of manual steps when I need to update something. Manual steps means that they might be forgotten or people make a mistake. So the objective is to avoid any manual labour as much as you can.

Instead of maintaining these files, I use Doctrine's SchemaTool. It takes care of this just fine because I annotated all my entities already.

To make use of this, I suggest to add the following to your test case:

<?php 
namespace MyCompany\Project\Test;

use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;
use Doctrine\Common\Persistence\PersistentObject;
use Doctrine\ORM\Tools\SchemaTool;

class MyTestCase extends \PHPUnit_Framework_Testcase
{
    protected $em, $tool;

    public function setUp()
    {
        $this->setUpDatabase(); // wrap this again
        $this->setUpSchema();
        /* more setup here */
    }

    public function tearDown()
    {
        $classes = array(
            $this->em->getClassMetadata('MyCompany\Project\Entity\SomethingImportant'),
        );
        $this->tool->dropSchema($classes);
        unset($tool);
        unset($em);
    }

    public function setUpDatabase()
    {
        $isDevMode      = true;
        $doctrineConfig = Setup::createAnnotationMetadataConfiguration(
            array('path/to/Entity'),
            $isDevMode
        );

        // database configuration parameters
        $dbConfig = array(
            'host'     => '127.0.0.1',
            'user'     => 'postgres',
            'password' => '',
            'dbname'   => 'testdatabase',
        );

        $this->em = EntityManager::create($dbConfig, $doctrineConfig);
        PersistentObject::setObjectManager($this->em);
    }

    public function setUpSchema()
    {
        $this->tool = new SchemaTool($this->em);
        $classes = array(
            $this->em->getClassMetadata('MyCompany\Project\Entity\SomethingImportant'),
        );
        $this->tool->createSchema($classes);
    }
}

Assumptions made (aka, room for improvement):

• your entities are in path/to/Entity
• PostgreSQL is used and runs on 127.0.0.1
• you're using a database called testdatabase

Tests

Once the setup is complete, I add the usual testFoo() methods into my test case class.

From within the testFoo() methods I have Doctrine's EntityManager available via $this->em. The entity manager allows me to do whatever I need or want within a test run.

After a test completes, the tearDown() method is invoked and destroys the tables in your testdatabase and then setUp() re-creates it. This will take some time but the side-effects of stale data are not to be neglected. Add to that, your tests should not rely on the order they are executed in anyway.

Another benefit of this setup are updated SQL tables each time a commit changes the annotations. No extra .sql files to maintain. :)

Fin

That's really all there is to running your test suite with Travis-CI and while you did all the above, you just added continuous integration to your toolkit because these tests run each time a pul request is opened or commits are pushed. :-)

As I mentioned early on, these steps apply to the open source offering as well — all but the link to login.

If PostgreSQL is not your game, have a look at the list of currently supported databases on Travis-CI. And if your database server is not on the list of supported applications, you might as well install it with a before_script (as long as it runs on Ubuntu). ;-)

Happy testing!

Since it's something which popped up time and time again and I never got around to figuring out why this happened in the first place, I decided to put it into a blog post.

Problem?

What I did was, I setup an svn:ignore in the root of my Subversion repository, in order to ignore modules which are installed in the app/modules structure:

dev% svn propedit svn:ignore .

My editor opened, and I put in app/modules/foo for the path I want to ignore.

Turns out this does not work:

dev% svn st
?       app/modules/foo

After trying out various things like app/modules/f* and */foo, I finally figured it out:

dev% cd app/modules
dev% svn propedit svn:ignore .

Solution

The revalation of the day is: svn:ignore does not work recursively — like svn:externals would.

Fin

And this is after almost eight years with Subversion. I am just glad to move on.

Apologies if this is too basic or written down somewhere — I did try to read the page on svn:ignore in the documentation, but decided to spend my time on something else after the first couple paragraphs.

I've tried Doctrine 1 a few years back and found it mostly too complicated for my needs. For all I know, I still find PEAR::DB and PEAR::MDB2 the perfect database abstraction layers (DBAL). Though of course they may or may not be totally up to date in terms of PHP etc.. The overall concept of an ORM never stuck with me.

Of course Doctrine has a DBAL too and it would be unfair to compare its ORM to another DBAL, but it seems like almost no one uses the Doctrine DBAL by itself. I haven't either.

The primary use-case for Doctrine seems to be ORM and here is what I think about it.

Little steps

RTFM is especially hard with Doctrine: Google usually points to older documentation, then there are a lot of out-dated links and the classic has not been written yet.

In case there is documentation I compare the experience to RPM (Redhat Package Manager):

• I need to learn about X
• But have to read about Y first.
• To understand Y I need to dive into Z.

Sounds familiar? Dependency resolution in documentation. However, of course there's always also light at the end of the tunnel — it's when it all works — I conquered a problem and defeated the proxies and caches. ;-)

Entities

Entities can be incredibly simple — take the persistent object:

<?php
use Doctrine\Common\Persistence\PersistentObject;

/**
 * @Entity
 * @Table(
 *   name="db.users"
 * )
 */
class Row extends PersistentObject
{
  /**
   * @Id
   * @Column(type="integer")
   * @GeneratedValue
   */ 
  protected $id
}

So what does this look like:

• database "db"
• table: "users"
• columns: "id" (most likely integer and auto_increment with MySQL)

I feel like that is feature a hidden gem because I could not find much about it in official documentation — but that's basically all you have to do.

Annotations are not my thing and XML or YAML or neither — I can live with either of the three though. What seemed annoying is that usually people start off by writing half a dozen getFoo() and setFoo() methods which of course depends on how many columns in your table are. With the persistent object, this is not necessary.

Abstraction

Abstraction is Doctrine's biggest strength and largest drawback in my opinion. Abstraction is nice within reason, but I think with Doctrine it's way over the top.

My personal opinion on SQL is that SELECT * FROM table WHERE id = 1 never hurt anyone. Not to suggest to do this in the wrong layer of your application, but actually understanding the tools I work with, made me a better developer today. A JOIN does not hurt either because then you and I know how to write one and what it costs to execute the sucker. LEFT JOIN, STRAIGHT JOIN and INNER JOIN — most developers don't know the difference. Doctrine also likes cascades but I bet not many people ever watched a busy database server while it's trying to process all them while a couple 1000 writes happen at the same time.

With Doctrine, SQL is so far away that developers really don't need to have an idea about what is going on and I think that is bad in particular.

My problem is that most of the time, the developer does not see their SQL at all and also doesn't analyze what it is doing. I have yet to figure out how to make the application log all statements produced. And yeah, it seems to be impossible because with the use of prepared statements no SQL is actually produced in Doctrine — it all happens on the server. But sure, prepared statements are pretty awesome because we are all idiots and don't know how to escape data. But I should leave this for another blog post.

About abstraction — we had a similar discussion at this month's PHP Usergroup in Berlin and usually someone will counter my argument that the benefits of faster development outweigh all the potential screw ups. And maybe they are right — for a while. And then they upgrade their database server to have more RAM, right? What if for a while is already past tense?

Example

I just found the following code in one of our projects and I think it's a good example of what I mean.

The basics (broken down):

• SQL:
  • a user table
  • columns: id, email, password, site_id
  • indices: PRIMARY (on id), UNIQUE (on email and site_id)
• an entity Doctrine\Model\User class describing the above
• a repository Doctrine\Model\UserRepository class to add convenience calls to the entity

So here is what I found:

<?php
// in class UserRepository
public function getUserById($id, $site_id = 1)
{
    $qb = $this->_em->createQueryBuilder();
    $qb->select(array('u'))
        ->from('Doctrine\Model\User', 'u')
        ->where('u.id = :id')
        ->andWhere('u.site_id = :site_id')
        ->setParameter('id', $id)
        ->setParameter('site_id', $site_id);

        /* ... */
}

And to the defense of all the people who worked on this (including me), it takes a while to see. :-)

Question to my reader: Take the setup into account, do you see the issue? Leave a comment if you figure out what the problem is.

So despite having an entity which defines the table setup, it's still possible to do something like that and there is no error/warning what so ever to let the developer know what they are doing. Not at run-time and not in the Doctrine CLI either. I'd argue that with the level of abstraction Doctrine adds to hide SQL from the developer, I expect that it would do enough analysis on what the developer is doing.

Or maybe the level of abstraction added here is actually counter-productive?

Fin

Food for thoughts. Hope it was not too ranty.

Disclaimer: This is with no offense meant to the people contributing to Doctrine or using Doctrine. I just had to raise my concerns somewhere.

To further illustrate how amazingly simple it is to get started on some Ruby, I'll convert the shell script from my last blog post to a little recipe for chef. Same objective, we install a PEAR package — but it could be anything really.

Follow me.

Shell

This is the shell script from before:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

Ruby

Create a cookbooks directory and create structure for your first cookbook in it:

$ mkdir -p my-cookbooks/first/recipes/

Create a default.rb file with the following content:

# my-cookbooks/first/recipes/default.rb
execute "apt-get update"

packages = ["php5", "php5-cli", "php-pear"]

packages.each|p| do
  package p
end

execute "pear upgrade-all"
execute "pear install -f HTTP_Request2"

The recipe is later referred to as first or first::default (name of the recipe directory, name of the .rb).

It's so simple it hurts. ;)

Step by step

1. I run apt-get update using Chef's execute resource.
2. I create an array of the packages (Arrays are ordered in Ruby, hashes are not (up until Ruby 1.9.x). Order is important here.)
3. I run pear upgrade-all using the execute resource.
4. I run pear install using the execute resource.

Vagrantfile

The Vagrantfile looks slightly different when you provision with chef-solo:

Vagrant::Config.run do |config|

  config.vm.define :web do |web_config|
    web_config.vm.box       = "lucid64"
    web_config.vm.host_name = "web"

    web_config.vm.provision :chef_solo do |chef|
      chef.cookbooks_path = "PATH-TO-YOUR-COOKBOOKS"
      chef.add_recipe "first"
      chef.log_level = :debug
    end
  end

end

The important bit: the path to the location of your cookbooks — could be an /absolute/path or ./../relative/path.

Because a Vagrantfile is essentially Ruby code, anything goes here.

Further reading

Getting started with Chef and Ruby can be intimidating or even frustrating at times. Google "chef cookbooks" and you know what I mean.

These links are what you need:

• Opscode's wiki with all available Chef resources
• A decent Ruby reference
• My blog ;)

Fin

Save, and enjoy — vagrant up.

Maintenance and setup of these environments is often not a trivial goal to achieve. Having worked with a couple different code bases and setups over the last decade, I often noticed things like environment specific hacks (if ($env == 'testing') { ... }) in application code and service configurations and a lot of manual labour all around. It's still very common that code deployed to staging does not work — but it worked for you right?

And it's also too common that members of a team do not know how something works because whoever set it up is currently out sick or on vacation.

My opinion is that the best setup currently available is something like: chef(-solo) on the server and Vagrant on the desktop. Something like because aside from chef, there is also puppet, cfengine and a couple others. Leaving specific projects aside, it really just boils down to automation (within reason).

Automation in my opinion is not just the easiest but the only viable way to setup development, testing, staging and production environments. Without some automation in place it's required that all team members know how all of it works when maybe that is not yet important and the end goal is that all environments actually resemble each other.

Enter Vagrant

Vagrant is a very nifty toolkit to bootstrap Virtualbox images. Bootstrapping means installing your application stack into one or multiple virtual machines in order to resemble production a lot better.

So a lot of times when I rave about how useful Vagrant and chef are, I get the crazy eye from PHP developers:

You want me to learn Ruby to setup my local development environment?

My response:

1. Do not fear the Ruby.
2. You don't have to. (Well, not a whole lot!)

Real talk

First off, of course you need Virtualbox — get a download from their website or use your package manager.

Then, it's not possible to avoid Ruby a 100% — after all Vagrant is written in Ruby.

When you're on a Linux or a Mac, this is usually enough (you may need sudo unless you RVM):

$ sudo gem install vagrant

For a sudo-less install, use RVM:

$ sudo gem install rvm
...
$ rvm install
...
$ rvm use 1.8.7
...

When set up, this is what you do:

$ gem install vagrant
...

RVM allows me to run multiple versions of ruby side by side and also leverage local gem installs — per Ruby version. Think of it as a nifty way to run multiple PHP versions and have a PEAR install per version — similar projects exist for PHP today:

• php_build
• phpfarm

To learn more about RVM, visit their website.

Once vagrant is installed, we can continue!

ShellProvisioner for fun and profit

So, in case you are more comfortable writing some shell script for the time being — Vagrant got you covered!

First off, create a new Vagrantfile for your project:

$ mkdir -p Documents/workspaces/blog-example/
$ cd Documents/workspaces/blog-example
$ vagrant init lucid64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

lucid64 (Ubuntu Lucid 10.04, 64bit) is the name of one of my local box files for Vagrant. In case you haven't got a box yet, head over to vagrantbox.es. There are a couple images you (aka box files) you can download. To get the lucid64 box, use the following command:

$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

Once you made it past vagrant init, you should have a Vagrantfile with a lot of stuff in there.

While it's not important for my tutorial, I recommend you review it some (other time). The created Vagrantfile contains examples for all the provisioners (puppet, chef and shell) and a couple other configuration options, etc..

Let's skip over this and get to the ShellProvisioner.

This is all you need

First off, let's create a shell script setup.sh in the same directory and put something like this in it:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

What does it do?

1. Update local package sources.
2. Install php5, cli interpreter and PEAR installer
3. reload environment
4. upgrade all installed PEAR packages
5. install PEAR HTTP_Request2

When you're done editing, make sure to chmod +x it.

Simple as that.

Putting your shell script to use

Replace your Vagrantfile with the following:

Vagrant::Config.run do |config|

  config.vm.box = "lucid64"
  config.vm.provision :shell, :path => "./setup.sh"

end

Then, run vagrant up and watch it provision. :)

Enter your virtual machine with vagrant ssh and verify HTTP_Request2 is installed:

$ vagrant ssh                                                                                                                                                               [14:16:08]
Linux lucid64 2.6.32-33-server #70-Ubuntu SMP Thu Jul 7 22:28:30 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.3 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
Last login: Thu Jul 21 14:08:15 2011 from 10.0.2.2
vagrant@lucid64:~$ pear list -c pear
Installed packages, channel pear.php.net:
=========================================
Package          Version State
Archive_Tar      1.3.10  stable
Console_Getopt   1.3.1   stable
HTTP_Request2    2.1.1   stable
Net_URL2         2.0.0   stable
PEAR             1.9.4   stable
Structures_Graph 1.0.4   stable
XML_Util         1.2.1   stable

Yay — your first steps into automation!

Fin

That's all for today — Vagrant with almost no Ruby. I hope this gets many more started!

It started off with a PR where I was asked to include code that would download (and install) a composer.phar if none existed. Not a bad idea in general, but also not something I want to impose on anyone, so I decided to create a new action (or in chef-lingo goal) instead.

setup

Up until now, my php_composer resource supported :install which runs php composer.phar install when a composer.phar is in the designated directory:

php_composer "/var/www/app" do
  action :install
end

Since an hour or so ago, it also supports :setup. And :setup will download composer for you if requested:

php_composer "/var/www/app" do
  action [:setup, :install]
end

Note: I'm not sure if this is a good idea for a couple of reasons — the most obvious one: getcomposer.org could be unavailable when you run your deployment. And then there may or may not be changes in the latest composer.phar which require you to make adjustments to your local composer.json. I prefer my deployments rock-solid to minimize side-effects! :)

Getting started

My resource will use composer but it's generally a chef-no-no to also install the tools required by the resource.

A small list of items you will need:

• php 5.3+
• the Phar extension
• git-core (and/or subversion)
• unzip (or PHP's zip extension)

For an idea how to set these up with chef, see my vagrant-test::default recipe.

Roadmap

I haven't really decided on a roadmap, but here are a few things I want to implement in the coming weeks:

• If you're feeling frisky: support for php composer.phar self-update.
• Support for php composer.phar update!
• An attribute to provide the location of a phar (e.g. local or network file-system, CDN, etc.).
• Support for --install-dir.

I'm open to contributions on any of these items, my requirements are:

• Code must be robust.
• Code must be robust.
• Your additions should work on Ubuntu and Linux in general. :)

Fin

That's all. Check it out on Github!

I think it took me (or us) a couple attempts to get this right — let me introduce you to Zf_Crud, a CRUD controller for the Zend Framework.

What's CRUD?

CRUD is an acronym and stands for:

• Create
• Read
• Update
• Delete

A general purpose for CRUD are administrative interfaces — view records, create them, update them or delete them. Think of phpMyAdmin as a very general purpose CRUD interface. One you need some SQL-fu for.

In my experience, such interfaces are most likely or often the very last item on a project. Not too many people (myself included) like to build these interfaces primarily because we have build them before. They are necessary non-the-less: not everybody on a project is a developer and feels comfortable writing SQL queries into phpMyAdmin to get data out (or in).

Zf_Crud

Zf_Crud aims to provide you with an interface for any table possible — think of it as a phpMyAdmin more tailored towards your data and (thanks to Twitter Bootstrap and the Easybib_Form_Decorator) prettier!

Example

Note: We haven't setup a PEAR package yet. I'll get to it soon, but feel free to submit a PR with a package.xml.

Install

The only technical dependencies are PHP 5.3+ and the Easybib_Form_Decorator:

$ pear channel-discover easybib.github.com/pear
...
$ pear install easybib/Easybib_Form_Decorator
...

The other dependency is a working Zend Framework (1) application and an idea what you're doing.

Setup

To setup Zf_Crud, clone (or export) it into your local vendor library:

Our Zend Framework projects have the following structure:

• app/modules/<module>/library/vendor

To export with git, just do the following:

$ cd app/modules/foo/library/vendor
$ git clone git://github.com/lagged/Zf_Crud.git ./Zf_Crud
...
$ cd Zf_Crud && rm -rf .git

Then check it into your own VCS.

If you don't have git (and cannot install it), get a download of Zf_Crud from Github.

Composer, you ask?

With the initial public release, we also added composer:

{
  "require": {
    "php": ">=5.3.0",
    "lagged/Zf_Crud": "0.5.1"
  }
}

Code

Working with Zf_Crud should be super-simple and easy!

In your module (e.g. foo), create a controller in app/modules/foo/controllers/AdminController.php.

<?php
use Lagged\Zf\Crud\Autoload as CrudAutoload;
use Lagged\Zf\Crud\Controller as CrudController;

require_once dirname(__DIR__) . '/library/vendor/Zf_Crud/library/Autoload.php';
CrudAutoload::register();

class Foo_AdminController extends CrudController
{
    protected $model = 'Foo_Model_SomeTable';
    protected $title = 'My Interface';
}

Next, create a plain model using Zend_Db_Table_Abstract:

<?php
class Foo_Model_SomeTable extends Zend_Db_Table_Abstract
{
    protected $_name = 'some_table';
}

The (naming convention in ZF1 is not PSR-0 and a little weird. In case it's not obvious: the) model should live in: app/modules/foo/models/SomeTable.php.

Last but not least: Zend_Db_Table means RDBMS. Zf_Crud expects a Zend_Registry-key called dbAdapter to work. If your's is called differently, skip to the "Convention over Configuration" section.

Anyway — once these two files created (and assuming the rest is setup correctly), you should see something like the following:

Convention over configuration

I'm a fan of convention over configuration and it's the approach we selected when we build Zf_Crud.

The idea is that it should work out of the box without setting up a huge application.ini or DIC, in case you want to tinker with it when you're up and running, here is how.

Since you're extending from the Lagged\Zf\Crud\Controller, this controller has a few configuration options. Configuration is probably too advanced since these are essentially a bunch of class-properties you can overwrite in your class or via init():

<?php
// setup here
class Foo_AdminController extends CrudController
{
    protected $model = 'Foo_Model_SomeTable';
    protected $dbAdapter = 'db';
}

Some of the gems are:

• $where: a WHERE-clause for the data query
• $order: column to order by
• $hidden: hide these columns from display
• $count: number of items per page

Fin

We've been using this code for a couple months now for various items. We recently tagged an early 0.5.1 which suggests that this code is still a WIP and a moving target. The configuration bits are not too great elegant yet. So there are a lot of rough edges to be aware of.

The bottom line is that Zf_Crud has been good for us since it allows us to take the pain out of building administrative interfaces. In most cases it's setup in an hour tops and then we can move on to build something more interesting than a couple forms and views to display and edit data.

If you have anything to add — comments and pull-requests welcome!

ssh tunnels — useful and powerful. They can help me with all kinds of trickery — e.g. usually for remoting through a tight firewall setup to access remote resources. So point taken there are a lot of GUIs for this, but if you spend a couple minutes with the ssh man page, you will realize how amazingly simple they are.

My example for an ssh tunnel put to good use is our production gearman setup: The gearmands runs isolated on EC2. On the upside: pretty secure, but the downside is that the service is unavailable when you need to a little live data for some local tests. When we added an interface to get more visibility into the processes we push through gearman, of course we couldn't access it.

A tunnel to the rescue!

Consider this:

till@macbook$ screen -S gearman-ssh-tunnel
till@macbook$ ssh -L 2222:127.0.0.1:4730 production-gearmand
Linux Ubuntu SMP AMD64

Last login: Sat May  5 16:22:47 2012 from YYY.YYY.YYY.YYY
till@production-gearmand:~$

("ctrl + a + d", to detach from screen.)

So what does that do?

First off, we are starting a session in screen: it's called "gearman-ssh-tunnel". You could use tmux as well, but screen works just as nice.

The consecutive command maps port 2222 on my (local) macbook to a service running on the server production-gearmand (this is via .ssh/config) but only listens on 127.0.0.1:4730.

Your .ssh/config could look like this:

Host production-gearmand
HostName ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com
User till
IdentityFile ~/.ssh/gearman.pem

If I wanted to connect to this server in a PHP-script on my macbook, I would use the following configuration:

<?php
$client = new GearmanClient;
$client->addServer('127.0.0.1', 2222);
var_dump($client->echo('ehlo'));
?>

Run it from the terminal:

till@macbook$ php gearman-ping.php
bool(true)

Success!

Need to stop the tunnel?

The following command lets you resume your session with screen:

till@macbook$ screen -r gearman-ssh-tunnel
...

Type exit twice, or hit "ctrl + d" to log off the server and "ctrl + d" again to kill the screen. Done.

Fin

Bonus points if you use this from your VM in vagrant. But otherwise, that's all for today.

This is my personal opinion, and not my employers.

Here are my thoughts on these.

Perspective

I think there is always a certain point of view when you read these things.

1. If you have to put this on your personal blog because your opinion and your employer's opinion differ so much, it might be time to find another job. For a second, let's focus on other areas in life. We are clearly not referring to the technical details of an implementation, programming languages or your open source license of choice. These are all non-issues where these kind of disclaimers are not necessary. So if you catch my drift — I don't even remotely understand how anyone can possibly spend eight hours a day working for someone who you disagree with on the really important issues.

2. If you happen to be the employer yourself or hold a management position in that company and still try to distance yourself from your own shop, then who are you kidding really? I find it hard to believe that for example people oppose war when they are at home, but during the work day they manage to run a company who's a weapon supplier.

Fin

There is absolutely nothing wrong with expressing your opinion. You can even donate all your money if you think that is really necessary. I really don't care if you're in favour or against something.

But please get real: your own statements will always reflect on the place you work at. In one way or another.

AZ is Amazon-speak for "availability zone", essentially a datacenter. RDS stands for: Relational Database Service.

Judging from my experience with our own setups where EBS is in the mix, I have to say that Amazon does an outstanding job hiding these potential issues with RDS from us. Looking at the price tag of the setup can be intimidating at first, but as far as TCO is concerned, RDS is the complete package: managed from every which way and painless for the most part.

RDS in a nutshell

RDS is pretty awesome — it's basically a highly available MySQL setup with backups and optional goodness like read-slaves. RDS is one of the best services as far as Amazon Webservices are concerned: 90% of what anyone would need from RDS, Amazon allows you to do with a couple clicks. For tuning, it gets a little less shiny and maybe even messy, but even changing parameters is possible.

Another pretty important feature is growing and shrinking RDS. Change your storage and either apply the change right away or wait for your next maintenance window. It should be noted that these changes are rarely instant (or "right away"), which doesn't make it any less awesome. So even though for example resizing the storage is not an instant operation (of course), it still puts a whole new definition into the word elastic.

The package

A standard RDS setup gives you a managed database instance with a DNS hostname and database credentials to log in.

Access from instances is granted using database security groups, which work just like the regular security groups (on AWS). In non-AWS-language, this translates to firewall policies.

Pricing

As far as pricing is concerned, AWS is always a little tough to understand: the baseline is 0.88 USD per hour for a multi-az deployment which totals to 633.6 USD a month (large instance class). Since we opted for reservation (a 1,200 USD one time fee for a three (3) year term), we were able to drop that price to 0.56 USD per hour.

Aside from instance costs there are storage costs as well: 0.20 USD per GB (100 GB will cost you and me about 20 USD) and 0.10 USD per million I/O requests (aka the "i/o rate"). On our multi-az RDS we selected 100 GB for total storage initially but since we currently use only about 60 GB, we just end up paying about 12 USD per billing period.

While storage costs are somewhat easy to predict, the "i/o rate" is not. But it's also not a major factor. I'm unable to provide exact numbers currently because we have three RDS servers (1 multi-az deployment, 1 read-slave and another single-az deployment) and the numbers are aggregated on the billing statement but our total is 368,915,692 IOs which runs at roughly 36 USD per month.

Vendor lockin

Anyway — if RDS is awesome, what's the catch? A closed environment.

The primary advantage and disadvantage of RDS is that we don't get access to the server and our own backups.

Of course there are backups and we can use them to restore (or rollback) our RDS setup from within AWS. There are options using the AWS console and I believe using their API as well. But in the end there is no way to export this backup and load it into a non-RDS-setup. And add to that: replicating from or into RDS is not possible either. Which makes migrations and backups an unnecessary pain in the butt.

Aside from not getting access to our own backup, we also don't get access to the actual instances. Which makes sense for AWS, but it means we need to rely on in my opinion questionable metrics like Cloudwatch. Questionable because there is no way for the customer to verify the data. AWS uses their own metrics (and scale) and it's often not obvious to me how well Cloudwatch works even on regular AWS EC2 instance.

I've seen instances which became unavailable, but Cloudwatch is reporting A-OK (green). I'm not sure how beta Cloudwatch is, but we decided on Silverline (more on that in another blog post) for instance monitoring. Since Silverline requires a local client, it's unfortunately not an option for RDS.

What's pain?

Aside from the monitoring and backup quirks, one of the real pain points of Amazon RDS is that a lot of the collective MySQL knowledge is not available to us. The knowledge which is manifested in books, blogs, various monitoring solutions and outstanding tools like Percona's backup tools are not available to people who run Amazon RDS setups.

Whatever the (technical) reasons for all this may be, they pain me at least a little and should be discussed when Amazon RDS is evaluated to replace MySQL infrastructure.

MySQL and backups

I mentioned backups! First off, I hope ya'll are doing them! ;-)

For many, the preferred way to do MySQL-backups is mysqldump. It's pretty simple to use:

$ mysqldump -u root -pPASS -A > my_backup.sql

This command essentially dumps all (-A) databases from your MySQL server.

mysqldump is an OK solution as long as you take a few things in mind:

• Create your backups during a period where there is the least activity — typically the night.
• There will be a slight bump, but hope that your database is small enough so no one notices.

With a larger database or most databases with lots of read and write activity, this is almost impossible to achieve. For a snapshot to be consistent, table locks are used and that usually stalls access to any application which relies on your database.

Of course there is Percona's xtrabackup (which is outstanding), but with RDS, that is not an option either.

Read-slave to the rescue

Typically people will use a read-slave with MySQL to offload read queries from the master. I haven't done any tests on how far these typically lag behind with Amazon RDS, but I am going to use my RDS read-slave for something else: backup.

Adding a read-slave is easy:

1. Log into the AWS Console and go to RDS.
2. Select your RDS server and click 'add read replica' above

The operation will likely take a while. The timeframe depends on the type of instance and the amount of storage provisioned. In my case I selected a small instance and it assigned 100 GB of storage to match my RDS. Small instances boot notoriously long — the entire operation completed in a little over ten minutes.

On a side-note: read-replicas allow you to scale RDS beyond availability zones (AZ). But you should be aware that traffic across different AZ is billed to the customer.

Costs

A small instance costs roughly 76 USD/month (excluding storage, I/O rate and bandwidth), which by itself is not bad for a fully managed server which I basically setup with two or three clicks. Since we plan to do backup on a regular basis, we will buy a coupon to reserve the instance which cuts down costs tremendously and generally makes the AWS cloud very affordable.

Amazon RDS, quo vadis?

I mentioned a little vendor-lockin with the service and the little visibility from the outside.

In theory, this should not matter — however there are more than a few issues you should be aware of. I don't want to mention them to stomp on Amazon — RDS is still in beta after all. But you should be aware of them to get a complete picture.

Pretty questionable is the way some of these issues are handle: not at all or in private messages. AWS is not always at fault here since I imagine pretty often the customer forgets to update the ticket when the issue is only temporary because their focus shifts to other areas.

But one of the core problem with customer service problem all over AWS is that customers have to resort to posting on a forum with no guaranteed response or have to buy a support contract which includes answers like "we fixed it". The first response is usually that more details are needed (Maybe customer accounts on the forum are not linked to AWS accounts on the inside?) and off it goes into private mode.

My wish is that these situations across all AWS services are handled more transparent in the future so people see development and evolution of the service which means that a trust-worthy platform is be build.

Fin

I've been thinking about my final statement for a while. If anything right now, I would be more in favour of Amazon RDS.

Amazon RDS is an extremely interesting product — the beta-tag is even more impressive. It'll be interesting to see what it will offer once Amazon pronounces it stable.

As for the future of our RDS-setups: they are not gonna go away soon. One of our objectives for 2012 is stabilizing across all products and infrastructure underneath. I think this will include some sort of integration with our external monitoring system to at least try to make sense of things like Cloudwatch and to be able to correlate with other events from all over production.

execute in chef recipes

In chef recipes, almost everything is a resource. In case you're just getting started with Chef, a list of current resources is available on the Opscode Wiki. It's a link I put in my browser bar since I frequently work on chef recipes.

Some examples for resources are:

• package (to install software)
• cron (setup a crontab)
• directory (create directories)
• template (install customized configuration files)
• user and group (to create users and groups)
• mdadm (to setup a RAID)

The above list are examples — so there is more. But if there isn't a designated resource, you can always use an execute block.

An example for an execute block is the following:

execute "discover a pear channel" do
  command "pear channel-discover easybib.github.com/pear"
end

This works pretty well, but it is also not very robust.

Fail hard

By default whenever a command fails, chef fails hard.

To illustrate what I'm talking about, let's test and execute the command from our execute block multiple times on the shell to see its exit status ($?):

till:~/ $ pear channel-discover easybib.github.com/pear
Adding Channel "easybib.github.com/pear" succeeded
Discovery of channel "easybib.github.com/pear" succeeded
till:~/ $ echo $?
0
till:~/ $ pear channel-discover easybib.github.com/pear
Channel "easybib.github.com/pear" is already initialized
till:~/ $ echo $?
1

So whenever a command returns not 0, chef will bail.

One solution is to brute-force your way through these things with ignore_failure true in your execute block. But that's usually not a great idea either because it hides other issues from you (and me) when we need to debug this later on.

For example, if this PEAR channel is unavailable during your next chef-run, it would be much, much harder to find the root cause as of why the install commands failed.

Another solution is using the not_if or only_if options with execute:

execute "discover a pear channel" do
  command "pear channel-discover easybib.github.com/pear"
  not_if do
    `pear channel-info easybib.github.com/pear`
  end
end

If the command wrapped in not_if succeeds (success is exit status), we would skip the execute block.

Optimize?

Since I discovered not_if and only_if, it allows me write recipes which work in most cases. More robust code, which allows me to re-execute recipes on already running instances. So for example when I update a recipe or configuration file which is distributed through a recipe I can re-run the entire recipe and it will not fail but instead complete successfully.

One problem remains with this approach I end up doing the same checks again and again.

In case you happen to dive into Wordpress plugins in 2012, you will see that this is unfortunately not a given. Even (or maybe especially) the commercially available plugins to extend Wordpress are a source for PHP worst practices and contain all kinds of examples how not to write PHP. Part of it, is the platform, the rest is unexperienced developers or general lazyness.

Anyhow! Enough the rant!

Disqus and the loop

In Wordpress, almost everything is a loop. If you'd like to display a list of posts somewhere (for whatever reason), a loop is executed. The advantage (vs. a straight SQL query) range from all kinds all things like caching to convenience. When said hook is used different actions (plugin hooks) and filters are run. So even when the objective is to just collect data (and not even display it) — for example for an Ajax request/response — all these things happen in the back.

So in this case, whenever a loop is used, disqus will attempt to add their JavaScript to it. Usually, to display a comment form, counts and what not.

Problem?

The problem is, that the JavaScript code is then added all over and especially with Ajax you end up with the same code multiple times but add to that the cost of transport.

Solution!

Here's a simple workaround to prevent that.

Stick the following code into your template when you do not want the code to appear:

<?php
if (has_action('loop_end', 'dsq_loop_end')) {
    remove_action('loop_end', 'dsq_loop_end');
}

Yay! Happyness!

Fin

Filters and hooks are really what make Wordpress worth while. Period. In case you happen to do development for it, I urge you to take a look at them. I have a couple more ideas for small posts — stay tuned.