PyPy has many advantages. It can run Python code faster then any existing interpreter, has a lower memory footprint then CPython, it allows you to choose the right level of language abstraction for your code depending on how fast you need it to be since it supports running Python, RPython and allows you to provide C and C++ extensions and It's 99% compatible with CPython.

One of the major differences and what makes PyPy adoption a non-trivial process is the why PyPy interfaces with CPython extensions. If you're interfacing with C code you need to ensure that you use the right bindings.

CPython extensions use Python's C API which PyPy does not have. PyPy can run some CPython extensions using cpyext but those extensions will run slower on PyPy due to the need to emulate reference counting. That also means that Cython generated extensions will be slower.

PyPy on the other hand has support for the not very commonly used ctypes and CFFI which can be run on both interpreters. CFFI runs much faster on PyPy than ctypes so if you're targeting PyPy it's better to use it. I was told that CFFI has been known to be a bit slower than C extensions on CPython because it does not access the C API directly but it wraps it. I could not find evidence for that claim although it makes sense.

If you're interfacing with C++ code, there has been a discussion to port boost::python to PyPy but I don't think it has been done yet.

PyPy provides cppyy which is written in CPython and thus able to JIT out most of the binding related code. cppyy has multiple backends it can use. In this post I'm going to cover Reflex since it's the default and it is currently the most stable backend.

In this post I'm going to demonstrate how to run Protobuf 2.5.1 on PyPy. Protobuf's compiler generates Python code which relies on a CPython extension in order to interface with the Protobuf implementation and is I mentioned before this is not going to be as fast as it should be for PyPy. Google provides a pure python implementation of Protobuf in version 3.0 which is not released yet and breaks compatibility in some aspects from Protobuf 2.x.

In order to use Protobuf with PyPy we have two options. Using CFFI or cppyy. Using CFFI requires us to be familiar with Protobuf's C API and writing the bindings manually which is not something we wanted since we needed to start working on our product as soon as possible. Fortunately Protobuf also generates C++ code so we can use cppyy in order to create a C++ binding to our Protobuf messages.

Installing the dependencies

Installing Protobuf

In order to install Protobuf type:

Installing Reflex

Reflex has some dependencies that are required in order to compile and use it.

In order to compile Reflex type:

In case anyone runs into trouble the full output of a successful installation can be found here.

If you want to always be able to use reflex add the export statements to your .bashrc file and restart your shell.

In order to test that the process succeeded type:

You'll get the usage information of the genreflex command line program.

In order to check if PyPy recognizes the Reflex backend type:

If something went wrong you'll see the following traceback:

Compiling the Protobuf messages

For this post we're going to use the following Protobuf messages:

In order to compile the mesages to C++ type:

If the compilation succeeded you'll see OK printed to your screen.

Generating the bindings

This is where things get interesting. The first time we run genreflex we got some errors:

This error is caused by what is probably a bug in GCCXML since the code can be compiled by g++. Googling reveals that ptrdif_t is defined in the standard C++ library.

The solution is to include the std namespace prefix before each usage of the ptrdiff_t type.

For your convinience here's a patch that does exactly that. Make sure to back up the original header.

Apply it by typing:

After that, generate the bindings again and you should not be getting any errors.

Don't rush into compiling this just yet. Since we're using strings and we haven't provided any reflection information on std::string, cppyy will segfault.

In order to resolve that we need to specify the --deep argument when invoking genreflex. Generaly speaking this is always a good idea to specify this argument if you want to avoid these kind of problems later on as I have found out.

After running genreflex with the --deep command line argument compile the generated bindings:

After that you may load the resulting shared object into PyPy and start working with it.

It's not all roses and peaches. There are still issues with the Reflex backend. Exceptions from C++ will just be printed instead of being raised on the Python side for example so test this carefully to see if it works for you too.

As you can see there are API differences between the Python Protobuf implementation and the C++ Protobuf implementation. I leave it as an exercise for the readers to implement the appropriate adapters. If you do so, please leave a comment.

There's also a good chance that we could create a Makefile that automates all these tasks. If you have written one, please also leave a comment.

General Information

The Templates Directories

We have two directories that interest us:

• The '/usr/share/git-core/templates/' directory on Linux and 'C:/Program Files (x86)/Git/share/git-core/templates/' directory on Windows (Note that on 32bit machines msysGit is installed by default on 'C:/Program Files/…') in which the default hooks are being copied from. If you installed Git using another configuration the installation might reside in a different folder. Adjust the path accordingly.
• The '.git/hooks/' directory is the directory in which the hooks templates are being copied to.

The hooked are being copied from the '[...]/share/git-core/templates/'  directory.  There are other types of templates but they are out of scope for this post.

Note:  If you change the templates directory the hooks directory  must be a subdirectory of the templates directory. Do not set the templates directory to the desired hooks directory instead.

Changing the global templates directory

If you'd like to change the global templates directory open the terminal and type:

You can also set the $GIT_TEMPLATE_DIR environment variable:

The global templates directory is exclusively for hook templates. Just as we usually don’t like to use site-packages for Python and therefore we set up a virtualenv for containment of each project/deployment/test suite we don’t want global business logic for our hooks, just framework code.

The .git/hooks/ directory should contain the business logic of the hooks.

Changing the repository's templates directory

You can change the template directory when you git init or git clone using the --template option to the desired templates directory specifically for the newly created repository.

For new repositories open the terminal and type:

For cloned repositories open the terminal and type: 

Order of precedence

The hooks templates folder will be chosen by the following precedence:

• The argument given with the --template option.
• The value of the $GIT_TEMPLATE_DIR.
• The init.templatedir configuration variable.
• The default templates directory.

Starting Fresh

The default templates directory contains some examples. Each example script ends with the *.example extension. If you remove the extension they become executable git hooks.

Since we are developing using Python lets create a new templates folder. Copy the templates into the python-templates folder.

Erase the contents of the hooks folder, download this gist and extract it into the empty hooks folder.

Architecture

Hooks are simple CLI programs. That's how git works. Every git command or sub system is a CLI program. 

Parsing Arguments

The Python standard library provides us with two libraries for parsing command line arguments: 

• Optparse - Simple but deprecated.
• Argparse - Has support for everything we need including accepting STDIN as an argument which is used by git's server side hooks. 

You can also use the wonderful docopt , a library that automagically parses your docstrings and generates a CLI for you. It has ports to different languages so if you are not using python, you can still use it.

Communicating with Git

Sometimes you need to access your git repository to fetch information or even modify it. Python has a library that do just that. It's called Dulwitch, A pure Python implementation of Git.

Testing Your Hooks

Testing your hooks is not hard but here are a few guidelines that might help you in doing so:

Unit Testing

Since we know git works we should mock out the dependencies that git hooks use. For example commit hooks provide the name of the file that contains the commit message as an argument.

You can do so by mocking open() as explained here.

If you're using a tool like pep8 in a commit hook mock it as well and ensure it is called.

Functional Testing

Fake the process object and invoke the commit hook with the correct arguments.

See how we did it with nose2. It's not complicated. 

Integration Testing

In order to verify that your hooks work exactly as expected you need to invoke git directly.

You're integration tests should git init when you initialize the suite and remove the .git folder when the suite is done running.

I prefer using what instead of the standard library to run sub processes because it has some assertions built into it.

That's it you're good to go. Your commits will be safer now that you write hooks that protect you from making silly mistakes.

How did I reach this conclusion? Keep reading.

What Is Coveralls

Coveralls is a web service to help you track your code coverage over time, and ensure that all your new code is fully covered.

There is but one prerequisite: Your code must be hosted on Github.

The Coveralls service is language-agnostic and CI-agnostic.

The Incident

At my day job we started  parallelizing our builds on Travis-Ci so that we will know faster if the unit and functional tests have passed in the meanwhile until the integration testsuite finishes up.

When the first parallelized build completed and succeeded we got the a notification from coveralls that our coverage decreased by 50%. That's a very drastic change. It's seems like a bug no? But it's an unintended feature and a great one.

The Cause

When you split the build by testsuites each testsuite's code coverage is being measured separately . Apparently, our integration tests covered much more code than our unit tests. It seems intuitive but it took me a minute or two to figure it out.

Conclusion

Parallelize your build. It's important not only because the build process is more responsive but because it helps you measure code coverage correctly.

Vagrant has a package manager for it's plugins which make them very easy to install, upgrade or remove but before Vagrant Plugin Bundler we had no way to ensure our Vagrantfile is using the right version of each plugin.

With Vagrant Plugin Bundler you can do so very easily. 

Installation

This is simply a must if you use Vagrant. To install type: 

Done? Great! 

This should be installed with every machine that runs Vagrant. Period. 

Usage

It's very easy to get started with Vagrant Plugin Bundler. Open up your Vagrantfile and define your dependencies in the following way:

If you have only one dependency you can simply add the following line to your Vagrantfile: 

Assuming that vagrant-omnibus is not installed you will get an error that says that you should install the version you specified in your Vagrantfile of vagrant-omnibus in order to proceed. 

Why Vagrant Plugin Bundler Is Useful

If you distribute your Vagrant boxes to be used as baselines you can specify the dependency requirements with the Vagrantfile that was included inside the box. This ensures that further development on that box will be made possible quickly. In fact you should include the plugin dependencies anyway as they will be required to run the box at times (if not at all times).

Assuming that Travis will be able to run Vagrant in the near future, with Vagrant Plugin Bundler you can ensure your build fails very early if you haven't installed all your plugin dependencies correctly.

What's Missing

• Specifying the plugin's version is currently a requirement. It should be optional.
  
• A plugins sandbox that will allow to install plugins per Vagrantfile  in order to run Vagrant projects with the same plugins but different versions.
• A command to install all required dependencies and a command to check that the current state of the dependencies matches the required state. 
• Flexible Version Constraints. 

Got more ideas? Create an issue.

Would you like to contribute? Fork it. 

And Finally

You should all thank the developer that made it happen. Thank you @tknerr!

This means that you should remove Distribute if you have it installed already and you probably do. 

You can find the instructions to do so here but I have encountered some problems during the process so I'll share my solution here.

Step 1 - Remove Distribute

You have to remove all Distribute versions that were already installed on all Python versions across the system.

To do so simply type:  

Step 2 - Upgrade Setuptools

You want to install Setuptools 0.7 and above for all Python versions.

To do so type: 

Step 3 - Reinstall pip

It might be necessary to reinstall pip. After upgrading on my machine pip stopped working.  It seems that there is an active pull request that obsoletes this step. The awesome developers that work on pip are attempting to resolve this.

In the meanwhile type: 

Step 4 - Install Distribute 0.7.x

I know I just said distribute is deprecated but some python packages still depend on it. Setuptools provides a compatibility layer that redirects all distribute calls to Setuptools.

Make sure you install Distribute 0.7.x after  you install Setuptools or you'll have trouble.

To install Distribute 0.7.x type: 

Step 5 (Optional) - Upgrade Tox

Tox 1.5.0 will now use Setuptools 0.7.x by default instead of Distribute. I filed the issue. You're welcome.

If you are using Tox it is strongly advised to upgrade. If you are not using Tox you can skip this step.

To do so type: 

All done. 

Goodbye Distribute. You won't be missed anymore. 

The Goal

I wanted to check that my vagrant box can be provisioned & packaged every single time for every single change I committed into the repository so I created a .travis.yml configuration file in order to do so.

What is Travis-Ci

If you are not familiar with Travis CI then you should be. Travis CI is the most popular Continuous Integration service for Github repositories and for a good reason: It's Free and Awesome™. 

Please consider using it for your current and future open source projects. It's really easy.

Before you continue go read a little bit about the .travis.yml configuration file format if you haven't done so already. I'll wait.

Done? Great, let's proceed. 

The Build Process

I configured Travis to execute the following build process and thought it should work as is.

I was wrong. So very wrong.  As you will see it's not trivial to implement at all.

The Problem

Travis CI uses OpenVZ virtualized containers to execute your build process. Vagrant uses a virtualization provider like VirtualBox in order to provision & package a box.

The result is that you have a virtualized container inside a virtualized container which is akin to placing a hamster wheel on top of another hamster wheel with both hamsters running in the opposite directions.

Even the OpenVZ twitter account maintainer says it's generally considered as a bad idea:

@the_drow @travisci running virtualization inside virtualization is hardly a good idea. Rent a dedicated server and run VirtualBox on that.

— OpenVZ Containers (@_openvz_) June 15, 2013

But not being able to assure that you can release your Vagrant box at any given time defies the whole purpose of Continuous integration. So we run into a deadlock here.

Bad Ideas can Still Lead to Working Solutions Right?

Not in this case. You see, OpenVZ containers have a custom made kernel based on version 2.6.32. VirtualBox requires the running kernel to be a standard kernel in order to be able to host the Vagrant boxes. If a standard kernel does not exist VirtualBox won't be able to use the current machine as a host for anything.

Here's the build log. Go through it. If you can find a solution please leave a comment. 

Possible Workarounds

1. As @_OpenVZ_ sugguested above, rent a dedicated server. After you installed VirtualBox & Vagrant develop a Fabric tasks that connects to that server through SSH, executes vagrant up  and vagrant package mybox. If the any of these tasks return a non-zero status code the build will fail. This solution certainly works but it is also very expensive to implement when you are developing a small open source project with no financial support. When you log in through SSH make sure your credentials are being passed over using secured environment variables.
2. Attempt to build VirtualBox for the OpenVZ customized kernel. I haven't tried yet but I have little hopes that it will work.
3. Wait until the awesome infrastructure team at Travis will find a solution. I will make sure to report when they will find one. Follow the Storify below for updates.

They are very easy to use and install. The learning curve is really low. 

Usage

Installing a provider is as easy as installing any other Vagrant plugin:

The workflow is exactly the same. When running a machine with a specific provider you only need to add the --provider option when you type vagrant up:

The rest is exactly the same.

Read more about providers at Vagrant's documentation site. 

VMWare Workstation & VMWare Fusion Support

Vagrant now provides a paid providers for VMWare Fusion (for Mac OS X) & VMWare Workstation (For Windows & Linux).

A license for one of these providers costs 79$ per person for a seat (two computers).

Why is this exciting?

Because VMWare outperforms VirtualBox in almost every aspect if you are managing a serious virtualization layer. 

I have never encountered a serious VirtualBox virtualization layer, it is just never done but not only for performance reasons. 

Both VMWare & Hashicorp  provide paid support for their software which is very important for major cooperations because they can get instant support when something goes wrong and someone can be held accountable.

If you are interested in an extensive comparison between VMWare & VirtualBox InfoWorld has already written an excellent and accurate one.

Providers For The Cloud

There are two official providers for: 

• Amazon Web Services 
• Rackspace 

Both cloud services are Infrastructure as a Service (IaaS) which means that they fit well to what Vagrant does, managing the virtualization layer configuration items and provisioning them.

What's Missing?

The only thing that in my opinion is missing is a provider for VMWare Player for those who prefer it over VirtualBox. It's not really a must but a very nice to have feature.

Another thing that is clearly missing is the ability to package multiple images into one box file in order to use the same box for multiple providers. This is evident if you read about the Vagrant's box format.

I have filed issues #1773  & #1774 on these subjects. Follow them and this post for updates.

Updates

As it turns out it is impossible to add VMWare Player as a provider due to missing features and licensing restrictions. 

Why Do I Think You Need One

​It lets you know exactly where you stand without typing anything.

It saves you time typing git branch to know what is the correct branch. It let's you know how many files are uncommitted/added, what revision are you looking at and much more.​

If it saves you time you should probably use it. Because if it does it allows you to focus on what's important - your product.

For Bash

git-sh​ is the best customized shell I have found for bash.

​First of all it turns all git commands and aliases to top level commands which saves you from typing git all the time.

It has bash completion built-in.​

It has a nice customizable prompt ​that allows you to display any relevant information you need about your repository.

Installing it is very easy. Open the terminal and type:​

To activate it type git sh and cd into your repository (if it's not your working directory yet).

For Powershell

Posh-Git takes a different approach. It only customizes the prompt. However, it provides almost anything that git-sh provides besides turning the git commands and aliases to top level commands and there seems to be a discussion to do just that.​

​To install it use chocolatey:

​And restart the shell.

Note that if you want to install Posh-Hg as well you'd have to install the Posh-Git-Hg package instead.

Happy hacking everyone :)​

Hooks are scripts that help you enforce policies on your repository by triggering scripts at certain events that enforce policies such as 'The commit message must be longer than 20 characters' or 'The test suite must be run before a commit and if it fails, reject the commit'.

Hooks are available in every VCS I have used before. They are also available in Visual Sourcesafe which is not a VCS.

Git Hooks

In Git the hook scripts can be implemented using any language but Shell, Ruby, Perl and Python scripts are the most common as far as I know.

The language of the script is determined by the shebang notation as it is usually in Linux based software. Note that this also applies to Windows because Git for Windows runs under MSYS.

If you supply the --no-verify argument to the git command then the command will still be executed even if one (or more) of the hooks failed (i.e. returned a non-zero status code). Please do this only if you know what you are doing.

These hooks reside under the .git/hooks/ directory of your repository.

The Git-SCM book already covered the various hooks available. It also provided an example of how those hooks can be written in Ruby.

For a quick reference you can open your favorite shell and type:

git hooks --help

Why should you be using hooks?

Because this is the last time we let Tim commit ASCII porn in the comments!

Controlling commit contents

Seriously now, no one wants to be the guy/girl who has to bring coffee & doughnuts at his/her expense for tomorrow’s standup meeting because he broke the build so we want to make sure our codebase does exactly what it should. If it happens to you too many times you might end up being broke :)

We want the test suite to run automatically every time we change something. It's one of the reasons we have a test suite, so we know we're on the right path and things aren't broken.

We also want to make sure that our test coverage is high enough, because untested code won't break the build but it will surely break your production environment.

These are good example of common hooks that you should implement in order to decrease build breakage and long, frustrating and lonely nights spent screaming “WORK GODDAMIT WORK!” at your development machine.

Allow me to paraphrase a line from the movie “Training Day” (It’s a great movie by the way)

I don’t know about you guys but I wanna’ go home. I implement these hooks for every project I work on.

Apart from testing, we want to make sure that our codebase fits our style guidelines so we need to run a lint tool.

We also want to make sure that the docs can be generated and that every public module, class & method is documented.

Controlling commit messages contents

Remember that time where you spent a day figuring out what changed in a specific commit that caused major downtime because the commit message was “Fixed this bug” for a commit that added new 20 files and modified 90 other files? If you don’t want to repeat that experience than we have to make sure that we know exactly what changed in each commit.

First of all, thank Linus, empty commit messages are not allowed in Git. If you do not provide a commit message your favorite editor will be opened for you. If you still haven’t typed anything in your favorite editor, the commit will be aborted.

But it’s not enough. We have to make sure the commit messages are meaningful.

It is recommended to reject very short commit messages. Commit messages that contain below 10 characters are usually useless.

All commit messages must be in English only because your team may speak another language but you never know when you’ll have to subcontract or hire a team member from abroad.

Since each commit that contains a new feature/bugfix will have its own tests you might want to append a diff of the test results & code coverage from the last commit. This will allow you to understand exactly what was added, removed or changed in each commit.

If you are willing to write a somewhat complex hook, a hook that verifies that every class & method that was added, modified or removed in the commit is mentioned in the commit message might be useful.

This list can grow very long and I just wrote the hooks I usually implement and some ideas I have. You are most welcome to share your own ideas in the comments.

Next Time

Next time I’m going to show you how hooks should be implemented in a much more structured way than what was suggested in the Git-SCM book.

The examples will be in Python but the theory is relevant for every scripting language.

It's a good start but in my opinion it's not enough.

I'm currently thinking about writing a new app that will integrate Django with vagrant in order to enable a better development workflow.

I'll update when I'll have something concrete.

P.S.

More posts coming up soon. I know I've been slacking off and I'm picking up the pace.

If you are interested in an introduction Scott Hanselman has already written a great blog post about it.​

It's called Chocolatey.

If you are not familiar with package managers here is the definition from Wikipedia:

In software, a package management system, also called package manager, is a collection of software tools to automate the process of installing, upgrading, configuring, and removing software packages for a computer's operating system in a consistent manner. It typically maintains a database of software dependencies and version information to prevent software mismatches and missing prerequisites.

Packages are distributions of software, applications and data. Packages also contain metadata, such as the software's name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database.

Package management systems are designed to save organizations time and money through remote administration and software distribution technology that eliminate the need for manual installs and updates. This can be particularly useful for large enterprises whose operating systems are based on Linux and other Unix-like systems, typically consisting of hundreds or even thousands of distinct software packages; in the former case, a package management system is a convenience, in the latter case it becomes essential.

Now all that's left is that puppetlabs will add a provider for it and than provisioning windows machines will be much easier.

1) Use librarian-puppet to manage your puppet modules

librarian-puppet is an excellent tool for managing your puppet module dependencies.

It is only supported on Linux at the current moment as far as I can tell so if you are running on windows there are two solutions:

1. Attempt to use it with MSYS which is something I haven't tried yet. YMMV.
2. Install it on the target machine you are currently testing and ssh to it in order to use librarian-puppet. It's not optimal but it works.

I have tested and used the second option mainly to save time and to allow me to develop my puppet projects more easily on windows.

You have to share the modules library in order to manage the puppet modules using librarian-puppet on the target machine and develop your project on windows which brings me to my next point.

2) Share the modules directory

In order to use the puppet modules globally share the /modules/ directory using:

config.vm.share_folder('modules', '/etc/puppet/modules', 'modules')

This way you can ensure that the dependencies are both on the development machine and on the target machine.

Make sure your module path in the provisioning configuration is set to 'modules' like this:

puppet.module_path = "modules"

3) Share the templates directory

In order to use templates with vagrant you have to share the project's template directory using:

config.vm.share_folder('templates', '/tmp/vagrant-puppet/templates', 'templates')

Make sure your templates path in the provisioning configuration is set to '/tmp/vagrant-puppet/templates' like this:

puppet.options = ["--templatedir","/tmp/vagrant-puppet/templates"]

4) When provisioning a puppet master share the required manifests

In order for the puppet master to provision the rest of the nodes share the project's manifests that are not related to the puppet master itself using:

master_config.vm.share_folder("puppet-master", "/etc/puppet/manifests", "puppet-master")

I selected puppet-master as the folder name where all the manifests reside but you can select any folder name you'd like.

Happy provisioning everyone.

Last time we talked about selecting and installing the OS. I have referred you to resources about preseeding and Kickstart in order for you to have a better grasp about unattended installations. You've probably attempted (and even succeeded) to perform one already. In this post I am going to further elaborate on how to improve the installation and configuration process of a new development machine.

What is provisioning?

As described on Wikipedia:

Server provisioning is a set of actions to prepare a server with appropriate systems, data and software, and make it ready for network operation. Typical tasks when provisioning a server are: select a server from a pool of available servers, load the appropriate software (operating system, device drivers, middleware, and applications), appropriately customize and configure the system and the software to create or change a boot image for this server, and then change its parameters, such as IP address, IP Gateway to find associated network and storage resources (sometimes separated as resource provisioning) to audit the system.

In our context provisioning is a set of actions to prepare a workstation with the appropriate software (such as IDEs, web browsers, Version Control Systems, language interpreters etc.), configuration and data and make it ready for development of software.

The required software, configuration and data vary between language ecosystems and target environment.

Push based automation systems vs. pull based automation systems

A push based automation system is a system that commands an endpoint (usually a workstation or a server in our context, but an endpoint can be any system that supports remote invocation of commands) to perform a specific set of tasks. Push based systems usually work using ssh or another protocol such as WS-Remoting. A pull based automation system is a system that relies on the fact that a client running on the endpoint will request a server for instructions on what needs to be done in order to keep the endpoint up to date.

You can read more about the topic here.

What is configuration management?

As described on Wikipedia:

CM is the practice of handling changes systematically so that a system maintains its integrity over time. CM implements the policies, procedures, techniques, and tools that are required to manage, evaluate proposed changes, track the status of changes, and to maintain an inventory of system and support documents as the system changes. CM programs and plans provide technical and administrative direction to the development and implementation the procedures, functions, services, tools, processes, and resources required for to successfully develop and support a complex system.

Provisioning tools

Puppet - A configuration management tool

Puppet is a pull based configuration management automation tool that uses a declarative language in order to define what software, data & configuration is needed in order to keep the system in a well defined and healthy state. I strongly recommend to read Pro Puppet in order to learn more about how configuration management is done using Puppet. For an example of a Puppet module which sets up a certain development box see here.

Blueprint - A configuration management reverse engineering tool

In most cases you already have set up at least one development workstation. Writing a puppet module that replicates the same configuration is a tiresome and erroneous work. This is where Blueprint comes in. It scans a machine and outputs the equivalent puppet module that is required to produce the same results on another machine.

Fabric - a tool for executing shell commands locally or remotely through SSH

Fabric is a push based automation tool that can be used to execute shell commands both locally and on a remote endpoint.

It is used for maintenance tasks that need to occur right away such as restarting a web server, initializing or cloning a repository from source control and more.

Vagrant - A VirtualBox configuration management and provisioning tool

Vagrant is a tool for developing & packaging VirtualBox virtual machines. It uses puppet as a provisioning tool.

Vagrant can be used to develop and test the development workstation configuration before provisioning the actual development workstation.

VeeWee - A tool for building Vagrant base boxes

VeeWee is a tool for building Vagrant Base Boxes.

VeeWee also allows developing, testing & packaging VMWare Fusion and KVM virtual machines.

VeeWee does not support windows at this time although some work has been done to add support to it. 

Foreman - A tool for machine lifecycle management

Foreman is a tool that is capable of performing many operations including unattended installations using Kickstart and Preseeding, acting as a web front of Puppet, DNS & DHCP management, virtual machines management and much more.

If you have a physical machine as the deployment target Foreman is optimal for deploying it and provisioning it.

Side note: Puppetlabs are developing Razor which aims to supreceed Foreman but it is currently not as stable and extensive feature-wise as Foreman.

Scripting the installation

If you already have an existing workstation use Blueprint to reverse engineer the current configuration. Adjust as necessary.

If you need a base box check here if someone hasn't already built one for you, if it does not exist create it with VeeWee.

If you are following my lead there are base boxes for Ubuntu 12.04 in 32bit version and 64bit version.

Write the configuration for the development workstation using puppet and test it with Vagrant. Refine and adjust as necessary. Make sure that the ubuntu-desktop package is required by your puppet module as vagrant base boxes come without one in order to allow faster downloading of base boxes. In order to edit puppet files more easily try to use Geppetto, an Eclipse based IDE for Puppet projects.

Optionally, use fabric to clone any active project the developer will be working on.

After all that provision to a physical machine using Foreman.

And that's it.

I haven't shown any code since all environments have different requirements and this post would never end if I had to cover them all. Make sure you check puppet-devbox to get a better grasp of how it's done.

Whenever a new developer arrives to the team he and/or the IT department usually spends some time on setting up their development workstation. This time is wasted as the man-hours spent on setting up a new workstation could have been spent on creating a better product (aka adding value).

The Task

Automating the installation and configuration of a new development workstation while still maintaining flexibility and room for personal customization and preferences using scripting.

Selecting an Operating System

For Python development I strongly suggest a Linux based OS. My personal choice is Ubuntu.

Why Linux?

• Most Linux distributions are shipped with Python already installed.
• Most if not all Linux distributions are shipped with a C compiler which allows us to compile Python C extensions with no effort.
• Some Linux distributions are shipped with package management utilities (such as apt-get and yum) which allows us to script the installation of the development environment required components.
• Native SSH.
• Some Python essentials are written exclusively for linux (e.g. supervisord, pythonbrew andvirtualenvwrapper) and some may need to be configured differently in order to work (e.g.virtualenv).
• Deploying disk images is by far more trivial.
• Linux is completely free.

Why Ubuntu?

First of all, this choice is based on personal preference. If you find an OS that suits for your organization go ahead and use it. With that being said, there are some rational choices for why one should Ubuntu as their development workstation OS.

The latest version of Ubuntu is shipped with a fully configured Python 2.7.2 which is the latest and the last 2.x Python version on which most of the modern Python code is currently targeted on.

You may think it is not much but what I learned from my experiment with CentOS is that it is depended on Python 2.4.

Python 2.4 is a very old version of python and most modern Python programs and libraries won't be targeting it at all.

The consequences are that switching the active Python version using pythonbrew system-wide may temporarily or permanently hurt the system's functionality. I haven't tried it though. I will check it out and let you guys know.

Ubuntu is also very easy to learn Linux with for those who aren't haven't used a Linux based OS yet.

Ubuntu has enterprise support which is important for companies who need to resolve issues quickly and can't afford to waste time that otherwise would be spent on adding value.

Why Not Windows?

• Multiple Python installations are harder to manage using a Windows operating system since pythonbrew is not available. I have yet to see an alternative solution.
• Compiling C extensions is far from trivial since Windows does not provide a C compiler by default and the C extensions must be compiled with the same compiler that Python was compiled on. Currently the Python installer distribution is compiled with MSVC 2008. Alternatively you can use MinGW but that is far from trivial. The implications are that if you are only working with a newer version of MSVC you may need to compile the Python sources yourself which is again far from trivial.
• I've had issues with x64 Python installations where C extensions compilation is not even possible since the compiler only ships with x86 version (See above for the same compiler note).
• Fabric's dependencies uses C extensions. As a result to the above, installing fabric is not trivial and sometimes not possible. Fabric is very important when automating the development process since (and I'm quoting) "it provides basic suite of operations for executing local or remote shell
  commands (normally or via 'sudo') and uploading/downloading files, as well as
  auxiliary functionality such as prompting the running user for input, or
  aborting execution.".
• virtualenvwrapper is based on shell scripts and although there is a port to PowerShell it is not compatible and I've had little experience with it so I can't say I recommend it. Another option is to use virtualenvwrapper with MSYS but I haven't tested how well it works with MSYS.
• As mentioned before, other Python essentials are written exclusively for Linux.
• Costs money.

Installing The Operating System

If your IT department has chosen to work with virtualization and you are working with VirtualBox you can get images for most Linux based OSes.

If your IT department are using VMWere, there are images you can use for some Linux based OSes as well.

Your mileage may vary.

If you listened to my advice and you chose Ubuntu, there is an automatic installer that can install itself on multiple computers. Ubuntu has only preliminary support and the automatic installer has some limitations and features that are not supported yet.

Again, your mileage may vary.

From now and on I assume you are have chosen to work with Ubuntu 11.10.

Images vs. Automatic Installers

Images are a "one size fits all" solution. If all 99% of your development workstations will be exactly the same, use an image because you can just deploy it and boot the system.

Further customization can be applied using boot scripts that will remove themselves after the first run but again, they script will be the same..

As mentioned, with Linux deploying images is by far more trivial. While incompatible hardware will often result in a crash on Windows won't crash Linux at all. All you need is that the operating system in the disc image will be the same as the deployment target's architecture and the disk image must contain all possible .

Automatic Installers are by far much more configurable and customizable but setting them up the first time can be quite consuming. If some of your development workstations will be different use an automatic installer.

Even if you are not planning to use an automatic installer currently due to the lack of time or resources, do take the time to learn what the different methods are and how to perform them.

Next time we will cover the scripting of the operating system installation.