I made a tutorial video (below) and have some documentation here: http://mattfordham.github.com/strap. If you find it useful, I'd love to hear about it. Anyone interesting in contributing to development is more than welcome to as well. Enjoy!

Strap from Matt Fordham on Vimeo.

One of the final pieces I’ve needed to figure out has been automated nightly MySQL and user upload backups to S3, which I typically setup for all projects. PagodaBox provides both Cron functionality and shared writable directories, so it seemed doable. The only question remaining was what commands were available to run via Cron. That question was answered by a Gist on Github.

The backup process has two steps. First, dump the database to a shared writable directory. Second, sync that directory to an S3 bucket. My usual MySQL dump script worked great. My go-to S3 sync tool, however, wasn’t an option, as Ruby doesn’t run on PagodaBox (I guess nothing is perfect). So, after a little hunting around, I found a PHP S3 sync option that did something similar. I tweaked the script a little, as it was using hashes for file naming, which didn’t work for me. Finally, here is the Boxfile needed to make this happen (your app will probably need more configuration than this).

I’ve got the whole package on Github, including my tweaks and some instructions. There you go, one step closer to PHP hosting nirvana.

Enter Capistrano. Capistrano is a Ruby tool that makes deploying any kind of website (including ExpressionEngine) a piece of cake. Put simply, it allows you to run a simple command in your local dev environment that in turn executes a more complex series of commands on a remote server (via SSH), such as deploying a website. Capistrano also makes rolling back to a previous version of the site a relatively easy thing.

As great as Capistrano is, it can be tough to get started with it. In this post, I’ll cover getting it setup and configured for an average ExpressionEngine site. Also, I made a little video walking through the process:

Local dependencies

As Capistrano is a Ruby tool, you’ll need Ruby running on your machine. For most OSX users, it should be there already. If not, try installing with RVM. Along with Ruby, you’ll need RubyGems, which will allow you to easily install Ruby projects.

Next up is Git. You’ll need to have Git running and have your project pushing to a remote repository. If you don’t already have a remote Git repo for your projects, give the free plan at BitBucket a try. BTW, Subversion works too if that’s your thing (Gods help you).

With Ruby and RubyGems in place, installing Capistrano is as simple as running “gem install capistrano” from the command-line. We’ll also be using the Railsless Deploy gem, which removes some Rails specific stuff from the default Capistrano deploy process (Capistrano was originally created with Ruby on Rails in mind). To install run “gem install railsless-deploy”.

Server dependencies

The server that you are deploying to will also need a little setup. I am assuming you have SSH access to the server and can install whatever you like.

The server, like your dev machine, will also require Git. Installing will depend on the specifics of your server, but a quick Google search should turn up the right commands to run. Also, you’ll need to add your server’s key to your repo host’s list of allowed keys (this is usually handled in the service’s “account” settings). Your repo hosting service should also provide docs on how to do this. Here is Github’s help docs on the topic, for example.

The server does NOT need to run Ruby, BTW. From the server’s point-of-view, Capistrano will seem like a regular SSH user logging in and running some commands. You will need to create a user for Capistrano to login as. You could also use the “root” user, but that is a little less secure. Finally, if you add your local dev machine’s ssh key to the server’s list of allowed keys, you won’t have to enter your password every time.

Capify

Ok, down to business. In the terminal, ‘cd’ into the directory of your project and run the following command:

Running this command will create the following:

The “Capfile” is the overall config for Capistrano. There will only be a few commands in there and they are basically “set it and forget it” type of things. For an ExpressionEngine project using the the railsless-deploy gem, I change my Capfile to look like this:

The “config/deploy.rb” file contains the specifics of your deployment, such as your server address, the Git repo and any custom tasks you want to run. We’ll come back to this file, as it contains the meat of the deployment.

Now, back in the terminal, run the ‘cap -T’ command and you’ll see the available default Capistrano “tasks”:

These are all the commands currently available. Some of the commands, such as “cap deploy” run a few of the other ones for you in series. OK, let’s pop open that deploy.rb file and replace ALL of the contents with the following:

So, those are all the basic settings I usually use. There are others, but I don’t often have to mess with them for a simple ExpressionEngine deploy.

For the deploy to work, Capistrano expects a certain structure within your application’s directory on the server. The “cap deploy:setup” command is supposed to get this prepared for you, but I have had issues getting it to work in the past, so I usually do it myself. On the remote server, run the following commands (change the paths for the specifics of your app):

That’s it. Capistrano expects a “releases” and a “shared” directory to be there. Now, run the following command from your local machine:

If your “deploy.rb” is configured up correctly, running this command will have Capistrano log in to your server and make sure it has everything it needs (permissions, the required directories, access to the Remote Git repo, etc). It’ll tell you if it was successful or not, and if not, why.

ExpressionEngine specific Capistrano tasks

Add the following to your deploy.rb file, making sure to adjust for your specific setup where noted:

When Capistrano runs a deploy, it checks out the latest code from the remote repo to a timestamped directory within the “/var/www/example.com/releases” directory. It then creates a symlink called “current” from the main deploy directory to the latest release. The “current” symlink is what you would point your server to. This way, the current release is what is served when someone visits your site.

The “create_symlinks” custom task above will create a symlink from the “images” directory in the “/var/www/example.com/shared” directory (which you will need to upload manually from your local environment) to the current release. The “images” directory is where any uploads via EE are stored. As uploads should not be in the repo, and they will need to be shared by each release, we create the “create_symlinks” task to ensure each release has access to the same shared files. You might also symlink some config files to the current release, if you are not wanting them to reside in the repo. The same thing is done for the “database.php” config file. Because the server database has different credentials than the local dev database, we symlink this file too (it too should not be in the repo). That said, I typically use a config bootstrap file instead of symlinking “database.php”, but I’m trying to keep things simple here.

The “set_permissions” task will ensure that some key ExpressionEngine files have the appropriate permissions with each deploy.

Deploying

Okay, we are now ready to deploy. First, commit local code changes and push to the remote repo:

Now run the deploy:

You should now see Capistrano do its thing. If any errors are encountered it will attempt to “rollback” to the previous deploy, making sure your site doesn’t go down. Congrats, you’ve deployed a new version of your site!

Conclusion

To sum up, our Capistrano deploy now basically does the following:

1. ssh into the remote server
2. Check out the latest code from the remote repo to a time-stamped directory in “/var/www/example.com/releases”
3. Create a symlink from “/var/www/example.com/current” to the latest release
4. Create symlinks for “images” and “database.php”
5. Sets the permissions of a few EE files and directories

Here is the complete deploy.rb file

Getting setup obviously takes a little bit of work (although it only takes 10 minutes once you’ve done it a couple times), but once its done, deploying couldn’t be much easier!

The Git Repo

The first step to a successful template for any software project is a version controlled repository. I, like many folks out there, use Git for this task. I have repos on Github, Beanstalk and BitBucket. I’ve had great experiences with all these services, but BitBucket provides free private repos for small teams, so you may want to check that out if you are just getting started. With my EE template under version control, I can make alterations to it and try out new things without the fear of losing anything. Note: As ExpressionEngine is a commercial product your template repo needs to be private.

The first step when starting new project with this system always involves updates to the template itself. With a clean copy of the template, I’ll make these updates and push the changes back to the main template repo. The updates may include updating the version of ExpressionEngine, updating an add-on I always use, adding some cool new goodness to the config bootstrap file, etc, etc. Usually experiences or troubles with the last project inform how the template gets tweaked (often times I’ll attempt to update the template in the midst of another project while such improvements are on my mind).

After the template itself is updated and feels as current as it can be, I’ll check out a new copy of the template and then kill the hidden Git directory that comes with it. By deleting the hidden Git directory (it is called .git and will be in the root directory of the project) I am breaking the connection with the template’s repo. I will then initialize a new Git repo in the directory for the project itself and can start committing project specific stuff.

The Details

As mentioned above, due to EE’s commercial license, I cannot share the repo itself. That said, below is a description of the overall layout and some additions and alterations of note. I’ll list out the structure first, and will then go over each piece (I have ordered things below for clarity…not for alphabetical perfection).

• __DISABLED.gitignore
• README
• Gemfile
• Guardfile
• Capfile
• ee_installs →
  • EE add-on zips go here
• ee_system →
  • codeigniter →
    • codeigniter stuff goes here… unchanged from the default install
  • expressionengine →
    • EE stuff goes here… mostly unchanged from the default install
  • index.html
  • index.php
  • config_bootstrap.php
• templates →
  • default_site →
    • main.group →
      • index.html
      • 404.html
    • shared.group →
      • _layout.html
      • styleguide.html
• public
  • static →
    • css →
    • img →
    • js →
    • fonts →
    • ee_images →
  • themes →
  • admin.php
  • index.php
  • .htaccess
• data →
  • ee-template_2012-07-08.sql
• config →
  • server config →
    • Some server scripts (automated backups, etc)
  • compass.rb
  • deploy.rb
• sass →
  • SCSS files
• coffeescript →
  • Coffeescript files

__DISABLED.gitignore

Including a .gitignore in a Git project will tell Git to not track certain files (here is a good start for EE ). In this case, I name the file “__DISABLED.gitignore” so that the template repo includes that stuff, so I don’t have to manually create it all at the beginning of every project. Prior to creating the repo for the actual project, I simply remove the “__DISABLED” part.

README

The README file contains some general usage notes about the template. This includes the above note about the .gitignore file and also the login credentials for a default admin user in the template’s database dump.

Gemfile

Many of the tools I use for development (more on the specifics below), regardless of the type of project, are Ruby tools. In the case of an EE project, none of the tools are used in production, they are just to aid in development. The Gemfile is used by Bundler to manage Ruby dependencies. By specifying all of my dependencies in the Gemfile, another developer can install them all with one command (as long as they have Ruby and Bundler installed).

Guardfile

Guard is a super-awesome tool for easily handling events on file system modifications, and the Guardfile is its config file. In the case of EE projects, it does the following for me:

• Reload the browser whenever I save a file
• Compile CSS files whenever I save a SCSS file
• Compile JS files whenever I save a Coffeescript file

The specifics of setting up and using Guard is out of the scope of this post, but check out the docs and maybe check back here for more related articles in the future. BTW, LiveReload and Codekit also accomplish similar things, if the command-line isn’t your thing.

Capfile

Yup, another config file for another Ruby tool. This time it is Capistrano. Capistrano is a tool for automating deployment of web projects. It was built for use with Rails, but it works great for any type of project. I can’t imagine deploying these types of projects without it (or something similar), and using FTP to deploy seems totally insane to me at this point (no offense).

ee_installs

This is simply a place to put zip files of EE add-ons. By keeping them in the template repo, I can ensure I always have the latest in a convenient spot when starting a project. I typically do not have them installed by default in the template (with a few exceptions), as every project calls for something different here.

ee_system

This is the default “system” directory with some tweaks. Firstly, it has been renamed. Renaming it is a good security measure and I prefer a less vague name. Secondly, it has been moved above webroot (which is the “public” directory), also for security and organizational purposes. The official EE docs describe how to do both of these things.

The “ee_system” directory also contains the “config_bootstrap.php” file. Mine is based on the NSM Config Bootsrap. This file allows for environment specific settings, such as database credentials for your dev environment vs. production. Not having to make any adjustments to settings upon deploying is a HUGE time-saver.

templates

This templates directory is pretty much what you’d expect, except that I’ve moved it to the root of my project directory structure… ‘cause I hate digging for it. Some basic starter templates are in the repo. The “_layout.html” template contains typical header and footer stuff. I use Stash and the template partials concept to pull this off.

The “styleguide.html” template contains some placeholder mark-up for most common HTML elements. The first step of my front-end development process includes creating the styles for these basic elements. I’d also throw other “modules” into this template too… basically chunks of design that will show up in multiple places.

public

This directory is where the web server points. “index.php”, “admin.php” and “themes” are the usual EE ones (edited slightly due to the move of the “system” directory). The “.htaccess” file removes “index.php” from URLs and does some other handy things.

I put all static assets under a “static” directory for a couple of reasons. First, it is a little cleaner. Second, it would make the move to a CDN, such as CloudFront, a little more straight-forward, if and when that happens. Generally, these directories should speak for themselves. The “ee_images” is the “images” directory provided by EE, but renamed in order to distinguish it from the “img” directory, which contains images used in the general design (as opposed to EE uploads, etc).

data

This directory starts out with one file only, an SQL dump of the database. The dump was created after running through the EE installer the first time and setting some basic preferences. The database dump is imported into a new database at the beginning of each project (I use SequelPro for such tasks). New dumps are added to this directory periodically once a project is started when importing/exporting from the development database to production.

config

This “config” directory is not the same as the one in the expressionengine directory. This one is for configuring some stuff outside of EE. The “server config” directory is a handy place to keep some scripts that usually end up on the server, but outside of the project. Among these are scripts for running automated nightly backups of the database and uploads to Amazon S3.

“compass.rb” is for setting some Compass preferences. Compass is a CSS authoring framework for creating stylesheets with SASS.

“deploy.rb” is the Capistrano deploy script for the project. It defines where the project gets deployed to, and what other deployment related tasks should be run and when, such as setting permissions and creating symlinks.

sass

As mentioned above, I typically use Compass for stylesheet development. SCSS (A SASS filetype) files are compiled into CSS files the “public/static/css” directory automatically whenever they are saved (thanks to Guard… mentioned above). A discussion of all the benefits of Compass and SASS is outside the scope of this post, but generally, SASS adds indispensable features such as variables, mixins and nesting to CSS.

coffeescript

Finally, we come to the Coffeescript directory. Coffeescript is kinda like SASS but for Javascript. It is an alternative way of writing Javascript that is MUCH cleaner, faster and way more fun. Guard automatically compiles the .coffee files into .js couterparts in the “public/static/js” whenever they are saved. Sidenote: I use Christian Johansen’s Juicer for combining and minifying all Javascripts prior to deploying.

Conclusion

Phew… are you still reading?… really? I glossed over a lot of the details, so be sure to post comments if there are pieces you’d like to hear more about. This template/workflow works really well for me, but I’ll be the first to admit that there may be some room for improvement… so please let me know what you think!

Usage

Add the following to your Gemfile:

Then run:

Then add the following to your model:

This will check to see if it is a properly formed CSV file.

In this case, csv_validator expects :my_csv_file to be an instance of ActionDispatch::Http::UploadedFile, which is created by default for Rails uploads.

Other validation options

There are a handful of other options for more specific validation too. A complete list is below… note that you wouldn’t use all simultaneously.

Check it out on Github