Michael van Rooijen

Deploying your static nanoc sites to Heroku's hosted platform with Nanoc::Heroku

2011-07-22T22:00:00Z

So you’ve created a new application and would like to launch it. It is a Ruby application, but you created the front pages using nanoc, a static site generator. Your Ruby application will be deployed to Heroku’s platform, but where will your static site run? A separate VPS? A shared host?

I personally dislike the idea of having to deploy parts of my application (the front pages, the application itself) to different servers/platforms. Shared hosting sucks, so let’s assume we get a VPS to host our static front pages. First of all, adding a new VPS costs money. You have to then set up a web server, and ensure it stays up. You have to keep the server up to date with security patches, and if things go wrong, then you’ll have to solve these issues yourself. All that for a simple static website.

I enjoy deploying applications to Heroku. It’s easy, it’s fast, it’s performant, it’s secure, it’s managed and you go from “nothing” to “deployed” in seconds. Wouldn’t it be nice if we could quickly create a static site using a great site development tool, and (optionally) leverage other tools like haml, compass, coffeescript, then render the html, and deploy it to Heroku, for free?

Nanoc::Heroku

Heroku currently supports Ruby, Node and Clojure, but not (directly) raw HTML sites. However, that doesn’t mean that we can’t use a little bit of Ruby and Rack to get around it.

Nanoc::Heroku is a little bare-bones nanoc application/structure that I set up. It is not a gem, you simply clone it from the repository and immediately get to work with a few commands. Let me highlight a few interesting points:

It has out-of-the-box support for:

HAML / SASS
Compass
CoffeeScript
Markdown
Builder

It auto-compiles your site when adding/updating files using:

Guard

It contains rake tasks for:

Optimizing Images of format: JPG and PNG
Compressing Stylesheets and Javascripts

It deploys to Heroku with:

rake deploy

after changing a constant in the Rakefile and running heroku create mysite --stack cedar.

Getting started

To get up and running quickly, all you have to do is clone the git repository, remove the origin remote, install the bundled gems, start guard and the nanoc server. Here’s how:

Clone the repository, remove remote, install dependencies:


  git clone git://github.com/meskyanichi/nanoc-heroku.git my_site_name
  cd my_site_name && git remote rm origin && bundle

In the current shell, run the following to do the initial compiling and start the server afterwards:


  nanoc compile && nanoc view

And finally, open a second shell tab and run the following as well:


  guard

Visit http://localhost:3000/ and your site should be up and running. Every time you add/update a file inside the ./content directory, Guard will run nanoc compile to deliver a fresh state of the latest changes, rendered in ./output. These will be immediately visible on http://localhost:3000/ after a browser refresh.

Refer to the nanoc documentation for more in-depth information about the static site generator itself.

Deploying to Heroku

First of all, let’s create the Heroku application. It must be running on the Celadon Cedar stack. Note: The Heroku gem updates very frequently, so it’s a good idea to run gem install heroku once in a while to ensure you’re on the latest version.


  gem install heroku
  heroku create mysite --stack cedar

This will add the heroku remote to your git repository. Next, open up your Rakefile, it contains a constant called BASE_URL. The value of the BASE_URL should be your domain url. For example: http://github.com. This is required because when you run the deployment task, it will update the config.yaml to ensure it compiles with the proper URL’s in your website, so your website’s link to link to http://localhost:3000/ urls.

Now, before we deploy we need to install the image optimization utilities, and the Java runtime for the CSS/JS compressor. On OSX using Homebrew, you can install the image optimization utilities by running:


  brew install jpegoptim optipng

On Linux (Ubuntu) with apt-get or aptitude, the equivalent is:


  apt-get install jpegoptim optipng
  aptitude install jpegoptim optipng

The master branch will always be used to deploy your website. You could tweak the Rakefile to change this behavior. Ensure that you are working in a clean working directory (that there is nothing to commit). Once you’re ready to deploy the master branch to Heroku, simply issue the following command:


  rake deploy

This will create a new branch called deployment, and checkout to it. It will then take the value of the BASE_URL constant (which is defined in the Rakefile) and apply it to the config.yaml file and will re-compile the whole website. It will remove the output entry from your .gitignore as well so it can commit the raw files. After that it’ll commit these changes to the deployment branch and push it to Heroku. Once deployed, it will checkout back to the master branch and remove the deployment branch again. You are now back at the state you were before you ran rake deploy, except that you actually deployed your website to Heroku.

You don’t have to worry about this procedure though. Just sit on a clean master branch and run the Rake task. If the master branch isn’t clean, it will tell you upon initializing the Rake task and abort the operation.

And That’s it! Your static site should now be running on Heroku! Ensure the web process is running. You can check the process list by running heroku ps. If it isn’t listed, be sure to run heroku ps:scale web=1. When you see that there is one process running, run heroku open to open up your browser to view the application.

You can of course use this for any static site, not just for front pages of your applications. You could use it for a blog, or other things. So far I have already moved this blog, as well as the HireFire front-page (both of which are using Nanoc::Heroku) to Heroku, and I’ve been very satisfied with the migration!

Automate your application's development environment with Guard

2011-06-18T22:00:00Z

Do you find yourself repeating various actions over and over again in your development environment? Examples of such actions are restarting your application server for your Rails app every time you update a files that are loaded in memory and require a server restart for the changes to take effect. Maybe you are using Nanoc, a static site generator and would like it to automatically recompile your site every time a file is saved. Or maybe you are using a testing library such as RSpec and want to automatically run your tests every time you save a Ruby file.

There are lots of things that could be automated. Some tools come with built-in utilities to take care of this (think of autotest, nanoc autocompile, etc), but they will all require their own separate terminal window.

Enter Guard

Guard is a RubyGem, written by Thibaud Guillaume-Gentil (@thibaudgg), that you could use to easily automate parts of your application’s development environment.

To show you what it can do, here’s an example: Let’s say we have a Ruby on Rails application, using Bundler to manage gem dependencies and RSpec as our testing library. Each of these bring with them repetitive actions. So let’s automate them.

We could use the following gems:

Guard::Rails
Guard::Bundler
Guard::RSpec

These gems are basically Guard “modules”. Simply add them to your Gemfile. Now create a Guardfile in your application’s root directory and add the following.


  guard 'rails' do
    watch('Gemfile.lock')
    watch(%r{^(config|lib)/.*})
  end

  guard 'rspec' do
    watch('spec/spec_helper.rb') { "spec" }
    watch('config/routes.rb') { "spec/routing" }
    watch('app/controllers/application_controller.rb') {
      "spec/controllers"
    }
    watch(%r{^spec/.+_spec\.rb})
    watch(%r{^app/(.+)\.rb}) { |m| "spec/#{m[1]}_spec.rb" }
    watch(%r{^lib/(.+)\.rb}) { |m| "spec/lib/#{m[1]}_spec.rb" }
    watch(%r{^app/controllers/(.+)_(controller)\.rb}) { |m|
      ["spec/routing/#{m[1]}_routing_spec.rb",
      "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb",
      "spec/acceptance/#{m[1]}_spec.rb"]
    }
  end

  guard 'bundler' do
    watch('Gemfile')
  end

Let’s go over what the above Guardfile + Guard Gems will do for us.

With Guard::Rails, if a file in the config or lib directories are updated, or the Gemfile.lock file gets updated, the rails app server will automatically be restarted. Normally, you would have to open the shell manually and restart the process yourself. With Guard::Rails, this will be done for you.

With Guard::RSpec, when you save for example a controller file, only that specific controller’s spec file will be run, rather than running all the tests. The same goes for models. You can customize this behavior to suit your needs. Another cool feature, which is particularly interesting when you’re working on gems that need to support multiple Ruby versions, is that you can pass in a hash of options to the guard 'rspec' method in the Guardfile. For example if you want to run the tests against MRI 1.9.2, 1.8.7, REE 1.8.7, JRuby and Rubinius, and you are using RVM, you could simply define the following:


  guard 'rspec', :rvm => ['1.9.2', '1.8.7', 'ree', 'jruby', 'rbx'] do
    ...
  end

Now every time Guard runs your tests, it’ll run them against the all defined Ruby versions and implementations.

With Guard::Bundler, it simply watches for changes in the Gemfile, and will re-bundle it when changes are made. Rather than having to manually go in to the terminal again to run bundle install, this will be done for you. Also, notice that the Guard::Rails has this line defined: watch('Gemfile.lock'). When Guard::Bundler runs the bundle install command, it’ll update the Gemfile.lock file, which means that dependencies have changed and that the Rails server will also be restarted automatically for you by Guard::Rails. Again, no need to go in to your terminal to restart it manually.

Another useful feature is that it supports growl notifications (or libnotify on Linux). So you could have a small notification pop up on your screen when tests run, or when the Rails server has been restarted, without having to check the shell.

Guard is very modular, and there are a bunch of modules available right out of the box that automate particular parts of your development environment. It is also easy to write your own Guard modules, refer to Guard’s README on GitHub for more information.

Managing and monitoring your Ruby application with Foreman and Upstart

2011-06-07T22:00:00Z

In my last post I briefly mentioned a gem called Foreman. Foreman is a gem that allows you to easily organize and horizontally scale your various processes in a centralized manner. For example, in a typical Ruby on Rails application you might have a few different processes: your application server (Thin, Unicorn, etc), a worker (Resque, Delayed Job, etc), a pubsub server (Faye, Juggernaut, etc). Whatever you’re using, you should be monitoring it to ensure that whatever you spin up, stays up.

There are various tools available to do this sort of thing. In Ruby you have God and Bluepill, in Python you have Supervisord, in C you have Monit, and there are plenty more. The one I want to talk about today however is called Upstart, because Foreman makes it easy to “export to” (setup) Upstart in your production environment. It even makes your development environment a lot more pleasant to work with if you have more than one process to run, which is likely the case.

The reason I started using Foreman is because I often have applications that consist of 3 or more processes. This means that I have to keep 3 or more shell tabs open at all times, and also have to switch between them to see each of their output. With Foreman, this is no longer the case.

Getting started in development

Foreman is very simple to set up in development. All you have to do is create a file called Procfile in the root of your application and define your processes inside of it. Here’s an example Ruby on Rails application using Thin as its app server, Resque as its worker process and Faye as its pubsub server.


  web:     bundle exec thin start -p $PORT
  worker:  bundle exec rake resque:work
  pubsub:  bundle exec thin start -p $PORT -R ./faye.ru

With this in place we are now able to start using Foreman. As you can see we’re executing every command from within the bundled environment using Bundler. This is always a good idea as it isolated each environment and avoids gem version conflicts.

Installing Foreman is as easy as running gem install foreman. With Foreman installed, we can now start all our processes with a single command from our application’s root.


  foreman start

You’ll see that Foreman starts up all the processes defined in the Procfile, and also give each of the processes a different color in the console for readability. It looks something like this:

One of the key features of Foreman is the ability to add concurrency to each of the processes by initializing Foreman with additional parameters. In development this might not be too interesting, but in production it sure is.


  foreman start -c web=4 worker=2 pubsub=1

This would spin up 4 Thin instances, 2 Resque instances and 1 Faye instance. By default Foreman will use port 5000 as a starting point. Then it increments by 1 for the corresponding process, and by 100 for each separate process. In the above code snippet that would mean:

Thin runs on ports 5000 5001 5002 5003
Resque runs on ports 5100 5101
Faye runs on port 5200

You can change the starting port using the -p option. It must be a 1000-value (1000, 2000, 3000, etc). Refer to Foreman’s the man page for more information.

From now on when you want to start up your development environment, all you have to do is to cd in/to/your/app and run foreman start. This is really nice, especially if each process has a couple of arguments you would normally need to remember and type in every time to start up your whole environment.

Converting Procfile to Upstart configuration files

Our application has now be deployed to our server, but we need to initialize the processes and keep them monitored. Some Linux distributions come with Upstart pre-installed. If yours does not, you can install it from source by downloading it from the official website.

Assuming your machine does have Upstart installed, we need to write configuration files to the init directory. On Ubuntu, by default, this would mean writing the configuration files to /etc/init. It also requires you to have root access. If you’re using Foreman, you’ve actually already indirectly written the configuration files: your Procfile in your application’s root. Now we need to export (convert) that Procfile to Upstart configuration files.

You can do so by running the following command:


  foreman export upstart /etc/init -a myapp

You should then see the following if all went well:


  [foreman export] writing: /etc/init/myapp.conf
  [foreman export] writing: /etc/init/myapp-web.conf
  [foreman export] writing: /etc/init/myapp-web-1.conf
  [foreman export] writing: /etc/init/myapp-worker.conf
  [foreman export] writing: /etc/init/myapp-worker-1.conf
  [foreman export] writing: /etc/init/myapp-pubsub.conf
  [foreman export] writing: /etc/init/myapp-pubsub-1.conf

If you’re running this command as a non-root user, then you must prefix the command with sudo


  sudo foreman export upstart /etc/init -a myapp

If it says it cannot find the foreman command, and you’re using RVM (Ruby Version Manager), be sure to use rvmsudo instead of just sudo


  rvmsudo foreman export upstart /etc/init -a myapp

If you want to have more concurrency for some processes, you must specify it during export as well:


  foreman export upstart /etc/init -a myapp -c web=3 worker=2 pubsub=1

Be sure to export it for the same user that’s deploying your application and has write access to your application path. If your deployments user name is “deployer”, you should specify it like so:


  foreman export upstart /etc/init -a myapp -u deployer

If you don’t set it to the deployment user, it might be unable to properly start up your application. You can overwrite the configuration files at any time by re-running the foreman export command.

Here’s a full example of I’m using for one of my applications:


  rvmsudo foreman export upstart /etc/init -a myapp -u deployer -c web=3 faye=1 worker=1

Start monitoring all your processes

Now that the configuration files are in place, all that’s left is to start the monitor and it’ll ensure your processes are always up and running, and restore them from crashes if necessary. To tell Upstart to monitor our processes, simply run the following command:


  sudo start myapp

myapp is the name of your application you set by specifying the -a option in the foreman export command. If you do not specify the -a option, it’ll name your application after the name of the root directory where your application resides.

That’s it. Now all your processes should be up and running! Try to kill off a process using kill <pid> and you’ll see that it’ll be instantly respawned by the Upstart utility!

If you want Upstart to automatically start up and monitor “myapp” when you restart your server, you can add this line at the top of your /etc/init/myapp.conf file


  start on runlevel [2345]

With this in place, each time you reboot your server, it’ll automatically spin up and monitor your application processes by itself so you don’t have to manually SSH in to do this.

Additionally you can also bind services other than your application-specific processes to Upstart via your Procfile, for example: nginx, mysql, postgresql, mongodb, redis, memcached, and so forth. You could also let Upstart handle that for you to ensure each of these is also always up and running, and starts up when your system boots. For example, let’s say we have Unicorn as our app server, Resque as our worker library, NGINX as our web server and MongoDB as our database. We would set up our Procfile to look something like this:


  web:    bundle exec unicorn -p $PORT -E production
  worker: bundle exec rake resque:work -e production
  mongo:  sudo /etc/mongodb/bin/mongod --journal
  nginx:  sudo /usr/local/sbin/nginx -c /etc/nginx/conf/nginx.conf

With that you can then re-export that to Upstart and run restart myapp. Upstart will now also start up and monitor your MongoDB and NGINX instances. Some of the commands that become available when exporting to Upstart are:

start myapp
stop myapp
restart myapp

These will start/stop/restart all processes (web, worker, mongo, nginx). If you only want to restart all the web (Thin, Unicorn) processes, because you just pushed changes to your application and need to restart the app instances for them to take effect:

restart myapp-web

These commands are available from anywhere on your machine, you don’t need to be in the root of your application or anything, but you’ll have to execute them as the root user, or with sudo.

As you can see, Foreman beautifully integrates in both your development, as well as your production environment. If you’ve set it up in your development environment, which should take less than a minute, you’re pretty much good to go when you deploy to production. I actually use 2 Procfiles: Procfile and Procfile.production. This is because I want to specify different flags for my production environment, such as -e production. You can tell Foreman to load a different Procfile, than the default.


  foreman start                           # defaults to ./Procfile
  foreman start -f ./Procfile.production  # uses ./Procfile.production

Conclusion

Foreman and Upstart can really save a lot of time, both in development, as well as in production. It’s a simple, reliable and awesome solution. It takes about a minute or less to set up locally for in development, and even less in production since all you have to do is export your Procfile and tell Upstart to run it. You have the joy of booting up your development environment with a single command to get to work quickly without having to remember all the command options/flags, and you can export your Procfile to Upstart in production and have it take care of keeping your processes up and running.

More concurrency on a single Heroku dyno with the new Celadon Cedar stack

2011-05-31T22:00:00Z

Heroku recently released the public beta of their new stack called Celadon Cedar. Check out their blog post for more information. There are a few minor adjustments to how their pricing model work. Free plans are not affected and it generally benefits the consumer. I am truly excited to see this because I’ve been using Foreman, a gem by David Dollar, which allows you to nicely organize, view and manage your processes. I’ll be doing an article on Foreman in the future. Heroku’s Cedar stack takes the same approach as Foreman, and you can (and should) actually use Foreman in development. If it works there, it probably works on Heroku’s new Cedar stack as well.

The newly mentioned capabilities of the Cedar platform however, are not the only thing I’m excited about. When someone told me about this stack prior to the release of the public beta, and showed me a Procfile, the first thing I asked was:

“Can I ride a Unicorn?” - And his answer was: “Yes”

Unicorn

I’m sure most of you know, or at least have heard about Unicorn. If you’re a Ruby on Rails developer, you’ve probably seen the Unicorn gem in your Gemfile. But, for those that don’t know, or never tried it out before, I highly encourage you to check out this post sometime, by Chris Wanstrath over at Github.

What I’d like to point out right now for this article is that Unicorn has a “master” process, and has one or more “worker” processes. The “master” process itself doesn’t see any requests, but instead, it manages the worker processes, which are its child processes. They get forked from the master process. The kernel handles load balancing, and you only connect via a single unix socket or port using an upstream in, for example, NGINX. Unlike with for example Mongrel and Thin, where you upstream to multiple ports.

In any case, what’s important to understand for now is the fact that there’s a master process which loads in your application’s environment, and forks off child processes to serve your application.

Cedar and Unicorn - More concurrency per Heroku dyno

At this point you may already know where I’m going with this. Let’s create a basic Rails application.


  rails new cedar-on-unicorn

Generate a simple controller which won’t interact with the database


  rails generate controller home index

Open up your Gemfile and ensure you have the following defined


  gem 'rails', '3.0.7'
  gem 'unicorn'

And now comes the interesting part, create a unicorn config file in Rails.root/config/unicorn.rb


  worker_processes 4 # amount of unicorn workers to spin up
  timeout 30         # restarts workers that hang for 30 seconds

And a Procfile for Heroku’s new Cedar stack in Rails.root/Procfile


  web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

Bundle it up


  bundle install

Initialize a new Git repository, commit it, and create your new Heroku Cedar Stack-based environment and push to it


  git init
  git add . && git commit -m "concurrecy"
  heroku create --stack cedar
  git push heroku master

Now, visit http://myapp.heroku.com/home/index and you should be up and running.. with 4 unicorn worker instances, on a single Dyno! Prior to Celadon Cedar, you could only deploy Thin application servers, which don’t have this kind of process management. Thin doesn’t have a master process which forks children, so this wasn’t possible. If you wanted more concurrency / instances, you had to spin up more Heroku dynos.

But don’t over-do it

“Too much” is never a good idea. A dyno is essentially it’s own little isolated environment. You probably get your own dedicated amount of CPU cycles, and there is a limit to how much RAM you may consume (for obvious reasons), even if you can’t see your memory usage. I believe that there is a 300mb memory usage limit per dyno. Also, the more instances you try to cram in to a single VM, the worse your throughput gets when your VM starts to reach its resource limits, in which case you’re better off with less instances.

Benchmarks!

Yup. I’ve done some basic AB runs to see the difference in throughput. Now, I’m not going to show every detail of these results, but I think you’ll get the main picture.

All tests were performed on a page where there is no interaction with a database, because that’s not what I’m testing. The tests were performed by doing 1000 requests, at a concurrency level of 100. The tests have been performed multiple times and I took the average results of each test.

1 Unicorn Worker (Roughly equivalent to a single Thin instance)


  Time taken for tests:   3.835 seconds
  Requests per second:    260.77 [#/sec]

2 Unicorn Workers


  Time taken for tests:   1.689 seconds
  Requests per second:    592.00 [#/sec]

3 Unicorn Workers


  Time taken for tests:   1.164 seconds
  Requests per second:    833.91 [#/sec]

4 Unicorn Workers


  Time taken for tests:   0.979 seconds
  Requests per second:    918.90 [#/sec]

5 Unicorn Workers


  Time taken for tests:   10.621 seconds
  Requests per second:    83.32 [#/sec]

See what happened? It seems that spawning 5 workers is “too much” and actually reduces throughput. 3-4 workers per dyno seems to be your best bet in this case. There were also times that spinning up 4 workers was too much. Notice how little 4 workers improves over 3 workers. This is clearly its limit. This is a bare-bones web application, if you actually wrote code, added gems and such, then 4 unicorn workers might be too much.

But! If having 3 Unicorn workers isn’t enough for you, go ahead and spin up 3 more:


  heroku scale web=2

This will spin up a second dyno that’ll spin up 3 more Unicorn workers, bringing it up to a total of 6 Unicorn workers spread across 2 dynos! Need 30 workers?


  heroku scale web=10

With this you have 30 Unicorn workers spread across 10 dynos, and so forth.

Imagine if you were using Sinatra instead of Rails, and your app consumes a lot less memory. You may even be able to spin up 7-8 Unicorn worker instances per dyno.

Conclusion

Before Heroku released the Celadon Cedar stack, this simply wasn’t possible. You’d have a dyno that should (in terms of memory) be able to spin up 3 or 4 instances, but you were only able to spin up a single Thin instance per dyno. With Thin, even after the introduction of Cedar, this is still the case. But now since the application server constraint no longer exists, you can use Unicorn to do it. With this, you should be able to improve your application’s throughput by a factor of 3 to 4 on the same dyno.

Setting up CoffeeScript in Rails 3

2011-05-27T22:00:00Z

About a year ago I attempted to start a blog, mainly to write about Ruby, basic server provisioning and other related technologies to share my knowledge and experiences to help other people out or start discussions. Unfortunately for various reasons I wasn’t able to spend enough time to do this back then.

Thankfully this is no longer the case and I’m excited to pick it up again!

Last week I decided to throw away the few articles I wrote up back then since they are pretty much obsolete now. I re-designed and rewrote my blog. Hope you like it!

Now, on to the actual article

As of late, more and more people are getting in to CoffeeScript. It was previously mentioned that CoffeeScript would ship with Ruby on Rails 3.1. Although you can easily switch it off if you prefer to write raw JavaScript. But it’s nice that they provide the option to quickly get up and running with CoffeeScript if you wanted to, I know I’ll be using it!

I recently released a new application I was working on and I had trouble getting CoffeeScript working in my production environment. CoffeeScript requires the V8 engine. I was using Node which enabled me to use CoffeeScript by installing NPM and then installing CoffeeScript via NPM. This was tragic. I previously somehow got it working in my staging environment, but when I tried to push it to production I simply could not get it to compile my .coffee files to .js files. I figured it had something to do with the load paths even though I set them up. I asked a bunch of people for help regarding this, but unfortunately no one I asked had a direct answer to my Node/NPM problem.

The solution

After spending a good amount of hours researching this, someone pointed out there is an open source project called The Ruby Racer. This is the V8 runtime, embedded in Ruby - sound cool? Well, it is. Since I only installed Node and NPM for the purpose of being able to use CoffeeScript, I figured that it’s pretty much overkill to use it.

I removed Node and read up on The Ruby Racer. It turned out to be incredibly easy to set up!

In a typical Ruby on Rails 3 app that ships with Bundler, all you define are the following gems in your Gemfile:


  gem 'therubyracer'
  gem 'barista'

That’s it. No Node, no NPM, just 2 gems defined in the Gemfile.

Barista is a gem that seamlessly integrates CoffeeScript support in to your Rails or other Rack-based applications. It only took me a minute to set up.

So if you want to use CoffeeScript, and nothing else Node related, you should definitely take a look at this solution before installing Node just for the purpose of being able to use V8 with CoffeeScript in Ruby.

Wrapping it up

This was a simple initial / introduction article, but it may save a couple of hours of headaches for developers trying to deploy their code base with unnecessary complexity as I did.

From here on out I’ll be frequently submitting new articles. They will mostly be related to Ruby, (App/Web/OS) server technologies, open source and perhaps various (actual) community related discussions/thoughts.

So if that interests you, be sure to keep checking back!