Alex Coomans

Building a Custom 15TB RAID Storage Server

2012-12-05T00:00:00-06:00

I recently built a custom 15TB RAID storage server to store all of my media files, and now that is has been running reliably for a few months, I figured I'd cover how I built it.

The Hardware

Before this, I stored all of my data across about 5 external hard drives that I've acquired and filled over the years, and I really wanted everything to be in one place. I'd decided on using ZFS so I started there. Thanks to a post I saw on HackerNews I knew some basic requirements, and I also looked into FreeNAS. After a fair amount of research, I figured that the following would be my best bet:

Intel Server Board S1200KP because it supported ECC memory in a pretty small package, also has dual gigabit ethernet
Intel Celeron G530 2.4GHz since I needed it to be low power (also really inexpensive)
Kingston ECC 2x4GB Memory because ECC was recommended in the YC comments about the article above. This was one of the harder (relatively) parts to find since the board needed ECC Unbuffered memory
Corsair 60 GB SSD which was originally planned to the ARC cache for ZFS, but I later scrapped that after the FreeNAS install didn't work (more later on). It ended up being the main OS drive.
5 x Seagate Barracuda 3TB SATA drives for the main storage
HighPoint Rocket 2720SGL HBA because I needed more SATA ports (the cables I used)
An Athena Power 300W Mini-ITX power supply

The Case

I've always wanted to build a custom computer case (and while this isn't a full case), this project gave me the perfect opportunity.

I started off by looking for places that would do custom CNC work, and hopefully allowed me to avoid a CAD software (I'd already tried a design and failed). I found eMachineShop and they had a Windows app to help design what you wanted. I booted up my VM and after about a month of designing, printing, and test fitting using the parts I already had, I finally placed my order. The only downside was the price - it ended up costing me a little over $500 for two sets of all the pieces I needed. After about two weeks of waiting I received a UPS tracking number another week later I had them in my hands.

At the same time I placed an order for all of the standoffs & screws I needed for the case from DigiKey. Overall the standoffs have worked great, the only time I've had problems is when a few snapped at the top when I had to ship it (I have yet to order the replacements though), but overall it is pretty strong.

The Software

I'd originally planned to use FreeNAS to power everything, but when I tried to install FreeNAS, there was a bug related to my HBA card, so it wouldn't recognize my five 3TB drives. After a bunch of tries suggested by other people using the card with previous versions, I decided to switch to Ubuntu. This ended up working out perfectly for me since I'm much more familiar with Debian-based systems than I am with FreeBSD ones.

I started off with a server install of 12.04LTS on the SSD. It instantly recognized my storage drives, and I started working on setting up ZFS. The ZFS PPA was really easy to get installed and afterwords I set up the ZFS pool. The following was all I needed to get started with one large storage tank setup in a ZRAID configuration:

$ sudo zpool create tank raidz /dev/disk/by-id/ata-ST3000DM001-serial ...
$ sudo zfs create tank/store

This ZRAID left me with about 10.72TB:

$ sudo zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
tank        3.47T  7.25T  46.3K  /tank
tank/store  3.47T  7.25T  3.47T  /tank/store

Sharing

To get sharing started, all I needed to do was install Netatalk and Avahi, and this article helped me configure everything. In my /etc/netatalk/afpd.conf config file I've got:

- -tcp -noddp -uamlist uams_guest.so,uams_dhx.so,uams_dhx2_passwd.so -nosavepassword -nozeroconf -advertise_ssh

And then to actually share that, I added the following into my /etc/netatalk/AppleVolumes.default

:DEFAULT: options:upriv,usedots
/tank/store Dashr veto:/temp/ options:ro
/tank/store Dash allow:alex options:usedots,upriv

Which gave me two shares - one for my own use (that excludes a temp folder I have some disorganized files in), and then a read-only share without authentication for anyone else to use.

Remote Access

So recently I decided I wanted to be able to access the server remotely (I've got nginx running in front of some other apps I run), and because I'm stuck in an apartment complex that has it's own router, and I have a router for my unit, I was in a bind. Then I realized that I could reverse SSH tunnel from my server to an EC2 micro server I've got running. To set that up, I created a cron script on my server that runs every five minutes, mainly too reconnect if something crashes (credit to this article):

#!/bin/sh
COMMAND="ssh -N -f -R 8080:localhost:80 ec2-user@"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND

And then on my EC2 server in my nginx configuration file, I added to following:

upstream dash {
    server 127.0.0.1:8080;
}

proxy_cache_path /var/www/cache  levels=1:2    keys_zone=STATIC:10m
                                 inactive=24h  max_size=1g;
server {
    server_name remote.example.com;
    location / {
        auth_basic "Restricted";
        auth_basic_user_file dash.htpasswd;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://dash;
    }
    location ~* ^.+.(jpg|jpeg|gif|png|css|js)$ {
        auth_basic "Restricted";
        auth_basic_user_file dash.htpasswd;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://dash;
        proxy_cache            STATIC;
        proxy_cache_valid      200  1d;
        proxy_cache_use_stale  error timeout invalid_header updating
                               http_500 http_502 http_503 http_504;
    }
}

I also added in some HTTP caching to speed some of the static assets up so those wouldn't need to also be sent over the tunnel. While not the speediest of options, it has worked reliably for me.

Conclusion

This was a really project to work on; I've wanted to build a custom case for a long time, and even though I spent hours and hours designing the case and quadruple checking that everything fit, I'm really pumped about the final result. Here are a few more pictures of my server today:

Deploying a Multi-Environment PHP site with Git

2011-05-20T00:00:00-05:00

I've been working as the System Administrator/Operations Engineer/etc. for a PHP based site and I implemented a fairly simple system for deploying to different application environments.

The Server

First, it will probably help to know a little about the server so that some of the later parts makes a little more sense. The server runs nginx to serve static files, and then proxies to php-fpm to run the PHP files. For staging, it gets a little more complicated, as all requests are proxied through a central-authentication system (which I'd love to write about), but it essence is the same.

Managing environments, however, isn't as simple as with Rails, mainly because Rails is engineered around the concept of environments. I didn't find anything for working with environments in PHP, so I coded up an Environment class to make all of this simpler (feel free to copy for your projects).

<?php
class Environment {
    public function production() {
            return $_SERVER['SYS_ENV'] === 'production';
    }

    public function staging() {
            return $_SERVER['SYS_ENV'] === 'staging';
    }

    public function development() {
        return !($this->production() || $this->staging());
    }

    public function db_settings() {
        if($this->production()) {
            return array("host" => "localhost", "database" => "db", "user" => "db", "password" => "pw");
        } elseif($this->staging()) {
            return array("host" => "localhost", "database" => "db-staging", "user" => "db", "password" => "pw");
        } else {
            return array("host" => "127.0.0.1:5000", "database" => "db-development", "user" => "db", "password" => "pw");
        }
    }
}
?>

The class simply checks against an environment variable that I set in our nginx file (shortened).

server {
    location ~ .php$ {
        fastcgi_param  SYS_ENV  production;
    }
}

I also added the db_settings function so that our database configuration would be in one place, and then I simply call the function and it returns the appropriate settings, and you could copy and paste that method for other environment-specific variables.

Deployment

For this project, the team has a privately hosted git repo on GitHub, so I built a custom post-receive hook on an internal application that manages the deploys for us, and authentication is handled by HTTP Basic Authentication (hence the grayed out part of the url).

The GitHub Post-Receive Hooks Management Page

The control application to which I referred to does a lot more than handle deployments, and I may open source the part that allows us to rollback deployments, but below is a simplified version of the Sinatra application. If you are unfamiliar, Sinatra is a Ruby web framework that helps you build and deploy simple applications quickly.

require 'rubygems'
require 'sinatra'
require 'grit'
require 'crack'
require 'time'

post '/github' do
  basic_auth!
  payload = Crack::JSON.parse(params[:payload])
  if repos.keys.include?(payload["ref"].split("/").last)
    `cd #{repos[payload["ref"].split("/").last].path.gsub('.git', '')} && git pull && chown www-data:www-data -R .`
  end
  "ok"
end

private
  def repos
    @repos ||= {
      "production" => Grit::Repo.new("/var/www/example.com"),
      "staging" => Grit::Repo.new("/var/www/staging.example.com")
    }
  end

  helpers do

    def basic_auth!
      unless authorized?
        response['WWW-Authenticate'] = %(Basic realm="Restricted Area")
        throw(:halt, [401, "Not authorized\n"])
      end
    end

    def authorized?
      @auth ||=  Rack::Auth::Basic::Request.new(request.env)
      @auth.provided? && @auth.basic? && @auth.credentials && @auth.credentials == ['github', 'auth']
    end

  end

If you've never taken a look at the Post-Receive hook documentation, it simply lays out the information GitHub will POST to your server. In this case, I check to see if the push was to either the "staging" or "production" branches, and if so deploys it. I also have to perform the chown command since the web server runs under a different user than the application. It Github, you would add a URL like: http://github:auth@example.com/github to your post-receive hooks so that they can correctly authenticate. If you don't need authentication, you can simply remove that part of the Sinatra server and the github:auth@ part of the URL

A git pull may not be the best command for this type of operation (I'd love to hear if something would work better), but because I know no one will have touched any of the files in those folders, I can't imagine having any problems. Also, you could change this server away from using the Grit library, but I have it since other parts of the application rely on those being Grit::Repo objects.

When code is ready to be pushed to staging we simply merge the master branch into the staging branch and push, and when changes are ready for production, we merge in the staging branch into the production branch and push. Normally within a few seconds the changes have been updated on the server, and you can continue hacking.

Any thoughts on other environment strategies or improving this process?