James Andres's Tips & How-To's

Subversion Deployment: DGEDeploy v2 and DGEPush

2008-09-17T16:14:38Z

Subversion Deployment: DGEDeploy v2 and DGEPush

Wednesday, September 17, 2008

Back in February I released a Ruby script called DGEDeploy . We use DGEDeploy internally at Donat Group to help manage several of our project deployments.

Besides being a misnomer, DGEDeploy had one major weak spot: actually pushing code to servers. That, and a few other bugs, are fixed now in this latest release. Heck, lets call it version 2.

The latest version adds a new script, DGEPush, that acts as a simple RSync front-end. When using SSH keys it’s a breeze to merge a development branch into staging and push to the server.

DGEPush features list: * Rsync’s code using the -C option to skip .svn and CVS directories. This saves server space, and is a better security practice. * Can use SSH keys for better security and ease of use. * Can deploy to multiple servers in one shot.

Here’s how you get it running:

Create a dgepush user, on your remote server

On Ubuntu / Debian this is as simple as:

# Log into your remote server
ssh remote-server.com

# Create an account called 'dgepush'
sudo adduser dgepush

If you like, set up key based password-less logins for the remote user:

# On 'local' machine, ie: your laptop
sudo adduser dgepush
sudo su dgepush
cd ~

# Create the SSH keys
mkdir -m 700 .ssh
cd .ssh
ssh-keygen -t rsa -f dgepush-key
# !! Don't enter a passphrase, just hit Enter and Enter.

# Copy the key to the remote machine, this syntax only works on Linux.
# Possibly only on Debian / Ubuntu Linux.
ssh-copy-id -i ~/.ssh/dgepush-key.pub dgepush@remote-server.com

Test your SSH keys ssh -i /home/dgepush/.ssh/dgepush-key dgepush@remote-server.com should log right in without a password.

Here’s some other tutorials on the subject: * Password-less logins with OpenSSH * ssh-keygen: password-less SSH login

Update your dgedeploy-projects.conf

Adding a ‘servers’ section to any workflow enables you to get started with DGEPush.

/etc/dgedeploy/dgedeploy-projects.conf

# YAML 1.1
# Settings for example-project deployment
example-project:
  # The workflow field is required, the order of the parameters specify which
  # branch merges with which branch.  eg:
  #     development --> staging --> live
  workflow:
    - development
    - staging
    - live
  development:
    # The order of the urls parameter is important.  The first in the list will
    # be used as the default choice whenever necessary.  Each url should be,
    # a directory inside a Subversion repository, of course.  All urls should
    # be in the SAME Subversion repository.
    urls:
      - http://example.com/svn/example-project/trunk
  staging:
    urls:
      - http://example.com/svn/example-project/tags/staging
    servers:
      # This server uses key based password-less logins.
      - "-e 'ssh -i /home/dgepush/.ssh/dgepush-key' dgepush@stage.com:/var/www/stage.com/"
      # You can push to multiple servers by adding more lines!
      - "dgepush@mirror-stage.com:/var/www/mirror-stage.com/"
  live:
    urls:
      - http://example.com/svn/example-project/tags/live
    servers:
    # This server will require a password every time DGEPush is run.
    - "dgepush@live.com:/var/www/live.com/"

DGEPush in action

If you have already got DGEDeploy up and running using DGEPush is one more step. Here’s a basic example:

# First merge trunk into staging branch
sudo dgedeploy example-project http://example.com/svn/example-project/trunk
# Next push the staging branch to the server
sudo -u dgepush  dgepush example-project staging

Important points: * sudo -u dgepush runs the command as the dgepush user * dgepush example-project staging copies the contents of the dgedeploy cace for ‘example-project/staging’ to remote server. In other words, DGEPush won’t work unless you have initialized your project with DGEDeploy at least once.

Download DGEDeploy-v2.0.tar.gz

new comment

Back on the blogging train

2008-09-17T05:51:32Z

Back on the blogging train

Wednesday, September 17, 2008

Wheew, it’s been a crazy summer. Excuses for my absence include:

I got married!

I moved!

Ahhh glad to be back.

new comment

HOWTO Compile MPD With AAC/M4A Support on Ubuntu

2008-11-12T20:00:21Z

HOWTO Compile MPD With AAC/M4A Support on Ubuntu

Monday, March 24, 2008

Linux’s Music Player Daemon, MPD, is an interface-less jukebox. It’s like iTunes, Winamp, or Rhythm Box, except there is no built in GUI. Think of it as the Apache or MySQL of the music player world.

MPD is great for remotely controlling your Linux powered entertainment system or, as in my office’s case, to allow 15 staff to each act as work-day DJs.

Our office music server runs Ubuntu 7.10, but the default mpd package for Ubuntu doesn’t have M4A (AAC) support for MPD. Here’s what we did to re-compile MPD.

This tutorial will set you up with a new MPD with support for OSS/ALSA that will play the following formats:

MP3
AAC
OGG Vorbis
FLAC
WAV

First, MPD requires libfaad2-dev and libmp4v2-dev to compile with AAC support. I’m using the debian-multimedia repositories to get these packages and they don’t have versions of libfaad2-dev and libmp4v2-dev suitable for Ubuntu 8.10.

No worries, this is a good excuse to learn how to build packages “gentoo style” using apt to build packages from source.

You’ll need the appropriate repositories enabled. Check your /etc/apt/sources.list file to ensure you have at least universe and multiverse enabled. Then add the debian-multimedia repositories. Append the below lines to your /etc/apt/sources.list.

## Debian Multimedia
deb http://www.debian-multimedia.org stable main
deb-src http://www.debian-multimedia.org stable main
deb http://www.debian-multimedia.org testing main
deb-src http://www.debian-multimedia.org testing main
deb http://www.debian-multimedia.org unstable main
deb-src http://www.debian-multimedia.org unstable main

Update your apt package database with an apt-get update. Next, we need to install the basic dependencies that will allow you to compile the software.

sudo apt-get install build-essential dpkg-dev \
     debhelper libasound2-dev libid3tag0-dev \
     ccache libxvidcore4-dev libid3-3.8.3-dev \
     libgtk2.0-dev libesd0-dev libsdl1.2-dev \
     liba52-dev liblame-dev libxt-dev \
     nasm libx264-dev dpatch \
     autotools-dev libfaad-dev libvorbis-dev \
     libimlib2-dev libfaac-dev texi2html \
     libtheora-dev libgsm1-dev libxvmc-dev \
     libamrnb-dev libamrwb-dev libsamplerate0-dev \
     libflac-dev fakeroot checkinstall \
     libmp3lame-dev quilt libdc1394-22-dev \
     libdirac-dev libschroedinger-dev

Once that has finished coming down the wire we’re almost ready to start compiling. Let’s get the MPD source code, and some initial dependencies.

sudo mkdir -p /usr/local/src
cd /usr/local/src
sudo wget http://www.musicpd.org/uploads/files/mpd-0.13.2.tar.gz
sudo tar zxvf mpd-0.13.2.tar.gz
sudo chown -R "$USER":admin mpd-0.13.2*

NOTE: The above snippet download’s mpd version 0.13.2. This is up to date as of November 12, 2008.

Time to compile! We’re using apt-get -b source PACKAGE-NAME to download and compile the source packages. This is a good way to do things since it will create DEB packages automatically, plus it’s good Debian form. I recommend learning how to use this feature of apt.

sudo apt-get -b source libavcodec-dev libavutil-dev
sudo dpkg -i *.deb
sudo apt-get -b source libmp4v2-0
sudo dpkg -i *.deb

Lastly we compile and install mpd. Run the following commands.

cd mpd-0.13.2
make clean
./configure
make
sudo checkinstall -D make install
sudo ln -s /usr/local/bin/mpd /usr/bin/mpd

If you haven’t received any errors, then you’re good to go!

The MPD Wiki is a good resource for help with the rest of the set up and configuration.

new comment

DGEDeploy Is A Subversion Merging And Release Control Script

2008-02-24T07:37:29Z

DGEDeploy Is A Subversion Merging And Release Control Script

Sunday, February 24, 2008

DGEDeploy is a Ruby script that helps make subversion merges a little easier. I wrote it at Donat Group Enterprises for use as a light weight tagging and release control solution.

Why?

I wrote DGEDeploy to:

Make tedious ‘svn merge’ commands easier to use.
Reduce the number of typos when running ‘svn merge’ commands.
Be extremely simple, its not meant to replace Capistrano, CruiseControl, ...
To improve my Ruby scripting skills

There are a lot of similar scripts out there, but if you find this one useful please let me know :-).

How does it work?

The script uses two configuration files to keep track of vital settings. The first, ‘dgedeploy.conf’, stores settings for the script itself (ie: subversion credentials). The second, ‘dgedeploy-projects.conf’ stores information about your codebase and projects.

All settings are stored in the YAML format.

Example: /etc/dgedeploy/dgedeploy-projects.conf

# YAML 1.1
# Settings for example-project deployment
example-project:
  # The workflow field is required, the order of the parameters specify which
  # branch merges with which branch.  eg:
  #     development --> staging --> live
  workflow:
    - development
    - staging
    - live
  development:
    # The order of the urls parameter is important.  The first in the list will
    # be used as the default choice whenever necessary.  Each url should be,
    # a directory inside a Subversion repository, of course.  All urls should
    # be in the SAME Subversion repository.
    urls:
      - http://example.com/svn/example-project/trunk
  staging:
    urls:
      - http://example.com/svn/example-project/tags/staging
  live:
    urls:
      - http://example.com/svn/example-project/tags/live

So, as shown in the configuration example above, the script is based on workflows. Merges always happen forward and must hit every level in the chain. This ensures your development code doesn’t go straight to live without being merged into staging first.

Make a merge

The syntax for the script is fairly straightforward. Once you have everything installed (see README.txt for details) you initialize your working copies like so …

dgedeploy example-project initialize

After the working copy caches are setup a merge can be performed using the same script. This example merges all of trunk into staging.

dgedeploy example-project \
  http://example.com/svn/example-project/trunk

#
# Translates to ...
#   svn merge \
#   http://example.com/svn/example-project/tags/staging \
#   http://example.com/svn/example-project/trunk \
#   /var/cache/dgedeploy/example-project/tags/staging
#

DGEDeploy will also merge just specific subdirectories. To send only a certain subdirectory from staging to live run a command like this …

dgedeploy example-project \
  http://example.com/svn/example-project/tags/staging/sub-dir

#
# Translates to ...
#   svn merge \
#   http://example.com/svn/example-project/tags/live/sub-dir \
#   http://example.com/svn/example-project/tags/staging/sub-dir \
#   /var/cache/dgedeploy/example-project/tags/live/sub-dir
#

Enjoy!

James.

Download DGEDeploy.tar.gz

new comment

Hook ordering in Drupal

2008-01-30T04:49:36Z

Hook ordering in Drupal

Wednesday, January 30, 2008

I use Drupal a lot. This has caused me to become atuned to some of it’s foibles. Hook ordering, the order in which hooks/callbacks are called, is a one issue that agrevates me quite often. There just doesn’t seem to be a Right Way ^TM to get out of a hook ordering jam.

The community has taken a few stabs at the hook ordering problem. Sometimes the concept of a “hook registry” has been used. Here’s two threads I found on the issue (if anyone knows of some newer ones please let me know!):

I haven’t seen any commited code relating to hook orderig or a hook registry in Drupal 6, though I think it may have found new life in the updated menu system.

Alas, I was really hoping hook ordering of some form would make it into Drupal 6. So over the next few paragrahs I’ll lay out my grand vision for hook ordering.

Here lies James’ receipe for hook ordering in Drupal

First, the links above contain some fairly drastic ideas on how to implement a hook ordering system. I propose something a little simpler.

Reasons why, I think, my solution is simpler:

It requires the addition of only one hook
It requires only a few lines of extra, unobtrusive, code to the core
It won’t affect any existing contrib or core modules

Also, please feel free to steal my ideas for something else if you like, I got the idea from the Compiz project anyway—namely their plugin dependancy system.

Addition of the hook_depends()

Sure, it sounds like reeling in a pair of adult diapers instead of trout. The API needs more humor anyway.

Ex:

function mymodule_depends() {
  return array(
    'hook_nodeapi' => array(
      'before' => array(
        'taxonomy_nodeapi',
        'audio_nodeapi'
      ),
      'after' => array(
        'image_nodeapi'
      ),
    ),
    'hook_form_alter' => array(
      'before' => array(
        'taxonomy_form_alter'
      ),
    ),
  );
}

The hook_depends is implemented by any module that cares about what order it’s hook(s) get called in. It should only be used when there is an issue with the standard hook ordering (more on that later). The idea should be quite straightforward from the example: the “mymodule” module needs its hook_nodeapi() run before a few other modules and after one, it also needs its hook_form_alter() run before one module.

How the core orders the hooks based on hook_depends()

When a new Drupal module is installed the core invokes all instances of hook_depends(). These should, in a perfect world, only exist in contrib modules because the core should take care of it’s own issues in a cleaner manner.

So, once a list of modules and their requested dependancies has been gathered the system can pour over them to figure out which order each hook should be run.

I’m sure someone else has a better algorithm than me, but here’s a shot.

Find all implementations of the hook in question (from all modules, including ones that implement hook_depends().
Split the list of hooks into two groups: (1) Hooks that don’t care when they are run, (2) hooks that care when they are run.
Keep making consecutive passes over the hooks in group (2) trying to fit them into group (1) as they desire to be placed.
If a cyclic relationship occurs then throw a nasty error (more on this later).

Once the ordered lists have been made for each hook cache the list to the database. I’m unsure if this should be done using cache_set or by keeping track of it in a seperate table entirely. Either way the result is the same.

Modifications to module_implements

Since Drupal is quite good at keeping it’s core code squeaky clean, adding hook ordering is a simple matter. Just modify module_implements() to grab the cached list of ordered hooks and everything should cascade from there.

What to do if there is a collision or cycle?

Nothing. Well throw an error, I guess. The real point is that while cyclic dependancies are easy to create it’s up to contrib module developers to avoid them. As long as the hook_depends() is used sparingly and only when there is a bonified reason cyclic dependancies shouldn’t happen.

Reasons to use hook_depends(), aka common reasons hook ordering is needed:

Some module up the food-chain from my module is messing up my data, and I want it to stop.
I want to mess with the data being supplied to some module down the food-chain.

Why not have ‘after all’ or ‘before all’ ordering classes?

Because somebody has to go first, and two hooks can’t both be first. It’s better to encourage developers to place their hooks directly before the module they intend to superceed. Also, wasn’t this what hook_init() an hook_exit() were created for?

All thoughts and feedback are welcome!

James.

new comment

Welcome!

2008-01-25T06:35:48Z

Welcome!

Friday, January 25, 2008

Hello Web-travlers and welcome to my new site. I’ll have some fresh content shortly, once I sand smooth all the rough edges on the site.

Until then, I’ve posted a few of the better posts from my blog on projectopus.com.

While you wait, why not check out the site in

/[e]?l(i|y)nks/

James.

new comment

Lesson learned: Apache2 MaxClients Variable

2008-01-19T18:42:50Z

Lesson learned: Apache2 MaxClients Variable

Thursday, November 30, 2006

This is an old post originally taken from my blog on Project Opus

Yea, I finally got around to reading about what the MaxClients variable in the apache2.conf is for. Please keep your laughter to yourself, but i thought what MaxClients meant was “how many clients can connect at one time.” So logically, I set it at 255. Well, I’m not sure if MaxClients is a cryptic name, or maybe I just missed the boat, but actually this variable controls how many server processes will run simultaneously!

Visually stated: MaxClients == MaximumNumberOfApacheInstancesRunningAtTheSameTime

So what? Well, when load got high, apache would start spawing “hefty” Drupal processes @ between 20MB and 80MB each.

Simple math: 255×60MB = 15,300 MB RAM usage (hint: we don’t have 15 GB of RAM)

As you can see this is a bad situation.

I changed the nubmer to 40 this morning and systems haven’t been happier :-)

Lessons learned … lessons learned

James.

new comment

PHP opcode caching and SMP (multiple processors)

2008-01-19T18:43:48Z

PHP opcode caching and SMP (multiple processors)

Thursday, November 23, 2006

This is an old post originally taken from my blog on Project Opus

Hello all,

Off-Topic Note

I’ve decided to start blogging on a bit more of a regular basis. I’m not much of a blogger, but who knows … it might just stick this time. I have a hunch problem was subject matter. So from now on I plan to stick to what I know best. Geek stuff!

Main Feature

Just learned an interesting thing about APC (the popular PHP opcode cache) today: APC hates SMP. I’ve been trying for nearly a month to get APC to run stably on our multi-CPU system. Invariably the extension will start seg. faulting within a few minutes or hours of starting up.

After many headaches, I gave up on APC. Running PHP without an opcode cache is a fairly crazy thing to do, so for the last month I’ve been looking high and low for a replacement. Recently I just stumbled across XCache and immediately loved what I saw. XCache is very similar to APC, the installation was nearly identical, but it has a few added options (I haven’t explored them all yet).

Best of all, it works great with SMP!!!

XCache Introcution Page ... before then I had used APC. Although it was quite stable for a php4 with flock() configuration it had become unstable once I upgraded my server to a dual cpu (4 threaded cpu) because it flock()ed badly so XCache was used instead and seems to have solved the problem.

I couldn’t have said it better my self.

James.

new comment