devroom.io

Why Asahi Linux is not for me

Wed, 23 Oct 2024 00:00:00 +0000

In previous posts I discussed some Asahi Linux specific setup to make Asahi Linux play nice with my MacbookPro M1 Max.

After running Asahi for about a week on my production laptop, I’m ready to do some evaluation.

Before I start of this rant, I want to take a moment to appreciate the enormous amounts of work the Asahi team put into their release. Reverse engineering all this hardware is not a small feat and I’m grateful for their work. The M1 Series Feature Support page gives you an indication of what to expect.

The good

Let’s start with the good.

I can run Linux on my Mac. I love open source software, so this is a big deal to me. The installer script is, by far, the easiest linux install I ever did.
I was able to re-use my (Arch focused) Ansible machine setup playbook for about 80%, so Gnome was set up the way I like it in no time.
The touchpad is working, albeit not as smooth as on macOS.
Brightness, volume and keyboard backlight keys work out of the box.
The webcam works out of the box.
Battery and sensors work out of the box.

The bad

Things that are not working as expected, but have viable workarounds.

Bluetooth works, most of the time. Sometimes it would just not work and my AirPods Pro would not connect. I think it’s related to sleep/suspend, but I’m not sure.
Wifi works out of the box, but it’s slow. On macos I get about 150-200mbps to the internet, on Asahi I get about 30-40mbps. It’s not a deal breaker, but I like my bandwidth.
TouchID is not supported. I can live without it, but I’m quite used to it to unlock my machine.
Some proprietary apps, like Spotify and Slack, have no aarch64 packages. Which means I have to use them in the browser.

The ugly

Things that are not working as expected, but are deal breakers for me.

Audio output is horrible. It’s working, but the subwoofer is clearly not working and the audio quality is sub par. For someone who like to listen to some music, this is disappointing.
The built-in microphone is not working. This means that any video call you will need a headset. If my Airpods Pods would just connect when I need them, that would not be a problem, but they don’t. A wired 3.5mm headset would work, but I just moved to going wireless.
Power management is not great. Where on macos I can go a full day without charging, on Asahi I need to charge after a few hours. Even when in sleep/suspend mode, the battery keeps draining. When I close the lid at 80% battery, an hour later, I’m at 70%. I feel that the battery is draining faster than it should and it makes it hard to depend on any charge remaining when I need it.

Conclusion

Should you run Asahi Linux on your Mac? If you are a Linux die-hard and can live with the bad and the ugly, sure.

But alas, this is a mobile work station and I really need those audio features and power management is important to me. If I was running this on a machine that was connected to a dock most of the day for power and an external audio interface, I would say: yes.

All in all, Asahi Linux is a monumental effort to make Linux run on Apple hardware.

If I really wanted Linux on my daily driver I would not go for Apple hardware.

Because I like AMD I’d probably go for one a Framework 16 or Tuxedo Computers Stellaris 15. The Dell XPS is a nice option too, but it only comes with Intel CPUs.

For now, I’m back to macos on the MacbookPro, but still running Linux on my desktop.

I run Arch, BTW.

MacbookPro M1 with Asahi Linux: Fractional Display Scaling in GNOME

Tue, 15 Oct 2024 00:00:00 +0000

The MacbookPro has a high pixels-per-inch screen, or as Apple likes to call it, a Liquid Retina XDR display. Including the notch area, it measures in at 3456x2234 pixels at 254 PPI. It’s a beautiful screen…

…until you run Asahi Linux and learn that you either get 100% or 200% scaling for your display.

What is scaling, anyway?

When using the 3456x2234 screent at a 100% scale - or no scaling. This means that things will be come incredibly small and hard to read. Normally, Apple runs at a scale of 200%, doubling the size of everything on screen. This scaling ensures that visual elements maintain clarity and are more appropriately sized for people on high-resolution screens.

In GNOME the options are 100% and 200% and nothing in between.

On Apple devices, you get a few choices ranging from Large Text to More Space.

Wouldn’t it be nice if we had the same options in GNOME?

Enabling fractional scaling in GNOME

Fractional scaling is not an officially supported feature in GNOME yet. Luckily, it’s available as an experimental feature. You just need to enable it:

gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']"

When you now go into GNOME Settings -> Displays you’ll see that the 100% and 200% buttons have been replaced with a dropdown, offering you 100% up to 400% in steps of 25%. I’m currently running at 175%, just like on macos, as I find it a good balance between readability and screen real estate.

The resolution is stated as 2160 high, instead of 2234. This is because I do not have the notch enabled (yet), so Linux is using a fractionally smaller screen to avoid the notch part of the screen altogether.

Fix for Chromium

Now, all apps I’ve used so far work brilliantly with the fractional scaling at 175%. The one exception was Chromium. It looked blurry and annoyed me enormously.

Chrome by default will not use Wayland as it’s platform backend so scaling will be a bit iffy. Go to chrome://flags and search for Ozone and select either Wayland or Auto so the correct backend is used. Restart Chromium and you’re done.

Saving for automation

Because I’m a lazy programmer I spend a disproportionate amount of time automating things that probably don’t need automating. Having an automated setup with Ansible that configures and sets up a new machine for me is very helpful indeed.

I’ll share more on this later, maybe, but my Ansible setup will configure anything from my user account, hostname and installed packages to configuring GPG, cloning dotfiles and tweaking GNOME settings.

Basically, when my Ansible playbook has finished, I have a working setup, across machines, to start working. My Fedora Asahi Linux machine is no different, since it need just a few changes from my Arch setup (mainly, a different package manager).

Anyway. Where you’d normally be able to use dconf to save settings for GNOME, that was not possible with Displays. Of course, the dislay settings are more fundamental than GNOME itself.

Luckily, I found out that a file named ~/.config/monitors.xml was generated. You can clearly see the scale entry here. (I guess using that arcane floating point number instead of 1.75 gives better results.)

So, this file was not quickly put into my Ansible playbook and will now be automtically restored with my setup.

<!-- ~/.config/monitors.xml -->

<monitors version="2">
 <configuration>
 <logicalmonitor>
 <x>0</x>
 <y>0</y>
 <scale>1.7489879131317139</scale>
 <primary>yes</primary>
 <monitor>
 <monitorspec>
 <connector>eDP-1</connector>
 <vendor>unknown</vendor>
 <product>unknown</product>
 <serial>unknown</serial>
 </monitorspec>
 <mode>
 <width>3456</width>
 <height>2160</height>
 <rate>60.000</rate>
 </mode>
 </monitor>
 </logicalmonitor>
 </configuration>
</monitors>

MacbookPro M1 with Asahi Linux: Fixing the touchpad

Mon, 14 Oct 2024 00:00:00 +0000

I’m a big Linux fan. On my Ryzen 3700X desktop rig I run Arch Linux and I like it. The only times I boot into Windows are to play Rocksmith 2014, because I still haven’t got that game to work properly on Steam/Proton.

My job provides me with nice “developer laptop”, which has been a high-end MacbookPro ever since I started there in 2007. The current iteration is a well spec’ed 16" M1 Max.

I run Arch, BTW!
…and now also Fedora on aarch64, apparently.

Now, I’ve never had any qualm with macos. It was a breath of fresh air when Windows was the only viable desktop OS. It’s really polished but still has that BSD underpinning that makes it a breeze using it as a development machine.

On previous Intel Macs I tried installing Arch and Ubuntu a few times, but always ran into unsupported hardware. Each time I gave up pretty quickly and accepted the fact that Apple hardware and macos are married.

Then a co-worker told me about his success in running Asahi Linux on his M1 machine. I had to try.

Although this is not a review of Asahi Linux: it’s amazing! There are a few quirks, as is expected, but nothing that is a real deal breaker.

In this post I want to address my issues with the trackpad and how I solved them.

Accidental tapping

Touchpads are great, just not when they register taps when you’re typing and shift focus to the most ungodly places. When you palms touch the touchpad, they can register ghost taps. With the MacbookPro’s touchpad being so huge, that’s quite a high risk.

The solution to this problem is quite easily fixed with some settings in GNOME.

First, Disable Touchpad While Typing. It will not completely fix the tapping issue, but it’s a great help. You find this in GNOME Settings -> Mouse & Touchpad and then the Touchpad tab.

Next, scroll down a bit and disable Tap to Click. You know with touchpads, where you can either click on it (it makes that clicking sound), or just tap it, like you would your phone. Well, this option turns of that tapping behaviour.

Fixing trackpad scroll speed

This is by far the most annoying problem. The scroll speed. It feels like the trackpad is on cocaine or something.

Unfortunately, GNOME cannot help us out here, because it simply does not have a setting for it. Long story short, GNOME is pointing to Wayland’s libinput to fix this problem, and libinput is pointing to windows managers (like GNOME) to handle it.

Luckily, there’s some great people out there who created kind-of-a-hack for libinput to allow scaling the scroll-speed.

You can find that code here: warningnonpotablewater/libinput-config.

Here’s a quick rundown of how to install it:

Install dependencies first: sudo dnf install meson ninja-build cmake libinput-devel
Clone that repo: git clone https://gitlab.com/warningnonpotablewater/libinput-config.git
meson build then, go into the build/ directory.
ninja && sudo ninja install to build and install the library.

Then, all you need to is create a config file in /etc/libinput.conf and reboot.

# /etc/libinput.conf

scroll-factor=0.5

In this case, the scrolling was too fast, so I multiply it with a factor of 0.5 to bring it down. You may want to try some other values if you’re not satisfied with the result.

If you want faster scrolling, use a factor >1.

Simply reboot your machine for the changes to take effect.

Peel before use: mistakes were made

Sun, 07 Apr 2024 00:00:00 +0000

The stock AMD Wraith cooler on my linux box was getting loud. Since I recently migrated the complete system from a (too) tight mini-ATX case to a larger Corsair 4000D Airflow, I decided to splurge on some new cooling.

Thermalright has a nice and affordable offering of CPU coolers and case fans.

So when my Peerless Assassin 120 arrived I immediately set out to install it. You know where this is going…

Upon booting up I ran stress-ng to put some load on my CPU and see what amazingly cool my Ryzen 3700X would stay. Well, it hit 94C within 10 seconds. Not good. Idling at about 50-60C.

So yeah. It turns out plastic “Peel before use” are not great thermal conductors.

With the plastic now removed and the perfect amount of thermal paste the CPU runs at 35-40C at idle and < 70C when at full load for 5 minutes. Noice!

Import Google Photos Takeout into Immich

Thu, 21 Mar 2024 00:00:00 +0000

Okay, importing into Immich from a Google Photos Takeout is super easy. Well, it is if you’re familiar with a CLI.

Prepare your takeout

There are two options for getting your Google Photos Takeout ready for importing:

Export as .zip (any size, bigger is better)
Export as .tgz (any size, bigger is better) and extract as per my previous post

Getting `immich-go`

immich-go is a single binary command line tool for importing photos and videos into Immich. It has awesome support for Google Photos takeout archives.

You should get the latest immich-go release for your platform from Immich github releases. I dropped the executable into where I extracted my takeout archives.

-rwxrwxr-x 1 ariejan ariejan 8.5M Mar 20 18:45 immich-go
drwxrwxr-x 3 ariejan ariejan 4 Mar 20 14:57 Takeout

Getting your Immich API key

Go to your Immich account settings (click your avatar at the top right). Then select API Keys and create a new key.

You probably already know your Immich url, which might optionally require a port number, depending on your setup.

Dry-run

First, let’s do a dry run. This will read your Takeout archives and test the connection to Immich. It will spit out a lot of info about what’s going to be imported, skipped, etc.

./immich-go \
-server=https://immich.devroom.io \
-key=Ir3za64rggrbK3zSIhDfskw8R9b97GTljPPK5NlIEY \
upload \
-dry-run \
-create-albums \
-google-photos \
Takeout/

-create-albums will re-create your Google Photos albums, which is probably what you want.

In this case I’m providing an extracted Takeout archive, so I need to specify the -google-photos option. If you have .zip files, you can supply takeout-*.zip instead, and the -google-photos option is not strictly necessary.

Importing for realz

If you’re happy what immich-go reported, go ahead and start your import. It can take a few hours, depending on how many photos and videos you want to import.

./immich-go \
-server=https://immich.devroom.io \
-key=Ir3za64rggrbK3zSIhDfskw8R9b97GTljPPK5NlIEY \
upload \
-create-albums \
-google-photos \
Takeout/

Result?

All my photos and videos are now safely in Immich (backed up to a separate 3TB mirror; offline backup to a Hetzner Storagebox).

How to extract multiple .tgz Google takeout archives

Wed, 20 Mar 2024 00:00:00 +0000

I love Google Photos for its easy of use and features. But, it’s Google. As you may knwo I like to selfhost all the things, but for the longest time I was not able to find a good selfhosted alternative for Google Photos.

Immich

Immich is a selfhosted photo and video management system. Sounds fancy, and it is. Aside from having a good mobile app for iOS that will do background backups, it sports facial recognition, hardware transcoding of videos, reverse geocoding and lots more. There’s a big disclaimer though, Immich is still under active development. I see that more as an encouragment to use it than a warning :-)

So, naturally, I want to move all my photos and videos (going back to 2006) from Google Photos to Immich.

Google Takeout

Due to great EU legislation Google Takeout exists. It allows you to easily create an archive of your data for export. Now, how useful that data export is going to be is another matter.

Because I have a lot of data, Google is offering me to create multiple .tgz files for export of 50GB each. I now have six Google takeout archives:

-rwxrwxr-x 1 ariejan ariejan 50G Mar 5 11:16 takeout-001.tgz
-rwxrwxr-x 1 ariejan ariejan 50G Mar 5 11:16 takeout-002.tgz
-rwxrwxr-x 1 ariejan ariejan 50G Mar 5 11:16 takeout-003.tgz
-rwxrwxr-x 1 ariejan ariejan 50G Mar 5 11:16 takeout-004.tgz
-rwxrwxr-x 1 ariejan ariejan 50G Mar 5 11:16 takeout-005.tgz
-rwxrwxr-x 1 ariejan ariejan 39G Mar 5 11:16 takeout-006.tgz

So, I was wondering, how do I unpack these files? Well, there are several ways I want to document.

cat | tar

With some linux-fu I could easily pipe all these files into tar to extract:

cat takeout-{001..006}.tgz | tar xzivf -

Or, I could just glob all the files, they will be ordered automatically. How nice is that!

cat takeout-*.tgz | tar xzivf -

Both these methods will pipe the data from the archives, in order, into tar, which will extract the provided data.

pv | tar

If you want to go fancy and have pv installed, you can use that as well. pv is a utility that monitors progress of data through a pipe.

pv takeout-*.tgz | tar xzif -
88.9GiB 0:19:19 [89.2MiB/s] [==========> ] 30% ETA 0:43:04

Importing into Immich

Now, with the data extracted, how am I going to import it into Immich? There are tools for that and I’ll post again on how that process went.

ZFS: Upgrade single disk to mirror

Thu, 07 Mar 2024 00:00:00 +0000

For a while now I’ve had a single 3TB WD Green in my NAS to store local Borg backups before syncing them to ~~Google Drive~~ a Hetzner Storage Box.

Because I recently replaced my 3TB Reds for 10TB HGST’s, I have some spare drives left.

My goals are:

Turn the single disk zpool for backups into a mirrored pair.
Replace the WD Green with a WD Red, eventually.

So, today is about upgrading a single disk ZFS pool to a mirrored pair.

Prepare the new disk

The 3TB Red came out of my tank pool. This means it has partitions and metedata suggesting it’s a disk in use by a ZFS pool. To prepare this disk, I’ll need to clean it fully. Note that I’m not going to zero out this disk, since it’ll be me who’s going to use it.

wipefs -af /dev/sdq

This will destructively clean any and all filesystem / ZFS information on the disk.

Upgrading the ZFS pool

The next step is to upgrade the pool by adding the new device into a mirrored pair.

zfs attach <pool> <existing hdd guid> <new hdd guid>

By using the zfs attach command like this, you’re creating a mirrored pair out of the existing and new HDDs. After this you’ll need to wait for the new HDD to resilver, but after that you’re all set.

Arch Linux: Improve boot time performance

Thu, 08 Feb 2024 00:00:00 +0000

I run Debian on all my servers. It’s a great stable OS and I love it. Proxmox, which I run on my homelab server, is also based on Debian.

However, on my desktop I run Arch Linux. It’s a great distro to tinker with. It comes with a lot of up to date packages, but it also has the AUR - Arch User Repository. So for any app you can find, there probably is an easy way to install it.

Slllooooowwww…

As of late, I noticed that boot times on my system were getting longer. Which is strange, because I run some pretty okay hardware.

As it turns out, cold booting this box takes 1min 7.538s, according to my logs.

Luckily, the Arch Wiki offers a nice guide on how to trouble shoot boot performance.

There’s systemd-analyze blame which will show the time it takes each service to start up. I’ve copied the top 10 here, which incidentally are also all >1 second start-up times.

❯ systemd-analyze blame
20.771s docker.service
3.514s dev-sdb3.device
2.459s systemd-journal-flush.service
1.880s upower.service
1.806s ldconfig.service
1.687s systemd-tmpfiles-setup.service
1.587s containerd.service
1.287s systemd-modules-load.service
1.032s systemd-fsck@dev-disk-by\x2duuid-96EB\x2d4C82.service
1.028s cups.service

Docker is a clear offender here. dev-sdb3 is also quite slow it seems.

Another command recommended in the wiki is systemd-analyze critical-chain. This will show you the critical chain to boot your system. Again, docker is here clearly a big offender.

❯ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
graphical.target @33.660s
└─multi-user.target @33.660s
└─docker.service @12.888s +20.771s
└─containerd.service @11.264s +1.587s
└─network.target @11.236s
└─wpa_supplicant.service @27.465s +268ms
└─basic.target @10.366s
└─dbus-broker.service @9.822s +541ms
└─dbus.socket @9.793s
└─sysinit.target @9.759s
└─systemd-update-done.service @9.722s +36ms
└─systemd-journal-catalog-update.service @9.375s +326ms
└─systemd-tmpfiles-setup.service @7.657s +1.687s
└─local-fs.target @7.587s
└─boot.mount @7.458s +128ms
└─systemd-fsck@dev-disk-by\x2duuid-96EB\x2d4C82.service @6.398s +1.032s
└─dev-disk-by\x2duuid-96EB\x2d4C82.device @6.397s

But wait, there’s more. systemd-analyze plot > plot.svg will generate an SVG image showing you the entire boot process in time. It’s big, but there are some clear red markers that indicate issues.

At the bottom right you’ll find graphical.target, where we want to end up as quickly as possible. And it’s clear docker is in the way.

Open the SVG in a new window to see more detail.

Fixed it!

So, with docker as a clear offender in slowing down the boot process, let’s fix that.

There are two systemd units: docker.service and docker.socket.

docker.service is there to start docker and make sure it is up and running.
docker.socket listens on /run/docker.sock (or /var/run/docker.sock through a symlink) and will start docker.service when needed.

I think you know where this is going. docker.socket is disabled by default and docker.service is enabled. Which makes sense, because when you boot your machine you want docker up and running as well. Especially for servers this makes sense.

For my desktop, not so much. I use docker, but not always and I prefer to login and check my email while docker is booting in the background anyway.

The trick thus is to disable docker.service from starting automatically and make sure docker.socket is enabled. That will take docker out of the criticial chain when booting and start docker when I’m logged in and ready to use it.

$ sudo systemctl disable docker.service
$ sudo systemctl enable docker.socket

So, what does that look like in systemd-analyze?

❯ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
graphical.target @3.893s
└─multi-user.target @3.893s
└─cups.service @3.672s +220ms
└─nss-user-lookup.target @3.763s

❯ systemd-analyze blame
2.152s systemd-modules-load.service
1.295s dev-sdb3.device
622ms boot.mount
385ms NetworkManager.service
310ms systemd-udev-trigger.service
280ms udisks2.service
258ms systemd-remount-fs.service
220ms cups.service
203ms user@1000.service
189ms systemd-tmpfiles-setup.service

Open the SVG in a new window to see more detail.

❯ systemctl status docker.socket
● docker.socket - Docker Socket for the API
Loaded: loaded (/usr/lib/systemd/system/docker.socket; enabled; preset: disabled)
Active: active (running) since Thu 2024-02-08 10:38:47 CET; 5min ago
Triggers: ● docker.service
Listen: /run/docker.sock (Stream)
Tasks: 0 (limit: 38400)
Memory: 0B (peak: 516.0K)
CPU: 1ms
CGroup: /system.slice/docker.socket

and

❯ systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; preset: disabled)
Active: active (running) since Thu 2024-02-08 10:39:33 CET; 5min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2522 (dockerd)
Tasks: 42
Memory: 222.1M (peak: 235.7M)
CPU: 797ms
CGroup: /system.slice/docker.service
└─2522 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Was it worth it?

Before:

Startup finished in 14.729s (firmware) + 6.386s (loader) + 12.761s (kernel) + 33.661s (userspace) = 1min 7.538s graphical.target reached after 33.660s in userspace.

After:

Startup finished in 13.735s (firmware) + 4.074s (loader) + 6.744s (kernel) + 3.893s (userspace) = 28.448s graphical.target reached after 3.893s in userspace.

Total boot time went down from 1m8s to 28s. I cannot explain the difference in kernel boot time, but the userspace savings are significant.

From here I could probably optimize more by compiling a customized kernel or using a different bootloader. Suspend to RAM would be even faster, but that feels like cheating against a hard boot.

Hopefully this will give you some pointers in how to troubleshoot slow boot times on your machine.

Troubleshooting zfs online not working

Sat, 27 Jan 2024 00:00:00 +0000

Yesterday the first 10TB recertified drive tested all okay. I’ve decided to upgrade the four 3TB drives in my pool to 10TB to expand capacity.

I have a 16 bay hot-swap chassis with 13 slots filled.

4x 3TB (raidz1, tank)
4x 8TB (raidz1, tank)
4x 14TB (raidz1, tank)
A single 3TB drive as ‘scratch’ disk for temporary backup data

I really want to have the same vdev drives stacked in the same column physically. I’m like that. So my first order of business was to move one of the 3TB drives to a free slot to make room for the 10TB disks.

So I performed a zfs offline tank <id> on the disk, took it out and stuck it in another drive bay. dmesg clearly showed me the new disk was ready an available:

sd 0:0:18:0: [sdi] Attached SCSI disk

Since it’s the same drive, with the same GUID, I expected ZFS to recognise this disk, online it and then resilver it as necessary. I’ve seen this work before with one of the 14TB drives when I had a bad SATA cable. But instead, sadness:

tank DEGRADED
raidz1-0 DEGRADED
wwn-0x50014ee2bb55b28f OFFLINE
wwn-0x50014ee265ff666a ONLINE
wwn-0x50014ee210aaa174 ONLINE
wwn-0x50014ee265ff7131 ONLINE

So, let’s online the disk manually, and get the vdev back in working order!

$ zfs online tank wwn-0x50014ee2bb55b28f
(no output)

That didn’t work.

$ zfs online tank /dev/disk/by-id/wwn-0x50014ee2bb55b28f
(no output)
$ zfs online tank sdi
(no output)
$ zfs online tank /dev/sdi
(no output)
$ zfs online tank <GUID>
(no output)

No dice.

The obvious solution would be to wipe the 3TB disk and replace it with itself, doing a full 3TB resilver on it. But, that would be too easy. Also, onlining a disk should work - so why isn’t it.

At this point, I had exhausted my Google-fu. I even asked ChatGPT, but it didn’t come up with anything I hadn’t tried yet.

Next step: IRC

I hopped on #zfs on Libera Chat and dropped my question. A few people were online and started giving the obvious answers. Together we decided that it’s not working. With one of the people (thanks, Rich) in #zfs I dove into partition tables, labels, cache files, etc. Turns out Rich does quite some work on ZFS and he was of great help investigating the issue.

Then, I thought to check the exit code for the zfs online command.

$ zfs online tank wwn-0x50014ee2bb55b28f
(no output)
$ echo $?
1

Aah! So, zfs online exists with an error. But why?

Well, it was helpful in the sense that zpool online now says it cannot online the drive, but the exact reason was still unknown. I then tried to, instead of the whole disk, online the partition. Let me explain.

Device Start End Sectors Size Type
/dev/sdb1 128 4194431 4194304 2G FreeBSD swap
/dev/sdb2 4194432 5860533127 5856338696 2.7T FreeBSD ZFS

So this is how FreeNAS (based on FreeBSD) partitioned the disk when it was added to the original pool. So now, instead of adding /dev/sdb I was going to try to online just the ZFS partition /dev/sdb2. Not that /dev/disk/by-id/wwn-0x50014ee2bb55b28f-part2 symlinks to /dev/sda2.

$ zpool online tank sdb2
cannot online sdb2: cannot relabel 'sdb2': unable to read disk capacity

In some way, this error makes sense, since I’m trying to online a partition and not a disk. Still, I threw the message into Google, which led me to an open issue on OpenZFS that says drives cannot be online’d when autoexpand=on for the pool.

Well, that would be easy to test:

$ zpool set autoexpand=off tank
$ zpool online tank <guid>
$ zpool set autoexpand=on tank

It took the better part of 15 seconds for the drive to sync up and resilver. Done! I think @behlendorf explained the root cause of this problem better that I can:

The core issue here is that since the pool was created on FreeBSD the partition layout differs slightly from what’s expected on Linux. This difference prevents ZFS from being able to auto-expand the device since it can’t safely re-partition it to use the additional space. Unfortunately, that results in the device being online at all.

There is nothing I can do to remedy this problem. The only way to “fix” it is to offline a disk with FreeBSD partitioning, wipe it and then replace it with itself so a new Linux compatible partition table is created.

For now, I’m going to replace the 3TB drives anyway. The new 10TB ones are getting the Linux compatible partition table anyway. That leaves my 8TB drives still in limbo, but since I now know the root cause of the problem, I can easily work around it when necessary.

Big shout out to Rich for assisting me with this issue! I probably wouldn’t have been able to figure it out with you.

Prepare new harddisk for ZFS/NAS

Wed, 24 Jan 2024 00:00:00 +0000

You can read more on my homelab and datahoarding problem here and here.

Today I scored two recertified 10TB HGST drives for very little. Normally I’d go for the brand new stuff, but this deal was too good to be true.

My main goal is to check if these recertified disks are worth the money/effort. Next up I want to experiment with a new ZFS pool setups. (You still cannot remove a raidz vdev from your pool in 2024¹)

Current state of affairs

Right now my main ZFS storage pool looks like this:

tank
- raidz1 4x 3TB WD Red
- raidz1 4x 8TB WD White
- raidz1 4x 14TB WD White

All in all good for 100TB raw storage space and roughly 75TB of usable storage. And yet, it’s getting full. Linux ISO’s take up a lot of space. I also have a 2TB (2x 1TB SSD mirrors) pool for VM storage and a single 3TB drive as a backup intermediate disk (i.e. backups are copied there, then uploaded elsewhere).

As stated, I cannot remove any of those raidz1 vdevs. My only options are to build a new pool with new disks or replace disks in this pool.

It’s all about trust

So, new drives I normally trust. If they don’t work, they don’t work. But if they spin-up, I’ve alwasy assumed they’d be okay. Yeah, I know.

But, since I now have a pair of 10TB recertified drives, I’d like to be sure they’re good to go. Recertified in this context probably means they were retired from a data center somewhere. Their power-on time is a little over 5 years with a production data of December 2017.

To make sure these disks are good to go, I’m going to run a bunch of tests against them to see if they hold up.

SMART conveyance test
SMART extended test
badblocks

S.M.A.R.T.

If you don’t know about different SMART tests, here’s a refresher. I’m skipping the short test, because I’m running a long test.

Short Checks the electrical and mechanical performance as well as the read performance of the disk. Electrical tests might include a test of buffer RAM, a read/write circuitry test, or a test of the read/write head elements. Mechanical test includes seeking and servo on data tracks. Scans small parts of the drive’s surface (area is vendor-specific and there is a time limit on the test). Checks the list of pending sectors that may have read errors, and it usually takes under two minutes.

Long/extended A longer and more thorough version of the short self-test, scanning the entire disk surface with no time limit. This test usually takes several hours, depending on the read/write speed of the drive and its size.

Conveyance Intended as a quick test to identify damage incurred during transporting of the device from the drive manufacturer to the computer manufacturer. Only available on ATA drives, and it usually takes several minutes.

Running these is as easy as:

$ sudo smartctl -t <short|long|conveyance> /dev/sda

If you want to know how long theses tests are going to take:

$ sudo smartctl -c /dev/sda
...
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1144) minutes.
...

So it’ll take a little of 19 hours to do a full extended SMART test.

Badblocks

Badblocks is a utility, well, searches a disk for bad blocks. It will write data to the entire disk an verify each block to good.

Naively I ran the following to do a full write test with progress indicator and verbose output:

$ sudo badblocks -wsv /dev/sda
badblocks: Value too large for defined data type invalid end block (9766436864): must be 32-bit value

As it turns out, badblocks uses a default block size of 1024, meaning it cannot - out of the box - scan disks 8TB or larger. Let’s figure out what blocksize our disk uses, and plug that information into badblocks.

$ sudo blockdev --getbsz /dev/sda
4096
$ sudo badblocks -t random -w -v -s -b 4096 /dev/sda
Checking for bad blocks in read-write mode
From block 0 to 2441609215
Testing with random pattern: 5.97% done, 39:45 elapsed. (0/0/0 errors)

And we’re in business. Well, now we wait a few hours / days for badblocks to complete. I might even do a second pass just for the fun of it.

What’s next?

After I’ve run at least two badblocks passes and both they conveyance and extended SMART tests on this disk, I’m going to do it on the other one as well. If that all goes well I’ll probably put them as a ZFS mirror pair in a new pool and do some testing there.

There are good technical reasons for that, I know. Wish I’d known about it before I built my pool, though. ↩︎

Volkswagen Golf Cabrio - Update

Wed, 24 Jan 2024 00:00:00 +0000

It’s been almost 18 months since I purchased a, let’s be honest here, beat-up Volkswagen Golf Cabrio from 1994. At the time my knowledge about car mechanics was pretty much zero. Did I tell you I like a good challenge?

I needed to start with the basics, like new brakes, struts and an obviously leaking exhaust to pass the annual safety and roadworthiness inspection. A big shout-out to ChrisFix for all his detailed videos.

Big jobs

With the basics out of the way I had to move on to the mechanical part of things. I really wanted to keep this thing on the road. Because I had a roadtrip planned with my co-workes from Kabisa I really needed to sort out the clutch, because it was gone. It was not completely dead yet, but almost.

This was my first big job. One that real mechanics and dealerships charge you hundreds of Euro for. It was daunting, to be honest. However, with good preparation, tips from friends (who actually know what they’re talking about), and a lot of curse words, I managed it.

Another big job, which I managed to do last year after retiring the car to storage for winter, was the timing belt. Another big job that feels impossible and is normally hugely expensive. Again, with some good preparation it was, actually, a breeze. Okay, the 1.8 liter, single cam, petrol engine is rather simple compared to modern engines. Since I needed to drain the coolant for the new water pump, I replaced some more coolant flanges at the same time. Those had sprung small leaks. Note for next time: replace them all in one go on a car of this age.

I also ran into a problem with the O2 sensor. In a 2001 Polo I used to drive, this was like a problem every six months. Turns out that you just need a new sensor (which are not that expensive), the right socket for an O2 sensor and the strength of a giant.

Looks

I also spent some time on making the car look better and just that bit more luxurious. I found an full leather interior set, which was easily swapped out and makes the whole car look soooo much better.

I also found a nice car stereo (Blaupunkt Milano 200 BT). It’s didn’t cost a fortune and works way better than the cheap unit it came with. I would have liked to get a vintage Beta or Gamma set from VW, but it was hard to find an affordable one.

What’s next?

Right now I have three main jobs left to do on the car:

Re-paint (part of) the front bumper where it’s scratched. This will probably need some sanding, priming and painting. I hope to do this early spring when the weather is a bit warmer.
Re-seal the ventilation blend doors. The foam is gone, so now it only blows warm air. Check HumbleMechanic, he can explain it better than I can.
Clean and polish the entire car and touch up the plastic trim. I tried Meguiars Ultimate Black Plastic Restorer and it seems to work wonderfully.

When all that’s done. Well, I think I’m going to drive it for a bit and put it up for sale. I’m going to miss this little car, but I also really want to move on to a new project. Maybe a Skoda Octaiva Vrs or some other nice VAG car to sink my teeth in.

Base2Tone Cave Dark for Slack

Wed, 17 May 2023 00:00:00 +0000

I like my colorschemes. The most recent one I discoved is Base2Tone Cave Dark.

So, naturally I wanted to have it in Slack too, so I created a custom theme. Here it is.

Preview

Installation

#222021,#1A2B23,#EBBC47,#1A1D21,#936C7A,#EBBC47,#EBBC47,#AD1F51,#222021,#EBBC47

Got to Preferences → Themes and paste the above in the appropriate box.

10GB Ethernet with Proxmox and Ryzentosh 3700X with macos Catalina

Wed, 18 Nov 2020 00:00:00 +0000

Please read my previous post on my DIY NAS here: 1 and 2.

10 gigabit. That is 10,000 megabit.

I come from a time when fast internet meant you had ISDN. For reference, that’s 192kbps, or roughly 24KB/s.

For the past eight years I’ve had CAT-5E / CAT 6 installed through-out my house and have enjoyed LAN speeds of 1000mbps or 1gbps. At full throttle that ways in at about 125 Megabyte/s.

Now, gigabit ethernet is fine for almost everything. Unless you either transfer large files or if you use that NAS as a direct access disk on a workstation.

The latter is pretty much my use-case. I make music and the occasional video, and storing and manipulating all this over a 1gbps line is not so fun.

Yes, I do have an SSD, yes I do have spinning rust in my workstation. The problem is that they lack redundancy and extra work to backup. Everything that’s on my NAS profits from the redudancy of my ZFS pool, periodical ZFS snapshots, decent drive health monitoring (thanks, Proxmox!) and automated backups to another set of disks and the cloud.

When I found out that there’s a really cheap option to add 10 gigabit networking between my Ryzentosh and Proxmox NAS - I went for it.

Total cost of this project:

$40 - 2x Solarflare SFN5122F shipped. yes, that is two network cards, with two 10 gbit ports each. For $40. Including shipping.
$20 - A SFP+ 10gbe copper cable (1 meter).

That. Is. It.

Proxmox setup

Okay, let’s start with the easy part. Proxmox. Or Debian Linux, more specifically. It’s just plug and play. ;-) Plug the card in any PCIe x8 slot and you’re good to go.

Next, you only need to configure your NIC. It’s a good idea to create a Linux Bridge in Proxmox. This will allow you to assign this interface to containers as well, as I’ll come back to in a minute.

You can pick any IP address you like in a private range. You don’t need a gateway - well I didn’t - because I have a point-to-point connection. Litteraly a cable between the two systems.

Be sure to enable advanced options and set MTU to 9000 - jumbo frames. This will enable you to reach those juicy 10gbps speeds later on.

Hackintosh / Open Core setup

Download Solarflare 10GbE Mac Drivers.zip

Solarflare is no longer supporting macos and has since removed these drivers from their website. The drivers they made still work in Catalina.

This is where things get a bit more tricky. I can’t vouch for this working on a real Mac, but then again, adding (non-Apple) hardware is not what you’re supposed to do with these anyway, right? ;-)

Any modern hackintosh should run Open Core these days. Open Core is awesome. It takes a bit of work to setup, but your mac will continue working after that, updates and everything. So, the next bit assumes you’re familiar with Open Core.

Download the archive above. It contains a signed installer for macos 10.9. Mount that disk image. Don’t bother using the installer.
Use “View contents” on the installer and find Archive.pax. Unpack that and find your kext and some Application Support stuff.
Drag folder under Application Support to /Library/Application Support if you need it.
Mount your EFI partition and drop the kext in the folder with your other kexts.
Then update your config.plist to load the kext - you can copy paste any other entry and change the filename. But again, if you’re already running Open Core, you probably know the drill.
Reboot

Next up, all you need to do is configure your NIC under Network Preferences. Again, pick an IP address in the same subnet as your server.

Make sure to select “Jumbo Frames (9000)” under “Hardware” here as well. Both sides need to have them enabled (or disabled).

Testing speed

Okay, time to do some benchmarks. Let’s break out iperf3.

On Proxmox: iperf3 -s to start the server
On the Ryzentosh: iperf3 -c 172.16.0.20 -f g
Wait 10 seconds (for 11.2 GIGABYTES to be transferred)

# iperf3 -c 172.16.0.20 -f g
Connecting to host 172.16.0.20, port 5201
[ 5] local 172.16.0.10 port 50113 connected to 172.16.0.20 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 1.00-2.00 sec 1.12 GBytes 9.64 Gbits/sec
[ 5] 2.00-3.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 3.00-4.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 4.00-5.00 sec 1.12 GBytes 9.64 Gbits/sec
[ 5] 5.00-6.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 6.00-7.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 7.00-8.00 sec 1.12 GBytes 9.63 Gbits/sec
[ 5] 8.00-9.00 sec 1.12 GBytes 9.64 Gbits/sec
[ 5] 9.00-10.00 sec 1.12 GBytes 9.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 11.2 GBytes 9.64 Gbits/sec sender
[ 5] 0.00-10.00 sec 11.2 GBytes 9.63 Gbits/sec receiver
iperf Done.

You can also adjust the output formatting to give you bitrates in GBytes/sec instead:

# iperf3 -c 172.16.0.20 -f Gbytes  1|2 ↵  1005  18:50:53
Connecting to host 172.16.0.20, port 5201
[ 5] local 172.16.0.10 port 50155 connected to 172.16.0.20 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 1.00-2.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 2.00-3.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 3.00-4.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 4.00-5.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 5.00-6.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 6.00-7.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 7.00-8.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 8.00-9.00 sec 1.12 GBytes 1.12 GBytes/sec
[ 5] 9.00-10.00 sec 1.12 GBytes 1.12 GBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 11.2 GBytes 1.12 GBytes/sec sender
[ 5] 0.00-10.00 sec 11.2 GBytes 1.12 GBytes/sec receiver

That is 1.12GB or 1120MB per SECOND. Imaging that. That’s a bit more than a full CD of data. Heck, that’s about 1000 floppy disks per second. Okay, I’m claiming that. 1000fdps. :-)

Samba

I run Samba in a LXC container on Proxmox. There’s now only a few things to do for Samba to use that fat pipe of networking goodness.

First up, add another Network device. Pick another IP address in the private subnet and use the Linux Bridge you created earlier. You can add this in addition to the network device you probably already have for your old and slow gigabit LAN.

You may need to check /etc/network/interfaces. Proxmox probably updated it for you already, but it’s missing the MTU setting. So, just add it and restart networking (or you’re container, it’s fast anyway) ;-)

iface eth1 inet static
address 172.16.0.22/16
mtu 9000

If you did it all correctly you should see:

# ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if82: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 8a:b3:db:74:b9:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0
3: eth1@if86: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ea:49:ef:01:82:80 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Then you need to configure Samba to listen on both your gigabit and 10 gigabit interfaces. Check /etc/samba/smb.conf, there’s probably a comment in there already about the interfaces flag. I simply added the following to enable both my interfaces.

interfaces = 172.16.0.22/16 10.0.2.245/16

Restart samba or reboot. And you’re good to go.

Accessing 10,000mbps Samba shares

You have to routes to your NAS / Samba shares now. One over your old gigabit network and one over your fast 10gbps point-to-point connection. Samba can handle both, so it’s only a matter of accessing it correctly from macos.

Open Finder and press ⌘-K (or select Go -> Connect to Server…). Then you’ll need to enter the proper URL for your Samba share like so. Remember to use the IP for your container - not the IP for your Proxmox host.

That’s all

That’s all there is to it. Mount those shares and enjoy blistering fast transfer speeds. Your HDD’s are once again the bottleneck now.

No, I’m not upgrading to a full SSD ZFS pool…

The Big DIY NAS Update

Thu, 12 Nov 2020 00:00:00 +0000

Please read my previous post on my DIY NAS here: https://www.devroom.io/2020/02/28/building-a-diy-home-server-with-freenas

A brief history

In May 2019 I decided it was time to jump on the NAS bandwagon. I started out with the following:

Fractal Design Node 804
4x 3TB WD Red (retail)
Gigabyte Z370M D3H
Intel i3 i8350K 4c/4t CPU
16GB Non-ECC Memory
128 GB Gigabyte SSD Boot drive
FreeNAS with a single RaidZ1 (4x 3TB) pool

That’s when it all started. Then things got wild.

Update November 2020

So, how are things today.

Norco 4U, 16x 3.5" LFF Hot swap case
4x 3TB WD Red (retail)
4x 8TB WD White (shucked)
4x 14TB WD White (shucked)
1x 3TB WD Green (retail)
2x 120GB Crucial BX500 SSD’s (because why not?)
SuperMicro X10SRi-F motherboard
Intel Xeon E5 2620v3 6c/12t CPU
64GB (2x32) ECC Memory
2x Dell PERC H200 HBA
250 GB Samsung SSD Boot drive
Proxmox 6

So, aside from the original 4x 3TB WD Reds, everything has changed. From enclosure to operating system.

Are you done now?! What more could you want?!

Nope. Never. I have some parts on the way to setup 10Gb ethernet between my NAS and Ryzentosh work station. This is one of the reasons I upgrade the motherboard and CPU - to accommodate 2x PCIe x8 HBA’s and 1x PCIe x8 10GB Ethernet.

Norco 4U Rack with hot swap bays

Great choice. It’s much easier to swap drives if need be. But I also have 4 empty bays right now. This is great to test a new disk or copy stuff over from old cold-storage disks. It also helped a lot migrating to the new SuperMicro motherboard.

Because the case is bulky, it also offers lots of room for 120mm fans. And 120mm fans (especially Noctua ones) are quiet.

Motherboard + CPU + Memory

My 4 core / 4 threads i3 8350K has served me well. It’s quite powerful too. But when running about 20 VM’s / Containers I feel it’s getting spread too thin. That, and the lack of PCIe slots on the micro-ATX board were a reason to look for something new.

There are two options if you want lots of PCIe slots: high-end gaming motherboards that support 3 or 4 GPUs or (used) server hardware. Since this is turning into a proper home-lab server, I decided to go with used server gear. It’s a bit cheaper and more suited to the task.

After scouring eBay and local classified sites I found a really good deal on a SuperMicro X10SRi-F. It has quite a lot to offer that makes me very happy:

IPMI remote management
10x SATA3 on board
Lots of PCIe slots:
- 1x PCIe 3.0 x16
- 2x PCIe 3.0 x8
- 1x PCIe 3.0 x4
- 1x PCIe 2.0 x4
- 1x PCIe 2.0 x2
Dual gigabit ethernet

The accompanying Intel Xeon E5 2620v3 6c/12t CPU has a lower clock speed of 2.4GHz (boost to 3.2GHz) as opposed to the 4.0GHz of my previous i3 8350K, but it does offer 6 cores instead of 4 and supports HyperThreading. So that offers a lot more flexibility with running a multitude of VMs and containers.

I also added 2 stick of 32GB ECC Registered memory for a total of 64GB. It’s enough for now, but with LRDIMMs, this motherboard has 8 DIMM slots available for a maximum of 512GB of memory. Quite enough for now and room to expand. Great!

Proxmox

Proxmox is great if you want to have a ZFS pool and run lots of services on top. As discussed in my previoust post, FreeNAS is great, but it’s not made for running additional services. Jails restrict you to FreeBSD and the bhyve hypervisor is not quite mature yet to easily run VMs.

Proxmox is just Debian linux with a lot of goodies for running a homelab server. ZFS on Linux being one of them. First class support for containers and virtual machines is another.

The only thing ’lacking’ from Proxmox, coming from FreeNAS, is easily setting up file sharing. But in the end all I needed was to spin-up a container, install Samba and configure the shares I wanted to create. Voila. Nice shares and TimeMachine backups.

10GB Ethernet

Yeah. That’s happening. I find myself storing more and more data on a network share on the NAS. But with lots of audio and video files, 1000mbit can be limiting (e.g. 120-ish MB/s).

I found myself a pair of Solarflare SFN5122F 10G Ethernet Dual Port (SFP+) cards on eBay. These should even work under macos, which is what I’d need. They’re still in the mail right now - but it should be pretty straightforward setting these up for a direct 10G link between my Ryzentosh and NAS.

Repairing a Xiaomi Roborock S5 Max

Sat, 03 Oct 2020 00:00:00 +0000

Many less-tech savvy people I know have a robot vacuum cleaner. So why didn’t I? I don’t know, so I bought one. I was able to get a good deal through a Chinese reseller that could ship directly from Spain. As it turned out, they shipped with Amazon.

The S5 Max has some nifty features, including mopping and LDS laser navigation. The reviews were good. The price was good. So, here we are.

The first few weeks went by without problem. But then disaster struck. The companion app reported a short circuit in the left wheel. I was surprised at this level of detailed error reporting, but that didn’t make the problem go away.

I contact support. Honestly, I was a bit wary about contacting support from this Chinese reseller as they don’t have a good reputation in general. But I got a friendly email back a day later, stating that the left wheel unit should be replaced. This is under warranty.

I was offered two options:

Ship the unit to Poland for repair. All shipping costs would be reimbursed. Nice!
I could buy a replacment left wheel unit from them, and they would refund that amount to me. I would have to replace the wheel unit myself.

As I’m not averse to opening up electronic devices, I chose option 2. The refund was processed before the left wheel arrived, which took only a week or so.

But how do I replace a left wheel unit on a Xiaomi Roborock S5 Max?

Youtube to the rescue! I found this 12 minute video (I have a copy archived if it ever goes missing) that shows you how to disassemble and re-assemble this model of Roborock.

Armed with this video it took about 30 minutes to disassemble the necessary parts, replace the wheel unit and put is all back together.

What surprised me most was how amazingly repairable this unit is. Okay, it takes you a good 15 minutes to take it apart - but it’s all philips screws and easy to remove modules. No glue. No difficult to remove parts, no weird screws. The fact that they’re modules makes them easily replaceable. Wheels, fan, LDS unit, 360° wheel, etc are all separate modules.

Looking around on AliExpress you can find many replacement parts for the Roborock - and it’s actually feasible for anyone who knows how to hold a screwdriver to replace the part.

This is such a breath of fresh air after looking to replace a battery and screen on my OnePlus 6. I didn’t even bother to buy replacement parts for that. Heck, even modern AV-receivers are harder to repair than this.

This goes to show that Chinese products nowadays sometimes do come with fast and helpful support and that repairing one of these units if rather easy to do.

Why I will not repair your amplifier

Sun, 29 Mar 2020 00:00:00 +0000

I like tinkering with electronics and one awesome way of doing that is by repairing things that are broken. I get about ten or more requests every month from random people from the internet to help them repair there device. My reaction is always the same: no, I will not. This posts tries to explain why I say no and what alternative I can offer them.

When I repair something (or anybody else for that matter), it will cost something. There’s a lot of things that go into a successful repair, especially of hardware you’re not intimately familar with. These are the steps I normally take:

Disassemble the device
Isolate and/or reproduce the problem
Find a service manual, datasheet, and other documentation
Theorize a solution to the problem
Find and order replacement parts
Perform the actual repair
Test the repair
Reassemble the device

Sometimes a problem is obvious. Like the repair of the HP 34401A, which turned out to be blown fuse. Sometimes the problem is more subtle and difficult to find, like with the repair of my Philips 3D LED TV.

When I’m looking for a repair project, I always do some research first - I’m not blindly buying things marked ‘with defect’. TV’s often have a broken screen - which I can’t fix. Some amplifiers have issues with HDMI ports, which I know will require me to purchase a rather expensive replacement circuit board. The repair will be expensive and not very interesting. If I’m looking to resell a device, those costs add up and may not make it worth the effort.

Now imagine some guy, let’s call him John, emails you out of the blue:

Hello,

I have a problem with my Denon some model or other. There is no sound at all. Can you help me repair this. I have no experience with electronics, but I do have a screwdriver.

Thanks!

There are quite a few problems with answering these kinds of messages.

Let’s take a look at some legitimate reasons why your amplifier might not produce and sound:

Did you turn the amplifier on?
Did you connect speakers?
- To the right outputs?
- That are not blown or shorted?
- With enough power?
- With the correct impedance for your amplifier?
Is your amp muted or is the volume set to its lowest setting?
Does the source contain audio?
Did you select the correct source?
Did you connect the source correctly?
Is the source using a digitial format your amp can decode?
Did you mess with assignable inputs and does the input use the correct audio connector?
If your source is phono, do you need a pre-amplifier?
Did you leave your headphones connected?
If using HDMI or optical, maybe you try a different cable first?

These are all legit reasons why an amplifier could not produce sound. And this is just the part to make sure there’s actually something broken in the first place.

Next, the person has little to no experience with electronics. At the very least you need to understand the dangers of opening an amplifier and what precautions to take to work with it safely.

Next, you will need at least a multimeter and maybe even an oscilloscope to find the exact problem. You will need to look for shorts, high resistances, check diodes and transistors.

After that you will need a soldering iron or SMD rework station. You will need solder, flux, solder wick or solder sucker. Do you know how to use these tools?

When finished, you need to verify your work - did you put everything back together correctly? Did you not make any shorts by accident?

It’s not feasible for me to personally take someone by the hand and guide them through this process.

Some people will then reply to me:

Can I just ship it to you, then?

Well, no.

I don’t have time for it. I’m not a repair shop. Also, I don’t want to risk damaging your stuff beyond repair - or it to be damaged in transit. And most of all, I’m not interested. This is not my day job.

So, what can I offer people who ask for my help.

First of all, if you device is under warranty, take that route. Just go back to the place you bought it or to the manufacturer and get them to repair it under warranty. You paid for that service, use it.

If your device is out of warranty, there are two options. Which you choose depends on your confidence level for repairing electronics.

If you don’t have any tools available and this is your first time, you’re probably better off finding a local technician to perform the repair. This will cost you money, and there’s a chance the repair cannot be carried out because parts are no longer available or the cost of the repair is higher than purchasing a new comparable product.

If you do have some tools, basic electronics knowledge, and a willingness to learn more, you should hop over to /r/AskElectronics and /r/audiorepair. There are over 80,000 people there who can help you out. Most of them will have more experience than me. Some might even know your particular device or the problem you’re having.

An added benefit is that your quest will be documented online for others to read.

Repairing can be great fun. Most people can do it, given you have the right tools and an eagerness to learn. There’s a great bunch of people out there who’re ready to help you out - for your benefit and that of the entire community.

Test Flutter on Drone CI

Tue, 17 Mar 2020 00:00:00 +0000

This post is specific to Drone CI, but is probably easily adoptable to other CI systems, like Gitlab.

Flutter is UI library based on Dart to create beautiful, natively compiled applications for mobile. Web and desktop targets are also in the works.

Recently I’ve been playing around with Flutter and Dart. As a backend engineer I have to say I quite like it. Dart is a breeze to work with, the static typing helps a lot. Flutter itself offers a react-like way of structuring your app, but it runs natively on Android and iOS. At Kabisa we’re in the process of adopting Flutter as our go-to solution for cross platform mobile apps.

Anyway, one of the first things I do is hook a new project up with my CI, currently Drone CI. I started looking for an already published docker image that contains the latest flutter-stable installation. I soon found cirruslabs/docker-images-flutter, but ran into an permission error. Turns out [I’m not the only one][cl-issues], but with a few simple steps you can test your flutter app with this image on Drone CI without issue.

.drone.yml:

kind: pipeline
name: default

steps:
- name: test
 image: cirrusci/flutter:stable
 commands:
 - sudo chown -R cirrus:cirrus .
 - flutter doctor
 - flutter test

The magic is the sudo chown -R cirrus:cirrus .. The flutter docker images are based on Cirrus Labs’ android SDK images, which runs commands as the cirrus user. This will change permissions and the flutter commands can now run without issue.

Any next steps in your pipeline will encounter files owned by cirrus. Since most drone plugins and images run as root anyway, this is not much of and issue. You could add another command to the snippet above to reset permissions back to root.

Note that I’m using cirrusci/flutter:stable. They’ve also got images available for beta and dev, just use the appropriate tag.

A Drone Hugo plugin that works

Wed, 11 Mar 2020 00:00:00 +0000

This article contains some background to why I wrote this plugin. If you just want to use it, see github.com/ariejan/drone-hugo for details.

I recently moved away from Gitlab + Gitlab CI to a Gitea + Drone setup. Both Gitea and Drone are lightweight and fast, and let’s be honest, more than enough for an engineering enthusiast like myself. They now also run on my ~~NAS~~ Homelab Server as docker containers, which helps to save a few bucks every month in hosting fees.

Hugo

This website (and a few others I manage) are built using the static site generator named Hugo. It’s written in Go andis super fast:

 ~/src/devroom.io $ hugo
| EN
-------------------+-------
Pages | 1209
Paginator pages | 0
Non-page files | 13
Static files | 249
Processed images | 0
Aliases | 450
Sitemaps | 1
Cleaned | 0
Total in 3073 ms

Because I like using .scss and some other goodies, I need to use hugo_extended.

Drone + Hugo

There’s an official Hugo plugin for drone, but it has two problems:

The official documentation is lacking on the use of the extended version of hugo
When you manage to get the extended version installed, it doesn’t work

As a good open source citizen, I opened up the code and took a look around. At this point I noticed a few more “problems” with this plugin.

The plugin needs to run on multiple architectures, that’s cool, but I’m happy with ye good ‘ole amd64.\

It contains a custom Go program to kick-off hugo builds. I can understand that this is useful for more complex plugins or plugins that are easier written in go because of available libraries. However, all this plugin needs to do is install Hugo and run it.

It supports a metric ton of command line options for Hugo. I understand why that’s important, but I have no use for any of them.

Can I do better?

I could have taken up the official plugin and try to make it work. The problem here is that it’s a rather complicated piece of software for a rather simple task:

Download the specified versio of Hugo
Run hugo to generate static HTML files

What I found is that a Drone plugin can be written in whatever. Settings from .drone.yml are passed on as environment variables with a PLUGIN_ prefix.

So, I opened up Vim and started to write a simple shell script that would download the proper, extended version of hugo and run it. This is all there is to it:

#!/bin/sh
HUGO_VERSION=${PLUGIN_HUGO_VERSION:-"0.67.0"}
HUGO_ARCH="64bit"

HUGO_URL="https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-${HUGO_ARCH}.tar.gz"

echo "Fetching Hugo ${HUGO_VERSION} from ${HUGO_URL}"
wget -q -O- ${HUGO_URL} | tar xz -C /usr/local/bin

hugo

This will take the specified Hugo version (or default 0.67.0), download and untar it and run it. That’s all I need.

It does need to be packaged up in a Docker image to be useful with Drone, so I needed a Dockerfile as well.

FROM alpine:3.11

RUN apk add --no-cache \
 ca-certificates \
 mailcap \
 git \
 wget \
 libc6-compat \
 libstdc++

ADD drone-hugo.sh /bin/
RUN chmod +x /bin/drone-hugo.sh

ENTRYPOINT /bin/drone-hugo.sh

Again, this is pretty straighforward. Use alpine for a small image size, add some dependencies and copy over the shell script.

Using ariejan/drone-hugo

In your .drone.yml you can add a build step. You’ll want to follow that up with something that uploads your HTML files.

steps:
 - name: build
 image: ariejan/drone-hugo
 settings:
 hugo_version: 0.65.3

At this time there’s no option to pass along any arguments to Hugo - as I don’t need that. Instead of exposing each option manually, I’m considering to add a single hugo_args for that purpose.

You can find the full source and install instructions on github.com/ariejan/drone-hugo.

The git submodule cheat sheet

Mon, 09 Mar 2020 00:00:00 +0000

A git submodule, in its essence, is a reference to another git repository. It’s a great way to include vendor code (like plugins or themes) into your own code base. This post contains some examples on how to use git submodules effectively.

Add a submodule

You need to know the remote git repository url and where you want to place that it in your repository.

git submodule add https://example.com/submodule-repo.git path/to/submodule
git add .
git commit -m "adds submodule path/to/submodule"

Cloning a project with submodules

When you clone a repository that contains submodules there are a few extra steps to be taken.

git clone http://example.com/repo.git repo
cd repo
git submodule init
git submodule update

If you’re sure you want to fetch all submodules (and their submodules), you can also use this fancy one-liner:

git clone --recurse-submodules http://example.com/repo.git

Update your submodule

If you’re simply tracking the master branch for the submodule, you can suffice with a simple fetch and merge.

cd path/to/submodule
git fetch
git merge origin/master

If you’re in a hurry, you can streamline this for all submodules in your repo with:

git submodule update --remote --recursive

Don’t forget to commit this change to your own repo, so others are locked to this new version of the submodule as well.

Track a specific branch of version

The repo for your submodule may have a specific branch (e.g. stable) or tag you want to track, instead of master.

git config -f .gitmodules submodule.path/to/submodule.branch stable
git submodule update --remote

Again, don’t forget to commit your changes to .gitmodules to send this change to other contributors to you repository.

Remove a submodule

Removing a git submodule consists of two steps: removing the reference and removing the locally cached version.

git submodule deinit path/to/submodule
git rm path/to/submodule
git commit -m "removes submodule path/to/submodule"

rm -rf .git/modules/path/to/submodule

Bonus: see submodule status in `git status`

You can configure git to show a submodule summary when you do a git status. There is a small performance trade-off here, but it might be useful to you.

git config status.submodulesummary 1

Building a DIY Home Server with FreeNAS

Fri, 28 Feb 2020 00:00:00 +0000

Keep reading! I’ve documented several upgrades to my DIY NAS at the bottom of this post.
After you’re doing reading this, hop over to my latest update on my homelab server / NAS.

This post is almost a year over due. I think it might still be relevant for people looking to start a NAS project or upgrade from a consumer-grade NAS like QNAP or Synology.

This is by no means a definitive guide on how to build a reliable network attached storage server. I made choices relevant to my use case, yours might be different.

For you eager people, there’s a full part list at the bottom of this post.

Some history

Up until May 2019 I stored my photos, old documents and other things on a single 3TB Western Digital Green drive in my Hackintosh.

Backups were already part of my setup by using Backblaze. This was great because it allowed me to use unlimited backup data for a fixed price.

But I was running into some limitations of this setup.

Filesize and volume of digital media

Every new phone has a better camera, and I like to shoot in the best quality available. With my current Oneplus 6 that means 3456×4608px per photo (roughly 4-8MB per photo). Not to mention 4k video at 60fps…

Having kids also means taking a lot more photos and videos.

Before I would automatically backup media on my phone to Dropbox, because that was convenient. But with a 5GB limit I would often be reminded to either upgrade or clean up some files, which means manually moving older photos and videso from Dropbox to my trusty 3TB hard drive.

Restoring backups sucks

I have not kept track on how many disks failed on me, but is has happened on a few occassions. Backblaze to the rescue! However, restoring 1TB+ of data is not a fun activity.

The reality is, disks fail. Especially if you buy the cheapest ones available and don’t take care of them.

Streaming video

I’ve always been a fan of Plex. With a collection of over 200 DVDs, there’s plenty to watch, but physical drives are disappearing all around. Plex is a great solution, but it works best if you media are available when you need them. Having to be at home and turn on your computer to watch something is a nuisance.

The plan

So, after much consideration, I decided I needed a NAS. Many of my friends have one and they seem to be happy with them. Of course, I could not run to a local electronics store and buy just any storage server. First I had to write down what I wanted out of this system.

Store 1TB+ of data safely
- Redundancy to cope with disk failure
- Extendable in the future
- Write/Read performance should be okay, but not a priority (I’m not editing 8k video on this thing)
- Problaby needs a UPS for safe shutdown in the case of power loss
- Share data/disks over Samba/AFS/NFS, Time Machine support would be nice
It needs to be always-on
- Energy efficiency is important
- It needs to be silent, will probably sit in the study
Needs to run Nextcloud and Plex
- Needs a companion mobile app to auto-backup photos and videos
- Needs sufficient memory
- Needs sufficient CPU power for transcoding 1, maybe 2 simultanious streams
- Room to grow to run more apps or VMs
Needs to be future proof
- More storage
- More memory
- Be able to use this a desktop computer if this NAS thing really sucks

What NAS solution fits best

What I had to figure out is what NAS solutions are available to me, and which one is most suitable for my usecase.

Consumer grade NAS systems like those from QNAP or Synology are quite popular. They do come with a few major drawbacks for me. They are not (or minimally) upgradeable. A simple Celeron CPU and 8GB of memory will probably not suffice. I’m also stuck with a custom operating system, something which I try to avoid like the plague.

At this point, my best option in this category was the Synology DS918+, retailing locally for €550. With room for four drives, it already felt limited. I want it to be expandable. The Intel Celeron J3455 (quad core @ 1.5Ghz) and 4GB RAM seemed especially limiting. Also the vendor lock-in is not very appealing.

Next I visited /r/DataHoarder, a nice corner of the internet where people come together to discuss hoarding data - and all the hardware needed to do that. Many here opt for used enterprise hardware from SuperMicro or Dell in 19" rack cases. The hardware can be very powerfull, but comes at a cost too. Space and noise are two considerations. This enterprise hardware is made for use in datacenters where noise is not much of an issue. Having to mod a case to be silent would be a major effort. Secondly, a 4u 19" server is quite bulky and not something you easily fit under your desk.

This left me to find the middle ground: consumer grade hardware, but with special, preferably open source, software. /r/DataHoarder quickly pointed me towards FreeNAS. _“Enterprise-Grade features, open source, BSD licensed” grabbed my attention. Based on FreeBSD, FreeNAS leans heavily on ZFS - which sounded really good to me.

At this point I setup a VirtualBox VM on my Mac with 4 “8GB Disks” to give FreeNAS a spin. The setup was rather easy (not how I remember installing FreeBSD 5.x years ago). I toyed around with ZFS, setting up a pool, removing / replacing / resilvering disks. It was awesome.

My mind was made up: I wanted a x86/amd64, FreeNAS compatible system with 8+ 3.5" drive bays in a small-ish form factor.

FreeNAS FUD

Here’s the deal with FreeNAS. It’s maintained by a company named iXsystems, who sell hardare and also offer a more professional version of FreeNAS called TrueNAS. The forums are filled with people who run NAS’es in professional settings.

If I were to setup a NAS for a company, yes, I would opt for the Pro/Enterprise SAS drives, Intel Xeon or AMD ThreadRipper CPUs and loads of ECC RAM.

However. I’m building a home NAS system with four users (this includes two pre-schoolers who can’t even read yet).

So, although many of the recommendations of helpful people on the internet are true, it does not necessarily mean you’re wrong if you do it differently.

Shopping time!

Time to go shopping! I always try to find the best value for money, so these choices were not always directly obvious.

Motherboard / CPU / RAM

When I built my NAS, Intel was still king. If I had to do a rebuild, I’d happily opt for some AMD gear, but alas, here I am. I was happy with the performance of my Intel i5-3570K and I looked for something comparable in a newer generation. The Intel i3 8-series is now quad core, and the i3-8350K goes up to 4Ghz and a maximum of 64GB DDR4 RAM.

To go with this, I opted for the Gigabyte Z370M D3H motherboard. It’s a decent board in µATX format with 6 SATA ports and 2 NVMe slots. It can accomodate a few expansion cards and has 4 slots for a maximum of 4x 16GB RAM. It also on the osx86 compatibility list, so if I need to, I could run macos on this board.

I wanted decent memory (but without all the RGB madness). I started out with 16GB (single module) Corsair Vengeance (DDR4 2400Mhz).

Many will swear on their mother that you need to have ECC memory when running ZFS. There is a point to be made for the benefits of ECC memory. But this is a pro-sumer system. As Matt Ahren (somewhat head of the OpenZFS project) once wrote:

There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.

I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.

I did not get a graphics card for this system.

Enclosure

I didn’t want a large 19" rack server. I also had a hard time imagening where I’d put a tall tower system. This led me to investigate some of the more “exotic” enclosures on the market. I was looking for something that could fit 8+ 3.5" drives and a µATX motherboard.

When I found the Fractal Design Node 804 I was sold on it immediately. It’s a beautiful case, fits ten (10!) 3.5" drives, has good cooling options, and it fits perfectly on the bottom shelve of my IKEA Ivar shelving unit.

Disks

I wanted as much storage as possible. But at a reasonable price. I had about 1TB of space in use already, so I would need at least four times that. I found a good deal on four 3TB Western Digital Red drives, which in RaidZ (the ZFS “equivalent” of RAID5) leaves about 9TB of usable space. That seemed plenty.

Updates 1 - Moar Memories!

I quickly realised that 16GB was fine, but since I was running more and more services on this system, and I was looking into running at least a Linux VM to play with Docker, It’d need more memory. An easy upgrade, since I was using only one of the four memory slots.

Update 2 - Moar Spinners!

More spinners, rust, disks, drives, whatever your want to call it. As it turns out, once you have 9TB of storage at your disposal, it quickly fills up. This is also around the time I discovered shucking.

It turns out that Red drives are kind of special in how the can deal with vibrations. Their firmware is also optimized to run drives 24/7. The thing is, they are rather expensive.

Now that’s a whopping 44TB Raw storage in a single pool. The pool consists for 2 vdevs, each with 4 drives in RaidZ1. I can lose one drive in each vdev without issue.

(Sorry, I have to remove identifiable drive data so other can’t abuse them)

The external USB drives (My Book and Essential) are much cheaper, and they often contain “white” drives, which are identical in specs to reds, but for “internal use” by Western Digital. Once the were on sale, it was easy to pick up four of those external drives, remove the disk and put them in the NAS.

For those looking to go the shucking route, keep in mind the following:

You void your warranty by removing the disks from their enclosure. WD might be lenient when you RMA them, but know what you’re doing.
Not all models have white drives and some models are known to have many issues (like the 6TB drives from WD). Check out /r/DataHoarder for up to date info.
Be sure to check the entire disk in its enclosure. You can easily RMA it if you find defects, and you know you’re putting something working into your machine.
White drives use a newer SATA spec, which re-uses a pin to disable power to the drive. If your system or power supply cannot handle that, the disk will not start-up. This issue can be easily fixed by using a piece of tape.

A real HBA Card

With only six SATA connectors on board, how did I manage to hook up 8 SATA drives? Well, I bought the cheapest SATA controller card I could find on AliExpress. It said it supports four drives, but I could only get it to work with two.

After some research I found a good deal on a use Dell H200 PERC card, which sports the LSI 9211-8i chipset. It has 2 SAS connectors, and when flashed into IT mode, supports 8 SATA drives - and it works flawlessly with FreeNAS.

Only problem is that it needs a bit of external coolingk as the card is used to sitting in a well cooled 19" server.

Yes, I cleaned out the dust, but after running for about 9 months I found the inside rather clean.

Parts list

This is an up-to-date parts list of my current Home Lab NAS / Server

Type	Item
CPU	Intel Core i3-8350K 4 GHz Quad-Core Processor
CPU Cooler	ARCTIC Alpine 12 CPU Cooler
Motherboard	Gigabyte Z370M D3H (rev. 1.0) Micro ATX LGA1151 Motherboard
Memory	Corsair Vengeance LPX 16 GB (1 x 16 GB) DDR4-2400 Memory
Memory	Corsair Vengeance LPX 16 GB (1 x 16 GB) DDR4-2400 Memory
Power Supply	be quiet! Pure Power 11 CM 400 W 80+ Gold
Case	Fractal Design Node 804 MicroATX Mid Tower Case
Storage (Boot)	Gigabyte 128 GB M.2-2280 NVME Solid State Drive
Storage	Western Digital Red 3 TB 3.5" 5400RPM Internal Hard Drive
Storage	Western Digital Red 3 TB 3.5" 5400RPM Internal Hard Drive
Storage	Western Digital Red 3 TB 3.5" 5400RPM Internal Hard Drive
Storage	Western Digital Red 3 TB 3.5" 5400RPM Internal Hard Drive
Shucked Storage	Western Digital ELEMENTS 8 TB External Hard Drive
Shucked Storage	Western Digital ELEMENTS 8 TB External Hard Drive
Shucked Storage	Western Digital ELEMENTS 8 TB External Hard Drive
Shucked Storage	Western Digital ELEMENTS 8 TB External Hard Drive
UPS	APC Back-UPS BX700U-GR 700VA/390W
Cables	CableDeconn 0.5M Red Mini SAS 36P SFF-8087 To 7P SATA
HBA	Dell Perc H200 Integrated Sas Pci-e 2.0 Raid Controller

PCPartPicker Part List

Future upgrades

There are some future upgrades I’m look at.

~~Add four 120mm Nocuta fans at the front to optimize air intake for cooling both the SAS Controller as well as the eight spinners. Still have figure out how to PWM control four fans.~~ I’ve added two front fans on the motherboard side to provide enough airflow for the passively cooled Dell Perc H200. I’ve also added another front fan on the harddisk side (at the top) to increase airflow across all eight HDD’s. Temperatures are much more stable now.
Add an additional mirror of two spinners for a seperate pool, maybe local Borg backups of the most important data.
~~Add an additional NVMe SSD.~~ My motherboard has two m.2 slots, so I added another (relatively cheap) Gigabyte 128 GB SSD and put that into a ZFS mirror configuration.
Add a mirror of two 2.5" SSDs to run my VM off.
Upgrade the 3TBs to something better (16TB, anyone?). Heck, upgrade all drives to 16TB for 128TB RAW storage capacity (96TB in the current 2x RaidZ1 setup) Additionally I might opt for a DAS enclosure with 4-8 more HDDs and hook these up through another SAS expander card.
~~Fill up the RAM slots for a total of 64GB~~

Want to build a NAS yourself or have any questions? Feel free to drop me a line.

Big August 2020 update

Right, so it’s time to write an update to my NAS project. Quite a few things have changed, so gear up!

The image above shows it already; I changed enclosures. But why? Let me start by saying that the Fractal Design Node 804 is an amazing case, but it has three drawbacks.

The airflow, even with 3 fans blowing across the hard disks, was not enough to keep the disks cool. Especially the 8TB Reds would easily reach over 50°C (~122°F) on hot days. That’s probably not something to worry about, but better safe than sorry.
The HDDs are not really easily accessible. Now I didn’t have any issues with disks yet, but with the amount of cable clutter and caddies with four drives each, it’s not trivial to replace a disk. This becomes more important now as failures are more likely to start happening and I’m still looking to upgrade the 3TB Reds sometime.
It can only hold 8 disks. Well, that’s not really a fault of this case.

So, what’s a reasonable upgrade for a Node 804 that makes disks easily accessible, accommodates more drives and can be fit under my desk. I could have gone with a Fractal Design Define 7 XL, which can house up to 18 HDDs, but it’s quite expensive at over €220. It also did not really make the drives any more accessible.

I soon ended up with a 19" rack mount form factor. I saw a few deals on cheap Dell machines with 8 or 12 bays and decent Xeon-based hardware. I passed on those, mainly because these things are power hogs and really LOUD. Besides, I already have all the hardware I need, I just want another case.

There are a few manufacturers out there that sell 19" rack mount storage chassis. The 2U versions house up to 8 HDD’s, which would be fine. But, if I’m going to invest in a new case, I want that little bit of extra room to expand in the future. This soon leads to 4U cases, which have 10, 16, 24 or even more HDD bays. The problem is, at this time, they’re really hard to find, especially if you want to stick consumer ATX hardware into them. Also, they are almost all sold out. And if you manage to find one, they’re not cheap either at over €500. Such is life in Europe, I guess.

Luckily, while browsing the classified ads section of Tweakers.net I came across a nice fellow who was selling his old Norcotek 4U 16 bay chassis. I informed about the price and we ma deal for €75. The only problem was, he lived at the other side of the country. And you simply don’t ship a 4U server easily or cheaply.

So, I took a day off work and drove over 3 hours to pick up my new NAS enclosure.

Map by OpenStreetMap.org

When I went to pick up the case, it turned out it came with a 520W power supply (already neatly cable managed), four Mini-SAS 8087 to 4x SATA cables and all the original screws in their originally labele bags. Bonus are nice scythe case fanse. Wow!

Installation was pretty straight forward, simply move everything to the new case. My single Dell PERC H200 card could accommodate 8 of the 16 hot swappable caddies. So I would need another one of those to handle the other 8. That would free up my motherboard SATA connectors for additional, internal SSDs and what not. Luckily, I could buy another one pre-flashed from a local ebay seller.

Enter Proxmox

Another item on my list was to move to Proxmox and virtualize FreeNAS. Let me explain why I want to do this.

First, FreeNAS / TrueNAS is a great system. FreeBSD is rock solid and ZFS offers an amazing “storage experience”. The only thing lacking is virtualization support. FreeBSD jails work fine, but I had some recent trouble with iocage commands being terribly slow, and I often struggle to convert linux install instructions to their FreeBSD equivalents. Updating jails has also proven to be a painful process for me. The solution was to run a virtual machine (with a fixed amount of RAM) and run docker on it. It all worked, but it didn’t feel solid. Also, I didn’t hear many great things about bhyve, although I haven’t gotten into any real trouble.

Well, enter Proxmox. Proxmox is a custom debian linux os that offers qemu virtual machines and LXC / Linux Containers. That sounds really awesome. With Proxmox it’s also possible to pass through PCI devices to a virtual machine. This means that I can map both my HBA cards to my FreeNAS virtual machine and run FreeNAS just as before.

The process was quite painless: backup my FreeNAS configuration, install Proxmox, create a VM for FreeNAS, and install it. Then restore your configuration (maybe make a few tweaks for changed device names, like your NIC, or scheduled SMART tasks), and you continue where you left off. It works really great.

I’ve migrated most of my jailed / dockerized services over to linux containers based on Debian. Some more complex ones, like Gitea and Plex are still where they were (a docker VM and FreeNAS jail, respectively).

For fun an giggles I added three old 500GB spinners to create a new ZFS pool with to try things out, like replacing disks, etc.

More RAM

All the while, the original Corsair Vengeance LPX 16 GB (1 x 16 GB) DDR4-2400 Memory were on sale at Amazon.de, so I decided to scoop up a few and extend my NAS’ memory to 64GB total. 24GB of this RAM has now been allocated to FreeNAS, which uses about half of that for services, the rest is used for ZFS cache.

So what about those disk temperatures?

My eight spinners are all located on the left side of the chassis right now, so they’re packed tight. But with the recent heat wave in the Netherlands with ambient temperatures rising to 32°C, no disk went above 44°C, which is a big win if you ask me.

Treebeard and Gandalf

treebeard and gandalf in my Lack Rack.

My NAS has the official hostname of treebeard. I also setup gandalf, a 1U 19" rack mount server with a simple ASRock mini-ITX board an i3 4130T processor. It’s fitted with 2x 128GB SSDs locally. gandalf also runs Proxmox. In fact, they form a cluster of two. This makes it super easy to migrate services between the two nodes. gandalf runs some essential home network services (all in Linux Containers), like pihole, wireguard, and home automation.

Update: gandalf was using over 150W of power while idle. It was also running quite hot, even though I upgrade the three 40x40mm fans with Noctua ones. For now, gandalf has been retired and removed from the cluster until I can figure out what’s causing this insane power consumption. For comparison, treebeard with its 11 spinners, is running at about 100W idle.

APC UPS woos

A year ago I purchased an APC Back-UPS 700. It’s been working great so far. However, after disconnecting it from the old server, and hooking it up again to the new one, I noticed that the USB connection would no longer work. Just nothing. I found out I needed to reset the UPS by powering it off, disconnecting the battery (with the large yellow plug on the back) and turn it back on again.

As that worked to get USB working again, it did yield some strange values in apcaccess, namely that the status was not ONLINE, but BOOST and that it was not reporting some critical values, like TIMELEFT. After a quick call to tech support I had to perform a battery callibration. Basically: you disconnect your load and instead hook up something that burns power, like an old light bulb or an electric heater. You then disconnect mains power and run the battery dry (this is safe, as the UPS will shutdown before you can damange your battery). Then, without any load, reconnect mains and let the unit charge fully.

After this the UPS was working great again.

Recently I read about how this budget line of UPS devices does not work well with no battery connection. E.g. if you disconnect the battery, your load will be disconnected from power, even if you have mains power connected. Read more at Fitzcarraldo’s blog post about his UPS experiences.

September 2020 update

This is starting to turn in a blog-in-a-blog kind of thing. But here’s another update for you.

More APC UPS woos

The reset I performed worked for a few days. Then, in the middle of the night:

STATUS : ONLINE REPLACEBATT

Of course, this happened in the middle of the night, accompanied by a loud beeping tone from the UPS. I purchased a replacement battery online (same model / ratings, just not the expensive, APC branded one). This fixed the problem and the UPS has been working as it should. Still very strange that the battery gave out after just one year of usage.

No more FreeNAS!

This may come as a shocker ;-) I dropped FreeNAS. As you may remember I started running FreeNAS in a VM on Proxmox with the HBA card passed through to FreeNAS. In order to share my pool/datasets with other VMs / containers in Proxmox, I needed to share them over NFS to the host system, which would then require my containers to run in priviledged mode to mount the NFS shares. The real issue was performance, many apps had issues with locking to the NFS mounts. Downloading large files would sometimes throw errors. It was not fun.

But then I thought, what am I really still using FreeNAS for? I don’t like the jails - they’re FreeBSD and I prefer linux. I have an NFS share - but only because I need to share with the host. Maybe a Samba share for time machine, but that’s really all. Well, and for ZFS of course.

But Proxmox support ZFS as well! A small container can run Samba for Time Machine backups. So, I took the plunge:

Shut down the FreeNAS Vm
Disable auto-start :-)
zfs import core-storage tank - importing and renaming the pool in one go

That. Was. It.

Moar drives!

My chassis can house 16 3.5" HDDs in hot-swap caddies. I ordered a second Dell PERC H200 HBA card from eBay. I now have the following drives in my pool:

4x Western Digital 3TB Red
4x Western Digital 8TB White (shucked from WD Elements)
4x Western Digital 14TB White (shucked from WD Elements)

Yes, you read that right. Amazon.de had the 14TB WD Elements on sale, so I grabbed four of them. My pool now consists of three RAIDZ1 vdevs:

# zpool status -v tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 0 days 20:08:21 with 0 errors on Thu Sep 17 05:25:13 2020
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdb ONLINE 0 0 0
sdg ONLINE 0 0 0
sde ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdd ONLINE 0 0 0
sdf ONLINE 0 0 0
sdh ONLINE 0 0 0
sda ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0

And some specifics:

# zpool list tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 90.8T 35.2T 55.6T - - 9% 38% 1.00x ONLINE -

That should be sufficient for a while, but don’t quote me on that. :-)

Smart UDMA CRC Errors

For a bit /dev/sdi had some UDMA CRC errors. These are, especially with new drives, often caused by cable issues. I offlined the drive, stuck in one of the four remaining free slots, and the problem was solved. ZFS of course had picked up on this as well and Proxmox immediately nofitified me of the SMART and ZFS issues by email. Yay for Proxmox!

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
199 UDMA_CRC_Error_Count 0x000a 100 100 000 Old_age Always - 16

Resetting the ZFS pool status is easy with zfs clear tank. For good measure I ordered a scrub, which finished without any further issue.

Next steps?

Nothing right now. I have several TB’s of storage left to hoard. CPU-wise I’m good. Memory is all max’ed out. I love Proxmox for being Linux and supporting ZFS.

One more thing…

Yeah, I know what I said last time I updated this post. I upgraded my motherboard/CPU/Memory even further.

Read more in my latest post on my NAS: https://www.devroom.io/2020/11/12/the-big-diy-nas-update

How to background a running process over SSH

Wed, 28 Aug 2019 00:00:00 +0000

Today I was logged in on my FreeNAS server to setup Duplicity backups. After initial success on a small dataset of about 100MB, I cloned my configuration to backup my long term storage dataset. It contains about 50GB of data - or so I thought. It turns out there’s over 500GB there.

I let the job run throughout the day, not minding it much. But then time comes to close my SSH connection, and I know what that means: the process will be killed. And I really do not want to kill this running backup.

I know about screen. And tmux. And nohup. Even about <command> &. But alas it’s too late for that.

Luckily, there’s solution to this (if you’re running Bash or Zsh at least, I didn’t do a compatibility chec).

1. Background your running process

Your process is probably running in the foreground, so you’ll need to background it. There are two steps to this.

Stop your process with Ctrl-Z. This basically pauzes the process, it does not kill it.
Background the process by issueing the bg command. This will continue the process where it left off in the background. No harm done.

2. Disown the process

The final step is to use disown to remove the job from the current shell. This means than when the shell is killed when you sign off, the process is not killed as well.

You can use jobs (or jobs -l on FreeBSD) to see a list of current jobs and their process ID. To disown a process, use disown -h <process id>.

That’s all. Your process has now been removed from your shell session and is running happily in the background.

Bonus action: Background your process from another SSH session

In case you can’t use Ctrl-Z, you can log in with another SSH session and use the kill command to stop and continue the process that way.

Sign in with another session and use ps to find the process ID.
kill -SIGSTOP <process id> to stop the process
kill -SIGCONT <process id> to continue the process in the background.
Use disown as described above to disconnect the process from it’s parent shell process.

Disclaimer

This approach does not with all processes, namely those that require access to the TTY to function. vim is a good example of this. If you background Vim, it will be stopped and you cannot disown it.

Bug severity explained

Fri, 08 Dec 2017 00:00:00 +0000

Recently I got an email asking me about how I classify bugs and issues and how the different categories for a bug’s severity, like Critical and Major can be explained clearly.

I’m a software engineer and for the longest time have I approached everything in my work as a software engineering problem. Bug reports are one of them. Any bug report is a report on how software does not behave like it should.

The overview page does not load < 2 seconds

Fix the calculation of the thing to reflect real world exchange rates

Bug reports are not about your software. Bug reports are about users and stakeholders using your software to perform a task. And a bug report tells you that a user cannot perform that task in an acceptable manner.

How do real engineers do it?

Computer programmers have a tendency to reinvent the wheel every few months. It’s surprising to see new ecosystems (looking at you, JavaScript) reinvent a lot of unix principles that have been around for over forty years.

So, let’s take a look at how the non-computer engineers tackle the categorization of bugs and issues. I found that ProQC has a nice write up on the topic. Note that their guidelines are aimed at physical products, but there’s a sensible parallel to software products.

Critical

Any condition found which poses the possibility of causing injury or harm to, or otherwise endangering the life or safety of, the end user of the product or others in the immediate vicinity of its use.

Luckily for most software engineers, nobody dies or gets injured from your software not working correctly (although many stakeholders want you to think otherwise). This is probably true for 99% of all web applications. That does not mean that bugs cannot cause harm in other ways.

Critical issues result in (most likely) financial damages for you or your customer.

Missed business opportunities
Waste due to product inavailablity (imagine an office of 100 people thumb twiddling because the website is down)
Fines because of failure to comply with rules and regulations
Customers leaving or seeking compensation for their damages

Basically any issue that results in real word damages should be classified as critical. Of course, this is related to your terms of service or service level agreement.

Major

Any condition found adversely affecting the product’s marketability and sale-ability or adversely affecting its required form, fit or function and which is likely to result in the end user returning it to the source from which is was purchased for replacement or refund.

Imagine you buy a new iPhone and when you unpack it, it has a scratch across the screen. You don’t want that, although the phone probably works just fine.

In software I would categorize this as any issue that prevents a user from correctly or efficiently performing a task. It’s closely related to critical issues, but the financial impact is limited.

Minor

Any condition found which while possibly less than desirable to the end user of the product, does not adversely affect its required marketability, sale-ability, form, fit or function and is unlikely to result in its return to the source from which it was purchased.

This will include most software issues that you would call annoyances or improvements.

Trivial

Trivial is not mentioned in the ProQC list, which kind of makes sense. A physical product is impossible to update without substantial cost. Software, however, can be updated frequently and strategies like continuous deployment make rolling out these changes almost painless.

I’m not a fan of the trivial category. It’s just too confusing for everyone involved. Trivial has too many different meanings to different people.

this is a huge issue and should be fixed asap
this fix is so easy
this issue is not a big deal, but I had to report it anyway

Any bug you’d classify as trivial has either a critical, major or minor impact on the user, so classify it as such.

Quantitative measurements

It’s tempting to add metrics to each category’s description. Of course you need to specify what you expect your application to do, but a description like, “It’s a major issue if the overview page takes longer than 10 seconds to load,” will not hold up. Aside from how and who measures this, what if the load time is a consistent 9.5 seconds?

The question should always be: in what way if the user impacted with this defect? Does it cost them money, will they stop paying for your SaaS subscription or is it an annoyance that can be optimized later?

I hope this post was helpful to you to get a grip on how to categorize bugs and issues. This is by no means a definitive list and I believe that it’s a good practice to regularly evaluate which categories you use and how you use them.

Tmux and Vim: Copy and Paste on macOS Sierra

Wed, 22 Mar 2017 00:00:00 +0000

As a developer, more than anything, you copy and paste things. Making copy and paste work in properly with Vim and Tmux is, unfortunately, not trivial.

There’s a lot of information to be found on the internet about setting up copy and paste with Tmux. There’s also lots of information on how to setup Vim. But then you run Vim 8. On macOS Sierra. And things break down quickly.

This short guide helps you setup Tmux and Vim on macOS Sierra for proper copy pasting glory!

Dependencies

First, you’ll have to install one dependency. I also recommend you install the latest and greatest Vim instead of using the (old) version bundled with macOS, and of course, tmux.

brew install reattach-to-user-namespace
brew install vim tmux

Configure Tmux

Tmux is a weird beast. Copy and pasting can be done in different ways, but I prefer the Vim style navigation.

# Configure your default shell, Zsh in my case.
set -g default-shell $SHELL

# Override the default command to use `reattach-to-user-namespace` for everything.
set -g default-command "reattach-to-user-namespace -l ${SHELL}"

# Remap prefix to ctrl-a (or caps-a for my mac)
set -g prefix C-a

# Vim style navigation in copy mode
setw -g mode-keys vi

# Setup 'v' to begin selection, just like Vim
bind-key -t vi-copy v begin-selection

# Setup 'y' to yank (copy), just like Vim
bind-key -t vi-copy y copy-pipe "reattach-to-user-namespace pbcopy"

# Update default binding of `Enter` to also use copy-pipe
unbind -t vi-copy Enter
bind-key -t vi-copy Enter copy-pipe "reattach-to-user-namespace pbcopy"

# Bind ']' to use pbbaste
bind ] run "reattach-to-user-namespace pbpaste | tmux load-buffer - && tmux paste-buffer"

This is how copy pasting will working after you’ve made the proper changes to your .tmux.conf. Note that I have remapped ctrl + b to ctrl + a, because that works pretty fast when you’ve already remapped Caps-Lock to Control.

ctrl + a, [ to enter copy mode. This allows you to use normal Vim motion keys to move around.
v to start selection, again use vim motion keys to move around.

y to yank the selection. enter also works.

Pasting works just as before:

ctrl + a, ]

Congratulation, you can now copy and paste with Tmux.

Configure Vim

The changes needed for Vim are minimal. All you need to do is unset the name of the default clipboard, so it will pass through to Tmux. In ~/.vimrc:

" Clipboard 
set clipboard=unnamed

Bonus features

So, copy and paste is now working from Vim, Tmux and Vim-in-Tmux. Pasting is also working as expected.

As a bonus, when you have two tmux panes or windows with different instances of Vim running, you can now easily copy and paste between them by using normal Vim commands!

Conclusion

Copy and paste with Vim and Tmux does not work out of the box, but once setup works like a charm.

Here are links to my ~/.vimrc and ~/.tmux.conf for your perusal.

I highly recommend tmux 2: Productive Mouse-Free Development, but feel free to check all my book recommendations.

Image Zoom with plain JavaScript and CSS

Mon, 20 Mar 2017 00:00:00 +0000

Note: I’ve moved away from the solution outlined in this post. Mainly because I did not want to write an entire gallery from scratch. I’ve since opted to use lightgallery.js which is a pure JS image gallery solution with some nice goodies.

The premise is simple. A post may contain images. These images are restricted in rendered size to keep the flow of the page in tact. Clicking an image allows you to zoom in. Here’s an example:

Go ahead, click that bunny!

The CSS

Let’s get the CSS out of the way first. The selector used is article img, which means any image in the post. By default I limit it to a maximum width of its parent container. Also, I change the cursor to a pointer, to indicate you can click on the image, like a link.

article img {
 max-width: 100%;
 cursor: pointer;
}

Then there are images with the zoomed class. This is still the same image element, but with an additional class:

article img.zoomed {
 position: fixed;

 top: 5vh;
 bottom: 5vh;
 left: 5vw;
 right: 5vw;

 max-width: 90vw;
 max-height: 90vh;

 margin: auto;

 border: 4px solid #000
}

Okay, that’s a bit more CSS, but this basically overlays the image on to the page and adds some whitespace around it.

The trick to zooming is adding the zoomed class to the img element. Zooming out means removing that zoomed class again.

Now, on to the JavaScript…

The jQuery solution

For years now jQuery has been my go-to tool for anything JavaScript, mainly because it comes bundled with Rails. (Yes, I used prototype as well in the old days.)

The jQuery solution is as you rather straight forward. Wait for the DOM to be loaded, and handle click events on all article img elements. When clicked, toggle the zoomed class.

$(function() {
 $(document).on('click', 'article img', function() {
 $(this).toggleClass('zoomed');
 });
}

Because I’m a keyboard user (Vim, not Emacs, thank you), I prefer to map ESC to also close any zoomed imaged.

$(function() {
 $(document).keyup(function(e) {
 if (e.keyCode == 27) {
 $('img.zoomed').each(function(idx) {
 $(this).toggleClass('zoomed');
 });
 }
 }
}

Again, hook into the keyup event, check if ESC was pressed and toggle the zoomed class for all zoomed in images.

But, using jQuery means adding an extra dependency: 1 extra HTTP request and 85kB download for you. Also, the few friends I have who practice JavaScript tell me that pure JavaScript is the way to go these days. So, let’s try!

The JavaScript solution

With some help from the You Might Not Need jQuery website, I managed to drop the 85kB big jQuery dependency and rewrite the above functionality in plain old JavaScript.

First, let’s write a function that waits for the DOM to load.

function ready(fn) {
 if (document.readyState != 'loading') {
 fn();
 } else {
 document.addEventListener('DOMContentLoaded', fn);
 }
}

This was taken straight from You Might Not Need jQuery to wrap any functions you want to run when the document has loaded fully.

Next I wrote a function to handle toggling the zoomed CSS class on the images:

function imageClick(e) {
 e.preventDefault();
 this.classList.toggle('zoomed');
}

It turns out that JavaScript is more than capable of toggling classes on its own.

While we’re at it, let’s also write the function that handles the ESC presses.

function handleEsc(e) {
 if (e.keyCode == 27) {
 var zoomedImages = document.querySelectorAll('img.zoomed');
 Array.prototype.forEach.call(zoomedImages, function(el, i) {
 el.classList.toggle('zoomed');
 });
 }
}

This is a bit more involed. I still check for the proper keyCode, and then proceed to find all zoomed images using document.querySelectorAll. It’s really that easy.

Next I use the Array prototype to map a function to each zoomed image. That function simply toggles the zoomed class, just like imageClick does.

What remains is nothing more than some glue to put the above fuctions together.

ready(function() {
 var images = document.querySelectorAll('article img');
 Array.prototype.forEach.call(images, function(el, i) {
 el.addEventListener('click', imageClick);
 });

 document.addEventListener('keyup', handleEsc);
});

Here I use the ready function I wrote. Just like handleEsc, I find all article img elements and add the event listener for clicks. Then I also add an event listener for the ESC key.

Conclusion

Rewriting a trivial piece of jQuery code to plain JavaScript appears to be more than worth the while. Besides the warm fuzzy feeling of dumping jQuery, it saves quite a few kilobytes from each page on devroom.io. Especially for mobile users this matters.

git says 11 deletions (bye, jQuery) and 27 additions (hello, JavaScript). This does not tell the full story, as one of these deleted lines is this one:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>

I can highly recommend you take a closer look at what your jQuery code is actually doing and consider moving away from unnecessary dependencies. Yay for lean and mean web pages!

Bitcoin Mining Anno 2017

Wed, 08 Mar 2017 00:00:00 +0000

Bitcoin was introduced in 2008 and has gained much popularity since then. At the time of writing the exchange rate for 1 BTC is about €1100 (or $1166), quite an increase from when you could get 1 BTC for just a few euro.

It is 29 November 2017, just eight months after writing this post, and the exchange rate has skyrocketed. The $10,000 barrier has already been broken and the €10,000 barrier is in sight. If you’re interest in mining bitcoin, this post is still relevant. Fees will be lower (as they are charged in $/€) and your net yield will be higher. Good times!

The problem with mining Bitcoin

Mining bitcoin can be quite lucrative, but it’s hugely dependent on the exchange rate and the difficulty of mining. The difficulty is something that is only going to increase. In the early days a fast CPU was enough to mine new Bitcoin blocks, followed by GPU mining and eventually specialized ASIC chips.

Besides investing in powerful hardware to mine blocks, there’s also the utilities bill. Calculationg millions of SHA-256 hashes per seconds is not very power efficient.

If you’re like me, you don’t want to have a machine in your study or living room generating noise and heat to mine Bitcoin. There are several options, like renting a powerful GPU machine at Amazon AWS, but there’s a huge upfront cost. For the past few years did not find mining bitcoin interesting or cost effective enough. Last week that changed.

Welcome to 2017

Last week I found Hashflare.io. The great thing about Hashflare.io is that you choose how much you want to invest in hardware by purchasing mining capacity in units of 10 GH/s, currently at a price of $1.20 per unit.

Your mining capacity is put towards mining Bitcoin in different pools (you can change these), resulting in a daily payout of mined Bitcoin.

The good part? This is a life-time purchase. You keep the mining capacity - forever.

The only cost is a fee for power and management of your hardware, which is substracted automatically from your daily payout.

Purchasing capacity is easy and straightforward. Payment options include credit card and Bitcoin. It’s also possible to reinvest your earned bitcoin into purchasing more mining capacity.

Return on Investment

Right now I’m seeing about 40% of my Bitcoin yield go to power and management costs. This means I’ll earn back my initial investment in ± 5 months. After that it’s pure profit.

The current exchange rate is also of importance here, because power and management fees are calculated in USD. Any increase (or decrease) in exchange rate will influence the amount of BTC that goes towards fees.

Since the value of Bitcoin has been steadily rising since 2008 (there have been some extreme fluctuations), it’s very well possible that I’ll break even before 5 months.

Let’s get mining!

There’s no reason not to start mining Bitcoin today. Hashflare.io makes it rediculously easy to purchase mining capacity and start minting Bitcoin.

Will you make your money back? Given time, probably. Is it fun to be part of this Bitcoin thing? Oh yeah!

Note: Besides SHA-256 hardware, for Bitcoin, you can also purchase capacity for mining with Scrypt, ETHASH and X11.

Disclaimer: The links to Hashflare.io in the article are referral links, I will get 10% of any purchases you make on Hashflare.io. I understand if you object to using a referral link, here’s a non-referral link instead.

Squash git commits when merging

Wed, 17 Aug 2016 00:00:00 +0000

Today I’ve been fighting to get our test suite to run against a newly delivered Oracle 12 database. Of course, that didn’t work out of the box, so there was some debugging, trial-and-error, and cursing involved. Finally, I managed to get the build back up and running. Yay!

The commit history for this pull request was horrible and some would call it unprofessional, looking at the various commit messages written in anger.

Normally I’d squash all commits together using rebase in the feature branch and create a nice pull request. In this case I just wanted to merge this to develop so the build was working. Turns out this is rather easy:

git checkout develop
git merge --squash fix_for_oracle_12

This applies all changes from the fix_for_oracle_12 branch to your working copy, ready for you to commit in a single commit.

git commit -m "Apply workaround for Oracle DB 12"

Done.

Music Streaming Setup

Sun, 24 Apr 2016 00:00:00 +0000

I love music.

Recently I got into vinyl and I love it. Although it’s not digital 24/96 studio quality, there is something relaxing about picking a record, putting it on my turn table and listening to the small diamond needle scanning the grooves of the record pressed 30 or 40 years ago. This post is not about vinyl, though.

Today I listen mostly to music on Spotify. It’s cheap, offers good quality and is available both at home and at work.

The thing is I have lots of great music lying around on CD from when I was a kid. It’s all sitting in a box in the attic, wasting away.

So, I thought to myself, what does it take to:

Archive my music collection in FLAC
Catalog my music properly
Stream high quality music over the web (to my laptop)
Possibly play directly (analog/digital) to my integrated amplifier

Catalog your music

I’ve only ever used iTunes before to manage my music collection (in low quality mp3, back in the day). These days I use a command line utility named beets. Beets is simple yet effective. Let me just copy the best features from their website:

Fetch or calculate all the metadata you could possibly need: album art, lyrics, genres, tempos, ReplayGain levels, or acoustic fingerprints.
Get metadata from MusicBrainz, Discogs, or Beatport. Or guess metadata using songs’ filenames or their acoustic fingerprints.
Transcode audio to any format you like.
Check your library for duplicate tracks and albums or for albums that are missing tracks.

It takes a bit of getting used to, but it’s really easy to use and powerful. I’m not moving to anything else anytime soon.

Enter Raspberry Pi

The first and most obvious choice was to purchase a Raspberry Pi (v3 in this case). It’s a cheap device with ethernet and audio playback capabilities. I also know there are expansions boards that offer digital out or analog out using high quality DACs. I had an old 32GB USB Thumbdrive to use for storage (until I need more).

My first install was using Raspbian and Koel. Koel looks a lot like Spotify and I had high hopes for it.

FLAC

I love FLAC for two reasons. It’s open source and it stores lossless compressed audio. The problem, however, is that FLAC is not as widely supported as I’d like. Mac OS X has no native FLAC support (because Apple obviously favours their own ALAC lossless format). Browsers also have limited support (if any) for FLAC.

This means that my web player, in this case Koel, should transcode FLAC files on the fly to a format that is supported, like mp3.

For this purpose I installed ffmpeg on the Pi and it worked. Sort of. There were two problems with my setup of Koel + Transcoding.

The transcoding could keep up with streaming, were it not that Koel first transcodes the entire file and then starts streaming.

Also, Koel, seemed not to deal with album artists vs. performer particularly well. For instance, Fun, by “Coldplay feat. Tove Lo” would show up as a separate album.

mp3fs

I got tipped off by a coworker that I should look into mp3fs. mp3fs is pretty awesome. It allows you to mount a directory/disk with FLAC files and use those files as if they were mp3s. Yes. Awesome.

The only thing is that everytime you access a file it will use ffmpeg to transcode the FLAC to mp3. Not a problem, until you start importing your collection into Koel. Because the Pi has limited memory, this would crash mp3fs with an out-of-memory error.

Meh. This is not the awesome stable solution I’m looking for.

File size

Well, FLAC is a lossless, compressed format. Despite it begin compressed, it’s still fairly large. Especially if you want to keep your files on 32GB USB drive.

At this point I decided to take another route: keep my high quality FLAC collection on my computer (where it’ll get backed up automatically as well), transcode it to high quality mp3 and use rsync to get it the Pi for streaming. Luckily I’m using beets, which has a convert plugin. This plugin makes life easy to keep a copy of my library available as mp3. For now I’m using mp3 V0 (highest quality variable bitrate), which suits me fine.

Sonerezh

I feel Koel is still too early in its development and I didn’t want some bloated music player like Ampache or Subsonic. Then I stumbled up Sonerezh. It has a clean iTunes-esque interface and just works.

Conclusion

I love my current setup. Keep and manage my FLAC collection on my Mac. Use beets to get proper metadata for each song and convert new or changed files to high quality mp3. These in turn are rsync’ed to my Raspberry Pi, where they become available in Sonerezh for a great streaming experience.

Am I cancelling Spotify? Not just yet, there’s too much music out there to be discovered and Spotify is still great for that. Am I going to expand my collection? You bet!

Hanami and Multi-Database Testing with Travis

Fri, 15 Apr 2016 00:00:00 +0000

This is a re-post of my article over at Kabisa’s The Guild.

I’ve been busy rewriting Firefly for a while now using Hanami. Hanami is a fascinatingly fresh ruby web framework with a strong opinion on Clean Architecture. Me like!

Why test against different databases

Initially I developed Firefly to use Sqlite for database storage. However, Sqlite is not always the best option. Running Firefly on Heroku for instance would be impractical, since Heroku’s architecture assumes you use a real database, like Postgres.

Firefly being open source also means that different users will want to use a database that they’re already familiar with, or one that’s already running for other apps.

Supporting multiple databases

The big three relational databases I want to support are Sqlite, MySQL and PostgreSQL. Hanami uses Sequel. This means my code is already abstracted from specific database implementation. As long as Sequel supports it, so can Firefly.

The only problem I encountered was the fact that MySQL datetime fields do not store fractions of seconds, which messed with some tests. This was easily taken care of.

Travis

Travis is an awesome CI-as-a-Service provider. Open source projects can even use their service for free! <3

Whenever a new commit is made (on master or in Pull Requests), Travis will check out the code, do some setup specified in a .travis.yml file and report back the test status.

The thing is that, with multiple databases, we need to tell Travis to run multiple sub-builds for each database. We also need to tell Travis how to configure / setup Firefly to use each database properly.

Hanami and databases

Before setting up multiple databases, let’s check how Hanami configures a database connection.

# lib/firefly.rb
Hanami::Model.configure do
 # * SQL adapter
 # adapter type: :sql, uri: 'sqlite://db/firefly_development.sqlite3'
 # adapter type: :sql, uri: 'postgres://localhost/firefly_development'
 # adapter type: :sql, uri: 'mysql://localhost/firefly_development'
 #
 adapter type: :sql, uri: ENV['FIREFLY_DATABASE_URL']

So, the actual database URI is set using an environment variable. Hanami makes use of the dotenv gem, which will load a .env or .env.test file depending on which environment Hanami runs in. Somehow we’d need different .env.test files for each database configuration

For Sqlite:

# .env.test.sqlite
FIREFLY_DATABASE_URL="sqlite://db/firefly_development.sqlite3"

For MySQL:

# .env.test.mysql
FIREFLY_DATABASE_URL="mysql://root@localhost/firefly_test"

For Postgres:

# .env.test.postgresql
FIREFLY_DATABASE_URL="postgres://localhost/firefly_test"

There’s also the issue of dependencies. For instance, when using PostgreSQL, the pg gem should be included in Gemfile. If you’re running with Sqlite, you do not want that dependency there. Here I’d like to take the same approach and create multiple Gemfiles that each specify their own dependencies as needed.

For Sqlite:

# gemfiles/Gemfile.sqlite
gem 'sqlite3'

For MySQL:

# gemfiles/Gemfile.mysql
gem 'mysql'

For Postgres:

# gemfiles/Gemfile.postgresql
gem 'pg'

Tying it all together

What’s left to is tell Travis about the different database and put the right files in place at the right time.

First, I setup the environment variables for each database. This triggers Travis to run a build for each combination of variables:

# .travis.yml
env:
 - DB=sqlite
 - DB=mysql
 - DB=postgresql

Travis will run the build three times, each time with a different DB value set.

The process of the Firefly build is like this:

# .travis.yml
install: bundle install --jobs=3 --retry=3 --without production

script:
 - 'HANAMI_ENV=test bundle exec hanami db create'
 - 'HANAMI_ENV=test bundle exec hanami db migrate'
 - 'bundle exec rake test'

What I want is hook into different places and setup the right .env and Gemfile for the specified database. As it turns out Travis provides before_install and before_script hooks. By making use of the specified DB environment variable, it really is just a matter of copying the right files into place.

# .travis.yml
before_install:
 - cp gemfiles/Gemfile.$DB Gemfile

install: bundle install --jobs=3 --retry=3 --without production

before_script:
 - cp .env.test.$DB .env.test

script:
 - 'HANAMI_ENV=test bundle exec hanami db create'
 - 'HANAMI_ENV=test bundle exec hanami db migrate'
 - 'bundle exec rake test'

That’s all it takes to run your tests against multiple databases with Hanami!

Bonus: test against multiple rubies

Testing against multiple databases is cool, but it’s also very well possible that end-users will not be using the greatest and latest ruby version. Firefly currently support the latest 2.2.x and 2.3. versions of Ruby. Travis supports this out of the box:

# .travis.yml
rvm:
 - 2.2.4
 - 2.3.0

This, in combination with our database setup, will trigger six builds. Sqlite, MySQL and PostgreSQL builds on ruby-2.2.4 and on ruby-2.3.0.

I hope you liked this post. Happy coding and keep testing!

Putting ariejan.net on a diet: a 69% reduction

Wed, 06 Jan 2016 00:00:00 +0000

You may have recently read Website Obesity by Maciej Cegłowski, as featured on Hacker News. There’s a lot of great stuff in that article. The most obivous and striking thing is that there is a lot of bloat on the web. Examples include a tweet (140 characters) that is served as a 900kB website. How to optimize a website and remove bloat is a topic in and of itself. I can highly recommend the article mentioned above.

Taking a look at ariejan.net

The article by Maciej triggered me to take another look at ariejan.net and see if I can make it any better (bloat-wise). But first, let’s take a measurement of the current situation. I’m using Yslow as a Chrome plugin for this.

The frontpage, which a list of all my posts, weighs in at 256.8kB. 170kB for the actual HTML document, but there’s also almost 100kB of CSS and JavaScript.

A change of scenery

The first thing I did was delete everything in my CSS file and remove all unneeded content from the frontpage, including tons of specific dates, word counts and reading times. In general I simplified the HTML quite a bit, removing unneeded elements. Feel free to take a look, you know you want to.

On the article pages I also made a few changes, most notably removing the Disqus comments. The reason for this is twofold: comments just don’t add much as most people tweet or mail anyway and it adds a reasonable amount of extra JavaScript. I might change my mind on this, but for now comments are gone.

I was also using Google Fonts to make things look pretty. Well, guess what, I don’t care that much about pretty. Out with the webfonts. Instead, I opted to for the ‘retro look’ with a nice web-safe monospaced font.

Because of this change in style, I also set up a new favicon, which turns to be 25% of its original size as well.

The result

From 256.8kB down to 78.8kB. If you have any kind of decent connection, you probably won’t experience a big difference. Still, the bloat on ariejan.net’s frontpage has been reduced by more than 69%.

The document (HTML) size has been greatly reduced, simply by omitting data I don’t need and removing unneeded (and over 300 times repeated) markup.

JavaScript is still at 26kB because Google Analytics. I’m not happy with that and I’m looking at other options to monitor popular content and visitor numbers, suggestions are welcome.

What’s next?

Nothing much. I might look in to replacing Google Analytics. Maybe I’ll restore comments if I get complaints about that. Other than that, I’m quite happy again with the new “design” and most importantly, my site’s weight.

El Capitan: Safari New Tab Slow Fix

Fri, 04 Dec 2015 00:00:00 +0000

Safari is a great browser. I know there are alternatives, like Firefox and Chrome, but I like Safari.

Recently I noticed a lag of about 3-5 seconds when opening a new tab. I use tabs often, so this started to get in my way. Because I did not feel like switching browsers, I did some digging around to see if I could get this problem fixed.

Doing a quick Google search (after waiting a few seconds for that new tab), there was one quick suggestion:

Disable iCloud for Safari

Alright, let’s try. No change. Still lag. This was the other common tip:

Set ‘New tab opens with’ to ‘Empty Page’

Ok, done. Still lag.

I went digging further in Safari’s settings and on a hunch I disabled “Include search engine suggestions”. The magic button. This makes opening new tabs fast again.

The reason why this fixes the issue is beyond me, unless the new tab automatically starts searching for suggestions even though it’s showing an empty page.

Rails generate model: be specific

Fri, 27 Nov 2015 00:00:00 +0000

Rails can generate a lot of things for you. Personally I use generate model often to quickly setup a new model, including test files and a database migration. In its simplest form it looks like this:

rails generate product

This will get you started with a naked Product model. To make things easier, you can also supply attribute names:

rails generate product name description:text

Optionally, you can tell the generator what type of attribute you want. You can choose from the same types as you’d normally use in your migration:

integer
primary_key
decimal
float
boolean
binary
string
text
date
time
datetime

Now, that’s not where it ends. Here are some more useful tricks:

Associations

You can specify that your model references another. Our Product might belong_to a Category.

rails generate product category:references

This generates the category_id attribute automatically. In some cases you might want to use a polymorphic relation instead, no problem:

rails generate product supplier:references{polymorphic}

Limits

For string, text, integer and binary fields, you can set the limit by supplying it in curly braces:

rails generate product sku:string{12}

For decimal you must supply two values for precision and scale:

rails generate product price:decimal{10,2}

Indices / Indexes

You can also set indices (unique or not) on specific fields as well:

rails generate product name:string:index
rails generate product sku:string{12}:uniq
rails generate product supplier:references{polymorphic}:index

Password digests and tokens

If you want to store password digests using has_secure_password, you can also use the digest type.

rails generate user password:digest

And the same goes for has_secure_token.

rails generate user auth_token:token

That’s it. Happy generating.

Building Golang CLI Tools Update

Mon, 12 Oct 2015 00:00:00 +0000

In my previous post I discussed how to use a Makefile to set version and build information at compile time. Although this approach may work fine for you, it has three drawbacks I want to discuss.

1. Simplicity

Andrew responded on the golang-nuts mailing list with the following comment:

To me it seems like you took something simple and cross platform “go generate” + “go install/build” and turned it into something more complicated and less portable.

Although I’m not sure go generate is relevant in this case, I agree that on some level a Makefile is complicating things unnecessarily. Let’s remove it!

2. Non-reproducable builds

Guilio responded with:

I only have an issue with buildTime: it makes the build not reproducible.

This is a valid point. The idea that if you compile a given version of your application, the resulting binary’s hash (be it MD5 or whatever) should be equal to that of any other binary build from that specific version.

By using BuildTime the binary is never the same.

Build time is also irrelevant. It does not matter when a binary was compiled, but it does matter which precise version was build.

Let’s replace BuildTime with the current git commit hash instead.

3. Why is there a `VERSION` file?

If you’re going to store version information under version control, you might as well put it right in the code, where it belongs.

Let’s remove VERSION and instead create a nice version.go to handle everything.

Let’s refactor

Okay, first things to go are Makefile and VERSION.

Next, I created a core/version.go which contains the necessary version information. I’ve also taken the liberty of creating a proper struct for the version information, including major, minor and patch numbers, as well as a label and release name.

package core

import "fmt"

type version struct {
 Major, Minor, Patch int
 Label string
 Name string
}

var Version = version{1, 2, 3, "dev", "Chuck Norris"}

var Build string

func (v version) String() string {
 if v.Label != "" {
 return fmt.Sprintf("Roll version %d.%d.%d-%s \"%s\"\nGit commit hash: %s", v.Major, v.Minor, v.Patch, v.Label, v.Name, Build)
 } else {
 return fmt.Sprintf("Roll version %d.%d.%d \"%s\"\nGit commit hash: %s", v.Major, v.Minor, v.Patch, v.Name, Build)
 }
}

As you can see, it’s quite easy to set and update the version numbers, label and release name.

Build is still set at compile time and contains the current git commit hash.

Because the go build command is quite long, I’ve put it in a nice build.sh file that makes building easier.

go build -ldflags "-X github.com/ariejan/roll/core.Build=`git rev-parse HEAD`" -o roll main.go

This will result in a build that reports version information like this:

$ ./roll version
Roll version 1.2.3-dev "Chuck Norris"
Git commit hash: b72b076af8b18ef4f6b10296f12840f23258acec

Check that SHA

If you want, you can grab the code and run ./build.sh yourself. The resulting binary has a SHA-1 of 3ad7509279690d99e4144332dc200ede732663fd. Yay for reproducable builds!

Naming variables in ldlags

A short note on Peter Kleiweg’s comment. He pointed out that I could use

"-X main.Build=`git rev-parse HEAD`"

This would be true if the Build variable is in the main package. But because it’s not (it’s in core) I have to specify the full package name.

Thank you!

Thanks to all the awesome gophers responding to my previous post! It’s great to get feedback and get to learn more about Golang. Keep the comments coming, please!

A makefile for Golang CLI tools

Sat, 03 Oct 2015 00:00:00 +0000

Note: I’ve received feedback on this post and written an update, which you can read here

It’s no secret I love the power and simplicity of Go. To further train my skills I wrote a simple app that will roll dice from the command line, because you know, that’s very useful.

There are two goals for me in this project right now: make it trivial to use compile time variables and have a Makefile for easy compilation, installation and clean up. I’m sure I’ll think of other features to try. These will get their own posts.

Compile time variables

The classic example for compile time variables is setting a version number and build date for the binaries you compile. You could manually edit code before each compile, because you would never forget to do that.

var (
 Version = "1.0.0"
 BuildTime = "2015-10-03T11:08:49+0200"
)

Luckily there’s a nice alternative provided by Go: the link docs command allows you to set string variables at compile time with the -X option. Let’s take a look at our code and build command.

var (
 Version string
 BuildTime string
)

Compilation would look like this:

go build -ldflags "-X github.com/ariejan/roll/core.Version=1.0.0 -X github.com/ariejan/roll/core.BuildTime=2015-10-03T11:08:49+0200" main.go

Note: this is the format used with Go 1.5.1, previous versions do not use the = sign, instead separate the variable and value with a space.

The Makefile

Makefiles have always been scary to me. Lot’s of magic and weird syntax and I’ve never had the need nor the desire to dive into them. As it turns out, Makefiles can be very useful. Let’s start by building one for the roll project.

First, let’s start with the build command that passes in Version and BuildTime and refactor it so it becomes more managable and we can easily set both variables to proper values.

# This is how we want to name the binary output
BINARY=roll

# These are the values we want to pass for Version and BuildTime
VERSION=1.0.0
BUILD_TIME=`date +%FT%T%z`

# Setup the -ldflags option for go build here, interpolate the variable values
LDFLAGS=-ldflags "-X github.com/ariejan/roll/core.Version=${VERSION} -X github.com/ariejan/roll/core.BuildTime=${BUILD_TIME}"

all:
 go build ${LDFLAGS} -o ${BINARY} main.go

Now, if you run make all your binary will be compiled with the proper variables passed in. There are a few caveats here, though.

Each Make target (we only named all) will check for an output file named all and decide if it needs compiling or not. This is how Make can speed up large builds - by not compiling things that don’t need compiling. In the case of our project I want to make two changes: properly name our target after the binary we create and make sure we only re-compile if any of the Go files have changed.

First, let’s rename our build target to the name of our binary. Also, set it as the default target and make sure make all will also compile the binary for us.

.DEFAULT_GOAL: $(BINARY)

$(BINARY):
 go build ${LDFLAGS} -o ${BINARY} main.go

Next let’s get a list of all go source files we want Make to watch, for this we’ll rely on find.

SOURCEDIR=.
SOURCES := $(shell find $(SOURCEDIR) -name '*.go')

Remember how each Make target corresponds to a file on disk? Make will check if that target file exists or was changed. This means we can add the list of source go files as dependencies to the build task. If any of the source files were changed, make will re-run the task.

This is the Makefile so far:

SOURCEDIR=.
SOURCES := $(shell find $(SOURCEDIR) -name '*.go')

BINARY=roll

VERSION=1.0.0
BUILD_TIME=`date +%FT%T%z`

LDFLAGS=-ldflags "-X github.com/ariejan/roll/core.Version=${VERSION} -X github.com/ariejan/roll/core.BuildTime=${BUILD_TIME}"

.DEFAULT_GOAL: $(BINARY)

$(BINARY): $(SOURCES)
 go build ${LDFLAGS} -o ${BINARY} main.go

For fun we can add two more tasks: install and clean. Because both of these do not result in a file in our repository (like the build command), we mark these targets with .PHONY, telling Make not to expect a file to appear.

SOURCEDIR=.
SOURCES := $(shell find $(SOURCEDIR) -name '*.go')

BINARY=roll

VERSION=1.0.0
BUILD_TIME=`date +%FT%T%z`

LDFLAGS=-ldflags "-X github.com/ariejan/roll/core.Version=${VERSION} -X github.com/ariejan/roll/core.BuildTime=${BUILD_TIME}"

.DEFAULT_GOAL: $(BINARY)

$(BINARY): $(SOURCES)
 go build ${LDFLAGS} -o ${BINARY} main.go

.PHONY: install
install:
 go install ${LDFLAGS} ./...

.PHONY: clean
clean:
 if [ -f ${BINARY} ] ; then rm ${BINARY} ; fi

It’s a basic Makefile that makes compiling your Golang command line tools a whole lot easier. Enjoy, and stay tuned for more posts on Golang and Makefiles.

You can find the code for Roll at https://github.com/ariejan/roll.

Postgresq error: type 'hstore' does not exist

Wed, 05 Aug 2015 00:00:00 +0000

Today I was programming and messed up my database schema in such a way that I just wanted to restore a recent backup and start over from there.

dropdb app_development
createdb app_development

Unfortunately I ran into an error:

PostgreSQL error: type ‘hstore’ does not exist

hstore is a Postgresql extension (available since postgresql-9.x) that allows you store sets of key/value pairs within a single Postgresql value (read column). Read more on hstore in the Postgresql documentation.

Anyway, the hstore extension is not loaded for my brand new database. No problem, simply add it with this command:

psql app_development -c 'create extension hstore;'

That’s all.

Bonus points

If you use hstore for almost any database, you could add it to the template1 database.

FYI: Postgresql creates a new databases by copying another. template1 is a default database provided by Postgresql for the purpose of creating new database.

Adding hstore to template1 is just as easy as adding it to your own database.

psql template1 -c 'create extension hstore;'

Whenever your use createdb now, the new database will have the hstore extension by default.

Testing with MiniTest

Tue, 07 Apr 2015 00:00:00 +0000

Ever since I started doing TDD I’ve used RSpec. It’s a great tool, and for a long time it was part of my standard testing stack. This stack also contains things like Cucumber and FactoryGirl.

Now, this stack works great. But that doesn’t mean it’s the best, it has its issues:

Cucumber

Cucumber is, in almost every project, added complexity without any benefit. The idea behind cucumber is that it allows you to write your features / user stories in plain English and prove that those features are functional. The process of writing plain English, and using a boat load of steps (regular expressions, really) to convert that to executable code is tedious. Unless those cucumber features actually find their way to a customer, I don’t see any added value in writing them. Developers know how to write code, why not omit the entire English-to-code conversion entirely.

FactoryGirl

Factory Girl is a great way of generating object to work with. Unfortunately, more often than not, these generated object are ActiveRecord models. When creating those, you will most likely hit the database. For simple apps that isn’t as much of a problem. When an application grows, models become dependent on each other. You can’t have an order without a customer, with a valid address and now without order lines and a valid products with sufficient stock. You could mock all these, but especially with integration and feature tests, you want the whole thing.

RSpec

The first thing I fell in love with with RSpec was its clean syntax. The RSpec DSL allows you to write your specs in such away the are very readable. This is how it’s suppost to look:

describe MyClass do
 context "with a valid email" do
 let(:email) { "john@example.com" }
 subject { MyClass.new(email: email) }

 it "reports number of mails sent" do
 expect(subject.email!).to eq(1)
 end
 end
end

But when an application grows, so do the number of specs. And the complexity of the boilerplate needed to set the stage for your test. I’ve seen spec files over over 2k lines where a spec on line 1982 gets setup by let statements and before blocks spread over all the preceding 1981 lines. Yes, this can be resolved to an extend by refactoring the test, splitting it up and what not. but still, it’s very difficult to read a spec and know what it’s actually doing.

What’s the alternative?

Curiosity drove me to investigate other testing frameworks and soon I discovered Minitest. What I love about minitest is the lack of DSL (unless you use minitest/spec, of course), it also feels much faster when running tests, but I have not run any benchmarks to support that feeling.

Here’s how I’d write that same test in Minitest:

class MyClassTest < MiniTest::Unit::TestCase
 def test_reports_number_of_mails_sent
 assert_equal 1, MyClass.new(email: "john@example.com").email!
 end
end

Note that everything needed to perform this test is contained in that single test method. You could do the same with RSpec, but it’s DSL seems to prefer another convention.

Assert, like, whatever

What I truely love about Minitest are the simple assert methods. In essence, that’s all you need to know about Minitest: assert.

Did you ever see this in Rspec:

expect(my_car).to be_drivable

There’s quite a lot going on here. Out of nowhere you have a my_car instance, which was probably declared in a let somewhere in this file. You could assume it’s of the type Car, but who knows. You also don’t know what else was done to this instance in some before block. Furthermore, this line assumes that the my_car object responds to a method named driveable?. Yes, with a questionmark. There is nothing explicit about what’s going on here. If you’re a novice Ruby developer, you will probably have some difficulty figuring out what’s going on here. If you’re an experience Ruby developer, you will as well.

Consider the following Minitest alternative:

my_car = Car.new(wheels: 4)
my_car.fuel!(type: :diesel, litres: 60)
assert my_car.driveable?

This is plain Ruby. If you know Ruby, you understand what’s going on here. After all, your tests are meant to drive your internal design. Well, here it is, in all it’s glory.

The only method you need to know, really, is assert.

assert(test, msg = nil)

You supply assert with a test value. When true, the test passes. When false, it doesn’t.

There are quite a few other assert* methods to help you do common assertions. You don’t have to use them, as the can all very easily be written with a common assert. They are only there for convience. The ones I use most are:

assert_empty - assert the supplied argument is empty
assert_nil - assert the supplied argument is nil
assert_equal - assert the supplied arguments are equal (using ==)
assert_match - assert the supplied regex mathes with something else
assert_includes - assert the supplied collections includes a certain object

Then, for every assert method, there’s also a refute method that fail when true (instead of passing). It is that simple. You can read more about these in the Minitest::Assertions documentation.

Fixtures or Fixture Replacements

Fixtures are nothing more than a pre-defined set of data to run your tests against. In the case of Rails fixtures are mostly data that go into the database. They get loaded once and each test is run inside a database transaction on that dataset.

FactoryGirl and Machinist are fixture replacements in the sense that you define how your data should look and then generate what you need for each test.

Using fixtures has two major benefits as opposed to FactoryGirl:

All data is loaded only once, before the test suite runs. With FactoryGirl it’s not uncommon that, for a specific test file, you need to create 10-20 records in the database, everytime, for 20 specs. This soon adds up an is mostly what makes test suites slow.
You test against a database full of data. This may sound strange, but with time you’ll create an extensive database of data used for testing. It’s easy to add (data-wise) edge cases and see how they behave in your application.

Capybara

I already mentioned Cucumber for feature testing. Under the hood, most of the time, cucumber will be using capybara to emulate a client browser. Minitest can handle this as well and it works just as you’d expect:

class TestCarCRUD
 def test_branded_car_listing
 visit "/cars/toyota"

 within("#cars") do
 # "Toyota Corolla Verso" from fixtures
 assert page.has_content?(cars[:toyota_corolla].name)
 end
 end
end

The minitest-rails-capybara gem might be of help here to make your setup easy.

Getting started with Minitest

I don’t want to go into too much detail about using Minitest with Rails. It’s already well documented elsewhere and basically all you need is the minitest-rails gem. For feature testing you’d also need minitest-rails-capybara.

The Testing Silver Bullet

There probably isn’t a silver bullet when it comes to testing Rails applications. I love Minitest for its ease of use, lightweightedness, explicitness, and the fact that’s just Ruby. In combination with fixtures it’s possible to write a very fast test suite.

I think that’s RSpec tries to do too much magic and DSL’ing which makes it much harder to write clean tests. As a developer I see little benefit from using a specific DSL for writing tests. Having paired up with a few junior developers, new to Ruby and Rails, they tend to pickup Minitest tests much faster than RSpec and Cucumber.

Be sure to give Minitest a try and let me know your thoughts!

Why I dropped fish in favour of Zsh

Thu, 20 Nov 2014 00:00:00 +0000

Developers are religious about two things: their editor and their shell.

After getting multiple recommendations from friends and co-workers I gave fish a try. It lists several benefits over other shells, among other there’s autosuggetions, colour support and web based configuration.

Installing it on a Mac is easy with Homebrew, so I figured why not give it a fair try.

And yes, fish has its benefits. I used the web based configurator. Once. The only feature that really stood out to me was the autosuggestions and more notably the fact that fish will suggest commonly used commands based on the directory you’re currently in.

Still, the most annoying thing is that fish is not POSIX compatible. Although I’m not a purist that insists on POSIX compatibility, it imposed some practical limitations.

Environment variables

The first and most annoying thing is environment variables. I’m not talking about the globally set variables, but those you use to run a command once.

Normally I’d do something like this:

TIME_COP=2016-02-24 rake app:send_daily_notifications

Well, that doesn’t work in fish as it needs you to explicitly run the env command:

env TIME_COP=2016-02-24 rake app:send_daily_notifications

It’s a subtle difference, but if you’re used to the old syntax and have quite some documentation with snippets like this, it’s quite a nuisance.

&& vs &

Running a second command conditionally is a construct I use often:

./analyse_data.rb && rake app:send_daily_notifications

If ./analyse_data.rb fails (e..g it has a non-zero return value), then the supplied rake task will not be executed. Again, this is something I commonly use.

Fish on the other hand uses a single &, the && operation is invalid.

Incompatibility with Vim

You’ll need to give Vim some extra love and attention when you’re running fish. Namely, you’ll need to explicitly tell Vim to not use fish, but bash instead.

It’s a simple fix, but one that should not be necessary.

Back to Zsh

The issues I have with fish are minimal and due to my own personal preference. I think fish is a nice shell, but it just doesn’t fit me.

So, back to Zsh! But no more Oh-My-Zsh-crap, please! More on how I’ve setup Vim/Tmux/Zsh in a later post.

Rails: Prevent Accidental Debugging Commits

Wed, 15 Oct 2014 00:00:00 +0000

Your Rails app has grown over time, multiple developers have worked on the code and you are about to make a small change in the code.

Unfortunately this change applies to code that allows you to view a financial overview of last year’s data. It’s legally not allowed to expose the current year’s data, but the change you need to make applies only to 2014. We should shift time to 2015. Problem?

There are several things you could do.

Change your system date to 2015. This has the downside that many apps get confused, including, most likely, your backup tools.
Use a gem like TimeCop to fake time. This works great for tests, but it feels a bit like cheating in development.
Change @financial_year = 1.year.ago to @financial_year = 2014.

Side note: This feature gets tested by both unit and integration tests using TimeCop. The problem here is viewing and testing the change in my browser, prototyping the change and showing it to the customer.

Option 3 seems like the easiest. In the scope of the financial overview, let’s pretend we’re always viewing data for 2014.

The real issue here is that you forget you made this change and commit it. Although there are several process in place to detect this: peer-review, QA-testing and custom acceptance. There still is a chance this change makes it to production. Whoops!

Because this is Rails there are lots of awesome goodies around, like rake notes. It finds comments that start with OPTIMIZE, FIXME or TODO and lists them for you. It’s not unlike what many IDE’s do you as well.

Instead of this change:

- @financial_year = 1.year.ago
+ @financial_year = 2014

I could make a change like this:

- @financial_year = 1.year.ago
+ # TODO: Change this back to 1.year.ago!!!
+ @financial_year = 2014

Then, before committing, I should run rake notes or rake notes:todo and check if there’s anything that needs to be reverted back. But I’m a programmer, dammit. I can automate this! I should automate this!

Using a combination of a git pre-commit hook and a custom annotation using Rails’ SourceAnnotationExtractor, the core of rake notes, it should be possible to simply reject a commit. Rejecting a commit for pending TODO’s doesn’t seem useful. Instead, let’s use a custom annotation: NOCOMMIT. The code change is like this:

- @financial_year = 1.year.ago
+ # NOCOMMIT: Change this back to 1.year.ago!!!
+ @financial_year = 2014

And with the following in .git/hooks/pre-commit:

#!/bin/sh

nocommit_notes=$(ANNOTATION=NOCOMMIT rake notes:custom)

if [[ -n "$nocommit_notes" ]]; then
 echo "Error: You have NOCOMMIT notes in your code."
 echo
 echo "$nocommit_notes"
 exit 1
fi

Using the NOCOMMIT tag will prevent you from accidentally committing changes you do not want committed. This is an extension to other practices, like interactively staging your changes and reviewing what goes into a commit, code reviews and acceptance testing. It’s yet another safe guard agains my own stupidity.

Check out railties/lib/rails/tasks/annotations.rake for details on the rake notes task.

Synchronize goroutines in your tests

Fri, 29 Aug 2014 00:00:00 +0000

I have been working on an emulator for the MOS 6502 Microprocessor, written in Go. As part of this package I have also implemented a minimal 6551 Asynchronous Communication Interface Adapter. The 6551 provides serial IO and is easy to use in combination with the 6502.

When the microprocessor writes a byte to the 6551 it is stored in the tx (transmit) register where it’s available for other hardware components to read.

In order to make my emulator available using websockets (more on that in a later post), I had to know when a new byte became available. In a real hardware design I’d probably have to use the \CTS (Clear to Send) input pin to signal the 6551 I’m ready to read the next byte of data.

But, my goal is to emulate the Microprocessor not a complete computer hardware design.o

The logical option in Go would be to use a channel that receives new bytes when they are written by the Microprocessor.

I quickly devised the following:

type Acia6551 struct {
 // ... ommitted for brevity
 output chan []byte
}

This would allow me to create an instance of type Acia6551 and supply it with an output channel.

The calling code would start a goroutine waiting for output on this channel.

output := make(chan []byte)
acia := &Acia6551{output: output}

go func() {
 for {
 for _, b := range <-output {
 // Handle output byte
 }
 }
}()

Awesome. This worked very well in my websockets prototype.

The problem is: how do you test this? How do you test that bytes written by the microcontroller to the Acia are actually sent out to the output channel?

The solution was to create a goroutine in the test and use another channel to synchronize the writing of the byte with the output appearing on the output channel.

func TestAciaOutputChannel(t *testing.T) {
 var value []byte // Used to store the output value from the channel

 output := make(chan []byte) // The output channel
 done := make(chan bool) // Channel signaling we're done waitign

 acia := &Acia6551{output: output}

 // Create a goroutine, waiting for data on the output channel,
 // and sending a signal when data arrived.
 go func() {
 value = <-output // Block, waiting for []byte
 done <- true // We're done!
 }()

 acia.WriteByte(0x00, 0x42) // Writes a single byte of data to the 6551

 <-done // Wait for the output to be received and stored in `value`

 if value[0] != 0x42 {
 t.Errorf("Expected output not received through output channel")
 }
}

This is an example of simple synchronization using a seperate channel, in this case chan bool. It worked great in my tests to verify data was actually written to the output channel.

GPG Sign Your Git Commits

Wed, 04 Jun 2014 00:00:00 +0000

I have written and talked before about GPG and the need for trust on the internet.

Getting started with GPG and using it on a daily basis is, when you’re using the right tools, not all that hard, but still quite technical. Today Google announced they are working on a Chrome extention to enable end-to-end encryption using OpenPGP.

As a developer, I do more than dispatching emails all day. On occasion I write code. And that code gets committed to a repository that will remember that commit forever.

Just as with emails it is remarkably easy to fake your identity when committing code.

git commit -a -m "Meh" --author "Chuck Norris <chuck@example.com>"

In theory, this would allow anyone to commit (malicious) code under your name. Meaning that you’ll get the blame for the back door you committed.

Git seems to offer a resolution by adding a Signed-off-by field, to allow a second developer to sign off on code that gets merged into the project. But this field suffers from the same trust issues as the Author field.

You cannot trust the git Author and Signed-off-by fields.

Again, GPG offers a solution to the problem of trust. By establishing trust based on public keys, wouldn’t it be cool if you could sign a git commit just the same way you’d sign an email?

Well, you can.

Signing commits

git commit -a -m "Cool new feature" --gpg-sign=F713697B

This will attach your signature to the git commit message, allowing others to validate your signature. Validating this signature is quite easy as well.

$ git log --show-signature
commit 3d53a4be3f6f955007dc056347d926067bbfa8de
gpg: Signature made zo 1 jun 20:42:58 2014 CEST using RSA key ID F713697B
gpg: Good signature from "Ariejan de Vroom <ariejan@ariejan.net>"
Author: Ariejan de Vroom <ariejan@ariejan.net>
Date: Sun Jun 1 20:42:58 2014 +0200

Any other developer (or your CI) can now validate your commit as coming from you, based on the trust they assigned to your public key.

Optionally to using --gpg-sign you can use -S. If you don’t specify a specific key, git will try to figure out what key to use based on your email address. It’s also possible to set a default signing key globally.

git config --global user.signingkey F713697B

Signing tags

Besides signing every commit you make, it’s also good idea to sign tags. That way you can be sure that the created tag was actually created by a trusted person.

git tag -s 1.2.3 -m "Release 1.2.3 including bug fixes."

Signing merges

If you are responsible for integrating features and bug fixes into the main branch of a project, you’d probably like to sign the merges you make. You have two options here.

The first is to merge and manually commit a sign the merge.

git merge feature-awesome --no-commit
git commit -m "Merge feature-awesome" -S

The second is merging and signing directly.

git merge feature-awesome -S

Be sure to check out this in depth guide by Mike Gerwitz.

Where to go from here?

Trust is a hard thing to come by on the internet and it really bites you when things go wrong. Just as with email, creating a web of trust can be helpful and someday save you from disaster.

Testing $HOME with Cucumber and Aruba

Tue, 15 Apr 2014 00:00:00 +0000

Cucumber and Aruba are awesome tools to write acceptance tests for your command line application. The allow you to do things like this:

Scenario: Exit with 0 when no examples are run
 Given a file named "a_no_examples_spec.rb" with:
 """ruby
 """
 When I run `rspec a_no_examples_spec.rb`
 Then the exit status should be 0
 And the output should contain "0 examples"

This example was taken from rspec-core.

Aruba basically does three things for you:

Create and inspect files and directories
Run commands
Inspect output and exit status

The problem

Now, if you are writing a CLI that interacts with a configuration file in the user’s home directory, you’d write a cucumber like this:

Scenario: use .my-app configuration
 Given a file named "~/.my-app" with:
 """
 awesome_enabled: true
 """
 When I run `my-app`
 Then the output should contain "AWESOME"

But how does your app and Aruba differentiate between this generated test file and your actual configration file in your home directory? It doens’t.

The answer

The solution is quite easy and elegant, in your Before set the $HOME environment variable to a custom location inside the tmp/aruba directory.

In features/support/env.rb:

Before do
 set_env 'HOME', File.expand_path(File.join(current_dir, 'home'))
 FileUtils.mkdir_p ENV['HOME']
end

This will set $HOME to tmp/aruba/home in the context of your cucumbers (and execute binary). current_dir automatically points to the right location for the aruba temporary directory.

Dealing With Technical Debt

Fri, 04 Apr 2014 00:00:00 +0000

The rails project I’m talking about is over three years old and has seen commits from 27 developers in that period. These developers were both co-workers, freelancers, off-shore developers and designers of different levels of expertise.

Technical Debt Inventory

Needless to say, as most projects of this size and age, this one has plenty of technical debt. Let’s make an inventory.

The test suite takes approximately eighty minutes to run. These are all RSpec tests, including features. Luckily we can split the entire suite up into smaller parts using Travis, but still the entire thing takes about forty minutes.

The main cause for these slow specs is a lack of understanding about how to write good tests. For example, testing if search and pagination works, someone thought it fine to create fifty ActiveRecord objects in a before block.

These big offenders are easily remedied. Others are a bit harder, as some specs require an elaborate tree of models to test functionality.

This project has no cucumbers to discuss and review features with Product Owners. These cucumbers would have been awesome to discuss features both with product owners and team members. But at the start of the project cucumber was deemed too much hassle and Rspec feature specs were used instead.

The main issue now is that the line between features specs and unit specs is a thin one and people often mistake one for the other. This is probably one of the reasons so many specs create many models and make the entire suite slow.

There are parts of the code that are not well tested - or not at all. This is a dangerous one. Because, from the start, nobody monitored test coverage or did proper code reviews, there are patches of code that are not well tested, and there are some that are not tested at all.

Pressure to finish a product has led to to cutting corners in code quality and test coverage.

Lots of smells and lots of noise. All the points above indicate there is a certain amount of technical debt in this project. Another pointer are code smells. To name a few I’ve seen over the past few months:

Long methods
Conditional Complexity
Data Clumps
Alternative Classes With Different Interfaces
Indecent Exposure
Uncommunicative Names
Divergent Change
Shotgun Surgery
Lazy Classes
Inappropriate Intimacy
Train Wrecks (or Message Chains)
Feature Envy

All of these sneaked their way into the project.

Why does this matter?

The product we ship works. Although the specs take a long time to run, and there are some code smells in there, the suite is actually green and we have confidence the product works as expected.

So why complain about technical debt? In my opinion as a software craftsman, software should

work
be clean and readable
allow for change
be well tested

Well, the code works. But clean, readable, easy to change and well tested are debatable.

Responding to change is still possible, but it’s a bit more painful than it should have been. There are some untested patches of code that we would really would like to see tested. And of course, code should be readable and easy to understand. We don’t always have these things and it’s holding back progress sometimes

So, you have some Technical Debt on your hands

Recently two new developers joined this project and they immediately asked the right question:

What can we do to make this project better?

In my opinion, as a software developer, this is the one magic question. What can you do, right now, to make this project better.?

Besides the fact that you ask yourself what you yourself can do, there’s another important component in that question: better.

It’s not about fixing all the things. It’s about improving the things you touch just a tiny bit. Over the course of a few weeks or months these small improvements start to add up and really make a difference.

It’s often called the Boy Scout Mentality. Leave things behind in a better state then how you found them when you arrived.

So, although the projects works fine, we as developers are now constantly improving the code and specs we touch.

Eliminating Technical Debt

Just like 100% Test Coverage, Low Technical Debt is not and end goal, but an ongoing mission.

Your application will always have some form of technical debt. Even if you have the best programmers in the world, it will happen.

What you can do is focus on minimizing technical debt.

The team currently working on this project now has formulated a few basic guide lines to help us bring down the technical debt of this project. We are not going to do it over night, but in a few months time we will have made quite a dent.

Guidelines

The overall objective is this:

Whatever you do, leave it in a better shape than you found it in. This goes for code, documentation, specs, whatever.

When implementing features or changes, we use a few simple steps to help us along:

Is this code I’m about to change well tested? If not, fix it now. Update those specs to be more efficient, handle the edge cases you see popping up right now. Write cucumbers.
With your improved specs green, refactor the existing code Your new specs suite is fast and elegant. Use it to refactor the existing codebase to make change easier.

Congratulations, you have just made your life much easier. Note that you have not written a single line of code yet for you new feature or change. But the cleaning up you just did will greatly benefit you and the time it takes to write this feature.

Write specs for your change or feature This is quite easy now, as it should add nicely to the changes you made in the first step.
Implement your features or change Because you refactored most of the code smells out in the second step, you should be able to easily change your code.

These are four easy steps to minimize technical debt in your project.

You should not wait until you hit the technical debt limit of your project. Minimizing technical debt is just as important as writing tests. It should be a team priority from Day 1.

Minimizing technical debt is not all too different from doing proper test driven development. It requires rigorous discipline and skills. And just like with tests, you will not handle that technical debt after this crunch period.

Next time you open a project, ask your self, “How can I leave this project in a better state than I just found it?”

Pretty Difficult Privacy

Thu, 03 Apr 2014 00:00:00 +0000

Disclaimer: I could write books about online privacy, freedom of speech and how encryption fits into that picture. There are others out there who can do a better job at that and who have way more experience in crypto than I have.

So, I got a keybase.io invite. Awesome. But what problem is keybase trying to solve?

I created my first GPG key in 2000 by way of checking out shiny new things. I’ve been happily generating GPG keys ever since, but I never really used GPG. There were three reasons for this:

The tools, like GnuPG, are difficult to use.
Integration with popular mail clients sucks.
I only know two to three people with a public key.

To start with that last point, there is barely anyone I know that uses GPG. And I know quite a few technically oriented people, like developers and sys admins. No body seems to bother for GPG and the hassle it entails. Most people do have a public key available, but they simple have not setup anything to read and send encrypted or signed emails on a daily basis.

This brings us to points 1 and 2. The tooling, GnuPG, is not very user friendly. I’m not saying it’s unusable, but compared to the CLI interface that keybase provides, it’s arcane magic. Even with GPG setup and integrated in [Sup][3], I routinely make mistakes and get feedback from people I encrypted an email wrong and it’s unreadable by them.

Using this arcane magic in conjunction with normal applications, like mail clients, is quite difficult as well.

Luckily there’s GPGTools for Mac, which provides you with everything you need to get started, including a Mail.app plugin, key management and a Mac version of GnuPG.

With this software installed, as a normal user, you probably still have no clue what you’re doing.

It took me a few evenings to get up to speed again with how GPG works and, more difficult, how to manage your keys. This last part turns out to be the most difficult thing, mostly because the plenhora of options and settings to choose from. All the terminology is not helping: private keys, public keys, fingerprints, trust (of keys), trust (of users), key rings, signing, ecrypting, cleartext, photoids, shortids and key ids, key length, algorithms, validity, passphrases and expiration dates. Not to mention subkeys, signing of keys, capabilities and revokation.

That’s quite a lot of matter to understand before you can use GPG properly.

I think what keybase is doing is great. There are also a few flaws in their plan, and I think it’s not for the better of GPG and privacy. First and foremose, Keybase is becoming a central repository that stores user identities. That does not sound good, not even to mention they are running a business and a closed-source site. Besides, there is already the web of trust and public distributed key servers for GPG to use. Proving identity with a twitter account or github account is also not very powerful IMHO, as it’s easy to fake.

These things aside, keybase does not solve the fundamental issue of cryptography in general and GPG in particular: it’s difficult to use and does mostly not integrate nicely with other software.

For me, to get GPG to the general public, I think the tooling and services around GPG should:

be understandable and usable by the average user
be open source
be distributed in nature
integrate easily with third party applications

If you can solve this problem, I thing you have gold. Get in touch with me if you have ideas on this topic. You can find info on my GPG key right here.

To Blog or Not To Blog

Thu, 03 Apr 2014 00:00:00 +0000

My last post was 154 days ago. That’s about five months without writing a single post on this blog. I’ve been thinking about what this means and what happened, and here is what I discovered.

First of all I love writing. It has something peaceful to write down your experiences and thoughts. I also love that I can share these experiences with you, like 90.000 times each month, if I’m to believe analytics.

But I didn’t write. I barely looked at my site and only once a month got reminded about it when the ad revenue email came in.

Finding where it had gone wrong has not been a high priority for me either. I’m not sure why or how. Recently I came across the A No-Excuses Guide to Blogging summary, which neatly lists all kinds of excuses to not blog and what to do about them.

This got me thinking about why I hadn’t posted a blog in such a long time.

I have been up to quite a lot lately, but still I struggled to write blog posts. From the A No-Excuses Guide to Blogging I picked up several tips that got me going again.

The three most important tips I got from A No-Excuses Guide to Bloggin were these:

1. Write about what you’re learning, while you learn.

I’m learning constantly. If it’s not using Vim more proficiently, then it’s about setting up servers with some shell script magic. Finding new tools to pair program, or write better tests.

Often I found myself not worthy of writing a post about a topic, like Bitcoin, GPG or Vagrant and chef because I don’t feel at the top of my game on these subjects. I’m no expert. I don’t want to look like a fool writing non-sense about these topics.

But I’m learning. I don’t have to be an expert to talk about what I discovered or learned. I can even ask questions and leave them unanswered in my posts, and who knows someone might find an answer.

2. Keep a list of topics and create focus

The seconds problem I had was the volatility of my ideas. I often had a great idea at night when learning or programming. The next day those ideas were mostly gone and forgotten.

Now, I’m keeping track of what I’m doing more actively using Evernote. This way I can later recall what I did and what that great idea for a blog post was all about.

When I started doing this, the list with ideas grew quickly. Too quickly. The list became quite long and I had no idea where to start or how to get any writing done.

Focussing on what’s most important sounds easy. But what is most important? What is most important to me? It turns out that there are subjects that get me excited and happy. There is no clear box to put these ideas. Some ideas are very techincal, others are more security or privacy oriented while other are not technical at all (like this post).

Optimizing for my own happiness was big factor in getting back to writing for this blog again.

It may be hard to believe but sharing was never a part of my daily workflow. And I honestly don’t know why it wasn’t.

Sharing comes naturally to me. When I’ve figured something out I want to talk to people about it. And more often than not they get excited as well.

Wouldn’t it be great if I could make this sharing a part of my daily workflow? I’m still experimenting with this (as I have to write a few new posts) but I think this will be a great motivator to keep writing.

To Blog or Not To Blog

After reading A No-Excuses Guide to Blogging I got excited again about my blog and writing. I’ve taken away a few tips from this guide that, hopefully, will keep me interested and focus on writing what I care about.

Keep a list of ideas
Keep notes when learning or trying new things
Technology doesn’t matter
Focus on what makes you happy and excited
Make sharing part of your workflow

With a fresh list of ideas for my blog I’m ready to start writing some blog posts again!

If you are a blogger and having some “issues”, make sure to read A No-Excuses Guide to Blogging by Sasha Chua.

Deploying with git-deploy

Thu, 31 Oct 2013 00:00:00 +0000

I’ve blogged before about deploying Rails applications. Normally I opt for using capistrano, as it provides all the features I need and is pretty easy to customize if needed.

One of my previous strategies was to use capistrano to checkout a branch on a remote server, and git fetch that branch upon a new deployment.

The problem with capistrano, however, is that it can be quite slow from time to time. It also has a lot of features I never or only rarely use, like multi-host deployments.

A few days ago someone pointed out git-deploy to me. I tried it out and it is fantastic. Let me explain that.

Light-weight

What could be the easiest way to deploy an app?

SSH to the server
Fetch code changes
bundle install, rake db:migrate, rake assets:precompile
Restart

Yes, you could write a script to do all that, but there’s an even easier way: git hooks.

Git hooks are triggered after certain events on a git repository. In this case, the flow becomes:

Push code to server

In the git hooks, we still need to handle:

bundle install, rake db:migrate, rake assets:precompile
Restart

Now, here comes git-deploy. What it does is add a few scripts to your project. Setup is rather easy:

# Install the gem, no need for Gemfile.
gem install git-deploy

# Add a git remote to your server
git remote add production "user@server.com:/home/myapp/app"

# Setup deployments
git deploy setup -r production

# Generate deploy scripts (locally)
git deploy init

After this, you should find the following files in deploy/:

after_push
before_restart
restart

The filenames are self explanatory, but here’s a quick run through:

after_push

This file is run after you’ve pushed your code to the remote server. It gathers some information about your changes, sets up the environment and kicks of the other two scripts.

before_restart

Before you can restart your server, you will probably need to update some things. By default git-deploy will check if it needs to run migrations, perform a bundle install and precompile your assets.

restart

Does exactly what it says, it will restart your Rails app. By default it’s configured for Passenger, so it simply touches tmp/restart.txt.

Server setup, RVM and Sidekiq

What follows is a step-by-step guide of how I setup a new server. In this case for a project using ruby-2.0 with RVM and Sidekiq for background processing.

0. Set your RAILS_ENV

I always declare RAILS_ENV en /etc/environment. This file gets loaded everywhere and saves you the headache of adding this env variable with every rails command you execute later.

I also set my editor here, so I’m not surprised by nano when doing visudo or crontab -e.

RAILS_ENV=production
RACK_ENV=production
EDITOR=vi

1. Create a user for the app

adduser myapp

2. Add user to sudoers

$ sudo vi /etc/sudoers

# Add the following line:
myapp ALL=(ALL) NOPASSWD: ALL

3. Setup SSH Keys

In this case all you need to do is add your own public key(s) to /home/myapp/.ssh/authorized_keys.

4. Setup RVM for myapp

I prefer to setup RVM for the specific user (instead of system-wide). The following command will directly install the latest ruby as well.

\curl -L https://get.rvm.io | bash -s stable --ruby

5. Apache + Passenger

Then I setup Apache with Passenger. It’s fine if you want to use Nginx instead.

sudo apt-get install libcurl4-openssl-dev apache2-mpm-worker libapr1-dev libaprutil1-dev apache2-threaded-dev
gem install passenger
passenger-install-apache2-module

Just follow the instructions of the Passenger install script. Depending on your system configuration, you may be asked to add some (temporary) swap space.

Passenger will spit out three lines that you should add to the bottom of /etc/apache2/apache2.conf:

LoadModule passenger_module /home/myapp/.rvm/gems/ruby-2.0.0-p247/gems/passenger-4.0.21/buildout/apache2/mod_passenger.so
PassengerRoot /home/myapp/.rvm/gems/ruby-2.0.0-p247/gems/passenger-4.0.21
PassengerDefaultRuby /home/myapp/.rvm/wrappers/ruby-2.0.0-p247/ruby

6. Configure Apache VirtualHost

Next, create and enable an Apache virtual host for your app:

<VirtualHost *:80>
 ServerName myapp.example.com
 DocumentRoot /home/myapp/app/public

 <Directory /home/myapp/app/public>
 AllowOverride all
 Options -MultiViews
 </Directory>
</VirtualHost>

And enable it:

sudo ln -sf /etc/apache2/sites-available/myapp.example.com /etc/apache2/sites-enabled/010-myapp.example.com
service apache2 restart

7. Install your database, tools, etc.

Install postgres, redis or whatever rocks your boat. Make sure to also install the proper tools (like imagemagick) and development headers.

Also make sure that you update config/database.yml in your project and commit these changes and the files in deploy.

I’m assuming you also create a database now, as we won’t be running rake db:create later on.

8. Add foreman for Sidekiq

I want to use foreman to manage starting/stopping sidekiq. Add foreman to your Gemfile:

gem 'foreman'
# Optional, but works well with a Debian server
gem 'foreman-export-initscript'

And add a line to Procfile, of course, add any parameters you need for running sidekiq here.

worker: bundle exec sidekiq

9. Enable RVM for git-deploy

In order to use RVM and rvmsudo properly, we need to set that up in deploy/after_push. Add the following lines before the run commands at the bottom.

export rvmsudo_secure_path=1
PATH=$PATH:$HOME/.rvm/bin

10. Sidekiq deploy configuration

There are two things we need to do. Firstly, we need to generate an initscript with foreman. We do this in deploy/before_restart:

run "rvmsudo bundle exec foreman export initscript /etc/init.d -a myapp -u myapp -l /var/log/myapp"
run "rvmsudo chmod +x /etc/init.d/myapp"

Secondly, we need to use this initscript to actually restart Sidekiq in deploy/restart:

echo "restarting foreman"
sudo /etc/init.d/myapp restart

11. Ready? Deploy it!

Now, deploying is rather easy:

git push production master

That’s it.

Future deployments

Well, from now on you can simply push your code with git and git-deploy will take care of it all:

git push production master

A few things to keep in mind:

You don’t need the git-deploy gem installed to deploy your project.
Keep your deploy/ scripts up to date and under version control.
Anyone with SSH access to your server can deploy the project.

Conclusion

I think that capistrano is a fine tool, but for many projects it’s overkill. git-deploy is light weight and easy to use and gets the job of deploying your app done quickly. It’s also easily extendable, if you need it.

Give it a try. You can easily spin up a VPS over at Digital Ocean to play around or host your next awesome project. (Disclaimer: referral link). If you use Digital Ocean, you may also be interested in my iOS App: Binary Deep.

Divide and Conquer

Thu, 23 May 2013 00:00:00 +0000

A few days ago I wrote about getting up early and getting stuff done (link).

After giving this a some more thought I figured out that there’s something else going on as well.

Getting up early and working during the quiet hours in the morning is not the only thing helping me be productive.

The other factor is having limited time.

The fact is that I have to make breakfast, shower and get out of the door for work on time. This means that my time has a hard limit put on it and I have to plan what I do accordingly.

I’ve noticed that each morning I try to pick a single task to focus on. This can be researching some new tech, or fixing a bug or implementing a new feature. Because my time is so limited, it’s much easier to estimate what task would fit into that time.

So, why does this work so well for me?

It forces me to break down features and bugs into smaller pieces that can be completed in about ninety minutes.

During a normal work day I don’t have these little units of time with hard limits and it’s much easier to just keep on going.

I’m going to undertake an experiment at work to slice my time into small units with hard time limits on them and see if it helps me to be more productive during a sprint.

Early Birds

Tue, 21 May 2013 00:00:00 +0000

Over the years I’ve often found myself behind my computer at two in the morning trying to finish a feature for a side-project and trying to keep my eyes open and focussed.

After a hack session until two or three in the morning, there would be a price to pay in the morning. Not even three alarms would wake me and I’d be grumpy all day long. The feature I wrote the evening before didn’t even look that good, so I’d be re-written or removed altogether. Another night wasted.

This rhythm was not rewarding and pretty hard to keep up. It tired me out and got me demotivated. It even caused me to give up on some projects because of it.

Late april I was invited to an early morning bird walk. The idea was to hit the woods at 5:30 and listen to all the birds waking up and singing their songs. To my surprise it was rather easy getting up at five, and when I got home at eight I was full of energy and still had an entire day ahead of me.

I went to bed early that evening – no late night hacking – and to my surprise I woke naturally at 5:30 the next morning. I got up, showered, got myself a cup of coffee and fired up my computer to hack on a feature I’ve been thinking about.

At 6:45 the feature was done and I was still thirty minutes ahead of my normal morning routine.

That day I had more energy. Knowing that I’d already completed that one feature put me at easy somehow. In the evening I didn’t do much – because I didn’t have to. I spend some time doing fun stuff and went to bed early.

I’ve been doing this for two weeks now and it’s been awesome. In the morning I’m more focussed and I honestly write better code.

It’s been a long while since I finished a side project, but the way things are currently progressing I might even finish this one and start another.

I'm here for an argument

Tue, 26 Mar 2013 00:00:00 +0000

When people make a request or proposal I often see them present their request and arguments using the following structure:

Make a request
Give a plethora of arguments why that request is a good idea.

An example:

I want to go to the zoo!
Because I want to see animal
Because I want to spend time with you
Because it has been a long time

Here I think, meh, the zoo. And then I read some arguments that may or may not change my opinion on actually going to the zoo or not.

You start with the what and then continue on to the why, but I’ve already lost interest.

Consider this request:

I want to spend time with you and see animals.
It has been a long time since we went to the zoo
Let’s go to the zoo!

Here you start with me agreeing on spending time together and it becomes easier for me to accept your request to go to the zoo. I’m not blinded by your request.

You start with the why and then continue to the what.

In addition to this article, please watch Simon Sinek’s TED Talk [18m05] How great leaders inspire action.

Automated nanoc deployments

Mon, 25 Mar 2013 00:00:00 +0000

I’ve migrated ariejan.net from a custom Ruby on Rails application to a statically generated site with Nanoc.

Publishing my site now goes like this:

Write stuff in Markdown
Commit and push
Watch how jenkins builds and publishes the site with nanoc and rsync.

Using nanoc

Nanoc is a very powerful static site generator. You can use ruby to create helpers, filters and what not. Fancy using Sass, Compass and Haml? No problem!

Simply write pages in Haml or Markdown and Nanoc will generate a complete site, including sitemap, robots.txt and RSS feed.

Once you’ve set everything up to your liking, simply run nanoc compile and your site is ready.

Nanoc also has some nice features to deploy your website through rsync, which will upload (or remove) files on a remote server as necessary.

Automate the crap out of it!

I’m a programmer. I like to automate repetitive tasks. So I did.

I’ve setup Jenkins with the RVM plugin and that’s all I need to have Jenkins generate and deploy my Nanoc site.

Here’s the build script I currently use:

#!/bin/bash
export NANOC_ENV=production
bundle install
nanoc compile
nanoc deploy -t public

If the compilation fails for some reason, the deployment is cancelled.

Enter gitlab

Now I use Gitlab to host tons of private repositories, including the one for ariejan.net.

I’ve setup a web hook that will trigger the Jenkins job when new commits are made or pushed.

This allows me to quickly edit a file, commit it and Gitlab en Jenkins will make sure the change is compiled and published to ariejan.net.

Where to go from here

For now, this setup suffices for me. I keep all my posts in version control, and backupped to S3 (by backing up my gitlab repositories). Adding a new story to git is sufficient for it to be published automatically on ariejan.net and in the event something goes wrong with compilation, Jenkins will notify me of the build failure.

Review commits in your feature branch

Fri, 08 Mar 2013 00:00:00 +0000

Github pull requests are awesome, but you can’t use them all the time, mostly when working on code not hosted at github.

The following snippet makes it easy to see the commits in your current (head) branch that are not yet in the base branch.

To see what commits are made in your current feature branch, but which have not been merged into develop yet:

$ gpr develop
* 5246248 <ariejan@ariejan.net> (HEAD, origin/feature-branch, feature-branch) Implements the awesome feature (50 minutes ago)
* 4f55b7c <ariejan@ariejan.net> Write specs for awesome feature (2 hours ago)

Snippets

To achieve this, add the following alias to ~/.gitconfig:

[alias]
 lg = log --graph --pretty=format:'%Cred%h%Creset %Cblue<%ae>%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative

And the following alias to your ~/.bashrc or ~/.zshrc:

alias gpr="git --no-pager lg HEAD --not $1"

Bonus tip

After reviewing the commits in your feature branch with gpr, use hub to attach your code to a github issue:

hub pull-request -i 42 -h ariejan:feature-branch -b you:develop

Instagram, Governments and Cypherpunks

Tue, 18 Dec 2012 00:00:00 +0000

The whole internet collectively fell over Instagram earlier this week when they released their new Terms of Service.

Some or all of the Service may be supported by advertising revenue. To help us deliver interesting paid or sponsored content or promotions, you agree that a business or other entity may pay us to display your username, likeness, photos (along with any associated metadata), and/or actions you take, in connection with paid or sponsored content or promotions, without any compensation to you.

Today Instagram officially responded to all the screaming and shouting, trying to control the damage.

From the start, Instagram was created to become a business.

Well, that was unexpected. It’s the same with Facebook, Twitter, Gmail, Hotmail, etc. All these services are here for you to use for free - awesome!

But nobody ever got rich from giving stuff away for free, and these are businesses we’re talking about.

Let’s see how this works over at Facebook.

Exhibit A: Facebook

Facebook is a good example of how you are the product being sold. We all know Facebook tries to show you ads that you are likely interested in. Based on what you put in your profile advertisers can target specific audiences.

That might sound all well and good to you, but did you also know that your friends might see a sponsored post in their newsfeed? And that that sponsored post appears as if it’s one posted by you?

Here you see a post by a friend of mine who apparently likes some mobile company’s ad. They didn’t like this add. I asked them.

So basically Facebook and an advertiser are using you, your name and your profile picture to promote an advertisement to one of your friends.

Exhibit B: Google

Another example, more in line with Facebook, is Google. All those free tools: Search, Gmail, Google+, Analytics, Web Master Tools, maps. They are all geared towards either exposing ads to you, or gathering data on you to expose more focussed ads.

The plus side here, I would say, is that Google actually delivers usable services, like search, mail, calendars and such. But still, look at the boatloads of money Google is making selling ads based on your private details and online behaviour.

Exhibit C: Twitter

Twitter is a company that has struggled for a long time to find a way to monetize itself. Hopes were high that they would find a new an innovative way, but they have not.

It remains to be seen what’s going to happen exactly with twitter, but we can all assume that our tweets are being analyzed and that Twitter will do anything to sell you to their advertisers.

Exhibit D: Instagram

To come back to Instagram, they’re in the same boat as Twitter and Facebook.

They have been growing their user base for quite some time now and now comes the time to move in and cash on all those users.

The new terms of service should not come as a surprise.

Instagram wants to use your data to match you up with advertisers and it even wants to show ads endorsed by you, just like Facebook already does.

So what’s happening out there?

Social Media are nice, they connect people and can start revolutions. But that is not why those media exist. They are marketing and data mining platforms on a massive scale.

Have you ever wondered why social media companies get multi million dollar investments? Because with a huge, world-wide user base, there is a huge advertising potential.

If you are not buying a product or service, but are receiving one for free, then you are the product or service begin sold.

I think that companies like Facebook and Instagram are wading in a gray area of privacy. Their practices may be perfectly legal, but they are ethically questionable.

On government spying and eavesdropping

You might sometimes worry about governments eavesdropping and keeping tabs on innocent citizens – and you should.

If you look at the sheer amount of data social media gather on you, imagine what kind of data a government agency with the proper tools and hardware can do.

But the issue is not really that companies and governments are gathering data - you are feeding it to them with every status update and photo you upload to the internet. With every email you sent.

So, what’s to be done?

Cypherpunks to the rescue?

Now, I won’t claim to know a whole lot about cypherpunks and the crypto wars, but these people have fought (and still are) fighting for our digital rights and freedom.

We are collectively giving away our personal data to companies and governments and most of the time we don’t even know it. But then some image sharing company changes its terms that will allow them to display adds alongside your name and we freak out.

Here’s quite from the Cypherpunks manifesto:

Privacy is necessary for an open society in the electronic age. … We cannot expect governments, corporations, or other large, faceless organizations to grant us privacy … We must defend our own privacy if we expect to have any. … Cypherpunks write code. We know that someone has to write software to defend privacy, and … we’re going to write it.

To be clear, I’m not a cypherpunk by any means. Here’s a snippet from Wired magazine, 1993:

The people in this room [cypherpunks] hope for a world where an individual’s informational footprints – everything from an opinion on abortion to the medical record of an actual abortion – can be traced only if the individual involved chooses to reveal them;

And this is where the problem lies. Yesterday Will Wheaton, of Star Trek fame, wrote about this as well.

Will has a great point that we’ve come to a point where it’s okay that somebody else shares your personal details with a third party on the internet – without your knowledge, without your consent. And there is nothing you can do about it.

I’m not sure about you, but this sure frightens me.

Binary debugging with git bisect

Thu, 29 Nov 2012 00:00:00 +0000

Part of resolving a bug is finding where and when that bug was introduced into your code. Not so much for blaming a specific person, but more for an understanding of how and maybe why the bug was introduced; and more over which versions of your app are affected.

Most of the time the bug was recently introduced and your CI notified you that stuff has been broken.

In order to find out when, how and by whom the build was broken, you’ll have to dig into your git history and run your specs to see if they pass or not.

Running your specs for every commit in your history manually is very time consuming and boring. Luckily there are better ways, using plain old git.

Binary search

Before I dive into git, it’s important you understand how binary search works. If you already know this stuff, skip right to the next section.

You have a sorted array. This means there is some order to the elements you have. Presume you have an array of ints:

a = [1, 3, 4, 7, 33, 42, 54, 76, 89, 91]

Now, we want to find the position (n) of 7 in this array using binary search so that a[n] == 7.

Binary search uses a divide and conquer strategy. You split the array in the middle. We have 10 elements, so a logical place would be to split the array at position n = 5, which has the value 42.

Comparing 7 <=> 42 tells us that, because we have an ordered array, the value 7 should be in the first half of the array.

We can ignore the right half of the array for searching, and repeat this step for the left part, specifically:

[1, 3, 4, 7, 33]

So, let’s split this part up again. We get 4. This is less than 7, so if we continue looking we should take the right part.

[7, 33]

Okay, again, we continue our search. We know have to split the array one way or the other, and we end up picking n = 4. We hit 33. Surely the value of 7 must be on the left part of this.

Note that although I don’t show the whole array, I’m still using the index positions for the entire a array.

[7]

Now, there’s not much to pick for us. This is 7. Right here at n = 3. Done!

Notice that the last step does not involve checking the value of the last element. We only have 1 element left, so we’re finished.

If we had been looking for a value of 18, we would have also found n = 3. This means that we can search for non-existing values, which then return the index right before where that number should be inserted. This works because the array was ordered, so we can safely make such assumptions. Nice, huh?

How does binary search relate to finding bugs?

Well, in the example above we were looking for an integer value. The test we use to evaluate is simple:

value <=> a[n]

When we want to find the commit that broke our build we need a more clever way of comparing values.

This is where your test suite comes in. You have a failing spec now, so basically we’re looking for the point in git history where that spec failed for the first time.

Git bisect

Because it’s not feasible to do a linear search over your entire commit history, we’ll have to start by marking a good and a bad commit.

First we’ll have to find a location in your git history where you know the app did not exhibit this bug.

More than likely, this place is your last stable release. If you used git tag to tag your release, you should be able to find out quickly if that release contains the bug.

git checkout v1.2.1
rspec spec/features/fancy_spec.rb
=> 0 failures

Good. We now our latest commit was broken, so let’s get started with that binary search!

You don’t have to keep track of the git history and binary search position all by yourself: git does this for you. All you have to do is compare each commit that is presented to you with an expected result. E.g. does the fancy_spec.rb spec pass or not?

Bisecting steps

First, let git know you want to do a binary search.

git bisect start

Next, let git know which commit is good, and which one is bad.

git bisect good v1.2.1
git bisect bad e8ab31

Git will respond with something like this:

Bisecting: 19 revisions left to test after this (roughly 4 steps)
[0e70ee6aefe428fa897ec7e48273c3fe4d0bf7fb] Did some funky stuff in `config/application.rb`.

Git has determined that commit 0e70ee is right in between v1.2.1 and ebab31. Check if this revision is broken and report back to git.

rspec spec/features/fancy_spec.rb
> 1 failure
git bisect bad

And git will respond:

Bisecting: 9 revisions left to test after this (roughly 3 steps)

You can continue this until git tells you the commit that first broke this spec.

When you’re done bisecting (or if you just feel like doing something else), just tell git:

git bisect reset

Usage in the field

Git bisect is a very powerful tool to find specific points in your code.

In this example we were looking for a commit that broke a specific test. You can look for all kinds of things:

When did the layout on that page break?
At what point did we upgrade that gem to a buggy release?
Who added this ugly piece of code?!

Doing a binary search over your git history is a fast and efficient way of finding these kind of things out.

CustoMac

Thu, 22 Nov 2012 00:00:00 +0000

Ever since Apple decided to put Intel processors in their Macs there have been attempts by enthusiasts to run Mac OS X on commodity hardware – with mixed results.

The key to installing Mac OS X on a non-Mac computer is using the right hardware. If your hardware is a close to Apple kit as possible, you have the best chance to succeed.

The so-called Hackintosh community has come a long way the past few years in making it easy for “normal people” to install Mac OS X on their PC.

Since I was tired of using Debian Linux on my Desktop, dual booting to Windows to play the occasional game of World of Warcraft, I decided to give installing Mac OS X a try. But there was a problem.

The hardware

My current PC is powered by an AMD processor (AMD Phenom II X6 1055T, to be precise). Simply put, installing Mac OS X on an AMD cpu is not going to work.

So, the first thing I did was go over to the OSx86 Project Hardware Compatibility List and see what hardware is most compatible with Mac OS X 1.8.2 (the most recent version of OS X at this time).

I already knew I’d need a shiny new Intel CPU, and thus also a new motherboard.

Choosing a CPU

Choosing a CPU for your CustoMac is not very difficult, because you are limited to Intel. Since the release of the new MacBooks Apple officially supports the Ivy Bridge architecture, which means about a 50% lower power consumption and a 5-15% speed increase (link).

The main candidates were:

Intel i5 3570
Intel i5 3570K
Intel i7 3770K

I won’t go into to too much detail here, but I chose the Intel i5 3570.

The i7 offers Hyper Threading at a price bump of about € 100,-. For me, this was not worth the money.

Then there’s the choice between the 3570 or the 3570K. The ‘K’ version has less features, but is unlocked, allow it to be easily over-clocked to higher speeds.

The price difference between the 3570 and the 3570 are minimal, but I’m not planning to over-clock my CPU, so I went for the slightly cheaper Intel i5 3570 processor.

Choosing a motherboard

Next came a more difficult decision. The motherboard. Again, the Hardware Compatibility List was a great help here. In the end I chose the Gigabyte Z77-DS3H.

The pro’s of this board are that it’s well supported by Mac OS X and the OSx86 community. This board is special because it features Gigabyte’s 3D UEFI Bios. This bios would make it easy to install Mac OS X untouched on your machine.

I didn’t end up using this UEFI feature, but nonetheless, support for this board is incredible.

The full hardware list

So, recycling other parts of my current PC I build the following CustoMac configuration:

Gigabyte Z77-DS3H Motherboard (But at Amazon)
Intel i5 3570 CPU @3.4Ghz (Buy at Amazon)
16GB RAM at 1333Mhz
1x 120 GB OCZ Agility 3 SSD
1x 1TB Western Digital HD
1x 2TB Western Digital HD
XFX ATI Radeon HD 6870 1GB (Buy at Amazon)

Note 1: The above Amazon links are affiliate links.

Note 2: I could have upgraded my memory to 1600Mhz units, which would be faster. But I have no use for the old memory, so I chose to re-use it for now.

The preparation

Before you get started, you should prepare an installation USB drive on another Mac. You’ll need at least 8GB of space on the drive.

Buy Mountain Lion from the App Store on you Mac and download the installer. If you already purchased Mountain Lion, re-download it.
Download UniBeast for Mountain Lion and follow step 1 and 2 from this guide.

At this point you have a bootable USB drive with the Mountain Lion install on it.

To prepare the actual installation, remove any devices you don’t need, like extra hard drivers, DVD/BluRay drives, etc. In my case I also pulled out the ATI 6870 and used the onboard Intel HD Graphics during installation.

Plug your USB drive into a USB 2.0 (black, not blue) port on the motherboard. Make sure to use one of the ports on the back of your computer, those are directly attached to the motherboard and have the greatest chance of succes.

Now, boot up the computer and enter the BIOS. There are two important changes you need to make.

Set SATA to AHCI mode.
Disable VT-d.

Then select the USB drive as the bootable device and boot.

Booting the installer

You’ll see the UniBeast boot screen which show a ‘USB’ option (and possibly other, depending on what’s on your disks). Choose ‘USB’ - but don’t press ENTER just yet. Instead, type -x, which will show up on the screen. Then, press ENTER.

After a few minutes you should have the Mac OS X Installer in front of you. Go ahead, install this baby.

Notes on Fusion drive

It’s possible to create a CustoMac Fusion Drive using an SSD and regular harddisk.

When you’re in the installer, choose ‘Terminal’ to open a terminal window and follow the steps in this fusion drive guide.

I was able to create and install Mac OS X on a Fusion Drive without problems. The only knack was that the custom bootloader you need is not Fusion Drive aware, which makes it difficult to use.

In the end I decided to not use a Fusion drive setup, and just install everything on the SSD.

Completing the installation

Now comes the tricky part. The installation is done and you’re CustoMac wants to reboot. Let it, but leave the USB drive connected.

Your machine will boot up with the same boot menu as before, but instead of ‘USB’ you should now be able to select ‘Macintosh HD’.

Select that, enter -x followed by ENTER.

You’ll now be taken through the final steps of installation, like setting up iCloud and creating a user account.

When finished, you should be on your new Mac desktop.

Installing custom kexts

Now, your CustoMac can only boot with the USB drive. Let’s change that by installing a bootloader and some kernel extensions.

Download MultiBeast for Mountain Lion.
Run the installer and select the following options: ** UserDSDT or DSDT-Free Installation ** Miscellaneous => FakeSMC ** Audio => Realtek ALC8xx => Without DSDT => Latest version for ALC8887/888b. ** Network => maolj’s AtherosL1cEthernet ** Disk => TRIM fix for 1.8.1+ ** Bootloaders => Chimera

That’s all. Install that stuff. Now, you should be able to reboot your CustoMac and boot it without the USB drive.

If things don’t work out (like a black or white screen, kernel panics, whatever), just plugin your USB drive again, boot from it and select your ‘Macintosh HD’.

Graphics

At this point, you should have a working CustoMac with sound and network working. The only thing missing is a proper graphics card.

You’ll need to make some tweaks to the chameleon plist file. Then shutdown your computer and install the graphics card.

Note: this works for my XFX ATI Radeon HD 6870 card. There may be subtle differences for different version and brands. Just use the Google to find hints, boot with the USB drive to get to your system and make updates as needed.

In /Extra/org.chameleon.Boot.plist make sure you have the following entries:

<key>AtiConfig</key>
<string>Duckweed</string>
<key>AtiPorts</key>
<string>5</string>
<key>Graphics Mode</key>
<string>1920x1080x32</string>
<key>GraphicsEnabler</key>
<string>Yes</string>
<key>PciRoot</key>
<string>1</string>

Depending on which slot you used for your graphics card, you may have to set PciRoot to 0.

I also had to add npci=0x2000 to Kernel Flags, but you may or may not need it.

<key>Kernel Flags</key>

Boot up!

Now, reboot one last time and everything should go smoothly.

If your system comes up without any troubles, start attaching those others disks and drives you had disconnected during the installation. There shouldn’t be any issues here.

You’re done!

Congratulations. You now have a CustoMac!

Keep in mind that you should not blindly install any update you see. Installing an update my change the bootloader or change kernel extensions that break your system.

A good tip is to create a full disk image of your SSD using a tool like Super Duper. In case of shit hitting the fan after an update, you can easily restore your disk to working order.

In my case, I’ve attached an old 500GB drive to store this disk image. It works great.

Shiny and fast!

Just as a side-note, my CustoMac is blazingly fast. It’s the combination of the fast Ivy Bridge architecture, the i5 processor, and the SSD.

I’ve measured boot-up time from pressing the power button to the Mac OS X login screen at about 11 seconds.

A Call to all CI Service Providers

Mon, 05 Nov 2012 00:00:00 +0000

As a professional developer I test my code. Every check-in I do is tested either on Kabisa’s Jenkins server or on Travis CI Pro.

For open source projects there’s Travis CI. It’s free and a great way to get to know Travis.

Now, as an individual I have some side projects, most notably my own site, Ariejan.net. I value well written and well tested code as much for my own stuff as for my professional work. However, I don’t have a Jenkins CI server in my basement.

Looking around the internets there are a few CI services that can test my Ariejan.net Rails application, among them Cirlce CI, Atlassian Bamboo and Semaphore.

Now, the thing here is that to test the few projects I have (about four) it will cost me more to have the hosted in a CI services than actually running these apps.

Travis - $129/mo for 2 concurrent builds.
Circle CI - $49/mo for 10 private projects.
Semaphore - $39/mo for 5 private projects.
Bamboo - $20/mo for 10 jobs.¹

¹ Bamboo advertises $10/mo, but a $10/mo Jira subscription is also required.

As an individual, these prices are pretty steep for the few commits I make during the weekend.

Looking at Github and Heroku

Let’s take a look at github and heroku here. At heroku I pay nothing to host a site. Performance is limited, but it more often than not is sufficient for a side-project. Adding a worker or extra dyno costs about $38/mo. Still less than half of the CI services I mentioned.

Having private repos with Github is also quite cheap. You’ll get 5 for $7/mo. Which, converted, is about €1 per repo a month.

Now, as an individual developer, I’m more than happy to use these services for these relatively low fees.

Where are the cheap, individual CI services?

As an individual developer, this is what I need:

Unlimited, or at least 5-10 jobs/private projects.
Low monthly cost: $5 - $10.

What I don’t need is:

Concurrency and paralellisation
Immediate execution
24/7 Support

I can do with a single worker doing one build at a time. I don’t write enterprise CRM software in my weekends, you know.

Also, I don’t expect my build to execute immediately. I don’t mind if there’s a job queue and I have to wait a bit longer to get back results.

Business opportunity?

IMHO there’s a business opportunity here. Github and Heroku are great examples of how giving something away for free (or cheap) to individual developers creates a market for their product.

I understand that CI takes a considerable amount of system resources and time, but having a good experience as an individual is more than likely to attrack people to your platform when they need a more profession plan.

Who’s stepping up to the plate?

Decorating Sorcery's current_user with Draper

Fri, 02 Nov 2012 00:00:00 +0000

I already wrote about how to apply your decorator to the current_user when you’re using Devise. However, the trick is a bit different when applied to Sorcery.

Instead of being nil when no user is signed in, Sorcery uses an explicit false value, no nil. In your ApplicationController at app/controllers/application_controller.rb add this:

def current_user
 UserDecorator.decorate(super) unless super == false
end

I’m using the most recent version of Draper by the way.

A static file server in Go

Thu, 04 Oct 2012 00:00:00 +0000

If you don’t know Go, you should really look into it. Today I was trying to figure out how to write a simple (and fast) static file server in Go.

As it turns out, this is very easy to do. Go contains (in the net/http package) a nice FileServer type that can server files from the directory you point it to.

Here’s a sweet and short example:

package main
import (
"net/http"
"log"
)
func main() {
err := http.ListenAndServe(":4242", http.FileServer(http.Dir("public")))
if err != nil {
log.Printf("Error running web server for static assets: %v", err)
}
}

By itself this is not very useful, but you can easily integrate this into any other http server you create, maybe for handling dynamic requests or doing web sockets.

Migrate git repositories

Mon, 01 Oct 2012 00:00:00 +0000

Sometimes you have to move your git repository to another host. In this case I want to move a privately hosted git repository to a brand spanking new github repository.

These are four easy steps to get that done:

git clone --bare git@yourserver.com:project.git
cd project.git
git push --mirror git@github.com:ariejan/project.git
cd .. && rm -rf project.git

That’s it. Don’t forget to update the remote of your working copy accordingly:

git remote set-url origin git@github.com:ariejan/project.git

Of course, this works with any git server or service, not just Github, although Github is awesome and you should use it.

Git: Interactive Revert

Tue, 04 Sep 2012 00:00:00 +0000

I recently made a commit in a project that, mistakenly, included changes to db/schema.rb. My local schema was out of date and this could cause trouble for the others in my team.

Luckily we use a successful git branching model so my changes were still up for review by the team.

The change I made was part of larger commit. But all I wanted was to revert serveral chunks from db/schema.rb that would cause trouble.

Interactive add

Now, there’s git add -i which allows you to selectively stage chunks for the next commit. Unfortunately, the process for reversing interactively is a bit more complicated, but not much.

Interactive revert

Well, let’s say you want to revert changes you made in commit f1e11c. Go to your feature branch and revert your changes for that commit but do not commit them yet, hence the -n option:

git revert -n f1e11c

Your working copy now contains staged changes to revert the entire f1e11c commit. You don’t want to revert everything, so unstage all those changes.

git reset

The key now is to interactively stage the changes you want to revert. You can do this like this:

git add -p

git add -i

Choose whichever suits you.

In my case I staged only the chunks that related to the changes in db/schema.rb that I wanted to revert.

With your reverting changes staged, commit them.

git commit

You’re now left with reverting changes you didn’t want to make. Just throw them away.

git reset --hard

And that’s all. You have now selectively reverted a commit.

Rails migrations: decimal precision and scale

Tue, 28 Aug 2012 00:00:00 +0000

I’m always confused when using decimal in a Rails migration. Normally I need to store a value that has 2 or 3 numbers behind the comma (or dot), or decimals.

Let’s say you have a Product model with a discount_percentage attribute. This attribute is currently an integer, only allowing non-decimal values. To allow 2 digit decimal values (e.g. 12.54), you can mak the following migration:

change_column :products, :discount_percentage, :decimal, precision: 5, scale: 2

This will allow you to store values like 80.00, 99.99 and 100.00. There are five digits in the entire number and of those five, two decimals.

It’s not necessary so specify precision and scale as your database will then allow any decimal value it can store.

Read more about the decimal data type in PostgreSQL here: http://www.postgresql.org/docs/9.1/static/datatype-numeric.html

Getting started with Arduino

Mon, 27 Aug 2012 00:00:00 +0000

I’m a software engineer. I don’t do hardware.

Well, that used to be the case. I did build some PC’s back in the day, but that’s where my involvement with electronics hardware ended. That’s all changed after I purchased an Arduino.

Arduino

Arduino is an easy to use hardare and software platform for designers, developers and tinkerers. What?

Well, Arduino is more than just a small piece of electronics hardware. It does include hardware. This is the Arduino Uno, the latest incarnation and the board you’ll most likely purchase if you want to break into the world of Arduino

This Arduino has some nice features you should know about. At its core it has the Atmel Atmega328 chip. It has 32Kb of flashmemory and usually runs at 16MHz. Now this may not sound like much, but note that this is not like your average Intel Core i7 processor. This chip does several things for you, but the most important part is that you can program it to interact with the outside word through its 14 I/O pins. It gets its power over USB or from a battery that can supply 7-12V.

As I said earlier, Arduino is more than just the hardware board. It also contains software. Software on the Atmega328 chip, called the bootloader, allows you to easily interact with the Arduino from your computer. Then there is the Arduino IDE, which you can install on Mac, Linux and Windows. This is the tool you write your code with and which allows you to upload that code to the Arduino.

Now the great part is that both the hardware designs and the software are open-source. This means that you’re free to buy all the separate components for an Arduino board and solder one yourself . You may even sell it if you like. The only catch is that you can’t use the name Arduino.

An example

The most basic thing you can do with Arduino is hook up a LED and resistor and make the LED blink.

This is what that would look like if you’d build it:

This is the electrical schematic for this setup. It’s very straight forward, as you can see, but enough to get you started. Even if you don’t yet know how LED’s work or what a resistor does, you can make this easily enough.

Note: if you do want to build this, I recommend you check out the Blink tutorial and buy an Arduino Starter Kit, which includes all the components to get started.

For those a bit tech-savvy, here’s the electrical schematic for the Blink tutorial. I’m not going to explain it in to detail, but note that the led and resistor are connected to D13 or pin 13 on the Arduino.

With the following, relatively simple code, you can make that LED blink.

int led = 13; // This refers to pin 13.

// setup() is run once during start-up
void setup() {
 // initialize the digital pin as an output.
 pinMode(led, OUTPUT);
}

// the loop routine runs over and over again forever:
void loop() {
 digitalWrite(led, HIGH); // turn the LED on (HIGH is the voltage level)
 delay(1000); // wait for a second
 digitalWrite(led, LOW); // turn the LED off by making the voltage LOW
 delay(1000); // wait for a second
}

After making this first step you’ll get to know more components and features of Arduino. You’ll start using sensors (buttons, light sensors, sound sensors, GPS, magnetometers, gas sensors) and you’ll start using actuators (motors, steppers, relais, LED matrixes). The possibilities are endless.

The next step

The great thing about Arduino is that it’s designed to be easy to use. But is also offers great possibilities.

For one, the Arduino Uno has only got 14 digital and 6 analog I/O pins. When you start using more other components you’ll run out of I/O pins quickly. Enter the Arduino Mega which sports 54 digital and 16 analog I/O pins. Need something smaller: Arduino Nano, Need something to integrate into clothing: Arduino Lilypad. Need something custom: it’s frickin’ open-source, so design your own PCB, add the Arduino components you need and you’re good to go.

The things that are being doing with Arduino are amazing. The most important thing is that it makes electronics tinkering available to more people. It’s pretty cheap to get started and learn about electronics. Then there are hundreds (if not more) possibilities to use Arduino, ranging for fancy LED matrix animations to the full-grown Lawn Bot 400:

What’s got all this to do with Ariejan.net?

Of course, I’m a Rails web developer, but Arduino has sparked my interest.

Over at Kabisa we’ve already done a small side-project using Arduino to scan and log RFID-tags - we’ll be deploying this at a Rails conference later this year (and we’ll be using it to give away some awesome prizes).

In the scarce spare-time I have I’m planning on going beyond the blinking LED and build a 4WD Robot. I’ll post my progress here at Ariejan.net. Keep an eye on the Arduino section for updates.

It’ll take me some months to get my robot driving and doing the stuff I want it to do. But my dream project is building a Quadcopter.

What about you?

You are still reading this? And you didn’t get started with Arduino yet? Come on, find your local Arduino distributor or head over http://arduino.cc and get one. Start doing the tutorials and be ready to be pulled into the world of tinkering and hobby electronics. It’s fun!

Images have been taken from arduino.cc with not so much permission.

Move your latest commits to a separate branch

Tue, 14 Aug 2012 00:00:00 +0000

The situation is pretty straightforward. You have been making commits for that new feature in your master branch. Naughty you!

Let’s assume you want to have this:

A - B - (C) - D - E - F

C was the last commit you pulled from origin and D, E and F are commits you just made but should have been in their own branch. This is what you wanted:

A - B - (C)
 \ D - E F

Step 1: Assuming you’re at F on master, create a new branch with those commits:

git branch my_feature_branch

Then, still on master, reset back to commit C. This is 3 commits back.

git reset --hard HEAD~3

Okay, you’re master is now back at C, which you lasted pulled, and the my_feature_branch includes D, E and F. Just checkout my_feature_branch and continue work as usual. I’m sure no one saw what you just did.

VPN too complicated? Use a IP-over-SSH tunnel instead

Wed, 11 Jul 2012 00:00:00 +0000

Some times you find yourself in a place where your Mac is safely tucked away behind a firewall. That’s great, but sometimes it is annoying as hell, because you need to access resources over FTP or contact people who’re on IRC.

The normal solution would be to setup a VPN with one of your servers elsewhere and connect to the outside world that way. Unfortunately, in all their wisdom, sys admins have probably closed up the proper ports to access your VPN server as well.

As a last resort you might consider setting up a SSH Tunnel for a specific service like this:

ssh -N user@server -L 3306:127.0.0.1:3306

But, this only works for a single port, and thus application. It may help, but it can become tedious pretty quickly. You also have to rewrite any configuration you had for connection to a remote host to use your localhost, most likely on some strange port.

Luckily for us, there’s this awesome tool called sshuttle.

Sshuttle allows you to setup what’s called IP-over-SSH. Basically it runs a local proxy server to a remote server over SSH and changes the routing for your machine to send everything through that proxy.

Besides giving you access to all the services you need, you also encrypt (e.g. hide) all your traffic from the prying eyes of any sys admins on the local network.

Installing sshuttle on your Mac is a breeze

brew install sshuttle

Then you can setup an IP-over-SSH connection to any remote server you have SSH access to. You’ll need your local admin password in order to setup routing properly.

sshuttle -r username@server 0/0 -vv

This routes all traffic over the tunnel towards server. Use on of those online ip checkers to see that you’re actually using your server’s IP address.

In the future you may want to change the -vv verbose option out and swap in -D to run in daemon mode.

The one thing this does not do is DNS. DNS is still done using your locally configured DNS server, mostly for speed.

Not to worry, you can go ‘full stealth’ with the --dns options, which also routes DNS over to the remote server:

sshuttle --dns -r username@server 0/0 -vv

To stop using your IP-over-SSH connection, simply press CTRL-C twice and sshuttle should restore your normal networking connections.

If sshuttle does not restore the connection properly, you can do so manually:

sudo ipfw -q -f flush

I’ve already create a few aliases in my ~/.zshrc:

alias tunnel='sshuttle -r ariejan@server 0/0 -vv'
alias tunnel_dns='sshuttle --dns -r ariejan@server 0/0 -vv'
alias reset_tunnel='sudo ipfw -q -f flush'

So, no need to setup complicated VPN contraptions, just use plain old SSH and off you go.

Bonus: you can also connect to a non-standard SSH port, in case port 22 has been blocked in the firewall as well:

sshuttle --dns -r username@server:port 0/0 -vv

Search and Replace in multiple files with Vim

Mon, 18 Jun 2012 00:00:00 +0000

I recently learned a nice VimTrick™ when pairing with Arjan. We upgrade an app to Rails 3.2.6 and got the following deprecation message:

DEPRECATION WARNING: :confirm option is deprecated and will be removed from Rails 4.0.
Use ':data => { :confirm => 'Text' }' instead.

Well, nothing difficult about that, but we have quite a few :confirm in this app.

Firstly we checked where we used them (note we use ruby 1.9 hash syntax everywhere):

ack -l "confirm:" app

Now you have a listing of all the files that contain the :confirm hash key. You can leave out the -l option to get some context for each find.

Now, we need to open Vim with those files:

ack -l "confirm:" app | xargs -o vim

Vim will open the first of these files. Here’s a snippet of what you may find:

= link_to "Delete", something_path, confirm: "Are you sure?"

Now, search and replace is easy:

:%s/confirm: ".*"/data: { & }/g

This will surround the current confirm with the data hash. Just the way Rails likes it. The & character will be replaced with whatever text matched the search pattern.

You could repeat this for every file manually. But, you’re using Vim.

:argdo %s/confirm: ".*"/data: { & }/g | update

This will perform the search and replace on each of the supplied arguments (in this case the files selected with ack) and update (e.g. save) those files.

Now you can quit Vim and enjoy the glory of all the disappearing deprecation warnings.

Note: to do this with the ruby 1.8 hash syntax, just update the search and replace texts accordingly.

SEO is bullshit

Thu, 24 May 2012 00:00:00 +0000

I get quite a lot of “oh we can SEO optimize your site for you” emails lately. I don’t know why, but SEO is bullshit. Really. SEO is one of those areas of expertise that are total rubbish.

Common SEO advice

The most common advise regarding SEO includes:

Getting links to your site
Add tons of keywords so the Google Crawler knows what you’re talking about
Add META data, lots of it
Structure your page in some way for crawlers to understand better
Add an XML sitemap

There might be some merit in this, but you are optimizing your site for a crawler, a computer program that interprets your site. Are crawlers your primary audience? If so, you have a sad site.

SEO Hacking is another common term I find in emails. SEO hacking tries to trick search engines to list your site higher for certain keywords. I won’t go into the details, but basically you try to cheat a search engine’s indexing rules.

Some shady SEO companies even offer you a guarantee that you’ll be the number 1 listen for certain keywords. Really? But how valuable is that?

The problem with doing a lot of SEO hacks on your site is this: search engines and crawlers are continuiously updated to prevent hacking. Although this is not disclosed by Google, there are rumours that if you do actually use ‘SEO hacks’, you’re site will be ranked lower, instead of higher.

The best SEO advice you can get

So, what can you do to make people find your site? Simple:

Write. Good. Content.

Use headers like <h2> and <strong> and <em> tags to highlight import terms or words in your text — this is valuable to your readers as well.

If your content is good, readers will find it. Link to it. Tweet about it. And crawlers will always be able to index your site properly. Of course, it helps to tell people about your article using social media and other news sites for your niche.

So, not META-tags you say?

No, that’s not whaty I’m saying. META tags do prodide information to crawlers, you should not assume they use that information in any way. For example, Ariejan.net has these META-tags:

<meta content='About Software Engineering, Ruby on Rails, Java, Git and the Cloud - by Ariejan' name='description'>
<meta content='ruby, rubyonrails, rails, git, svn, mysql, mac, ios, apple, web, web2.0, development, dev' name='keywords'>

In Google, this results in a view like this:

Do note that the posts listed are considered popular by Google. Those are the post that are most visited and linked to on my site.

Conclusion

If you get an email, promising you that for $200 you’re site will be ranked nr. 1 for specific keywords: throw it away.

The best SEO is writing awesome content people want to read.

How Star Trek has shaped our technology and future

Wed, 23 May 2012 00:00:00 +0000

You have probably watched Star Trek. Even if you don’t like it, I bet you’ve seen at least a few episodes of The Next Generation or a movie.

Regardless of you opinion on Star Trek, its cool to see that current technology was only Science-Fiction ten or twenty years ago.

Let’s start with a simple example - the Communicator. Press a button and talk to anyone wirelessly, wherever they are. Reminds you of that mobile phone you carry around all day, doesn’t it?

Well, if that doesn’t convince you, then check out the Vocera B3000 Communication Bagde

But there’s more. Are you afraid of needles? Wouldn’t it be totally cool if they didn’t have to stick metal tubes into your arm to give you meds? Well, the can! Enter SonoPrep (Sorry, their site appears to be down). It’ll weaking your sking in 15 seconds and you’re ready to receive meds. You won’t feel a thing.

Warp Speeds? Try Hyperdrive.

Cloaking devices? Got it

Phasers? Check

Medical Tricorder? Sure, we can

Transparent Aluminium? Yes, please

Okay, tractor beams then? Oh, yeah

A huge library where you can just play about any tune in history? Sweet mother of god, yes

Hand-Held devices that tell you your current location? We have those too

PADD’s? Oh, yes!

Warp Core’s and photon torpedo’s? We’d need Antimatter for that, so we’re getting there.

I must admit that I was quite surprised to find some of these things already exist in current technology! I wonder how long it will take before we build our first “USS Enterprise”.

Well, about 20 years.

Build the Enterprise has thought about building a huge, Enterprise like, star ship. It’s entirely feasable to built on in the next 20 years with proper funding (about half of what NASA got during its Apollo missions).

Theoretically the Enterprise could be ready in 20 years. It could then take you to Mars in 90 days. The moon would take you ‘only’ 3.

The future is now.

Note: Probably no one can tell if a specific technology or device was inspired by Star Trek or not. But it’s very fascinating to see that stuff that was once thought-up by Gene Roddenberry (and others) are already a reality.

Ruby: regex scanning in a case statement

Tue, 22 May 2012 00:00:00 +0000

Here’s a handy ruby snippet that might come in handy one day.

When the regex matches (input should end with " today"), you can directly grab the matched value using the special $1 variable.

case input
when /(.*)\stoday$/i then
 puts "Today: #{$1}"
end

I think you can see how you can bend this to your own needs.

What's causing all that disk I/O on my Mac?

Mon, 21 May 2012 00:00:00 +0000

After doing a full re-install of my MacBook Pro a few weeks back to combat system slowness, I’m again struck with a slow system.

The problem is that there’s a lot of disk I/O going on. CPU and memory are fine. I’m just not sure what’s causing this trouble.

Luckily, there’s an easy solution to find out what’s doing disk I/O on you mac.

Open up a terminal and run

$ sudo fs_usage
09:08:45.749208 read F=12 B=0x400 0.000210 W Sparrow.3845544
09:08:45.749274 lseek F=12 O=0x0602e000 <UNKNOWN> 0.000002 Sparrow.3845544
09:08:45.749279 read F=12 B=0x400 0.000005 Sparrow.3845544
09:08:45.749341 lseek F=12 O=0x06038c00 <UNKNOWN> 0.000002 Sparrow.3845544
09:08:45.750358 RdData[async] D=0x025aaf38 B=0x1000 /dev/disk0s2 0.028830 W Sparrow.3845547
09:08:45.750379 pread F=16 B=0x30 O=0x00b32730 0.028895 W Sparrow.3845547
09:08:45.750390 pread F=16 B=0x30 O=0x0de9a810 0.000008 Sparrow.3845547
09:08:45.750396 pread F=16 B=0x30 O=0x1f5ec5c0 0.000006 W Sparrow.3845547
09:08:45.750399 pread F=16 B=0x30 O=0x236c84a0 0.000003 Sparrow.3845547
09:08:45.750405 pread F=16 B=0x4b O=0x236c84b6 0.000002 Sparrow.3845547
09:08:45.750701 pread F=16 B=0x30 O=0x00c95f00 0.000007 Sparrow.3845547
09:08:45.758502 RdData[async] D=0x04f0b608 B=0x1000 /dev/disk0s2 0.009148 W

What you see here is Sparrow doing mad reads ot my disk. I’m not sure what it’s doing, but it must stop. After killing Sparrow disk I/O goes down to normal, workable levels again.

Now, to pick another mail client for Mac…

Why Diablo 3's DRM is the best kind there is

Sun, 20 May 2012 00:00:00 +0000

I just tried to play Diablo 3, but could not log on due to server maintenance. Blizzard requires you to be always online to play even single player campaigns. I’ve seen a lot of people rage about this, but Blizzard actually has very good reasons to do this. And they even kept you, as a player, in mind!

First of all, what Blizzard is doing is not really DRM.

Digital Rights Management is making sure you have authorization (from the providing party) to do something off-line. The best example is music. After authorizing your computer you can play your DRM protected music.

What Blizzard requires is you logging on to one of their servers to play Diablo 3. This sounds like DRM, but you are actually using Blizzard’s servers to play. And for good reason!

Diablo 3 contains several features that allow you to interact with other players. For example, you can easily interact with your friends and you can even join their games. There’s also the Auction House, where you can trade items.

These features are nothing new to World of Warcraft players, and it’s not surprising they require a central server to make sure everything is fair.

An added benefit of this always online model is that Blizzard can closely monitor how players behave in the game, allowing them to fine-tune game balance with every update the release.

So, there are many wins for the player here. The big win for Blizzard is that they can be selective about whom they allow to enter the server: only people with a valid license key.

I know it’s frustrating that you can’t play Diablo 3 on a rainy sunday afternoon when you want to. But with a modern, social game like Diablo 3, being online is an absolute must.

What happened to downloading games from The Pirate Bay?

Sat, 19 May 2012 00:00:00 +0000

This post is on piracy. Piracy of software, games, movies, music and other virtual goods.

Before I continue writing this post let me confess to you that I used to use pirated software. I played games downloaded from the internet. And movies I watches ‘on demand’ by downloading them when I wanted to. There were times when there were no legally purchased songs on my iPod.

After reinstalling my Mac a few weeks ago I noticed something remarkable. There is not a single piece of pirated software on my Mac. Everything has been paid for or is free / open-source. The same goes for my gaming rig, all games are legally purchased.

This is in shrill contrast to a few years ago. What the fuck happened?

Then I saw Diablo 3 was released. I’m not a huge fan of the franchise, but wanted to play Diablo 3 anyway. So, what to do.

Checking with local and online retailers I can get a fair price of €49 euro - but the game is either sold out or will not be delivered in the next few days. I want to play Diablo 3 ** RIGHT THE FUCK NOW**.

Okay, let’s check with Blizzard then, the Battle.net store. They have the digital download version of Diablo 3, but for €59. It is instantly available, though.

Clickety-Click, Diablo 3 purchased. Download ready in 16 minutes. Playing 5 minutes later.

It’s been said many times before, and it will be said many times again: buying digital goods should be easy, fast and secure.

I’ve amazed myself by forking out an additional €10 to get easy access to Diablo 3. But I’m a happy customer and I bet Blizzard is also happy to have sold another Diablo 3 copy.

Hopefully the entertainment industry will get this message soon. Because movies are the only thing that are still available to me when I want them through The Pirate Bay.

Running a different ruby with Passenger 3.2 and RVM

Fri, 11 May 2012 00:00:00 +0000

Passenger 3.2 will have quite some nice new features. 1 2

The features I’m looking forward to most is the ability to specify - per virtual server - which ruby to use.

Before, you installed passenger and specified the required ruby version using passenger_ruby, like this in your nginx.conf:

http {
passenger_root /opt/passenger;
passenger_ruby /usr/local/bin/ruby;
server {
server_name ariejan.net;
passenger_enabled on;
}
}

Now, if you added another server it would be forced to use the same ruby version. This might be okay for most servers, but for me not so much. I have several side-projects running on a single machine, and using only one ruby version is not optimal or even impossible.

Now, with the upcoming Passenger 3.2 you can select a ruby version per server. This is awesome. All you have to do is move the passenger_rubydirective into the server block and all is set. Of course, you can leave the globally set ruby just as is.

http {
passenger_root /opt/passenger;
passenger_ruby /usr/local/bin/ruby;
server {
server_name ariejan.net;
passenger_enabled on;
passenger_ruby /home/deployer/.rvm/rubies/ruby-1.9.3-p194/bin/ruby;
}
}

As you can see in the example above, I’m referencing ruby-1.9.3-p194, installed with RVM.

Installing “experimental” Passenger

The installation is easy, as usual, but you must checkout the passenger source from Github and use the experimental branch.

Warning: do not install the experimental branch of Passenger on your production server unless you are absolutely sure what you’re doing and you know how to rollback quickly and easily to a stable version of passenger.

cd /opt
git clone https://github.com/FooBarWidget/passenger.git
cd /opt/passenger
git checkout -b experimental origin/experimental
./bin/passenger-install-nginx-module

Decorating Devise's current_user with Draper

Sat, 14 Apr 2012 00:00:00 +0000

I’ve become a big fan of decorators, especially Draper.

Decorators allow you to move view related functionality for your models in to separate decorator classes. This keeps both your models and views clean and readable.

Anyway, if you use Devise you’re provided with a current_user helper. However, this helper returns an instance of User - without your decorators. To enable decorators for your current_user by default, simple add this to app/controllers/application_controller.rb:

def current_user
 UserDecorator.decorate(super) unless super.nil?
end

Now, anywhere in your views where you call current_user you’ll get a decorated version instead.

Check here to see how to use Draper with Sorcery

Eindhoven.rb Lightning Talk: Gitlab

Fri, 06 Apr 2012 00:00:00 +0000

These are the slides of my Eindhoven.rb lightning talk. The topic is Gitlab, why it’s awesome and how it relates to Github.

If you have any questions about Gitlab, feel free to contact me or visit the Gitlab Mailinglist

Showing Ruby, Rails and git info in your app

Mon, 02 Apr 2012 00:00:00 +0000

Some people’ve asked me how I show rendering information on ariejan.net.

There are a few things going on here, let me explain them one by one.

Rails version

The current Rails version is probably the easiest you see here. Rails exposes its version information like this:

Rails.version

Ruby version

Ruby also exposes version information, albeit using constants:

RUBY_VERSION
=> "1.9.3"

You may know that ruby also has different patch levels for each release. You can also retrieve that information:

RUBY_PATCHLEVEL
=> 125

The you may also want to know which engine you’re using. This may be “ruby” or something different:

RUBY_ENGINE
=> "ruby"

Combine these to get a sexy ruby version string:

puts "#{RUBY_ENGINE}-#{RUBY_VERSION}-p#{RUBY_PATCHLEVEL}"
=> "ruby-1.9.3-p125"

Current process ID

I like to know which Unicorn produced a certain page. Retrieving the process ID is easy:

Process.pid
=> 9473

Current git revision SHA

Knowing which version of your app is currently running can be useful information. To do this I created the following initializer:

# config/initializers/git_revision.rb
module AppName
REVISION = `git log --pretty=format:'%h' -n 1`
end

This will expose the current short SHA in your application’s namespace:

AppName::REVISION
=> "ac6d3a0"

You can now use this info through-out your app to show version information.

From 11.34s to 0.625s for opening a .rb file in Vim

Wed, 21 Mar 2012 00:00:00 +0000

Would you believe me if I told you that opening a simple Ruby file on my 2011 MacBook Pro takes 11.34 seconds?

To test this, I’ve used this command:

$ vim --startuptime log-before.txt app/models/user.rb

This command will time everything that Vim does until the file is ready for you to edit down to the millisecond. This is a great way to find out what’s slowing things down.

I’ll highlight the most interesting parts of log-before.txt here:

 000.028 000.028: --- VIM STARTING ---
6643.597 5496.976 5496.976: sourcing /usr/share/vim/vim73/ftplugin/ruby.vim
11262.686 3963.993 3963.993: sourcing /Users/ariejan/.vim/bundle/vim-css-color/after/syntax/css.vim
11263.907 3976.912 004.334: sourcing /usr/share/vim/vim73/syntax/html.vim
11263.997 3977.361 000.449: sourcing /usr/share/vim/vim73/syntax/xhtml.vim
11340.533 000.004: --- VIM STARTED ---

These are the big spenders of loading a ruby file. Firstly there is ruby.vim taking about 5.4 seconds to load. Then there is css.vim taking another 3.9 seconds - and this file doesn’t even include CSS!

These two time sinking hogs are keeping me back – 11 seconds at a time.

Let’s see, vim-css-color. This plugin shows color hashes in their actual colour. So #00f will have a blue background. Great when editing CSS files, but not all that import. I removed vim-css-color.

Note: the reason vim-css-color is slow with terminal vim is that is has to pre-compile colour hashes to ther xterm escape code equivalents. This is pretty time consuming.

Next up: ruby.vim. Why is this so bloody slow?

As it turns out, Vim has trouble finding the right ruby for me. This can be remedied by adding the following snippet to your ~/.vimrc. It sets a logical search path for ruby:

if !empty($MY_RUBY_HOME)
let g:ruby_path = join(split(glob($MY_RUBY_HOME.'/lib/ruby/*.*')."\n".glob($MY_RUBY_HOME.'/lib/ruby/site_ruby/*'),"\n"),',')
endif

Again I ran my timer command (full output):

$ vim --startuptime log-after app/models/user.rb

Now look at that:

000.034 000.034: --- VIM STARTING ---
107.182 000.834 000.834: sourcing /usr/share/vim/vim73/ftplugin/ruby.vim
625.001 000.003: --- VIM STARTED ---

Yes, that is 0.625 seconds! I’m a happy ruby coder again.

Redis using 2GB of memory on 70MB data set - the fix

Wed, 21 Mar 2012 00:00:00 +0000

For Ariejan.net I use redis to cache pages and shards. This works great and all, but today I noticed something alarming:

redis Running 3d 12h 4m 0.0% 45.2% [1829556 kB]

Yes, that’s about 1.7 GB of RAM. That’s way too much for what I cache. Let’s see what redis has to say for itself:

$ redis-cli
redis 127.0.0.1:6379> info
redis_version:2.2.12
...
connected_clients:5
connected_slaves:0
used_memory:71626608
used_memory_human:68.31M
used_memory_rss:1873465344
mem_fragmentation_ratio:26.16

Well, that’s awkward. The OS is reporting 1.7GB memory usage, while Redis claims to store a mere 68MB. What’s happening here!

You may have noticed that I included mem_fragmentation_ratio in the snippet above as well. It’s at a whopping 26.16, meaning that for every byte I store, 26.16 bytes of memory are used. This explains the 1.7GB memory usage.

But, how do I get rid of this? My system has enough RAM to cope redis as is, but it’s not a comforting thought to leave Redis running like this.

As it turns out, there isn’t a lot you can do about this. Redis 2.2 uses malloc which causes the fragmentation. One alternative is to add a slave redis-server, migrate your data and then switch the slave to master. Although this is reported to work well, it’s not a good solution to the problem.

Fortunately, Redis 2.4 on Linux by default does not use malloc anymore. Instead it uses jemalloc. From the redis README:

Redis is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was picked because jemalloc has proven to have fewer fragmentation problems than libc malloc.

The only logical step to take is to upgrade to Redis 2.4.

If you’re already running (or just upgraded to) 2.4 you can easily check if your redis is using jemalloc:

$ redis-cli
redis 127.0.0.1:6379> info
redis_version:2.4.9
mem_fragmentation_ratio:1.11
mem_allocator:jemalloc-2.2.5

I’ve taken this step and the mem_fragmentation_ratio samples I’ve measured have all been in the 1.1-1.4 regions.

Open Source is a privilege. Not a right.

Tue, 20 Mar 2012 00:00:00 +0000

Edit: this is not a Gitlab specific issue. I’ve seen it before and I’m seeing it again at the Gitlab project.

The past few months I’ve been contributing to the Gitlab project. This has been a great experience, mostly because Gitlab has become a very popular project with over 2.3k watchers right now.

It gives me great satisfaction knowing that my code is being used by hundreds if not thousands of people right now. But there is also this little thing that has been bugging me lately.

There is a small core team of developers devoting time to the development of Gitlab. They do this passionatly and they have been cranking out releases and new features every 22nd of month for the past six months.

A lot of effort has gone into making Gitlab what it is today.

But with the rise in popularity there has also been an increased number issues, bugs and feature requests. The mailing list is also more active than ever. Of course, all of this is a good thing for an open source project, you would think.

Most users appreciate the effort that goes into developing an application as complex as Gitlab. They appreciate the fact it’s available for free. The appreciate somebody made time to develop a tool they can put to use on a daily basis.

This appreciation is great motivator for the developers. Knowing their code is being put to good use only drives them write more code, more features.

Unfortunately, I read tickets and threads of this kind too often lately:

Add [this feature]. My company needs this and if you don’t add this we won’t use Gitlab.

Also this:

We are expecting [this feature] in the next release.

Or:

We must have [this feature]. Gitlab is useless without it and we’ll go back to using Github.

Why do people think they can make demands like these?

Is it too hard to read the installation guide that comes with the project?

Don’t get me wrong, it’s awesome people are willing to try and install Gitlab. It’s even more awesome people post their ideas and feature requests to further improve Gitlab.

However, the lack of respect for people who spend their free time developing an open source application for you to use for free is sickening.

No, really. Sickening.

There really is no reason to threaten or pressure the people who are dedicating the free time, parts of the lives, to an open source project to implement something for you.

Of course, some of you will say, “Just ignore those ass holes. They’re not worth your time.” I agree, but after reading two or three of these responses I don’t feel like spending my evening off coding some awesomeness.

This is a request to anyone who’s ever going to post a feature request, a bug report or even a post to the mailing list of an open source project:

be polite,
don’t make threaths or demands, and
show some respect for the souls who dedicate their time to making something awesome.

Open Source is a privilege. Not a right.

Removing untracked files and directories with git

Tue, 10 Jan 2012 00:00:00 +0000

I just tried writing some new code, but it was no success. This happens, but it left me with a working copy littered with new and changed files.

# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: config/routes.rb
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# db/migrate/20111231131752_create_validations.rb
# vendor/assets/images/
# vendor/assets/javascripts/
# vendor/assets/stylesheets/custom.sass

So, how do I get rid of this mess?

1 - Be absolutely sure you want to delete your work

I sometimes commit my work although it’s of no use to me at the moment. It’s in a separate branch anyway and can be easily ignored until I come back to it.

So, if you’re really sure you want to delete your changes and files, continue to step 2.

2 - Delete untracked files and directories

Now, delete all files that are not yet tracked by git. You could do it like this:

rm -r db/migrate/20111231131752_create_validations.rb vendor/assets/images \
vendor/assets/javascripts/ vendor/assets/stylesheets/custom.sass

This requires you to copy/paste or type all the files and directories you want to delete. There’s a much easier way: ask git to do it for you:

git clean -df

The d option tells git to include directories, f says that you really want to perform the delete. Optionally, you can first run with n instead of f to see what’s going to happen - a so called dry-run.

If you use this command frequently and don’t want to specify the f options every time you can set clean.requireForce in your ~/.gitconfig to true to omit the f options.

3 - Reset changes to tracked files

With all the new stuff out of the way, let’s clean up the files that are tracked by git. You want to revert everything back to the last commit you made:

git reset HEAD --hard

That’s all. You’re now back to a clean working directory at HEAD. Start over now.

Recursively fixing file and directory permissions

Tue, 13 Dec 2011 00:00:00 +0000

While working on a Gitlab installation I noticed that all repository file permissions were off. Fixing recursive file and directory permissions can be quite hard. Or so I thought.

Using the following commands (in plain Bash) allow you to recursively set permissions for files and directories. So, to fix the proper read permissions on your Gitlab repositories you can use this:

# Go to your git repositories directory (as git or the gitlab user)
cd /home/git/repositories

# Fix ownership
sudo chown -R git:git *

# Fix directory permissions
sudo find -type d -exec chmod 770 {} \;

# Fix file permissions
sudo find -type f -exec chmod 660 {} \;

After this, your Gitlab should have no trouble accessing your code (e.g. in the tree browser).

Upgrade postgresql-8.4 to postgresql-9.1 on debian

Tue, 22 Nov 2011 00:00:00 +0000

Today I upgraded a production PostgreSQL 8.4 database to version 9.1. This was on a Debian server. ~ The first step is to make a full dump of your data. I personally like to store that dump somewhere safe before upgrading. As root:

su - postgres
pg_dumpall > dump.sql
exit
cp ~postgres/dump.sql /root/

Now you can safely remove the postgresql-8.4 and install postgresql-9.1:

aptitude purge postgresql-8.4
aptitude install postgresql-9.1

Next check the postgresql configuration in /etc/postgresql/9.1/main. If you make any changes, make sure to restart postgres with /etc/init.d/postgresql restart.

Postgresql 9.1 is now up and running, let’s import our data back into it.

su - postgres
psql < dump.sql

That’s all. You’re now fully upgraded to PostgreSQL 9.1.

Deploying a third-party Rails application - like Gitlab

Fri, 18 Nov 2011 00:00:00 +0000

We all know how to deploy our own Rails projects. (If not, read this guide.) But how do you handle deploying a third-party application that may require some customisation on your part?

A good example would be Gitlab

Gitlab is an open source Github clone, build using Ruby on Rails. It’s a nice project that uses Gitosis under the hood to manage your git repositories. There are several good installation guides available on the web, but they all assume you want to deploy gitlab verbatim - without any modification or configuration

I have also setup Gitlab, but I want to use capistrano and unicorn. I also want to tweak some configuration. In some rare cases I want to fix an annoying bug and not wait for the Gitlab team to pull it.

Doing a git pull on my remote server just won’t cut it.

Fork! Fork!

What I did was clone Gitlab and use the same principles describe in my Contributing to open source with Github to apply my own changes and merge any upstream changes when they are available.

Here’s how I set everything up.

First, clone the official Gitlab repository and name it upstream.

git clone --origin upstream https://github.com/gitlabhq/gitlabhq.git my_git_server

Next I made all the changes I want. I updated config/gitosis.yml and unicorn to Gemfile and setup Capistrano.

I then pushed this to my own git server. This is the same server Capistrano will use to pull changes from.

git remote add origin git@git.ariejan.net:my_git_server.git
git push origin master
cap deploy

That’s all there is to deploying Gitlab from my own repository.

Merging upstream changes

Now, the Gitlab crew is pushing out new features at an amazing rate. So, how do I get those new features (and the occasional bug fix) into my copy of Gitlab for deploying?

git fetch upstream

Remember how we named the official Gitlab repository upstream earlier? With this fetch we get all changes from their repository (but we don’t apply them to anything yet).

Then, merge the upstream changes with your own branch.

git merge upstream/master

There may be merge conflicts, just resolve them and commit your merge. Then again to deploy:

git push origin master
cap deploy

Why do this?

The reason I use this approach is that it’s easy to merge upstream changes and have my own customizations and deployment tools at the same time.

Contributing to Open-Source with Github

Wed, 09 Nov 2011 00:00:00 +0000

You want to contribute to an open-source project, but are scared away by all the git-complexity involved? This small guide will help you out.

Step 1 - Fork, fork, fork

First create a fork of the original project. You can do this easily by clicking the “Fork” button on the top of the Github project page. This will give you your own copy (fork) of the entire repository.

Then, check out your fork:

git clone git@github.com:ariejan/repo-name.git

Step 2 - Contribute

Before you start writing code there are a few tasks you need to perform:

Are there tests for this project? Are they all green? If not, fix this first. (using this same step, of course)
Create a new branch: git checkout -b fix_for_this_or_that
Red-Green-Refactor (e.g. write code)
Commit your changes (in your fix_for_this_or_that branch).

With your contribution done, don’t merge it back into master. master is your way of receiving changes from the original (upstream) repo (see step 4).

First push your branch to Github, so you can share it with others.

git push origin fix_for_this_or_that

You should now see this branch in your Github project page. You’ll also notice there’s a “Pull Request” button at the top. Click it if you want the project maintainer to pull your fix_for_this_or_that branch into the main project.

Step 4 - Keeping up-to-date

Over time the master of your fork will start lagging behind. Because you did not merge any of your code changes into the master of your fork, you can update it easily.

Before you can pull in changes you must add a git remote for it. You can use the Git Read-only URL for this.

git remote add upstream https://github.com/some_one/some-repo.git

Now, for raking in the changes;

git checkout master
git fetch upstream
git merge upstream/master

Step 4.5 - Keeping your feature branch up-to-date

If you pulled in changes from upstream and did not yet share your feature branch, you can rebase your feature branch. This makes sure you have the latest code from master. This also makes merging your pull request easier.

Do not rebase if you already pushed your branch to Github. Read why.

git checkout fix_for_this_or_that
git rebase master

If any conflict arise, fix them. And continue your rebase:

git add conflicting_file
git rebase --continue

You may also abort the rebase:

git rebase --abort

Big picture

This small guide will help you to fork repositories on Github and use them to contribute code. By using the approach you lighten the task of the project maintainer who can easily checkout your specific changes and decide to include them into the main project or not.

Fixing a slow starting Terminal or iTerm2 on Mac OS X

Tue, 08 Nov 2011 00:00:00 +0000

For some time I have been annoyed with how slow my Terminal (in my case iTerm2) starts up. It would take 5-10 seconds before I was presented a prompt. Being in the console for the better part of the day, this was unacceptable.

The first thing I did was upgrade Bash to Zsh. I’d heard great things about Zsh, so I thought I’d give it a try. Zsh is really awesome, but it did not fix the start-up delay I was experiencing previously with Bash.

Doing a little digging around I found a working solution. Apparently Apple keeps system logs in /private/var/log/asl/*.asl. Removing these files made my shell fast again.

Do not remove all files in /private/var/log/asl as there are other files there that are not related to the shell. Leave them in there.

With the above warning in the back of your mind, open your terminal (slowly) and issue the following command:

sudo rm /private/var/log/asl/*.asl

Now quit and restart Terminal or iTerm2 and your prompt should present itself quickly again.

Installing Node.js and NPM on Ubuntu/Debian

Mon, 24 Oct 2011 00:00:00 +0000

This is just short snippet on how to install Node.js (any version) and NPM (Node Package Manager) on your Ubuntu/Debian system.

Step 1 - Update your system

sudo apt-get update
sudo apt-get install git-core curl build-essential openssl libssl-dev python

Step 2 - Install Node.js

First, clone the Node.js repository:

git clone https://github.com/nodejs/node.git
cd node

Now, if you require a specific version of Node:

git tag # Gives you a list of released versions
git checkout v13.10.1

Then compile and install Node like this:

This might take a while, depending on your hardware.

./configure
make
sudo make install

Then, check if node was installed correctly:

node -v

Step 3 - Install NPM

Simply run the NPM install script:

curl -L https://npmjs.org/install.sh | sudo sh

And then check it works:

npm -v

That’s all.

Updated 2020-03-11

Use the new nodejs/node repo
Install correct dependencies, including python\
Update example node version to v13.10.1_

Automatically switch between SSL and non-SSL with Nginx+Unicorn+Rails

Sat, 22 Oct 2011 00:00:00 +0000

Scroll down for setup instructions. Or, read this bit about SSL in the real world first.

SSL or Secure Socket Layer is a nice way to secure sensitive parts of your Rails application. It achieves to goals.

Firstly is encrypts all traffic between you and the remote server. Consider the passwords and personal information you submit to websites. When unencrypted (using HTTP), all this data is sent over the internet for all to read. With SSL (HTTPS) enabled, all traffic is encrypted, making it very hard for a third-party to eavesdrop.

Secondly, a proper SSL connection can give you trust in that you’re communicating with the right people.

For example, Rabobank (a Dutch bank) uses SSL for its website. When you open their site, you’ll notice the green ‘Rabobank Nederland’ in the address bar.

This tells me I am communicating with Rabobank Nederland. This is why SSL certificates are so expensive - the SSL authority needs to verify the identify of Rabobank before they issue the certificate.

In the example above Rabobank uses an EV SSL Certificate. EV stands for Extended Validation. This means that the SSL authority has verified (among other things) that Rabobank is a legitimate business and that they are the legal owner of the domain rabobank.nl

The cost for such an EV SSL Certificate is $200 - $1000 per year. You probably don’t need it for your site.

SSL for you and me

When you are looking to secure the back-end of your site (where you login etc.), you only require the encryption part of SSL. There are two routes you can take

Self signed SSL

You are able to create a working SSL certificate yourself. This will give you encryption, but no identity validation. When you use a self-signed SSL certificate all browsers will warn you about this.

Encryption
No validation
Warnings from your browser
Free

For me, that’s a reason not to use self signed SSL for any other than development and testing purposes.

Standard SSL

Most SSL authorities provide you with a Standard SSL product. These certificates only check if you own the domainname. They also offer encryption and work (without warnings) in your browser. You can get one of these for as little as $9 a year.

Encryption
Domain validation / trust
No warnings from your browser
Cheap ($10 - $20)

Setting up SSL for your Rails application

Setting up SSL is a web server thing. It does not involve your Rails appliation directly (but more on that in a moment).

If you followed my nginx+unicorn guide, you’ll have Nginx and Unicorn setup already.

Create your private SSL key (key)

First you need to create a private key. Do this on you server. The following command will generate a 2048 bit key. When it asks you to set a passphrase, do so.

$ openssl genrsa -des3 -out example.com.key 2048
Generating RSA private key, 2048 bit long modulus
..................................+++
.................................................................................+++
e is 65537 (0x10001)
Enter pass phrase for example.com.key:
Verifying - Enter pass phrase for example.com.key:

Creating a key and Certificate Sign Request (csr)

Okay, you now have your key. Next step, create the Certificate Sign Request. The sign request is the part you’ll send to the SSL authority to sign. You will need to provide some information here. Make sure you enter everything correctly.

Country Name - Your 2 letter country code. I.e. NL, UK, BE
State or Povince Name. I.e. Noord-Brabant, New York
Locality Name - Your city. I.e. Eindhoven, London
Organization Name - Your company or site name: Ariejan.net, Apple Inc.
Organization Unit Name - The section of your company. You may leave this blank. I.e. Online Services, Finance
Common Name - The domain this SSL certificate will be used for. If you want to run this on https://www.example.com, you enter www.example.com here. This is not a wildcard. example.com will not work on www.example.com and vice versa.
Email Address, challenge password and optional company name should normally be left blank. Just hit enter.

Here’s the full version:

$ openssl req -new -key example.com.key -out example.com.csr
Enter pass phrase for example.com.key: <<passphrase>>
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:NL
State or Province Name (full name) [Some-State]:Noord-Brabant
Locality Name (eg, city) []:Eindhoven
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Ariejan.net
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:example.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Great, you now have two files:

example.com.key - your private key. Keep it secret, keep it safe!
example.com.csr - sign request

Get your certificate (crt)

Now, go to your selected SSL authority, order your Standard SSL Certificate and upload the contents of example.com.csr when requested.

After doing some validations, which may require you to click some links in emails, you’re certificate should be ready for download. Save this file as example.com.crt.

Intermediate certificates

Some SSL authorities work with so called intermediate certificates. This requires you to include an intermediate certificate with your own certificate. If your SSL provider requires this, save the intermediate certificate as intermediate.crt.

For usage with nginx, you must place both your own and the intermediate certificates in a single file. This is easy:

cat example.com.crt intermediate.crt > sslchain.crt

Remove the passphrase from your key

Now, this is not recommended, but many people do this. The reason is that when your private key has passphrase, your server requires that passphrase everytime your (re)start it. This could mean that your server cannot boot up without manual interaction from your part.

cp example.com.key example.com.key.orig
openssl rsa -in example.com.key.orig -out example.com.key

You now have example.com.key.orig, which is your original private key with the passphrase. And you have example.com.key, which is the same private key, but without the passphrase.

Setup nginx for SSL

Finally, you can setup Nginx for SSL. Normally I add both a SSL and non-SSL configuration. Setup is very easy.

First of all, become root and copy your keys and certificate to /etc/ssl.

cp example.com.key example.com.crt /etc/ssl

or if you use an intermediate certificate:

cp example.com.key sslchain.crt /etc/ssl

Next, you take your non-SSL server configuration and duplicate it. Then you add the following lines.

listen 443; # Instead of Listen 80

ssl on;
ssl_certificate /etc/ssl/sslchain.crt; # or /etc/ssl/example.com.crt
ssl_certificate_key /etc/ssl/example.com.key;

location / {
 # Add this to the location directive.
 proxy_set_header X-Forwarded-Proto https;
}

Most of this is pretty straight forward. The proxy_set_header directive is needed to let your Rails application know if the request came in over SSL or not. Normally this shouldn’t matter, but you’ll need it for the next part of this guide.

Save, restart nginx and your SSL connection should be available.

Automatically switch between SSL and non-SSL with Rails

To take this site as an example, I don’t want to run the front-end through SSL. First of all, you can’t submit any data to my server. Second, I include several external resources (Disqus, Twitter, AdSense), that will give you warnings about using “insecure” content on an encrypted page.

What I do want is to encrypt traffic to the backend, where I log in and write posts like these.

I need to make sure that your browser knows exactly when and when not to switch to SSL. This is where the rack-ssl-enforcer gem comes in.

First, update your Gemfile:

# Gemfile
 gem 'rack-ssl-enforcer'

After you’ve run bundle install, update config/application.rb (or config/environments/production.rb is you only want to configure this for your production environment).

# config/application.rb or config/environments/production.rb
config.middleware.use Rack::SslEnforcer,
 :redirect_to => 'https://example.com', # For when behind a proxy, like nginx
 :only => [/^\/admin\//, /^\/authors\//], # Force SSL on everything behind /admin and /authors
 :strict => true # Force no-SSL for everything else

With the following statement, you achieve the following:

When you access any URL with a path that starts with /admin or /authors, you’ll be redirected to the SSL site.
Because we’re behind a proxy and want to do proper redirects, we specify the correct SSL domain.
We set strict to true. If you access something that is not /admin or /authors, you will be redirected to the non-SSL version of that page.

There’s a lot more possible with the rack-ssl-enforcer gem. Checkout their README on Github for details.

Note: if you find yourself getting into an infinite redirect loop, make sure have the proxy_set_header directive set correctly in your Nginx configuration.

Wrapping up

You now know how you can setup an SSL certificate with Nginx and how you can make your Rails application automatically switch between SSL and non-SSL whenever you want to.

Rails 3: Customized exception handling

Fri, 14 Oct 2011 00:00:00 +0000

Exceptions happen. There’s no way around that. But not all exceptions are created equally.

For instance, a 404 “Not found” error can (and should) be handled correctly in your application.

Let me give you an example of how to handle a ActiveRecord::RecordNotFound exception. Let’s assume you have an application that could show a user profile:

# GET /p/:name
def show
 @profile = Profile.find(params[:name])
end

Now, it may happen that the :name paramater contains a value that cannot be found in our database, most likely because someone made a typo in the URL.

If Profile#find cannot get a proper result it will throw ActiveRecord::RecordNotFound.

Now, instead of showing the user the (by default ugly) 404 page from public/404.html we want to do something more fancy.

Action-specific exception handling

Here’s one solution:

# GET /p/:name
def show
 @profile = Profile.find(params[:name])
rescue
 render :template => 'application/profile_not_found', :status => :not_found
end

You can now create app/views/applicaiton/profile_not_found.html.haml and give a nice custom error message to your user.

You may try to find some matching profiles to :name or show a search box.

Global exception handling

The above example only works for the specific profile show action. It’s also possible to hanlde exceptions on the application level.

Your show action still looks like this:

# GET /p/:name
def show
 @profile = Profile.find(params[:name])
end

Then, in your app/controllers/application_controller.rb add this:

class ApplicationController < ActionController::Base
 rescue_from ActiveRecord::RecordNotFound, :with => :rescue_not_found

 protected
 def rescue_not_found
 render :template => 'application/not_found', :status => :not_found
 end
end

Whenever an ActiveRecord::RecordNotFound exception is thrown (and not handled by the action itself), it will be handled by your ApplicationController.

Custom exceptions

It’s possible to throw your own custom exceptions and handle them in different ways. Like this:

# Define your own error
class MyApp::ProfileNotFoundError < StandardError
end

# GET /p/:name
def show
 @profile = Profile.find_by_name(params[:name])
 raise MyApp::ProfileNotFoundError if @profile.nil?
end

And add this to your ApplicationController:

rescue_from MyApp::ProfileNotFoundError, :with => :profile_not_found

Optionally, if you don’t want to write that custom profile_not_found method, you may also supply a block:

rescue_from MyApp::ProfileNotFoundError do |exception|
 render :nothing => "Profile not found.", :status => 404
end

Fast specs - Run your specs in less than 1 second

Thu, 13 Oct 2011 00:00:00 +0000

Okay, let me clarify that title first. I, as most of you, have two sets of tests for my Rails application: rspec and cucumber. rspec heavily focusses on testing models and business logic while cucumber focusses on testing the entire application stack and user interaction.

The problem is that as your app grows, your test set grows - and so does the time it takes to run those tests.

This post is inspired by Corey Haines’ talk at Arrrrcamp (Oct 2011) and my own experience with writing fast specs.

Red - Green - Refactor

If you do TDD/BDD you most likely follow the Red-Green-Refactor pattern:

Write one test, and see it fail (red)
Write the most minimal implementation to satisfy that test, and see it pass (green)
Refactor your code to look/perform better (refactor)
Repeat

But what if your test suite takes >30 seconds to run. You write a test, then wait 30 seconds to see the test fail. You then write the simple implementation, wait 30 seconds. Oops, you made a typo - fix it, wait 30 seconds. Now it passes. Refactor, again wait 30 seconds.

I think this scenario is very familiar for many rails developers.

Take a closer look

So, let’s take a closer look at a real-world example.

This is a Post model, it belongs_to and author and it can give you a summary of an article by returning the text above a ‘~’ marker.

# app/models/post.rb
class Post < ActiveRecord::Base
 DELIMITER = "~\n"

 belongs_to :author

 def summary
 summary = if body =~ /#{DELIMITER}/i
 body.split(/#{DELIMITER}/i).first.strip
 else
 body
 end
 end
end

This is a spec that’s defined for the Post model:

# spec/models/post_spec.rb
require 'spec/helper'

describe Post do
 context "summary" do
 it "should return the summary" do
 post = FactoryGirl.build(:post, :body => "Summary\n~\nNo summary.")
 post.summary.should == "Summary"
 end
 end
end

Running this spec would take at least 10 seconds. Running time rspec spec/models/post_spec.rb woudl output something like this:

Finished in 0.05523 seconds
real 0m10.387s

This means that running the actual spec took 0.05s, but running the entire command took 10 seconds. What is slowing us down?

Dependencies

As you see in the spec above I use FactoryGirl.build instead of FactoryGirl.create to prevent interacting with the database. I do this because hitting the database slows down your test.

I removed the database dependency in order for my test to run faster.

What other dependencies are there that could be removed in order to test the summary functionality?

Any idea?

Yes!

Rails.

Rails is a dependency to your app

You have a Rails-app. But Rails is not your app, it’s a dependency, just like the pg and haml gems are dependencies.

Loading the entire Rails stack takes quite some time. And just like with the database dependency we must ask ourselves: do we really need Rails to perform this test?

There are a lot of scenarios where the answer to that question is NO.

Specs without Rails

This may seem a bit weird at first, but let’s take another look at the Post model:

# app/models/post.rb
class Post < ActiveRecord::Base
 DELIMITER = "~\n"

 belongs_to :author

 def summary
 summary = if body =~ /#{DELIMITER}/i
 body.split(/#{DELIMITER}/i).first.strip
 else
 body
 end
 end
end

The summary method does not interact with the Post model at all, except that it access the body attribute. But the body attribute is just a String.

So, if we wanted to test the summary method and remove all Rails dependencies, we’d have to remove ActiveRecord.

Consider this:

# app/logic/myapp/summary.rb
module MyApp
 class Summary
 def self.for(text, delimiter)
 summary = if text =~ /#{delimiter}/i
 text.split(/#{delimiter}/i).first.strip
 else
 text
 end
 end
 end
end

I think you can quickly see that this method does exactly the same as Post#summary. But it does not have any dependency to ActiveRecord.

Could we rewrite our test for this new class?

# fast_spec/myapp/summary_spec.rb
require 'myapp/summary'
describe MyApp::Summary do
it "should return the summary" do
MyApp::Summary.for("Summary\n~\nNo summary.", "~\n").should == "Summary"
end
end

That looks good! Note that this spec does not include spec_helper. spec_helper is responsible for loading up your test environment, which normally includes all your app dependencies, including Rails.

Your new Post model

The Post model should also be updated, of course to utilise this new class.

# app/models/post.rb
class Post < ActiveRecord::Base
 DELIMITER = "~\n"

 belongs_to :author

 def summary
 MyApp::Summary.for(body, DELIMITER)
 end
end

The spec for Post should also be changed. Since we already have tested that MyApp::Summary#for returns the right summary for a given text and delimiter, all we have left to do is make sure that Post#summary calls it correctly.

# spec/models/post_spec.rb
require 'spec/helper'

describe Post do
 context "summary" do
 it "should return the summary" do
 post = FactoryGirl.build(:post)
 MyApp::Summary.should_receive(:for).with(post.body, Post::DELIMITER)
 post.summary
 end
 end
end

Running fast specs

The files above are located in fast_spec and app/logic. I do this because I want to separate my fast_specs so I can run them independtly from my normal specs.

Running fast specs works like this:

rspec -I app/logic fast_spec

Try it out with time rspec -I app/logic fast_spec:

Finished in 0.03223 seconds
real 0m0.421s

That’s your same spec, down from about 10 seconds to 0.5 second.

The big picture

By now you have seen that you can extract business logic into a seperate class. You’ve also seen that you can write tests that don’t depend on Rails and run amazingly fast.

Of course you ask, how does this relate to normal RSpec and Cucumber tests?

My opinion is that your fast_specs test business logic that does not relate to anything Rails specific, like making calculations, processing text, et cetera.

When you make changes to your business logic, you can test is very quickly. This shortens your TDD cycle and allows you to focus more on the task at hand (instead of waiting for your tests).

RSpec tests are there to test integration with your dependencies like Rails. This is the place where test scopes and mailers.

Cucumber still has its place to test user interaction with your app. Can a user still click the ‘Order’ button and get the proper response from our app.

RSpec and Cucumber will now be ran less often. Maybe only before you commit code - or maybe have your CI run them for you?

Added benefit - Design

An added bonus of TDD is that your tests dictate the design of your app. By using fast_spec you force your business logic into separate classes. This makes them even more re-usable than normal Rails models.

Your ActiveRecord models now become cleaner and are more a configuration of Rails than anything else.

By doing this your app will become way more maintainable, easier to test, faster to test and your code will be more re-usable. What’s not to like?

Where to go from here?

As Corey Haines put it:

Try to extract one single method from your app into a fast_spec. Just one. When you have that one spec running so fast it’s easy to add more.

This would also be my advise to you. Try to extract one piece of functionality from a model into a fast_spec. It may be difficult at first, but try it and see the results.

Then, when you have that one example working, add more.

Testing Rails 3 scopes revisited

Sun, 09 Oct 2011 00:00:00 +0000

In my previous article I told you about how I like to tests my scope. There was a fair amount of criticism on that post and after considering it all (and hearing Corey Haines’ talk on Arrrrcamp last friday), I’m convinced it’s the wrong path.

In essence Rails allows you to create a scope to generate a custom database query for you. Now, if you only test your configuration of Rails (as I did in my previous post), you don’t know if that scope works or not. If a piece directly relies on your database, you must test it against your database.

Also, your configuration may be correct, but what happens when you upgrade to a newer version of Rails or PostgreSQL? Does that configuration still work as advertised?

So, this is the proper way of testing your scopes:

class Post < ActiveRecord::Base
 scope :latest, order("created_at DESC")
end

describe Post do
 context 'scopes' do
 before(:all) do
 @first = FactoryGirl.create(:post, :created_at => 1.day.ago)
 @last = FactoryGirl.create(:post, :created_at => 4.day.ago)
 end

 it "should return posts in the correct order" do
 Post.latest.should == [@first, @last]
 end
 end
end

A note on speed

So, you now have to utilize the whole Rails stack and hit your database to perform this test, which is slow. But then you don’t upgrade Rails or your database very often, so you don’t have run this spec with every TDD Baby Step you take. (More on that in a later post.)

Properly testing Rails 3 scopes

Sun, 25 Sep 2011 00:00:00 +0000

The content of this post is no longer correct. Please read this article for details.

Testing scopes has always felt a bit weird to me. Normally I’d do something like this:

class Post < ActiveRecord::Base
 scope :published, where(:published => true)
 scope :latest, order("created_at DESC")
end

describe Post do
 context 'scopes' do
 before(:all) do
 @first = FactoryGirl.create(:post, :created_at => 1.day.ago, :published => true)
 @last = FactoryGirl.create(:post, :created_at => 4.day.ago, :published => false)
 end

 it "should only return published posts" do
 Post.published.should == [@first]
 end

 it "should return posts in the correct order" do
 Post.latest.should == [@first, @last]
 end
 end
end

This test is okay. It tests if the named scope does what it needs to do. And there’s also the problem. Scopes are part of ActiveRecord and are already extensively tested there. All we need to do is check if we configure the scope correctly.

What we need is a way to inspect what where and order rules are set for a particular scope and make sure those are the correct ones. We can then trust ActiveRecord to keep its promise to execute the proper SQL query for us.

Here’s another test that utilizes some Rails 3 methods you may not have heard of before.

describe Post do
 context 'scopes' do
 it "should only return published posts" do
 Post.published.where_values_hash.should == {:published => true}
 end

 it "should return posts in the correct order" do
 Post.latest.order_values.should == ["created_at DESC"]
 end
 end
end

The where_values_hash and order_values allow you to inspect what a scopes doing. By writing your test this way you achieve two import goals:

You test your code (instead of the ActiveRecord)
You don’t use the database, which is a significant speed boost

Happy testing to you all!

RSpec speed-up (24.6%) by tweaking ruby garbage collection

Sat, 24 Sep 2011 00:00:00 +0000

[Today I learned][1] that Ruby garbage collection can be of huge importance to performance. More precisely, if Ruby does a lot of garbage collection it may slow down your code. Running garbage collection only every 10 or 20 seconds when running specs may increase performance dramatically.

At this time specs for Ariejan.net take an average of 25.29s to run. This is not bad, but in my opinion faster specs are better. After tweaking ruby garbage collection I got my specs to run in 19.05s, a 24.6% speed increase! [1]: http://37signals.com/svn/posts/2742-the-road-to-faster-tests ~ So, how do you speed up your tests?

The key is that you don’t want ruby to decide when to do garbage collection for you. Instead, we’ll tell Ruby when to do this, and we’ll let it do it every 15 seconds.

To set this up for RSpec create a file spec/support/deferred_garbage_collection.rb:

class DeferredGarbageCollection

 DEFERRED_GC_THRESHOLD = (ENV['DEFER_GC'] || 15.0).to_f

 @@last_gc_run = Time.now

 def self.start
 GC.disable if DEFERRED_GC_THRESHOLD > 0
 end

 def self.reconsider
 if DEFERRED_GC_THRESHOLD > 0 && Time.now - @@last_gc_run >= DEFERRED_GC_THRESHOLD
 GC.enable
 GC.start
 GC.disable
 @@last_gc_run = Time.now
 end
 end
end

Next, add the following to your spec/spec_helper.rb so rspec will use our deferred garbage collection.

config.before(:all) do
 DeferredGarbageCollection.start
end

config.after(:all) do
 DeferredGarbageCollection.reconsider
end

Now, when you run rake spec you should see a nice speed increase. Try to alter the threshold value a bit to see what gives your best performance:

DEFER_GC=20 rake spec

Enjoy!

Capistrano and the custom maintenance page

Mon, 19 Sep 2011 00:00:00 +0000

Randuin posted a comment on my previous Lighting fast, zero-downtime deployments with git, capistrano, nginx and Unicorn post asking how I handle database migrations. This is a good question.

Database migrations, especially with large datasets, take a long time to run. They also lock your database tables which may cause all kinds of trouble.

There’s a quite an easy solution for this, offered to us by Capistrano. Unfortunately it will cause downtime for your site while the migration is running.

Setup nginx

The first thing you should do it update your Nginx configuration in /etc/nginx/sites-available/default and add the following snippet, just before the location /:

if (-f $document_root/system/maintenance.html) {
 return 503;
}

error_page 503 @maintenance;
location @maintenance {
 rewrite ^(.*)$ /system/maintenance.html last;
 break;
}

As iGEL and Anlek Consulting pointed out in the comments, it’s good practice to send a 503 Service Temporarily Unavailable HTTP code back to the client. A normal user won’t notice the difference, but spiders do. Sending the 503 code will tell search engines, like Google, that your site is not available and that they should not re-index your maintenance message as the new site content. Instead, they’ll come back later when your site returns a HTTP 200 code again.

You can try this fairly easily by putting your site in maintenance mode (per the instructions that follow) and do a HEAD request:

$ curl -I http://ariejan.net
HTTP/1.1 503 Service Temporarily Unavailable
Server: nginx/0.8.54
Date: Tue, 20 Sep 2011 18:22:35 GMT
Content-Type: text/html
Content-Length: 1276
Connection: keep-alive

Use Capistrano

Now, you can run (from your own machine):

cap deploy:web:disable

This task will upload a generic placeholder and place it in public/system/maintenance.html. If nginx sees this file exists, it will render it and abort further processing. So, as long as the maintenance.html file is present, your app is not accessible.

When you’re migrations are done, you can run:

cap deploy:web:enable

This will have remove the maintenance.html file, and thus making your app accessible again.

Piecing it all together

What you probably want is a separate task to deploy new code and run migrations. Here’s the task I used:

namespace :deploy do
 desc "Deploy and migrate the database - this will cause downtime during migrations"
 task :migrations do
 transaction do
 update_code
 web:disable
 migrate
 web:enable
 end
 restart
 end
end

Customize your maintenance page

Of course you want to customize the maintenance page, because, frankly, it’s kind of ugly by default. This does require you write your own deploy:web:disable task:

namespace :deploy do
 namespace :web do
 task :disable, :roles => :web, :except => { :no_release => true } do
 require 'erb'
 on_rollback { run "rm #{shared_path}/system/maintenance.html" }

 reason = ENV['REASON']
 deadline = ENV['UNTIL']

 template = File.read("./app/views/layouts/maintenance.html.erb")
 result = ERB.new(template).result(binding)

 put result, "#{shared_path}/system/maintenance.html", :mode => 0644
 end
 end
end

Now you can create a maintenance page for your app at app/views/layouts/maintentance.html.erb. See the original maintenance template for inspiration.

Note that you can use the environment variables REASON and UNTIL to give some info about why the app is down.

Keep in mind that when you put up the maintenance.html none of your files is accessible. This includes stylesheets and js files. Make sure to host any images and other assets on a remote server like Amazon S3.

Lighting fast, zero-downtime deployments with git, capistrano, nginx and Unicorn

Wed, 14 Sep 2011 00:00:00 +0000

Everyone who has ever developed a web app has had to deploy it. Back in the day you simply uploaded your files with FTP and everything would be good. Today we have to clone git repositories, restart servers, set permissions, create symlinks to our configuration files, clean out caches and what not.

Doctor, what’s wrong?

In my opinion there are two critical problems with deployments today:

They are slow
They cause downtime

Both topics have been discussed by the likes of Twitter and Github. They have optimized their deployment process to allow for fast and continuous deployments. But, you are probably stuck with a default Capistrano install. As it turns out, with a little work, it’s quite easy to setup the same deployment process as Github and Twitter use.

For Ariejan.net, I’ve managed to get zero-downtime deployments that run in under 10 seconds. Yes, you read that right. ~

Let’s go!

This guide will help you setup your server and Rails 3.1 project for fast, zero-downtime deployments. I’ll be using Nginx+Unicorn to serve the application, git+capistrano for fast deployments. ~

The shopping list

Here’s a list of ingredients you’ll need:

A recent Ubuntu server (I used 11.04 Netty)
Your Rails 3.1 app
A remote git repository that contains your app

Assumptions

I’m making some assumptions about your app:

Ruby 1.9.2
Rails 3.1 app using Postgres named my_site
You want to use RVM and Bundler

Setting up your server

There are a few things you need to setup before diving in. The first bit is run under the root user.

Here’s the full apt-get command list I used.

apt-get update
apt-get upgrade -y
apt-get install build-essential ruby-full libmagickcore-dev imagemagick libxml2-dev \
 libxslt1-dev git-core postgresql postgresql-client postgresql-server-dev-8.4 nginx curl
apt-get build-dep ruby1.9.1

You’ll also need a separate user account to run your app. Believe me, you don’t want to run your app as root. I call my user deployer:

useradd -m -g staff -s /bin/bash deployer
passwd deployer

To allow deployer to execute commands with super-user privileges, add the following to /etc/sudoers. This required deployer to enter his password before allowing super-user access.

# /etc/sudoers
%staff ALL=(ALL) ALL

Ruby and RVM

With that done, you’re ready to install rvm, I performed a system-wide install, so make sure you run this as root.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Next up install the required ruby, in this case ruby-1.9.2-p290 and rubygems:

rvm install ruby-1.9.2-p290
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
tar zxvf rubygems-1.8.10.tgz
cd rubygems-1.8.10
ruby setup.rb

Create a ~/.gemrc file, this sets some sane defaults for your production server:

# ~/.gemrc
---
:verbose: true
:bulk_threshold: 1000
install: --no-ri --no-rdoc --env-shebang
:sources:
- http://gemcutter.org
- http://gems.rubyforge.org/
- http://gems.github.com
:benchmark: false
:backtrace: false
update: --no-ri --no-rdoc --env-shebang
:update_sources: true

Also create this ~/.rvmrc file to auto-trust your .rvmrc project files:

# ~/.rvmrc
rvm_trust_rvmrcs_flag=1

Note: do this for both root and the deployer user to avoid confusion later on.

Because you’ll be running your app in production-mode all the time, add the following line to /etc/environment so you don’t have to repeat it with every Rails related command you use:

RAILS_ENV=production

Postgres

I know not everybody uses Postgres, but I do. I love it and it beats the living crap out of MySQL. If you use MySQL, you’ll know what to do. Here are instructions for setting up Postgres. First create the database and login as the postgres user:

sudo -u postgres createdb my_site
sudo -u postgres psql

Then execute the following SQL:

CREATE USER my_site WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE my_site TO my_site;

Nginx

Nginx is a great piece of Russian engineering. You’ll need some configuration though:

# /etc/nginx/sites-available/default
upstream my_site {
 # fail_timeout=0 means we always retry an upstream even if it failed
 # to return a good HTTP response (in case the Unicorn master nukes a
 # single worker for timing out).

 # for UNIX domain socket setups:
 server unix:/tmp/my_site.socket fail_timeout=0;
}

server {
 # if you're running multiple servers, instead of "default" you should
 # put your main domain name here
 listen 80 default;

 # you could put a list of other domain names this application answers
 server_name my_site.example.com;

 root /home/deployer/apps/my_site/current/public;
 access_log /var/log/nginx/my_site_access.log;
 rewrite_log on;

 location / {
 #all requests are sent to the UNIX socket
 proxy_pass http://my_site;
 proxy_redirect off;

 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

 client_max_body_size 10m;
 client_body_buffer_size 128k;

 proxy_connect_timeout 90;
 proxy_send_timeout 90;
 proxy_read_timeout 90;

 proxy_buffer_size 4k;
 proxy_buffers 4 32k;
 proxy_busy_buffers_size 64k;
 proxy_temp_file_write_size 64k;
 }

 # if the request is for a static resource, nginx should serve it directly
 # and add a far future expires header to it, making the browser
 # cache the resource and navigate faster over the website
 # this probably needs some work with Rails 3.1's asset pipe_line
 location ~ ^/(images|javascripts|stylesheets|system)/ {
 root /home/deployer/apps/my_site/current/public;
 expires max;
 break;
 }
}

All dandy! One more then:

# /etc/nginx/nginx.conf
user deployer staff;

# Change this depending on your hardware
worker_processes 4;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
 multi_accept on;
}

http {
 sendfile on;
 tcp_nopush on;
 tcp_nodelay off;
 # server_tokens off;

 # server_names_hash_bucket_size 64;
 # server_name_in_redirect off;

 include /etc/nginx/mime.types;
 default_type application/octet-stream;

 access_log /var/log/nginx/access.log;
 error_log /var/log/nginx/error.log;

 gzip on;
 gzip_disable "msie6";

 # gzip_vary on;
 gzip_proxied any;
 gzip_min_length 500;
 # gzip_comp_level 6;
 # gzip_buffers 16 8k;
 # gzip_http_version 1.1;
 gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

 ##
 # Virtual Host Configs
 ##

 include /etc/nginx/conf.d/*.conf;
 include /etc/nginx/sites-enabled/*;
}

Okay, that’s Nginx for you. You should start it now, although you’ll get a 500 or proxy error now:

/etc/init.d/nginx start

Unicorn

The next part involves setting up Capistrano and unicorn for your project. This is where the real magic will happen.

You’ll be doing cap deploy 99% of the time. This command needs to be fast. To accomplish this I want to utilize the power of git. Instead of having Capistrano juggle around a bunch of release directories, which is painfully slow, I want to use git to switch to the correct version of my app. This means I’ll have just one directory that is updated by git when it needs to be.

Let’s get started by adding some gems to your app. When done run bundle install.

 # Gemfile
 gem "unicorn"

 group :development do
 gem "capistrano"
 end

The next step is adding a configuration file for Unicorn in config/unicorn.rb:

# config/unicorn.rb
# Set environment to development unless something else is specified
env = ENV["RAILS_ENV"] || "development"

# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.
worker_processes 4

# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen "/tmp/my_site.socket", :backlog => 64

# Preload our app for more speed
preload_app true

# nuke workers after 30 seconds instead of 60 seconds (the default)
timeout 30

pid "/tmp/unicorn.my_site.pid"

# Production specific settings
if env == "production"
 # Help ensure your application will always spawn in the symlinked
 # "current" directory that Capistrano sets up.
 working_directory "/home/deployer/apps/my_site/current"

 # feel free to point this anywhere accessible on the filesystem
 user 'deployer', 'staff'
 shared_path = "/home/deployer/apps/my_site/shared"

 stderr_path "#{shared_path}/log/unicorn.stderr.log"
 stdout_path "#{shared_path}/log/unicorn.stdout.log"
end

before_fork do |server, worker|
 # the following is highly recomended for Rails + "preload_app true"
 # as there's no need for the master process to hold a connection
 if defined?(ActiveRecord::Base)
 ActiveRecord::Base.connection.disconnect!
 end

 # Before forking, kill the master process that belongs to the .oldbin PID.
 # This enables 0 downtime deploys.
 old_pid = "/tmp/unicorn.my_site.pid.oldbin"
 if File.exists?(old_pid) && server.pid != old_pid
 begin
 Process.kill("QUIT", File.read(old_pid).to_i)
 rescue Errno::ENOENT, Errno::ESRCH
 # someone else did our job for us
 end
 end
end

after_fork do |server, worker|
 # the following is *required* for Rails + "preload_app true",
 if defined?(ActiveRecord::Base)
 ActiveRecord::Base.establish_connection
 end

 # if preload_app is true, then you may also want to check and
 # restart any other shared sockets/descriptors such as Memcached,
 # and Redis. TokyoCabinet file handles are safe to reuse
 # between any number of forked children (assuming your kernel
 # correctly implements pread()/pwrite() system calls)
end

Okay, as you can see there’s some nice stuff in there to accomplish zero-downtime restarts. Let me tell you a bit more about that.

Unicorn starts as a master process and then spawns several workers (we configured four). When you send Unicorn the ‘USR2’ signal it will rename itself to master (old) and create a new master process. The old master will keep running.

Now, when the new master starts and forks a worker it checks the PID files of the new and old Unicorn masters. If those are different, the new master was started correctly. We can now send the old master the QUIT signal, shutting it down gracefully (e.g. let it handle open requests, but not new ones).

All the while, you have restarted your app, without taking it down: zero downtime!

Capistrano

Now for Capistrano, add the following to your Gemfile.

# Gemfile
group :development do
 gem "capistrano"
end

And generate the necessary Capistrano files.

capify .

Open up config/deploy.rb and replace it with the following.

This deploy script does all the usual, but the special part is where you reset the release paths to the current path, making the whole release directory unnecessary.

Also not that the update_code is overwritten to do a simple git fetch and git reset - this is very fast indeed!

# config/deploy.rb
require "bundler/capistrano"

set :scm, :git
set :repository, "git@codeplane.com:you/my_site.git"
set :branch, "origin/master"
set :migrate_target, :current
set :ssh_options, { :forward_agent => true }
set :rails_env, "production"
set :deploy_to, "/home/deployer/apps/my_site"
set :normalize_asset_timestamps, false

set :user, "deployer"
set :group, "staff"
set :use_sudo, false

role :web, "123.456.789.012"
role :app, "123.456.789.012"
role :db, "123.456.789.012", :primary => true

set(:latest_release) { fetch(:current_path) }
set(:release_path) { fetch(:current_path) }
set(:current_release) { fetch(:current_path) }

set(:current_revision) { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
set(:latest_revision) { capture("cd #{current_path}; git rev-parse --short HEAD").strip }
set(:previous_revision) { capture("cd #{current_path}; git rev-parse --short HEAD@{1}").strip }

default_environment["RAILS_ENV"] = 'production'

# Use our ruby-1.9.2-p290@my_site gemset
default_environment["PATH"] = "--"
default_environment["GEM_HOME"] = "--"
default_environment["GEM_PATH"] = "--"
default_environment["RUBY_VERSION"] = "ruby-1.9.2-p290"

default_run_options[:shell] = 'bash'

namespace :deploy do
 desc "Deploy your application"
 task :default do
 update
 restart
 end

 desc "Setup your git-based deployment app"
 task :setup, :except => { :no_release => true } do
 dirs = [deploy_to, shared_path]
 dirs += shared_children.map { |d| File.join(shared_path, d) }
 run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"
 run "git clone #{repository} #{current_path}"
 end

 task :cold do
 update
 migrate
 end

 task :update do
 transaction do
 update_code
 end
 end

 desc "Update the deployed code."
 task :update_code, :except => { :no_release => true } do
 run "cd #{current_path}; git fetch origin; git reset --hard #{branch}"
 finalize_update
 end

 desc "Update the database (overwritten to avoid symlink)"
 task :migrations do
 transaction do
 update_code
 end
 migrate
 restart
 end

 task :finalize_update, :except => { :no_release => true } do
 run "chmod -R g+w #{latest_release}" if fetch(:group_writable, true)

 # mkdir -p is making sure that the directories are there for some SCM's that don't
 # save empty folders
 run <<-CMD
 rm -rf #{latest_release}/log #{latest_release}/public/system #{latest_release}/tmp/pids &&
 mkdir -p #{latest_release}/public &&
 mkdir -p #{latest_release}/tmp &&
 ln -s #{shared_path}/log #{latest_release}/log &&
 ln -s #{shared_path}/system #{latest_release}/public/system &&
 ln -s #{shared_path}/pids #{latest_release}/tmp/pids &&
 ln -sf #{shared_path}/database.yml #{latest_release}/config/database.yml
 CMD

 if fetch(:normalize_asset_timestamps, true)
 stamp = Time.now.utc.strftime("%Y%m%d%H%M.%S")
 asset_paths = fetch(:public_children, %w(images stylesheets javascripts)).map { |p| "#{latest_release}/public/#{p}" }.join(" ")
 run "find #{asset_paths} -exec touch -t #{stamp} {} ';'; true", :env => { "TZ" => "UTC" }
 end
 end

 desc "Zero-downtime restart of Unicorn"
 task :restart, :except => { :no_release => true } do
 run "kill -s USR2 `cat /tmp/unicorn.my_site.pid`"
 end

 desc "Start unicorn"
 task :start, :except => { :no_release => true } do
 run "cd #{current_path} ; bundle exec unicorn_rails -c config/unicorn.rb -D"
 end

 desc "Stop unicorn"
 task :stop, :except => { :no_release => true } do
 run "kill -s QUIT `cat /tmp/unicorn.my_site.pid`"
 end

 namespace :rollback do
 desc "Moves the repo back to the previous version of HEAD"
 task :repo, :except => { :no_release => true } do
 set :branch, "HEAD@{1}"
 deploy.default
 end

 desc "Rewrite reflog so HEAD@{1} will continue to point to at the next previous release."
 task :cleanup, :except => { :no_release => true } do
 run "cd #{current_path}; git reflog delete --rewrite HEAD@{1}; git reflog delete --rewrite HEAD@{1}"
 end

 desc "Rolls back to the previously deployed version."
 task :default do
 rollback.repo
 rollback.cleanup
 end
 end
end

def run_rake(cmd)
 run "cd #{current_path}; #{rake} #{cmd}"
end

Now there is one little thing you’ll need to do. I like to run my apps, even on the server, to use their own gemset. This keeps everything clean and isolated. Login to the deployer account and create your gemset. Next run rvm info and fill the PATH, GEM_HOME and GEM_PATH variables accordingly.

Don’t forget to install bundler in your new gemset

Database configuration

I always like to keep the database configuration out of git. I’ll place it in the shared directory.

# /home/deployer/apps/my_site/shared/database.yml
production:
 adapter: postgresql
 encoding: unicode
 database: my_site_production
 pool: 5
 username: my_site
 password: password

First setup

Now setup your deployment like this:

cap deploy:setup

This will clone your repo and link your database.yml file. Optionally, you may want to run migrations or upload an SQL dump to get started quickly with your app.

Deployments

Whenever you have a new feature developed in a feature branch, this is the process of deploying it:

Merge feature_branch into master
Run your tests to make sure everything is dandy.
Push master
Run cap deploy

For Ariejan.net, step 4 takes less than 10 seconds. Unicorn is restarted with zero-downtime so users don’t even notice the site was updated.

What’s next?

You now have fast, zero-downtime deployments working for your app. There are still some things you should to (which I might cover in some later post):

Tweak the nginx and Unicorn settings (especially number of workers); Perform some tests and see what works best for your app/server combination.
Add caching to nginx (or add Varnish)
Enable some monitoring, maybe Monit
Crank up security a notch
Integrate other deployment tasks like whenever or prepare your assets

Git: checkout a single file from another commit or branch

Tue, 13 Sep 2011 00:00:00 +0000

I recently worked on a new feature in a separate branch. It didn’t work out well, so I branched master again and tried another solution. However, I needed a specific filesI committed in the first feature branch. To avoid placing those files back in my working copy by hand, I used git to checkout the specific file from the other branch.

git checkout feature_1 -- path/to/file/iwant

This will not checkout the feature_1 branch, but instead checkout the most recent version of path/to/file/iwant in the feature_1 branch. Very handy indeed!

Git: remove, reset and rollback commits

Thu, 08 Sep 2011 00:00:00 +0000

We’ve all been there, you committed changes you now regret. If you didn’t share those commits with anyone yet, you’re safe. Let me show you how to remove commits from your local repository. I’ll also include an example how to roll back commits you already did share with others. ~ Use git log to see your most recent commits. Let’s say you want to revert the last three commits, you can run the following command:

git reset --hard HEAD~3

If you only want the last commit to be removed:

git reset --hard HEAD~1

HEAD~1 is a shorthand for the commit before head. But, it’s also possible to roll back to a specific commit using the SHA hash.

git reset --hard d3f1a8

Please note that all your uncommitted changes will be lost when you perform git reset --hard. You might want to use git stash to save your uncommitted changes.

In case you already pushed your changes to a remote repository, you can’t use git reset, because it will wreak havoc with other people’s repositories later. Instead, you could revert your commit (e.g. create a new commit, undoing a previous one).

Note that git revert does not walk back into history, but only works on a specific commit or range of commits. To use my previous examples:

git revert HEAD~3..HEAD
git revert HEAD~1..HEAD
git revert d3f1a8..master

Optionally specify the --no-commit option to see what’s being reverted.

Git Log: What did I do yesterday, exactly?

Wed, 24 Aug 2011 00:00:00 +0000

Sometimes you have to take your git repository’s log to see what you did the day before (ideal in preparation for the daily stand-up). What I want is a clean overview of each commit messages, their author and the time. The output result should be easily grep-able so I can filter stuff I don’t need out. ~ To do this, I use the following custom git log command:

git log --pretty=format:'%Cred%h%Creset - %C(yellow)%ae%Creset - %Cgreen%cd%Creset - %s%Creset' --abbrev-commit --date=iso

The result:

5d6ef1e - ariejan@ariejan.net - 2011-05-02 10:36:43 +0200 - Bumped to version 1.5.2
afede9e - ariejan@ariejan.net - 2011-05-02 10:35:53 +0200 - Fixed #29 - Sharing to facebook without a title now works properly.
d9985a1 - ariejan@ariejan.net - 2011-04-23 00:13:06 -0700 - Added Travis build status to README
3e1149c - ariejan@ariejan.net - 2011-04-22 13:44:21 +0200 - Bumped version to 1.5.1
fcab3bf - ariejan@ariejan.net - 2011-04-22 13:43:41 +0200 - Don't test on ruby 1.9.2 for now. See issue #27
93843ec - ariejan@ariejan.net - 2011-04-22 13:42:29 +0200 - Fixed issue #28 - Share-to-* double-encodes titles
ec56315 - ariejan@ariejan.net - 2011-04-22 12:43:41 +0200 - Fixed up utf-8 encoding for ruby 1.9.2
2146134 - ariejan@ariejan.net - 2011-04-22 12:39:37 +0200 - Bump Sinatra to 1.2.3
c5efbf4 - ariejan@ariejan.net - 2011-04-22 12:07:41 +0200 - Truncate long URLs in the back-end to maintain a correct layout.

Now, it’s easy to see what I did on april 22nd using grep

git log <snip> | grep 2011-04-22

You can also filter on email address to select only specific user etc. Of course, all this could be done with git, but I’m way more comfortable using grep.

To make life easy, you can create a shortcut for this log command. Add this to you ~/.gitconfig:

[alias]
 timelog = log --pretty=format:'%Cred%h%Creset - %C(yellow)%ae%Creset - %Cgreen%cd%Creset - %s%Creset' --abbrev-commit --date=iso

You can now use git timelog in any git repository.

Git: Squash your latests commits into one

Tue, 05 Jul 2011 00:00:00 +0000

With git it’s possible to squash previous commits into one. This is a great way to group certain changes together before sharing them with others. ~ Here’s how to squash some commits into one. Let’s say this is your current git log.

* df71a27 - (HEAD feature_x) Updated CSS for new elements (4 minutes ago)
* ba9dd9a - Added new elements to page design (15 minutes ago)
* f392171 - Added new feature X (1 day ago)
* d7322aa - (origin/feature_x) Proof of concept for feature X (3 days ago)

You have a branch feature_x here. You’ve already pushed d7322aa with the proof of concept of the new feature X. After that you’ve been working to add new element to the feature, including some changes in CSS. Now, you want to squash your last three commits in one to make your history look pretty.

The command to accomplish that is:

git rebase -i HEAD~3

This will open up your editor with the following:

pick f392171 Added new feature X
pick ba9dd9a Added new elements to page design
pick df71a27 Updated CSS for new elements

Now you can tell git what to do with each commit. Let’s keep the commit f392171, the one were we added our feature. We’ll squash the following two commits into the first one - leaving us with one clean commit with features X in it, including the added element and CSS.

Change your file to this:

pick f392171 Added new feature X
squash ba9dd9a Added new elements to page design
squash df71a27 Updated CSS for new elements

When done, save and quit your editor. Git will now squash the commits into one. All done!

Note: do not squash commits that you’ve already shared with others. You’re changing history and it will cause trouble for others.

Your Mac slow? Disable Spotlight in Snow Leopard

Mon, 27 Jun 2011 00:00:00 +0000

For some time now I’ve experience my mac to be very slow. Opening Vim would take minutes. Creating a new Tab in iTerm would take more than 20 seconds. What the hell is going on - this is a brand new MacBook Pro! ~ After investigating, using Activity Monitor I discovered the following:

CPU’s idling at 3-5% usage
+500M free RAM
+200G free disk space
Negligible amount of network traffic
Very high amount of disk I/O writes (> 450w/s)

What’s causing disk I/O?

So, something is using my disk. But what? The solution is to use the iotop utility:

sudo iotop -C 5 12

A common entry here is the mds procoess, which has an insane amount of BYTES. So, this mds process is causing a lot of I/O, causing things to get slow.

What’s the `mds` process do?

A quick Google search reveals that the mds process is actually the Spotlight indexer service.

Disable Spotlight

I don’t use Spotlight at all, so let’s disable it - preventing the disk I/O.

sudo mdutil -a -i off

That’s all. Spotlight indexing disabled. After a few seconds the disk I/O dropped from ± 450w/s to 0w/s. Vim starts ups again within a seconds. I’m happy.

Enabling Spotlight

If, for some obscure reason, you want to re-enable Spotlight, use the following command:

sudo mdutil -a -i on

Git: What files were changed since the last release?

Fri, 24 Jun 2011 00:00:00 +0000

Sometimes it handy to get a list out of git log that tells you which files were changed since your last release. It’s not straight forward, but very doable with the help of git log and grep. ~ Let’s say you want to view all the changed files since the last tagged release, v1.3.1:

git log --reverse --name-status HEAD...v1.3.1 | grep -e ^[MAD][[:space:]]

As you’re used to, this shows each files that Added, Modified or Deleted. This command does not squash file changes. So it’s possible for a file to first be added, the deleted, then added again and later modified. The --reverse option shows file changes historically, so the first file changed after the v1.3.1 release is shown first.

Crowd sourcing your BitCoin mining

Thu, 23 Jun 2011 00:00:00 +0000

[BitCoin][1] is a hip new currency using peer-to-peer networking to process transactions. You can either buy it from others, or mine it by solving math puzzles. Would it be possible to crowd source the mining process to visitors of your website? [1]: http://bitcoin.org ~

What’s BitCoin anyway?

BitCoin (BTC) is not distributed by a central bank, but is generated by its users. In order to mine BTC, your computer must solve math puzzles. When you solve the puzzle, you are awarded 50BTC. How difficult the puzzles are and how big the reward is, is all determined by the network itself - it cannot be influenced by a single entity or person.

So, let’s turn on that computer and start generating BitCoin! With a current exchange rate between $8 - $30 per BitCoin, gaining 50 BTC would be quite lucrative.

So, how long would it take to solve that math puzzle and score 50BTC, or $400 - $1500?

Given the current difficulty (how difficult is the puzzle to solve) of 877226 and a hash rate (the speed at which you’re trying to solve the puzzle) of 6 mHashes/s you can calculate how long it would take on average before you are rewarded 50BTC.

Using a calculator I can compute that with my current AMD Phenom II 1055T X6 (6 cores @ 2.8Ghz) it would take an average of about 20 years to generated those 50BTC. That’s 0.007 BTC a day, which translates to $0.056 - $0.21 a day. That doesn’t even cover the power bill for running your computer each day.

Can we do better than a 6 core CPU?

Yes, we can! My system is also equipped with an ATI Radeon HD 6870 video card. I love playing games, you know ;-). The thing is, a video card is basically a very large and very fast calculator. It can perform simple operations, like the hashing required for BitCoin mining, very vast. It can do 1120 hashes at the same time. Yes, that is in parallel.

Where the CPU could do 6 mHashes/s my GPU can do about 270 mHashes/s. Yes, my GPU is about 45 times faster at calculating hashes than my CPU.

But, what does that mean for the time it takes to get those 50BTC? It will take 161.5 days on average. Yielding about $2.48 - $15.50 per day. That’s more like it! Still, not worth leaving your day job for.

More power, please!

It’s possible to rig up a machine that uses even faster video cards to squeeze more mHashes/s out of it. Also, a system is not even limited to one video card, you can add up to two or four in the same computer! Imagine having four ATI Radeon 6870’s. That’s a whopping 1080 mHashes/s.

1080 mHashes/s gives you an average of 40 days and 9 hours, which translates to about $9.90 - $37.14 per day. You could be making $1100 a month with that (excluding the cost for purchase and power)

How about The Cloud?

You might wonder about the cloud. That big virtual pool of unlimited hardware. How about throwing some very big machines at this problem and get rich instantly.

Well, it won’t work. As we saw, CPUs are bad a generating those hashes and are thus not really a viable option for mining.

Amazon does offer an instance type that provides two Nvidia Tesla GPUs. Unfortunately Nvidia cards don’t perform well generating hashes. The cost of running one of those instances at about $50 a day. Because it performs worse than a single ATI Radeon HD 6870 you’ll only lose money with this approach.

Team up!

It’s also possible to team up with others to mine BitCoin faster. It’s been done before and the most famous example would be Seti@Home.

Users run a program in the time their CPU is idle, contributing in the analysis of signals from space. If enough people contribute their spare CPU time, you’ll get the processing power of multiple time the fastest super-computer in the world.

Great, so let’s do that for BitCoin! In the BitCoin world these are called pools. You can join a pool and try to solve the math puzzle together. When that happens, every participant gets a proportional share of the 50BTC for their work done.

Imagine there are a hundred people with an ATI Radeon HD 6870 (thus doing equal amounts of work). This would result in 2.7 gHashes/s. 50 BTC would be mined in about 1 day and 14 hours. The result per day would be about the same as a single person mining on his own.

The big difference is that when mining in a pool you get BitCoins on your balance much faster. After just a day an a half you have 0.5 BTC sitting in your account, yours to spend. A single person would have to wait 162 days for that to happen.

Use the Crowd, Luke!

Now, this got me thinking. What if you could harness the power of a thousands or hundreds of thousands of CPUs, even for a short period of time? How fast would mining be? Would it be profitable?

Then I noticed there is a website that offers a web miner. This is a Java applet that runs in the background of a web page. While a visitor is on your site, the applet will use spare CPU power from that users computer to generate hashes and effectively participate in a private pool.

I ran an experiment on Ariejan.net during june 2011. I have implemented the web miner and generated BitCoin for a month using my visitors as (mostly unknowingly) test subjects.

Ariejan.net gets between 55.000 - 65.000 page views each month. The average time per page is 40 seconds. This would yield me more than 700 hours of computing time each month.

Using the web miner I got between 60 and 100 mHashes/s for an entire month. Doing all the mining on my own it would take 1 year and 2 months to generate 50 BTC. That’s $0.75 - $2.84 per day on average.

Because that would be too long, I participated in another pool, contributing the immense power of my websites visitors. During june you generated 1.98 BTC. About $15.85 - $59.40 when exchanged right now.

In comparison, my ATI Radeon HD 6870 could have made between $75 and $280 in the same period of time.

This is a nice chunk in the monthly revenue stream generated by Ariejan.net, but it would not be enough to dedicate time on writing articles on a more regular basis.

Also, stealing a visitors CPU time is ethically debatable. Personally I think it would be okay to ask a visitor to opt-in. Running the miner secretly in the background as I did for this test would not be acceptable.

Conclusion

If you’re in the business of mining BitCoin, crowd sourcing most likely is not an option for you. You probably don’t have a web site big enough to generate enough CPU power. Using commodity hardware would yield the same or better results.

I would be interested to see others continue this research. What would be the results:

if you asked users to opt-in only
if you asked users to opt-out only
on a site with 1+ million visitors
if a miner were integrated in a game or piece of software
if Google put a JavaScript miner on google.com that performed only 10 hashes/view.

Got an opinion, feel free to leave it below.

Vows and CoffeeScript

Fri, 10 Jun 2011 00:00:00 +0000

CoffeeScript is a really nice way to write JavaScript code. Combined with NodeJS you are empowered by a very fast platform to develop server-side applications. Of course, you want to test these apps as well. Vows is really great way to do this. Unfortunately it’s not straight forward (yet) to set up NodeJS + CoffeeScript + Vows.

~ First off, make sure you have CoffeeScript and Vows installed. Here I install them globally so you can use the coffee and vows command line utilities.

npm install -g coffee-script
npm install -g vows

Next up, in your product directory, create a directory named test. Here we’ll create (classic) example: division-test.coffee

vows = require 'vows'
assert = require 'assert'

vows
 .describe('Division by zero')
 .addBatch
 'when dividing a number by zero':
 topic: -> 42/ 0

 'we get Infinity': (topic) ->
 assert.equal topic, Infinity

 'but when dividing zero by zero':
 topic: -> 0 / 0

 'we get a value which':
 'is not a number': (topic) ->
 assert.isNaN topic

 'is not equal to itself': (topic) ->
 assert.notEqual topic, topic

 .export(module)

I’m not going to explain the intimate details of Vows here, suffice it to say that you calculate a value and store it in topic. Then you perform a set of expectations.

The magic part is the last line, .export(module). In all other examples you’ll see the last command is .run(). This run() command runs your vows immediately.

When using coffee-script, you don’t want to directly run you vows, but gather them all together, convert them to JavaScript and then have vows run them.

With the test/division-test.coffee saved, try running vows from your console. Here’s the output from vows --spec:

vows --spec
♢ Division by zero

 when dividing a number by zero
 ✓ we get Infinity
 but when dividing zero by zero we get a value which
 ✓ is not a number
 ✓ is not equal to itself

✓ OK » 3 honored (0.002s)

Another great tip is vows -w. This will keep vows running and monitor your test files for changes. When a file changes, it will re-run your vows for you.

Happy testing!

Yes, I know there are issues with Coderay parsing UTF-8 characters.

Rake with namespaces and default tasks

Mon, 04 Apr 2011 00:00:00 +0000

Rake is an awesome tool to automate tasks for your Ruby (or Rails) application. In this short article I’ll show you how to use namespaces and set default tasks for them. ~ Let me first tell you what I want to accomplish. I have a Rails application that needs to be cleaned up occasionally. Essetially we delete old data from the database.

There are several models that each implement a cleanup! method which takes care of cleaning up data. Now all I need is a few rake tasks to kick off the clean up process.

This is what I’d like:

rake cleanup # Cleanup everything
rake cleanup:clicks # Aggregate click stats
rake cleanup:logs # Clean old logs

Here’s what I put in lib/tasks/cleanup.rake:

namespace :cleanup do
 desc "Aggregate click stats"
 task :clicks => :environment do
 Click.cleanup!
 end

 desc "Clean old logs"
 task :logs => :environment do
 Log.cleanup!
 end

 task :all => [:clicks, :logs]
end

desc "Cleanup everything"
task :cleanup => 'cleanup:all'

Notice that the cleanup:all task does not have a description. Without it, it won’t show up when you do a rake -T to view available tasks.

The task cleanup has the same name as the namespace and calls the (undocumentend) cleanup:all task, kicking off the entire cleanup process.

Rails 3 + Devise + Uploadify = No Flash Session Hacks

Sun, 27 Mar 2011 00:00:00 +0000

Uploadify is a great project to provide file uploads in your project. The problem is, it’s written in flash.

Besides the point that it is flash, there’s something else that has been bothering me a lot: sessions. ~ The problem is like this. When a browsers opens a connection to your Rails app, it has a session. Normally, session information is stored a cookie that is sent with every request. This session also contains information needed for you to stay logged in as a particular user.

If you embed Flash on your site and have it communicate with your Rails application, it will start a new session. Basically, Flash just acts as a whole other browser.

Because of this behaviour, when a logged in user uploads files with Uploadify, the uploads will appear to come from an unauthenticated user. There is no session information provided by flash to identify that a user is signed in.

The old solution

The old solution I’ve found around the web is to work around this session problem by forcing flash to send the browser’s session information. This solution also requires you to add a piece of rack middleware to your app that takes any requests from a flash client and hacks the session to look like it came from the browser.

Besides the fact that I couldn’t get this hack to work, it still is a big hack. So, what to do?

A word on devise

I happen to use Devise in my app to authenticate users. As you may know, Devise is very complete. It includes everything ranging from sign up, forgotten passwords and even account locking. It also features token authentication.

Token authentication means that a user can authenticate himself by providing a unique token, in most cases a string of random characters.

A common use case for this token authentication is to provide users with a secret link to an RSS feed, or to allow quick access to the application from link sent in an email. You click the link and you’re automagically logged in.

The new solution

So, this token authentication got me thinking. Instead of sending an encoded string with session information to flash, which in turn sends it to the server, which in turn hacks it into an actual session, I could just send the user’s authentication token along! No sever-side hacks required - it’s all built in into devise already!

Let me show you. First check that your (devise-powered) user has an authentitication token:

@user.authentication_token
=> "4R2bzzQRdoT_iz-ND4Bb"

In case your authentication_token is nil, you should generate one with @user.reset_authentication_token!

The next step is to tell Flash to use this authentication token in its request to the server (while uploading files). Nothing fancy here either. Not that this is a snippet from JavaScript, embedded in a HAML template:

$('#image_file').uploadify({
 // I omitted all other config options, since they're not relevant.
 'script' : '#{images_path(:auth_token => current_user.authentication_token, :format => :json)}'
)

Rails will generate a URL like this: /images.json?auth_token=4R2bzzQRdoT_iz-ND4Bb.

The final step is to protect your ImagesController#create action with devise.

class ImagesController < ApplicationController
 before_filter :authenticate_user!

 def create
 # Handle your upload
 end
end

That’s all. You dont’ even need to add rack middleware or hack Uploadify to allow an authenticated devise user to upload images through flash.

Easy, huh?

Narf: A Ruby Micro Test Framework

Fri, 11 Feb 2011 00:00:00 +0000

I’m a happy user of RSpec, Cucumber and sometimes Steak. Great tools to write specs and features and prove my application does what it’s supposed to do. But sometimes I have the need for something more light weight. ~ For example, sometimes I need to write a single ruby method. Just something ‘quick’ to import a file, convert some data or whatever. Being a good citizen I want to test that method.

Using RSpec or Cucumber here just seems wrong here. So, I’ve implemented my own Ruby Micro Test Framework: Narf

def assert(message, &block)
 puts "#{"%6s" % ((yield || false) ? ' PASS' : '! FAIL')} - #{message}"
end

(Yes, that’s it.)

With this simple method, added to your ruby file, you can test your method. And example:

Let’s say you’re going to write a method that counts how often the word ‘ruby’ occurs in a given String:

def count_rubies(text)
 # TODO
end

#--- Ruby Micro Test Framework
def assert(message, &block)
 puts "#{"%6s" % ((yield || false) ? ' PASS' : '! FAIL')} - #{message}"
end
#---

#--- Tests
assert "Count zero rubies" do
 count_rubies("this is Sparta!") == 0
end

assert "Count one ruby" do
 count_rubies("This is one ruby") == 1
end

assert "Count one RuBy" do
 count_rubies("This is one RuBy") == 1
end

Now, simple save this file and feed it to ruby:

$ ruby my_method.rb
! FAIL - Count zero rubies
! FAIL - Count one ruby
! FAIL - Count one RuBy

Now, implement your method…

def count_rubies(text)
 text.match(/(ruby)/i).size
end

And re-run your tests:

$ ruby my_method.rb
 PASS - Count zero rubies
 PASS - Count one ruby
 PASS - Count one RuBy

So with the addition of just a single method you can fully TDD/BDD your single method Ruby code. Pretty need, huh?

Need codez?

Using your Firefly URL Shortener with Twitter for iPhone

Mon, 07 Feb 2011 00:00:00 +0000

The Twitter for iPhone app (itunes) can be configured to use your very own Firefly URL shortener! Here’s how.

Actually, it’s quite simple. First, setup your own Firefly instance, possibly using Heroku.

When that’s done, simple open Twitter for iPhone, then follow this path:

Tap the three dots
Open “Accounts & Settings”
Tap the “Settings” button at the bottom
Open “Services”
Choose “URL Shortening”
Then pick “Custom…”

All right, you’re now ready to enter your magic URL. Simply replace the YOUR_DOMAIN and YOUR_API_KEY placeholders with your actual domain name and API key (you can find them in your config.ru)

http://YOUR_DOMAIN/api/add?api_key=YOUR_API_KEY&url=%@

Then hit “Save” and you’re set! Twitter for iPhone will now automatically use your Firefly instance to shorten URLs!

Looking for the Firefly source code?

Hot: Firefly 1.3.0 URL Shortener released

Tue, 01 Feb 2011 00:00:00 +0000

I’ve been doing some work on Firefly lately and tonight I’ve released version 1.3.0. If you’re using Firefly it’s recommended you upgrade to the latest and greatest version. ~ For those who missed it: Firefly is a simple URL shortener gem that can be used to host your own personal (or corporate) URL shortner on your own domain. Since Firefly is rack-based, it’s easy to deploy to Heroku or other ruby hosting providers.

What’s new

Since version 1.1.0 (released october 2010) these features have been added. I’ll just summarize those and ignore the 1.2.x release I made.

Make Barby optional to save some gems for those of us who don’t need QR code generation. [mboeh]
Added a CodeFactory to generate unique short codes (and prevent collisions with custom short codes) [ariejan]
Support for specifying one’s own preferred short codes via the API. Also, invalid URL creations return a 422 code, not 200. [mboeh]
Added GUI support for custom short codes. [ariejan]
Improved error handling / reporting [ariejan]
Minor tweaks to the web GUI [ariejan]
Use dm-mysql-adapater as default, dm-sqlite-adapter is optional [ariejan]

How to upgrade

IMPORTANT: Always make a backup of your data before you upgrade.

Simple install the latest gem version and restart your server.

gem install firefly -v1.3.0

Then restart your server. DataMapper will take care of migrating your database for you.

NOTE: Firefly is now using MySQL by default. If you were using sqlite3 before, please install the appropriate DataMapper adapater: gem install dm-sqlite-adapter

Code, Issues, Forking

Want to fork? http://github.com/ariejan/firefly

Rake task to sync your assets to Amazon S3/Cloudfront

Sat, 01 Jan 2011 00:00:00 +0000

With my move to Heroku I felt bad about having Heroku’s app servers serve static content for me. It’s not really a problem, but I just like to use the best tool available for the job.

Because Ariejan.net is a rack app, it has a public directory with all static assets in once place. There are, however, a few problems that need adressing. ~

These are the problems I want to resolve:

Keep my S3 Bucket in sync with my public directory

The first and foremost is to keep my S3 bucket in sync with the content of public. I don’t care about file deletions, but I do care about new and updated files. Those should be synced with every deployment to S3.

Don’t re-upload the entire public directory with every deployment

Over time the size of public has grown. New images are added all the time. I don’t want to re-upload them with every deployment. So, my sync script must be smart enough to not upload unchanged files.

Hook the S3 sync into my current deployment rake task

My current rake deploy task should be able to call assets:deploy or something to trigger an asset sync.

Minimal configuration

I don’t want to configure anything, if possible.

The script

Well, this is the rake task I currently use:

require 's3'
require 'digest/md5'
require 'mime/types'

## These are some constants to keep track of my S3 credentials and
## bucket name. Nothing fancy here.

AWS_ACCESS_KEY_ID = "xxxxx"
AWS_SECRET_ACCESS_KEY = "yyyyy"
AWS_BUCKET = "my_bucket"


## This defines the rake task `assets:deploy`.
namespace :assets do
 desc "Deploy all assets in public/**/* to S3/Cloudfront"
 task :deploy, :env, :branch do |t, args|

## Minify all CSS files
 Rake::Task[:minify].execute

## Use the `s3` gem to connect my bucket
 puts "== Uploading assets to S3/Cloudfront"

 service = S3::Service.new(
 :access_key_id => AWS_ACCESS_KEY_ID,
 :secret_access_key => AWS_SECRET_ACCESS_KEY)
 bucket = service.buckets.find(AWS_BUCKET)

## Needed to show progress
 STDOUT.sync = true

## Find all files (recursively) in ./public and process them.
 Dir.glob("public/**/*").each do |file|

## Only upload files, we're not interested in directories
 if File.file?(file)

## Slash 'public/' from the filename for use on S3
 remote_file = file.gsub("public/", "")

## Try to find the remote_file, an error is thrown when no
## such file can be found, that's okay.
 begin
 obj = bucket.objects.find_first(remote_file)
 rescue
 obj = nil
 end

## If the object does not exist, or if the MD5 Hash / etag of the
## file has changed, upload it.
 if !obj || (obj.etag != Digest::MD5.hexdigest(File.read(file)))
 print "U"

## Simply create a new object, write the content and set the proper
## mime-type. `obj.save` will upload and store the file to S3.
 obj = bucket.objects.build(remote_file)
 obj.content = open(file)
 obj.content_type = MIME::Types.type_for(file).to_s
 obj.save
 else
 print "."
 end
 end
 end
 STDOUT.sync = false # Done with progress output.

 puts
 puts "== Done syncing assets"
 end
end

This rake task is hooked into my rake deploy:production script and generates the following output (I added a new file just to show you what happens.)

 $ rake deploy:production
 (in /Users/ariejan/Code/Sites/ariejannet)
 Deploying master to production
 == Minifying CSS
 == Done
 == Uploading assets to S3/Cloudfront
 ......................................U.........
 == Done syncing assets

 Updating ariejannet-production with branch master
 Counting objects: 40, done.
 Delta compression using up to 4 threads.
 Compressing objects: 100% (27/27), done.
 Writing objects: 100% (30/30), 4.24 KiB, done.
 Total 30 (delta 17), reused 0 (delta 0)

 -----> Heroku receiving push

Conclusion

It’s very easy to write your own S3 sync script. My version has still has some issues/missing features that I may or may not add at some later time. There’s no support for file deletions and error handling is very poor at this time. Also, public is still under version control (where I want it), and is pushed to Heroku. This is non-sense, because most of the assets in public are not used (except robots.txt and favicon.ico)

Now powered by Heroku

Fri, 31 Dec 2010 00:00:00 +0000

This is just a quick note to let everyone know Ariejan.net is now proudly hosted by Heroku. Since this site is powered by a toto (a rack-based blog engine) and no database, Heroku is the perfect hosting solution.

All articles are simple text files (with some YAML front-matter to specify title, tags, etc.) and are kept under Git version control. Deploying Ariejan.net is a simple git push to Heroku.

Want to run your own toto-powered blog on Heroku? Checkout Dorothy (code) to get started in “10 seconds”.

Public Readable Amazon S3 Bucket Policy

Fri, 24 Dec 2010 00:00:00 +0000

Amazon S3 allows you to set per-file permissions to grant read and/or write access. This is nice, but sometimes you just want to share your whole bucket with the world.

Luckily, Amazon features bucket policies, which allow you to define permissions for an entire bucket. ~ This example will give read access to Everyone on all files in your bucket.

{
 "Version":"2008-10-17",
 "Statement":[{
 "Sid":"AllowPublicRead",
 "Effect":"Allow",
 "Principal": {
 "AWS": "*"
 },
 "Action":["s3:GetObject"],
 "Resource":["arn:aws:s3:::bucket/*"
 ]
 }
 ]
}

Make sure you replace bucket in arn:aws:s3:::bucket/* with your bucket name.

After setting this bucket policy (see ‘Bucket -> Properties -> Add Bucket Policy’), all your files will be publicly readable.

Why did error_messages_for disappear from Rails 3?

Wed, 15 Dec 2010 00:00:00 +0000

Today I learned that error_messages_for has disappear from Rails 3. When I tried using it I got the following deprecation warning:

DEPRECATION WARNING: form.error_messages was removed from Rails and is now available as a plugin.

What happened? Why was this pulled from Rails 3? ~ The reason error_messages_for was pulled from Rails 3 is a new guideline that says that nothing in Rails Core should dictate the look and feel of an app.

Okay, that’s all well and good. How do you fix this problem? There are two ways. The first is to implement your own error handling. A HAML snippet:

- if @post.errors.any?
 .errors
 %h2 There was a problem saving this post

 %ul
 - @post.errors.full_messages.each do |msg|
 %li= msg

This is by far the best way to go about it. However, if you have a current Rails 2 app you’re upgrading, writing your own error handling may be rather difficult or time consuming. In that case you can install a plugin that restores the original functionality of error_messages_for.

rails plugin install git://github.com/rails/dynamic_form.git

Just make sure to restart your server.

Hide 'Last login:' on bash login

Thu, 25 Nov 2010 00:00:00 +0000

Everytime I open a new Terminal on my Mac, I get a line like this:

Last login: Thu Nov 25 09:07:55 on ttys004

This annoys me. I don’t care when I last opened a local Terminal.

To hide this “Last login” message when logging in to bash you need to create a file in your homedirectory.

touch ~/.hushlogin

With this .hushlogin file in place you won’t see the “Last login” message and go directly to your prompt, where you want to be.

Clear your MySQL password

Tue, 26 Oct 2010 00:00:00 +0000

Most people need their MySQL database protected with at least a decent password. I agree, but in development this often causes conflicts - and I prefer to work with my MySQL datbase without all the password-hassle.

On Ubuntu I recently installed MySQL and set a password. Here’s how to remove that password so you can skip all the password stuff during development. ~ First, connect to MySQL and check what permissions are currently set:

$ mysql -u root -p
use mysql;
select Host, User, Password from user;

You’ll probably see three entries for the root user: localhost, 127.0.0.1 and your hostname like hostname. To clear the password for the root user issue the following query:

update user set password = '' where user = 'root';

Optionally, you may only want to reset the password for localhost:

update user set password = '' where user = 'root' and host = 'localhost';

Keep in mind that using no MySQL password is insecure. Always protected your MySQL database with at least a strong password in production.

Setup a PPTP VPN connection on Mac OS X Snow Leopard

Tue, 12 Oct 2010 00:00:00 +0000

In my previous tutorial I explained how to setup a PPTP based VPN server on Ubuntu. Now I’ll show you how to configure Mac OS X Snow Leopard to use that VPN connection.

~ Just watch the movie.

How to setup a PPTP VPN connection from Mac OS X 10.6 from Ariejan de Vroom on Vimeo.

Note: You may open port 1723 (TCP) on your router to allow you to connect to your VPN server from the Internets. In that case, use your public IP Address.

Setup a Ubuntu VPN server

Mon, 11 Oct 2010 00:00:00 +0000

I recently installed Ubuntu Linux on a home server (I hate that word, but it best describes what it is, so). Anyway, I’d like to be able to create a VPN network between my home server and my MacbookPro, which might be anywhere in the world.

This first part of the tutorial describes how to setup a VPN server in Ubuntu. ~ First, install the pptpd package. pptpd offers a PPTP-type VPN which is supported by Microsoft and other network vendors. This is also the easiest to setup.

sudo apt-get install pptpd

Next up, edit /etc/pptpd.conf with sudo vi /etc/pptp.conf. At the bottom add the following lines:

localip 192.168.1.10
remoteip 192.168.1.230-239

Here localip references the IP of my home server. The remoteip variable configures which IPs remote clients may use when the connect through VPN to my network. In this case I reserve 10 IP address: 192.168.1.230 through 192.168.1.239.

With that out of the way, let’s tell PPTP which users to allow. Edit /etc/ppp/chap-secrets, just like you did before using sudo.

# client server secret IP Address
ariejan pptpd somepassword *

That’s all! Yes, seriously. Just restart the pptpd daemon and you’re good to go.

Now, go and watch my PPTP VPN setup for Mac OS X Snow Leopard

Note: This was tested on Ubuntu 10.10-amd64

Firefly 1.1.0 adds QR Codes for your shortened URLs

Sat, 02 Oct 2010 00:00:00 +0000

I just pushed Firefly 1.1.0 (code) to Rubygems containing a very nice new feature: QR Codes.

Why would you care? Well, almost anything can scan QR Codes nowadays. Simple add .png to the end of your shortened URL and you’ll get a nice QR Code that you can print or embed on somewhere on the web. Give it a try: http://aj.gs/2Y.png

Simply scan the image and it will yield your short URL, ready for use.

Updating your current Firefly installation is easy, simply run gem install firefly -v1.1.0 and restart your server.

Precompile SASS to CSS for deployment to Heroku

Tue, 28 Sep 2010 00:00:00 +0000

If you have deployed apps to Heroku you know that you cannot write to the read-only file system that Heroku offers. For file uploads you have to use some storage provider like Amazon S3 or RackSpace CloudFiles.

Now, if your application (I’m assuming you’re already on Rails 3), is using Haml + Sass, you’re in some trouble. Sass is set to generate CSS files on the fly and save them to public/stylesheets so the can be served like static content. On Heroku, that is not possible.

Luckily, there are a few solutions to this problem. I’ll describe one of them. ~

Use the Force (of git)

In my solution I’m using the power of git to generate the necessary CSS files and commit them automatically. Here’s how it works.

In a normal situation the Sass plugin will compile the SASS or SCSS files in public/stylesheets/sass/*.scss and store the generated CSC files in public/stylesheets. What we need to do is generate those CSS files by hand and commit them just like other versioned files in our repository. To do this, you need to write a pre-commit hook for git. This sounds more difficult than it really is.

Here’s the pre-commit hook I’m currently using to take all public/stylesheets/sass/*.scss files and store the resulting CSS files in public/stylesheets.

#!/usr/bin/env ruby
Dir['public/stylesheets/**/*.scss'].each do |sass|
basename = sass.gsub(/public\/stylesheets\/sass\//, '').gsub(/\.scss$/, '')
next if basename.match(/^_/) # skip includes
css = "public/stylesheets/#{basename}.css"
puts "Compiling #{sass} -> #{css}"
system "sass #{sass} #{css}"
system "git add #{css}"
end

Store the above Ruby script in .git/hooks/pre-commit, then give is execute permissions with chmod 755 .git/hooks/pre-commit.

This script will find all *.scss files and save their *.css equivalents. Then it also stages those files for the commit. Since this is a pre-commit script, the scenario is like this:

You stage your files to commit, like usual and run the git commit command.
Before git makes the actual commit, it runs the pre-commit script, which generates the necessare CSS files. You’ll a message like Compiling public/stylesheets/sass/app.scss -> public/stylesheets/app.css.
With any changes to the CSS stages, your commit is made.

Won’t this spam a lot of CSS commits?

No, it won’t. Git is smart enough to see there are not changes to the content of the CSS file.

Does it work?

Yes, it works. The generated CSS files are deployed to Heroku like normal, static CSS files and will be served as such.

One more thing…

By default, Sass is set to generate CSS files when needed. Since the CSS file is already there Sass probably won’t try to generate it again in production. But, it might try so anyway and cause an exception.

To prevent Sass for generating CSS in production completely, add the following line to your config/environments/production.rb.

Sass::Plugin.options[:never_update] = true

Notes

There is one problem with this approach. If you have multiple developers or machines you work on, each and every one must have this pre-commit script installed to make it work. My advise would be to include the script in your project’s doc directory.

Mass convert WMA to MP3 using ffmpeg and ruby

Sat, 11 Sep 2010 00:00:00 +0000

Today I found myself in a situation where I have a few (200+) WMA audio files. Due to personal preference I want MP3, not WMA. So, let’s convert that lot.

What I want to do is convert all those WMA files to constant bitrate 192kbps, stereo mp3 files. (In my case, the WMA files have the required quality to use this settings). ~ The first tool you need is ffmpeg. If you’re on Mac, simple run brew install ffmpeg. The second tool is irb or _Interactive Ruby.

With ffmpeg setup and in your path, create a directory and stuff you WMA’s there. Then open irb and run the following Ruby command:

ext = ".wma"
Dir.glob("*#{ext}").each {|f| m = f.gsub(ext, '.mp3'); `ffmpeg -i '#{f}' -ab 192k -ac 2 -ar 44100 '#{m}'` }

When done, you’ll find the WMA files converted to MP3 (having the same filename, except for the extention).

Happy transcoding!

Note: If you need to add ID3 tags to your files, I can highly recommend MusicBrainz Picard. It can fingerprint your music and find the proper data online. It’s free!

Update: Added a separte ext variable, this works great for converting FLAC to MP3 as well! Just use ext = '.flac'.

Resque: how to requeue failed jobs

Mon, 23 Aug 2010 00:00:00 +0000

Today I found about 100k Resque jobs in the failed queue. Due to a small error in some user content, those jobs all failed. After fixing the problem, how do you reprocess all those jobs?

Option one: go to the resque-web backend and click retry about 100.000 times.

Option two: do some cool ruby commands. ~ Resque offers you direct access to the failed queue and also provides a method to requeue jobs. How easy can it be, right?

With Resque::Failure.all(0,1) you retrieve the first job from the failed queue. Resque::Failure.all is simply a ruby Array.

To requeue jobs, you can use Resque::Failure.requeue(index) where index corresponds to the index of the job in the Resque::Failure.all Array.

To requeue all jobs in the failed queue, you can simply run the following commands:

# Requeue all jobs in the failed queue
(Resque::Failure.count-1).downto(0).each { |i| Resque::Failure.requeue(i) }

# Clear the failed queue
Resque::Failure.clear

That’s all there is to it, really. Happy processing!

Rename a git branch

Mon, 09 Aug 2010 00:00:00 +0000

In git, branching is cheap and easy. You do it all the time (you’re not? Well, you should). Sometimes, though, you create a new feature branch, only to come to the conclusion later that the name you gave it does not cover the stuff you’ve been doing.

No problem for git! Renaming a branch is really easy:

git branch -m old_branch new_branch

That’s all.

Using multiple clipboards in Vim

Mon, 09 Aug 2010 00:00:00 +0000

On of the first things you learn when using Vim activly is that when you delete something using x or dd that content is actually cut and put on a clipboard. Later you can retrieve that content using the p or P commanands.

One thing that makes Vim more awesome than let’s say, uhm TextMate, is that it actually has more than one clipboard! Yeah! In fact, in Vim terminology those clipboards are called registers. ~

To view your current registers type :reg. You’ll notice that every register is prefixed with the symbol ".

How do you use these things? Let’s say you want to copy a line into a specific register: "kyy will yank the current line into register "k. If you later want to paste register "k you can do this: "kp. Easy, huh?

The beauty of registers is that they keep around until you explicitly copy something else into them. The normal clipboard or paste buffer is replaced everytime you cut or copy something, registers are not affected unless you specify to use them.

Now, go on, code more productively.

How to enable SSH Forwarding on Mac OS X Snow Leopard

Thu, 29 Jul 2010 00:00:00 +0000

The other day I was toying with Rubber to deploy a Rails3 app to Amazon EC2. I host the project code in a private Github repository, accessible only with my own SSH key.

In order to checkout your code an any EC2 instance you can do one of two things:

Copy your private SSH key to the instance - This sounds easy enough, but has serious security implications. You do not want to be sending out your private SSH key, do you? That leaves you with option 2.
Let SSH forward the authentication request to your local machine. This is call Forwarding and requires ssh-agent to be running on your system. You’re in luck, ssh-agent is started automatically on your mac.

Now, the problem is that in Leopard (10.5) SSH Forwarding was enabled by default. You guessed it, in Snow Leopard it has been disabled by default. So, it’s up to you to enable SSH Forwarding manually. Here goes:

Open Terminal.app.
sudo vi /etc/ssh_config
You will be asked for your password now. Feel free to use your preferred editor here.
Add the following two lines to the top of the file:
Host *
ForwardAgent yes
Save the file and exit your editor.

All right sparky, you now have enabled SSH Forwarding. Have fun!

PS. If you have enabled “Remote Login” under Sharing Preferences, make sure to stop and start that service to notify it of the changes you just made.

Screencast: Firefly URL shortener in less than 2.5 minutes

Mon, 12 Jul 2010 00:00:00 +0000

Firefly is a light-weight URL Shortener app. It’s written in Ruby/Sinatra and can be easily deployed to Heroku, as I’ve written before. This screencast illustrates how easy it really is to setup Firefly and give it a spin. Did I mention that both Firefly and Heroku can be used for free?

Firefly URL shortener in less than 2.5 minutes from Ariejan de Vroom on Vimeo.

~ As always I’m keen to hear how you are using Firefly. Got some coding talents? Feel free to contribute features and fixes through github.

Cherry-Picking specific commits from another branch

Thu, 10 Jun 2010 00:00:00 +0000

I’m often asked how to merge only specific commits from another branch into the current one. The reason you’d want to do this is to merge specific changes you need now, leaving other code changes you’re not interested in right now behind.

First of all, use git log or the awesome GitX tool to see exactly which commit you want to pick. An example:

dd2e86 - 946992 - 9143a9 - a6fd86 - 5a6057 [master]
 \
 76cada - 62ecb3 - b886a0 [feature]

Let’s say you’ve written some code in commit 62ecb3 of the feature branch that is very important right now. It may contain a bug fix or code that other people need to have access to now. Whatever the reason, you want to have commit 62ecb3 in the master branch right now, but not the other code you’ve written in the feature branch.

Here comes git cherry-pick. In this case, 62ecb3 is the cherry and you want to pick it!

git checkout master
git cherry-pick 62ecb3

That’s all. 62ecb3 is now applied to the master branch and commited (as a new commit) in master. cherry-pick behaves just like merge. If git can’t apply the changes (e.g. you get merge conflicts), git leaves you to resolve the conflicts manually and make the commit yourself.

Cherry picking a range of commits

In some cases picking one single commit is not enough. You need, let’s say three consecutive commits. cherry-pick is not the right tool for this. rebase is. From the previous example, you’d want commit 76cada and 62ecb3 in master.

The flow is to first create a new branch from feature at the last commit you want, in this case 62ecb3.

git checkout -b newbranch 62ecb3

Next up, you rebase the newbranch commit --onto master. The 76cada^ indicates that you want to start from that specific commit.

git rebase --onto master 76cada^

The result is that commits 76cada through 62ecb3 are applied to master.

Uploading files with Curl

Mon, 07 Jun 2010 00:00:00 +0000

I’ve always trouble uploading files with Curl. Some how the syntax for that command won’t stick, so I post it here for future reference.

What I want to do is perform a normal POST, including a file and some other variables to a remote server. This is it:

curl -i -F name=test -F filedata=@localfile.jpg http://example.org/upload

You can add as many -F as you want. The -i option tells curl to show the response headers as well, which I find useful most of the time. ~

Firefly 0.4.3 and Firefly Client 0.4.0 released

Sun, 06 Jun 2010 00:00:00 +0000

Today version 0.4.3 of Firefly was released with some minor updates. To complete the package, a new gem firefly-client has been released.

The client library allows your Ruby application to easily shorten URLs with a remote Firefly server. It’s very easy to use and lightweight.

Firefly 0.4.3 Changelog

Handle invalid API keys correctly.
Added a fix for MySQL users to update the code column to use the correct collation. Fixes issue #9

Firefly Client

Using the Firefly Client is very easy, read the following snippet from the README:

require 'rubygems'
require 'firefly-client'

firefly = Firefly::Client.new("http://aj.gs", "my_api_key")
firefly.shorten("http://google.com")
 => "http://aj.gs/8ds"

Nice, huh? Get more info over at github

Setup your own Firefly URL shortener in 2.5 minutes

Sun, 06 Jun 2010 00:00:00 +0000

By popular demand I’ve setup a guide to setup your own personal URL shortener with Firefly and Heroku. I’ve timed it an you can do it in under 2.5 minutes. How’s that for fast?

I’m going to assume that you have git installed on your computer and that you have an already working Heroku account. If not, sign up here for Heroku first.

I’ll be referencing to a my_firefly_dir in this guide. You can just pick a name during step 2 and it will be created for you.

gem install heroku
git clone git://github.com/ariejan/firefly-heroku.git my_firefly_dir
cd my_firefly_dir
heroku create
Use the URL returned by heroku create and use the FQDN (xxxx.heroku.com) to configure the hostname in config.ru. Like: set :hostname, 'xxxx.heroku.com'
Change the API key in config.ru to something more secure.
Commit your changes with git commit -am 'Updated hostname and apikey'
git push heroku master
Done! Open your app with heroku open

So, there’s no excuse not to try Firefly now! Use the included bookmarklet to quickly shorten URLs from your browser or use the firefly-client gem to enable shortening in your own application!

Please, let me know if you did a succesfull install of Firefly and what you think about it.

Upgrading to Mongoid Beta 6

Sun, 30 May 2010 00:00:00 +0000

If you are working with Rails 3 and Mongoid, you’re likely to upgrade to [mongoid-2.0.0.beta6][1]. That’s okay, but you will run into a few problems. Among others, one will be:

Database should be a Mongo::DB, not NilClass

Mongoid::Errors::InvalidDatabase: Mongoid::Errors::InvalidDatabase

Another, Mongoid-related problem is the error uninitialized constant OrderedHash.

Luckily, these problems can be solved quite easily. [1]: http://rubygems.org/gems/mongoid

The first thing you need to do is make sure you use the right version of bson_ext. Beta 6 requires you to run bson_ext-1.0.1 or you’ll get the OrderedHash error. Okay, with that out of the way, let’s focus on the MongoDB errors.

The problem is that Mongoid is accessed/used before it is properly initialized. To resolve this issue, add the following line to your other requires at the top of config/application.rb.

require 'mongoid/railtie'

With that, you initialize Mongoid correctly. Hope it helps.

Update: I also ran into keys must be strings or symbols errors. This is now a known issue with mongoid-2.0.0.beta6 and has been fixed in master. If you are using Bundler (you are, aren’t you?) then you can use the master branch instead of the gem:

# gem 'mongoid', '2.0.0.beta6'
gem 'mongoid', :git => 'http://github.com/durran/mongoid.git'

Bundler + Passenger with Rails 2.3.5? Yes, please!

Mon, 17 May 2010 00:00:00 +0000

Bundler allows you to define the gems your application uses, resolve dependencies and load everything up. This is great, because you don’t have to manage all those different gem versions yourself any more.

There is a little problem, though. When you want to use Bundler with Rails 2.3.5. you need to do a bit of extra work. You’ll need to create a file config/preinitializer.rb that contains the following:

require "rubygems"
require "bundler"

if Gem::Version.new(Bundler::VERSION) <= Gem::Version.new("0.9.5")
 raise RuntimeError, "Your bundler version is too old." +
 "Run `gem install bundler` to upgrade."
 end

 # Set up load paths for all bundled gems
 Bundler.setup
rescue Bundler::GemNotFound
 raise RuntimeError, "Bundler couldn't find some gems." +
 "Did you run `bundle install`?"
end

Then you deploy your app with Capistrano (as root) and find that Passenger can’t find your gems. True, you need to install them, so you add a Capistrano task to run bundle install after you update your code. Still, passenger can’t find the gems.

The problem is that bundler installs the gems to your ~/.bundle. When you run bundler as root, passenger won’t be able to find the gems in /root/.bundle.

A solution is easy: bundle install .bundle will install the gems to ./.bundle, which should be your rails root directory. That solves your problem with passenger! Here’s the full Capistrano task:

desc "Install bundled gems into ./.bundle"
task :bundle do
 run "cd #{release_path}; bundle install .bundle"
end

Firefly 0.4.1 released

Fri, 30 Apr 2010 00:00:00 +0000

I just pushed Firefly 0.4.1 to Rubygems. Updating is easy:

gem update firefly

Don’t forget to restart your server, that’s all.

The 0.4.1 release covers the following changes:

Normalize URLs before shortening them. This prevents false duplicates.
Validate URLs to be valid HTTP or HTTPS, don’t accept others. Closes #8
Don’t ask for the API key after shortening a URL with the bookmarklet.
Show the highlighted URL separately. Closes #7.

If you are interested in contributing to Firefly, please fork the project on github. Pull requests are very welcome.

Ruby version and gemset in your Bash prompt? Yes sir!

Sun, 25 Apr 2010 00:00:00 +0000

RVM is an easy way to switch between different ruby implementations and gemsets. If you don’t know about it, go check it out. If you do know about, you’ll know how annoying it is never to know which ruby version and gemset you’re currently using. Here is a nice .profile hack that shows your current ruby version and optional gemset in your prompt.

Firstly, add the following function to your ~/.profile:

function rvm_version {
 local gemset=$(echo $GEM_HOME | awk -F'@' '{print $2}')
 [ "$gemset" != "" ] && gemset="@$gemset"
 local version=$(echo $MY_RUBY_HOME | awk -F'-' '{print $2}')
 [ "$version" != "" ] && version="@$version"
 local full="$version$gemset"
 [ "$full" != "" ] && echo "$full "
}

Next, you can use this function in your prompt. Like this:

export PS1="\$(rvm_version) \w \$(parse_git_branch) \$ "

The results? For standard ruby 1.8.7

@1.8.7 ~ $

Or with the rails3 gemset enabled:

@1.8.7@rails3 ~ $

So, now you always know which ruby you’re using! Happy coding!

A new day, a new Firefly

Wed, 14 Apr 2010 00:00:00 +0000

Get ready for 0.4 (or, 0.4.0.1 to be exact)! This release has some interesting new features. Read more to find out, I’ve included a screen shot for your pleasure.

Twitter integration
Easier copying of short URLs
Highlight the last shortened URL
Several fixes, see HISTORY for details

Coming from Digg? Check out the source and README at http://github.com/ariejan/firefly

The 0.4.0.1 release fixes file permissions for the twitter.png image so your server can read it properly.

Get ready for Firefly 0.3!

Tue, 13 Apr 2010 00:00:00 +0000

Hot off the press is Firefly 0.3! Firefly is a simple URL shortener application intended for personal use. It’s core features are:

Very light-weight - based on Sinatra
Shorten URLs using 62-Base encoding
Offers an easy to use API
Keeps track of URL clicks
Supports most popular database backends, including MySQL and Sqlite3
Includes a ready-to-use Bookmarklet
Works with any Rack capable web server

Interested? Give it a go! Here’s how in ht

1. Install Firefly

gem install firefly

2. Ready?

Create a new directory and place the following config.ru file in it:

require 'rubygems'
require 'firefly'

disable :run

app = Firefly::Server.new do
 set :hostname, "localhost:3000"
 set :api_key, "test"
 set :database, "sqlite3://#{Dir.pwd}/firefly.sqlite3"
end

run app

3. Go!

Start your engines! In this case I use thin:

thin start -R config.ru

Now, when you visit http://localhost:3000/ you’ll be asked for your API key to login. Then, start shortening!

Updating

If you were already running Firefly 0.2, upgrading is easy. Simple update the Firefly gem to 0.3 and restart your server.

When you update from 0.2 you’ll notice your click stats are all indicating 0. Don’t worry, no data was lost! The solution is quite simple.

Remove the clicks field from firefly_urls

ALTER TABLE `firefly_urls` DROP `clicks`;

Rename the visits field to clicks

ALTER TABLE `firefly_urls` CHANGE `visits` `clicks` int;

All done! No restart required.

Detect browser Web Sockets support

Mon, 05 Apr 2010 00:00:00 +0000

HTML5 Web Sockets are awesome! I’ve been toying around with it for a bit today and noticed that not every browser supports native HTML5 Web Sockets yet.

Google Chrome 5 has native support for web sockets, FireFox 3.6 does not. This poses a problem if you’re building something awesome that does require web sockets. ~

Luckily, it’s easy to detect web sockets support through JavaScript. All you really need to do is check if WebSocket is defined or not.

Here’s a simple example, note that I’m using jQuery here.

$(document).ready(function() {
 if( typeof(WebSocket) != "function" ) {
 $('body').html("<h1>Error</h1><p>Your browser does not support HTML5 Web Sockets. Try Google Chrome instead.</p>");
 }
});

Maybe there are better ways, but it works for me.

Announcing Firefly, a ruby URL shortener

Mon, 29 Mar 2010 00:00:00 +0000

Here it is! Firefly! The easiest URL shortener there is in Ruby land!

How easy? Install the gem, copy and paste a config.ru and you’re good to go!

Read the documentation or the source for more details. I’ve included the README in this post as well for your convenience.

FYI: I’m currently running aj.gs on Firefly 0.1 with MySQL.

FireFly

FireFly is a simple URL shortener for personal use.

Installation

sudo gem install firefly

After you have installed the Firefly gem you should create a config.ru file that tells your webserver what to do. Here is a sample config.ru:

require 'rubygems'
require 'firefly'

disable :run

app = Firefly::Server.new do
 set :hostname, "localhost:3000"
 set :api_key, "test"

 # Use Sqlite3 by default
 set :database, "sqlite3://#{Dir.pwd}/firefly.sqlite3"

 # You can use MySQL as well.
 # Make sure to install the do_mysql gem:
 # sudo gem install do_mysql
 # set :database, "mysql://root@localhost/firefly"
end

run app

Next you can start your web server. You may try thin:

thin start -R config.ru

Configuration

All configuration is done in config.ru.

:hostname sets the hostname to use for shortened URLs.
:api_key sets the API key. This key is required when posting new URLs
:database a database URI that DataMapper can understand.

It’s possible to use all kinds of backends with DataMapper. Sqlite3 and MySQL have been tested, but others like MongoDB may work as well.

Usage

Adding a URL is done by doing a simple POST request that includes the URL and your API key.

curl -d "url=http://ariejan.net" -d "api_key=test" http://localhost:3000/api/add

If you’re on a MacOSX you could add the following function to your ~/.profile to automate URL shortening:

shorten(){
 URL=$1
 SHORT_URL=`curl -s -d "url=$URL&api_key=test" http://localhost:3000/api/add`
 echo $SHORT_URL | pbcopy

 echo "-- $URL => $SHORT_URL"
 echo "Short URL copied to clipboard."
}

After you restart Terminal.app (or at least reload the .profile file) you can use it like this:

$ shorten http://ariejan.net
-- http://ariejan.net => http://aj.gs/1
Short URL copied to clipboard.

Bugs, Feature Requests, etc.

Feel free to fork Firefly and create patches for it. Here are some basic instructions:

Fork the project.
Make your feature addition or bug fix.
Add tests for it. This is important so I don’t break it in a future version unintentionally.
Commit, do not mess with Rakefile, VERSION, or HISTORY. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
Send me a pull request. Bonus points for topic branches.

License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Really? Another Sinatra URL Shortener in Ruby?

Sun, 28 Mar 2010 00:00:00 +0000

With my recent interest in Sinatra I decided to make some us of it and write a URL shortener service for with it. This also gave me an excuse to polish up my DataMapper skills a bit.

Now, it’s finished! I put the app on http://github.com/ariejan/firefly. Feel free to fork!

Not sure if it works? Try this: http://aj.gs/2

Questions? Please use the comments!

Installing the Nokogiri ruby gem on Debian

Thu, 25 Mar 2010 00:00:00 +0000

To install Nokogiri on a Debian system you need some system packages in place. This snippet will get you going quickly. ~ First, install the necessary debian packages if you don’t have them already:

apt-get install build-essential libxml2-dev libxslt1-dev

Then you can install nokogiri without any problem with gem install nokogiri

Ariejan.net now in valid HTML5

Wed, 24 Mar 2010 00:00:00 +0000

As most developers already know: HTML5 is very awesome! So, with my move from WordPress to Toto I’ve made sure Ariejan.net is HTML5 compliant.

I’m not going to tell you how I did it all, there’s enough about writing HTML5 on the web already.

Here’s the proof:

Or, if you don’t believe the screenshot, validate it yourself.

Note: The warning is about using the expiremental HTML5 Conformance Checker

How a little varnish changed my life

Wed, 24 Mar 2010 00:00:00 +0000

Okay, it’s a bit of an exaggeration to say varnish changed my life, but it sure did change the speed of my site!

I got from a rotten 6 requests per second with WordPress to a whopping 9500! If you’re on Linux and running Apache, installing varnish is a breeze! Especially if you’re hosting a well cacheable site like a blog.

I’m running on a Linux Debian virtual server (provided by Kabisa) with 1 VCPU core and 512MB of memory. Nothing fancy.

Before I was running Apache 2 with PHP and WordPress. Doing the occasional test this gave me (uncached) a performance of about 6 request/second (using apache’s ab with 10 concurrent connections). With some caching plugins I was able to crank that up to about 15-20 requests a second.

After upgrading my blog to Toto, I got quite a boost to about 30 requests/second. But, my blog doesn’t contain any dynamic elements any more and Toto + Rack give you all the handles to implement caching (ETags, Cache-Control headers, etc.). FYI: I’m running Apache2 + Passenger to run Toto.

Since I heard about Varnish a few times before I decided to give it a try and apt-get install varnish‘ed it on my Debian box (I’m running squeeze, thank you).

Now I have a few other sites running on my vps which I don’t want to cache just yet. The problem was how do I tell Varnish to only cache ariejan.net, and skip the rest.

Here’s the entire configuration for Varnish to accomplish just that:

backend default { .host = "127.0.0.1"; .port = "8080"; }
sub vcl_recv { if (req.http.host !~ "ariejan.net") { return(pass); } }

Yes, that is just two lines! What this does is forward everything you throw at varnish to the server at port 8080. The vcl_recv makes sure that if the hostname does not include ariejan.net varnish passes the request forward - no caching.

The second thing I had to do was configure Apache to listen on port 8080 instead of 80 in /etc/apache2/ports.conf. Then also make sure to have all your virtual hosts (even those you don’t want cached) configured for port 8080 too in /etc/apache2/sites-available

Restart apache, restart varnish and you’re golden!

When I first ran my ab benchmark with 10 concurrent connections I got to about 150 requests per second. But when I really pushed it to a 1000 concurrent connections (ab couldn’t handle more), I got to a whopping 9500 requests per seconds doing 60k requests! That is epic! Oh, and my VPS didn’t even break a sweat - system load got up to 1.08 for second or so.

For the record:

$ ab -c 1000 -n 60000 http://ariejan.net/2010/03/22/shields-up-rrrack-alert/

Server Software: Apache/2.2.15
Server Hostname: ariejan.net
Server Port: 80

Document Path: /2010/03/22/shields-up-rrrack-alert/
Document Length: 5117 bytes

Concurrency Level: 1000
Time taken for tests: 6.290 seconds
Complete requests: 60000
Failed requests: 0
Write errors: 0
Total transferred: 331434376 bytes
HTML transferred: 307460062 bytes
Requests per second: 9539.34 [#/sec] (mean)
Time per request: 104.829 [ms] (mean)
Time per request: 0.105 [ms] (mean, across all concurrent requests)
Transfer rate: 51459.38 [Kbytes/sec] received

Connection Times (ms)
 min mean[+/-sd] median max
Connect: 1 56 364.9 12 6209
Processing: 2 22 76.7 16 3073
Waiting: 2 19 76.6 13 3070
Total: 3 78 374.6 29 6223

Interested? Check out Varnish now or ask us at Kabisa to help you out!

Shields up! Rrrack alert!

Mon, 22 Mar 2010 00:00:00 +0000

For a very long time I have been dissatisfied with WordPress. Although it’s the best PHP blogging engine out there, it sucked.

Firstly, it’s slow. I got the front end a bit snappier by caching the living crap out of it, but that did not help things for me in the backend. So, what to do? ~

I’ve been looking towards ruby-based blogs a lot. Typo and Mephisto were two of the first ones out there, but they just didn’t feel right. I did try Mephisto for a while. However, the project didn’t appear to be under active development and I had a really hard time migrating my WordPress content.

Then came the first static site generators into my view. Jekyll being the obvious choice because of its affiliation with Github. But again, I had a hard time migrating my content (which consists of 192 blog posts, excluding this one).

Then along came Toto. Toto is epic! Without much effort I dumped my posts into files and I had Toto up and running! It worked! All my content was here!

There are two things that make Toto so great. Firstly all content is stored in files, so they are under version control and no database is needed. This also means that pages are served very quickly! Secondly it’s rack-based, which means I can deploy it easily to my VPS at Kabisa under passenger (and Unicorn when I migrate to that).

Well, so here I am! Still fiddling a bit with Toto (especially the HTML) as you can see. My goal is to make everything HTML5-compliant, but that’ll have to wait a few days.

Thinking about moving away from WordPress yourself? Feel free to comment or ask questions!

How to order your Kindle from the Netherlands

Tue, 02 Feb 2010 00:00:00 +0000

I get a lot of questions about how I bought my Kindle and what it cost to get it shipped to the Netherlands. So, for all those Dutchmen (and Dutchwomen) who are considering a Kindle, here’s a short how-to in Dutch:

Eindelijk heb je dan besloten dat je een eReader wilt en je keuze is gevallen op de Amazon Kindle. Goede keuze! Maar, hoe bestel je nu zo’n Kindle? Wat heb je nodig en waar moet je op letten.

Stap 1 - Shoppen bij Amazon

De eerste stap is dat je de Kindle (of Kindle DX) in je winkelwagentje laadt, eventueel samen met accessoires zoals deze cover.

De Kindle kost $259 (de DX gaat voor $489 over de virtuele toonbank). Helaas zijn er nog bijkomende kosten:

Verzenkosten – Amazon bracht bij mijn bestelling zo'n $21 dollar aan verzendkosten in rekening. Voor dit bedrag wordt je Kindle via UPS Express bezorgt.
Invoerrechten – Omdat je de Kindle van uit de V.S. importeert moet je hier invoerrechten over betalen. In het geval van de Kindle komt dit op een bedrag van $53.

De totale kosten van de bestelling bedragen ±$333. Afhankelijk van de wisselkoers komt dit uit op een bedrag van €240,-.

Stap 2 - Maken van een Amazon account

Als je de bestellig wilt gaan plaatsen zal Amazon je vragen in te loggen of een account te maken. Een account maken is snel en eenvoudig.

Stap 3 - Afleveradres

Amazon zal je vragen waar het apparaat afgelevert moet worden. Zorg er voor dat je voor ‘Netherlands’ kiest bij land en vul ter volledigheid bij ‘State’ je provincie in.

Stap 4 - Betaling

Om bij Amazon te betalen heb je een VISA of MasterCard creditcard nodig. Amazon biedt geen iDeal of bankoverschrijving aan, het is immers een Amerikaans bedrijf.

Heb je zelf geen creditcard, dan kun je misschien een deal maken met een vriend of kennis die wel een creditcard heeft. Amazon biedt je de mogelijkheid om de bestelling in dollars of euros te betalen. Het voordeel van het direct betalen in euros is dat je exact weet hoeveel je crediccard provider in rekening brengt.

Let op als je de creditcard van een vriend of kennis gebruikt! Je moet dan een apart factuuradres (billing address) opgeven dat klopt met de adresgegevens van de creditcard. Het afleveradres kan gewoon die van jezelf zijn.

Stap 5 - Afronden van de bestelling

Als je de bestelling plaats krijg je hiervan per email bevestiging van Amazon. Nu is het wachten tot UPS je Kindle komt afleveren.

Stap 6 - Levering

Zodra Amazon je Kindle verstuurt heeft krijg je hiervan een email. In deze mail staat de verwachtte lever datum (meestal ± 5 werkdagen) en een track-and-trace code van UPS. Met deze code kun je op ups.com zien waar je bestelling zich bevindt.

UPS bezorgt niet op zaterdag. Als je niet thuis bent zullen ze het pakketje bij de buren proberen te bezorgen, je krijgt hiervan een briefje in de brievenbus.

Stap 7 - Uitpakken en gebruiken

Als je je pakketje hebt uitgepakt kun je direct gaan lezen! Als je een boek koopt (via de Kindle of via je computer) dan wordt dat boek direct via de draadloze 3G verbinding naar je Kindle gestuurd!

Veel lees plezier!

Bonus tip!

Via project Gutenberg zijn duizenden boeken gratis te download! Op alle boeken die via deze weg beschikbaar zijn rust geen copyright meer, en zijn dus helemaal legaal. Zie hier de top 100 meest gedownloadde boeken.

Download altijd het Mobipocket formaat. Zet deze file in de documents directory op je Kindle. (Dit doe je door de Kindle met de USB kabel aan je computer aan te sluiten).

Mocht je via deze ‘guide’ een Kindle kopen, dan hoor ik graag (via de comments) hoe hij bevalt!

Amazon Kindle - $259

Amazon Kindle DX - $489

Sign the petition: Stop EU Software Patents

Tue, 19 Jan 2010 00:00:00 +0000

From stopsoftwarepatents.eu:

Our petition aims to unify the voices of concerned Europeans, associations and companies, and calls on our politicians in Europe to stop patents on software with legislative clarifications.

The patent system is misused to restrain competition for the economical benefit of a few but fails to promote innovation. A software market environment is better off with no patents on software at all. Healthy competition forces market players to innovate.

European court decisions still accept in many cases the validity of the software patents granted by national patent offices and the European Patent Office (EPO) that is beyond democratic control. They not only continue to grant them, but also to lobby in favor of them. Despite the current deep crisis of the patent system, they are unable to reform and put at risk too many European businesses with their soft granting policy.

On 2005 the Commission appeared to be more supportive to the interests of major international conglomerates than of small and medium sized enterprises from Europe - who are a major driving force behind European innovation. The European Parliament rejected at the end the software patent directive, but has no rights for legislative initiatives.

We urge our legislators

to pass national legal clarifications to substantive patent law to rule out any software patent;
to invalidate all granted claims on patents that can be infringed by software run on programmable apparatus;
to also strive to propagate these rules to the European level, including the European Patent Convention.

Sign the petition now!

The epic e-reading experience: Amazone Kindle

Sun, 17 Jan 2010 00:00:00 +0000

For some time I have been eyeballing Sony’s e-reader in the local bookstore. I tried it a few times, but I didn’t like it - actually I had serious doubts about e-books in general because of the experience. Sony’s e-reader was not really easy to use with only one next-page button in a not-so-easy to access place. It also had a slow e-ink screen. It took a second to a second-and-a-half to show the next page. I didn’t like it and had serious doubts about buying any e-reader at all.

But, reading about all the “epic win”-stories of the Kindle by @johnnybusca I just had to give it a try - I bought one.

Since receiving the Kindle last monday, I haven’t felt any regret! The Kindle easy beautiful, fast with screen updates and of course integrates neatly with Amazon’s Kindle Store. The whole product is designed to be easy to use - and it is.

I had already purchased The One That Got Away for Kindle for iPhone. Syncing it to my Kindle was done in about 30 seconds and I was ready to start reading!

For me the Kindle has three big advantages:

The Kindle is easier to hold with one hand that a paperback. With a "Next page" button on either side of the device, it doesn't matter if you hold it with your right or left hand!
I don't need to lug around paperbacks. I can always have my stack of novels with me. Also, Kindle takes up a lot less space than all my paperbacks do now!
The built-in dictionary is very easy to access while reading. This gives me, as a non-native English speaker, the great benefit of quickly looking up words I don't yet know and actually learn them while reading.

There is one little problem with Kindle, and that is the battery. You can use Kindle a long time on battery, even with wireless enabled, but eventually you’ll have to recharge. It’s not a huge task to hook up the Kindle to my Mac, but it might be a little troublesome on holidays where no computer is available. The US power connector is utterly useless in Europe.

To make a long story short. If you’re considering to buy an e-ready, go for Amazon’s Kindle. Don’t buy the Sony or any other product that might be cheaper, go for quality and ease of use! Now, enough with the blog writing, back to reading on my Kindle!

Epic TextMate Theme

Tue, 24 Nov 2009 00:00:00 +0000

Okay, I’ve wanted to make a custom TextMate theme since I first installed the thing on my MacBook in 2006. Today I present you with ‘Epic’.

Installation

Grab the theme here: EpicBlue.tmTheme.zip (1.5k). Unzip it and open it with TextMate. That’s all. Enjoy!

Samples

How to create and apply a patch with Git

Mon, 26 Oct 2009 00:00:00 +0000

Creating a patch file with git is quite easy to do, you just need to see how it’s done a few times.

This article will show you how to create a patch from the last few commits in your repository. Next, I’ll also show you how you can correctly apply this patch to another repository.

Before you start

To make creating patches easier, there are some common git practices you should follow. It’s not necessary, but it will make your life easier.

If you fix a bug or create a new feature – do it in a separate branch!

Let’s say you want to create a patch for my imdb gem. You should clone my repository and create a new branch for the fix you have in mind. In this sample we’ll do an imaginary fix for empty posters.

git clone git://github.com/ariejan/imdb.git
cd imdb
git checkout -b fix_empty_poster

Now, in the new fix_empty_poster branch you can hack whatever you need to fix. Write tests, update code etc. etc.

When you’re satisfied with all you changes, it’s time to create your patch. FYI: I’m assuming you made a few commits in the fix_empty_poster branch and did not yet merge it back in to the master branch.

Creating the patch

Okay, I’ve made some commits, here’s the git log for the fix_empty_poster branch:

git log --pretty=oneline -3
* ce30d1f - (fix_empty_poster) Added poster URL as part of cli output (7 minutes ago)
* 5998b80 - Added specs to test empty poster URL behaviour (12 minutes ago)
* aecb8cb - (REL-0.5.0, origin/master, origin/HEAD, master) Prepare release 0.5.0 (4 months ago)

In GitX it would look like this:

Okay, now it’s time to go and make a patch! All we really want are the two latest commits, stuff them in a file and send them to someone to apply them. But, since we created a separate branch, we don’t have to worry about commits at all!

git format-patch master --stdout > fix_empty_poster.patch

This will create a new file fix_empty_poster.patch with all changes from the current (fix_empty_poster) against master. Normally, git would create a separate patch file for each commit, but that’s not what we want. All we need is a single patch file.

Now, you have a patch for the fix you wrote. Send it to the maintainer of the project …

Applying the patch

… who will apply the patch you just sent! But, before you do that, there are some other steps you should take.

First, take a look at what changes are in the patch. You can do this easily with git apply

git apply --stat fix_empty_poster.patch

Note that this command does not apply the patch, but only shows you the stats about what it’ll do. After peeking into the patch file with your favorite editor, you can see what the actual changes are.

Next, you’re interested in how troublesome the patch is going to be. Git allows you to test the patch before you actually apply it.

git apply --check fix_empty_poster.patch

If you don’t get any errors, the patch can be applied cleanly. Otherwise you may see what trouble you’ll run into. To apply the patch, I’ll use git am instead of git apply. The reason for this is that git am allows you to sign off an applied patch. This may be useful for later reference.

git am --signoff < fix_empty_poster.patch
Applying: Added specs to test empty poster URL behaviour
Applying: Added poster URL as part of cli output

Okay, patches were applied cleanly and your master branch has been updated. Of course, run your tests again to make sure nothing got borked.

In you git log, you’ll find that the commit messages contain a “Signed-off-by” tag. This tag will be read by Github and others to provide useful info about how the commit ended up in the code.

That’s all folks!

Are there any other git topics you’d like covered here? Please let me know!

They are just tools, people!

Sun, 25 Oct 2009 00:00:00 +0000

Codaset is openly asking its users to comment on what pricing strategy they would like most. I’ve spotted this before, but again, there are two types of users. Those who see a great service and know that it will make their job easier, so they are willing to pay for it. There are also those who want a trillion repositories, unlimited disk space and what not for $1 a month (or less). This post is for the latter group of people.

Some developers claim they need to use all of 37Signals’ apps, have the biggest Github plan available and buy that new shiny 17" MacBook Pro (or that 27" iMac, I know). With all those tools and hardware available, how can your brilliant plan not succeed? All the successful people you’ve heard of use them. So, with all that setup, you’re golden! Right? Then they check the price tag. It’s huge!

In my opinion, the problem started with shared hosting. Shared hosting was the business to be in about 5-10 years back. Everyone wanted it and everybody wanted it cheap. So, what you got (and still have) are big, over-selling hosting companies. You get 5 Terabyte of storage, unlimited bandwidth and you can host unlimited websites! The price? Only $1 a month! A bargain!

Did you really think that there is a big server with 5 Terabyte of space is reserved for you? Or that you really have unlimited bandwidth? I can’t remember how many people I’ve heard bitching about them hosting MP3’s and then having their accounts shutdown for excessive bandwidth usage.

Those companies were over-selling like crazy. They basically sold every megabyte of disk space they had available 50 times or more! Of course, they attracted vast numbers of customers and business was booming. With the number of customers rising, the quality of services went down the drain. Too often have I heard people complaining about their over selling shared hosting provider. Shame.

But why do people choose for the $1 over selling package when for $7 they could have had better service and quality, albeit for a higher price? Simply because we like the feeling that we can grow big. Every one wants to be the new Google and store all his home-made (or other wise gathered) music online.

Luckily, most “web 2.0” service providers (which 37Signals and Github really are) have come to another idea. Sell less features for a reasonable, maintainable price. Yes, Basecamp has less features than the competition, but at least the price is fair and service is good.

So, here we are in 2009 and every small company can get good project management, code hosting, issue trackers, invoicing tools and what not for quite reasonable prices. Truly awesome!

But, here come the people that want to “grow big” and become the next Google. These are mostly young developers who don’t have a lot of business experience, and are still in school most of the time. They read and hear the great success stories and how those people used Basecamp and Github. Those must be magic tools!

And look, every one is using a Mac and TextMate! I’ll need those too!

No offense to Github, Basecamp, TextMate or Apple, but that is not how it works. Businesses are not created by using a certain set of tools. The reason a successful business turns to Basecamp is because they have a problem with their current project management. When you start freelancing and you only have one or two client, you probably don’t need the biggest Basecamp plan to get along. Oh, you are developing the next iPhone killer-app? Then you probably don’t need Basecamp at all.

The same goes for Github. When you’re working on a project that you believe in, it’s worth the $7 for the basic plan.

At Codaset I read some user comments and the story is the same. Some serious freelancers or companies explain what’s valuable to them and what they’d like to pay for it. All reasonable, because they know the cost of doing business.

Then, there are also some people who want it all for free. Why, because they really need Codaset and all its features.

People, sorry to burst your bubble, but they are just tools. It’s like buying a 112-piece screw driver kit when all you want to do is switch the occasional light bulb.

Here’s my advice. Start your project. Just on your old Mac (or Dell) and work on it for a few weeks. Use the tools available to you to do your time tracking and invoicing. Push your git repo that old Pentium 4 Dell you have gathering dust in the basement.

Then, when you find that communicating with your clients is getting difficult by phone and email only, then give Basecamp a try. When you feel that your code is getting so valuable that you don’t want to risk putting it only on an old machine in the basement, store it at Github.

All those web 2.0 apps are really just tools that you can use. Tools are there to solve a problem. If you don’t have a problem, you don’t need the tools.

Only use the tools that you really need, and when you do, they’ll be more than worth the price you pay for them.

Git problem: error: unable to create temporary sha1 filename

Thu, 15 Oct 2009 00:00:00 +0000

I got git problem: error: unable to create temporary sha1 filename when pushing to a remote repository. The fix is rather easy.

On both your local and remote repositories perform the following magic:

git fsck
git prune
git repack
git fsck

The last fsck should not report any problems.

Epic vs. Awesome

Tue, 13 Oct 2009 00:00:00 +0000

There’s bit of a discussion between me and @ludooo about which word has the most significance when measuring the greatness of something.

I say epic is bigger than awesome. @ludooo says awesome is bigger than epic (he’s not right, of course).

Please help us decide!

Valerii: 32-base string encoder and decoder

Tue, 13 Oct 2009 00:00:00 +0000

You have probably seen URL shorteners that use short, seemingly random strings to identify sites. These strings are not random, they are encoded integer values. My valerii gem allows you to easily and quickly encode and decode integer values. Let me show you.

To understand the shortening of integer values to strings you first need to know how integers work.

We use a base-10 system. This means that for any number, every digit can hold 10 different values: 0 through 9. If you want to express more than 10 values, you need to add a digit: 26 means (6 * 1) + (2 * 10).

You may be familiar with hexadecimal numbers. Hexadecimal has a base of 16, meaning every digit can hold 16 distinct values. 0 through 9, a through f.

Converting an integer to a 32-base string

The key to shortening a number to a string is that you need to store as many different values in a single digit. Because we want the digits to be part of an URL, we can only use valid URL characters. Characters like ‘/’ and ‘?’ are a no-go in this situation.

Another reason for sticking with 32 (and not 52 (a-z, A-Z, 0-9)) is that I’m going to use bit encoding. But, first we need to define an alphabet to use to represent the 32 different values:

ENCODE_CHARS =
 %w( B C D F G H J K
 M N P Q R S T V
 W Z b c d f h j
 k m n p r x t v )

For a value of 3 we’d use ‘F’. Easy, right?

Let’s say we want to encode the number 123. What valerii does first is convert 123 to a binary string.

123.to_s(2) # => "1111011"

Cool, now split up this string in blocks of 5 bits. 5 bits can contain 32 different values. We need to start ‘chopping’ from the right to the left, so first we reverse the binary string and split it up.

"1111011".reverse.scan(/.{1,5}/) # => ["11011", "11"]

Now we convert the binary strings to 10-base numbers and map those to the characters defined in ENCODE_CHARS. The resulting characters are reversed again and joined to one string:

123.to_s(2).reverse.scan(/.{1,5}/).map do |bits|
 ENCODE_CHARS[bits.reverse.to_i(2)]
end.reverse.join # => "Fp"

Converting a 32-base string to an integer

Converting a 32-base string back to its original integer value is quite easy now. The only trick is to create another hash that maps each character to its integer value:

DECODE_MAP = ENCODE_CHARS.to_enum(:each_with_index).inject({}) do |h,(c,i)|
 h[c] = i; h
end

This looks scary, but it’s actually a common way to reverse key/values in a hash.

Next is taking each character from the string and pushing the 5 bits we read in a variable. This variable is the original integer value.

"Fp".split(//).map { |char|
 DECODE_MAP[char] or return nil
}.inject(0) { |result,val| (result << 5) + val } # => 123

Notes

Once you have established your way of encoding/decoding you should not change the alphabet you’re using, since it redefines the meaning and value of the encoded strings.

There are a lot of ways to store an integer value into a string. Another common methods is to use a 52 characters alphabet (a-z, A-Z, 0-9). You can’t use the bit encoding here, but there are quite a few algorithms out there that let you do that.

More info on how to get the valerii gem can be found at Gemcutter.

Also checkout my other gems

Codaset.com: Github, but better

Tue, 08 Sep 2009 00:00:00 +0000

Today I had the very pleasure of giving Codaset.com a try! Codaset is being developed by Joel Moss in Ruby on Rails and could be a real Github killer!

When I asked Joel what Codaset was and how he come to his idea he answered this:

Codaset [...] was born out of my experiences with several tools and services.

I've used Trac for years, and admire its powerful issue management and milestone features.

I've used Github from the start, and love how social it makes open source software (as it should be).

I have fully embraced Git, and have no love for any other [...]. It makes for fast and efficient source control.

And I have used lots of other tools that help me in my day to day coding work.

Codaset is a tool that I created because I wanted the powerful issue and milestone management of Trac; with the social features of Github; mixed in with a heavy dose of Git […] but I have so much more to include.

Sounds like a brilliant app to me? I was invited into the beta, and this is what I saw:

First of all, Codaset looks very slick. I’m no designer, but Codaset adds just enough visual sugar to make the app more usable, without distracting you. It just feels right.

When compared to Github.com there are similarities and differences. Let me focus on those difference, since in my view those make Codaset really cool:

As you create a new project you have the option of forking an existing repository. Not just a Codaset repository, no any git repository you have access to! I had no trouble creating a new project an importing my existing github project for example.

The next thing I noticed when setting up a project is the public and private settings. No surprises there, but there was a third option: semi-private. This option will make your project closed-source, but expose the other features like the issue tracker, wiki and blog. It’s not something an open-source developer would get excited about, but it is quite nice to have this option.

If you’re looking to add collaborators to your project, Codaset provides a nice role/permission based system for this. It allows you to give more fine-grained permissions to users, which is always a good thing.

Codaset has, as Joel Moss tells you, great similarities with Trac when it comes to tickets. First you’ll notice that you can define milestones in your project. To these milestones you can add tickets which in turn can have customized fields and states. Ticket-spam is battled with a built-in spam filter for tickets. The only thing missing is tags, but Joel tells me those are in the making.

Just like github.com, codaset provides web space for you and your project. Setting this up is quite easy. Besides web space, every project has a blog as well. Hopefully this blog can be used to automatically post news about new tags in your git repository etc.

Another nice touch is your personal dashboard. This dashboard show you your own and bookmarked projects and recent activity, but also an overview of open and assigned tickets for you. Really handy!

The best thing ever is Twitter integration (which I have to try out), but it promises to push project activity to twitter, which would be a great way to keep people informed about your project.

The only things I’m missing are tags for tickets and rubygem building/publishing. But, Joel Moss is still very hard at work, so I’m looking forward all the new great things he has to offer.

For now, check out Codaset.com (and blog) or sign-up for a beta invite. Also make sure to checkout Joel Moss’ blog.

I haven’t published any images or screenshots from Codaset in this post, because I’ve not yet asked Joel Moss for permission that.

Git Tag Mini Cheat Sheet Revisited

Sat, 05 Sep 2009 00:00:00 +0000

Just as a kind of mini cheat sheet for using git tags. Jörg Mittag had some great additions that weren’t in the original post which warrant a new post.

Git has three different type of tags:

Lightweight tags
Annotated tags
Signed tags

Let’s start with lightweight tags.

Lightweight tags

In the previous cheat sheet only the lightweight local tags were discussed. A lightweight tag is nothing more than a reference to a particular revision or SHA1 object name in the repository. This kind of tag is quick and easy and very usable for local development to mark places in your commit history.

Creating a lightweight tag is easy:

git tag tag_name

Viewing available tags is done with -l:

git tag -l

Annotated tags

Annotated tags are almost like lightweight tags, the big difference is that they contain a message. Normally this message indicated why this tag is interesting. Use the -a option to create an annotated tag.

git tag -a tag_name

Since a message is required for annotated tags, you will be prompted with an editor to enter a message, or you can use the -m option to specify one directly.

git tag -a -m "Tagging release 1.0" v1.0

To view annotated tags you can use the same -l option as before, but you have to instruct git to show the annotation messages as well:

git tag -l -n1

This will not only show the messages for the annotated tags, it will also show the commit message of the revisions tagged with lightweight tags as well. Quite useful!

Signed tags

Signed tags take annotated tags a step further, they include an OpenPG signature to provide trust. While gits SHA1 tags provide integrity for the repository, the OpenPG signature makes sure that a trustworthy person created the tag.

To create a signed tag you’ll need to have GPG or some other OpenPG tool setup and use the -s option to sign the tag:

git tag -s -m "Tagging release 2.0" v2.0

The -s options implies the -a option, so here too a message is required.

To verify a signed tag you can run the following:

git tag -v v2.0

Deleting tags

There are times when you want to remove tags as well. This quite easy:

git tag -d tag_name

To remove a tag on a remote repository, you should do a special push:

git push origin :refs/tags/tag_name

Pushing tags

To push your tags to a remote repository, use the following command to push all tags:

git push origin --tags

Happy tagging!

Git Tag Mini Cheat Sheet

Fri, 04 Sep 2009 00:00:00 +0000

Just as a kind of mini cheat sheet for using git tags:

Adding a tag:

git tag tag_name
git tag
Should show your new tag.
git push origin --tags or git push origin :tag_name
Because git push doesn't push tags.

Removing a tag:

git tag -d tag_name
git tag
Should no longer show your tag.
git push origin :refs/tags/tag_name
Because git push --tags doesn't push deleted tags.

Hope that helps.

Rails + MySQL: Case-Sensitive strings in your database

Thu, 03 Sep 2009 00:00:00 +0000

When using Rails + MySQL, you’ll find that normal string (or varchar(255)) fields are case insensitive. This can be quite a nuisance, but it’s easy to resolve. You need to set your table to the utf8_bin collation. By using the binary variant, you’re basically enabling case sensitivity.

create_table :posts, :options => 'ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin' do |t|
 t.string :title, :limit => 100
end

That’s all. The title field is now case sensitive.

Another question I get a lot is how to change the collation and charset for an existing column. That’s easy with the following query. Just make sure to pick the right column and data type:

ALTER TABLE posts MODIFY `title` varchar(100) CHARACTER SET utf8 COLLATE utf8_bin;

JInput Mac OS X 64 bit natives

Tue, 01 Sep 2009 00:00:00 +0000

Yesterday I ran into a little problem running Slick 2D on Java 6 64bit on my Mac. It’s a MacPro, which has a 64 bit processor and is running Leopard. The problem I encountered was related to the native libraries provided by LWJGL.

If you’re a user of LWJGL, you’ll be using JInput as well. Unfortunately, JInput does not currently have any 64 bit native libraries as it is only providing for PPC and i386 (32bit). This is a problem because Mac OS X users are somewhat bound to Java 6 64bit. There is no 32 bit version of Java 6 for Mac OS X.


$ file libjinput-osx.jnilib
libjinput-osx.jnilib: Mach-O universal binary with 2 architectures
libjinput-osx.jnilib (for architecture ppc): Mach-O dynamically linked shared library ppc
libjinput-osx.jnilib (for architecture i386): Mach-O dynamically linked shared library i386

Thus, the default JInput natives provided by LWJGL, and subsequently by projects like Slick 2D and jME are of no use on a system running Java 6 64 bit. Also games and other apps based on those libraries won’t run on Java 6 64 bit!

To solve this, I recompiled the JInput natives from CVS with 64 bit support:


$ file libjinput-osx.jnilib
libjinput-osx.jnilib: Mach-O universal binary with 3 architectures
libjinput-osx.jnilib (for architecture ppc): Mach-O dynamically linked shared library ppc
libjinput-osx.jnilib (for architecture i386): Mach-O dynamically linked shared library i386
libjinput-osx.jnilib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64

I packaged the Mac OS X native (libjinput-osx.jnilib) and the resulting jinput.jar and jinput-test.jar up and you can download them here:

jinput-20090901-osx+ppc+i386+x86_64.tar

Note: I’m not running Linux or Windows at the moment, so I can’t provide any recompiled binaries for those platforms.

Question: now, how do I package this up nicely so I can use it in my Slice 2D based app?

Once and for all: Rails migrations integer :limit option

Thu, 20 Aug 2009 00:00:00 +0000

I literally always have to look up the meaning of :limit in migrations when it comes to integer values. Here’s an overview. Now let’s memorise it (oh, this works for MySQL, other databases may work differently):

:limit	Numeric Type	Column Size	Max value
1	tinyint	1 byte	127
2	smallint	2 bytes	32767
3	mediumint	3 byte	8388607
nil, 4, 11	int(11)	4 byte	2147483647
5..8	bigint	8 byte	9223372036854775807

Note: by default MySQL uses signed integers and Rails has no way (that I know of) to change this behaviour. Subsequently, the max. values noted are for signed integers.

IMDB Ruby Gem 0.4.0 Now available at RubyForge!

Sun, 14 Jun 2009 00:00:00 +0000

I just released version 0.4.0 of the IMDB Ruby Gem into the wild. There are only a few minor updates:

Changes in 0.4.0

Updates to the console 'imdb' utility

Show the IMDB ID
Show the full IMDB URL

Installation or upgrade

$ sudo gem install imdb

$ sudo gem update imdb

Issues, source or contributions

You can find the code at github, as well as the issue tracker. Feel free to fork and contribute!

Speaking at Rails Underground

Sun, 14 Jun 2009 00:00:00 +0000

I haven’t seen a schedule yet, but I’ve been told by Mark that I’ll be speaking at Rails Underground this year.

My talk will be on the topic of Git. In about 45 minutes time I’ll show you all the basic git features you’ll need on a daily basis. Not only that, but I’ll also explain how git manages all those commits and branches so you can be on your way to become a git power user.

Are you attending or speaking at Rails Underground? Please let leave a comment or find me on Twitter (@ariejan).

Oh, and does anyone have a clue if Mark has the schedule ready yet? ;-)

Best Practice - The Git Development Cycle

Mon, 08 Jun 2009 00:00:00 +0000

Git is quite an awesome version control system. Why? Because it’s lightning fast, even for large projects (among other reasons).

But, how do you use Git effectively for development on a daily basis? Let me explain to you.

Branches

With git you normally have a ‘master’ branch. This is also the branch you use to sync your code with other repositories. That is also the reason why you should never code in the ‘master’ branch. Always create a new branch and develop your code there.

$ git checkout -b new_feature
# add, commit, repeat

Rebase

Now, while you are working hard on your new feature, other developers complete theirs and push their changes to the remote master branch. When you’re done with your project, you need to first get the most recent version of the project’s code.

$ git checkout master
$ git pull

Now, to make merging your new feature easy, you should rebase your new_feature_branch. What this does is add all the commits you just pulled in to your new_feature branch. Any conflicts that arise will happen in your new_feature branch as well, leaving your master branch clean and in order.

$ git checkout new_feature
$ git rebase master

Merge

Now, you have resolved all (if any) conflicts with the current code and your new_feature, you can now merge your new_feature into the project’s master branch without any problems.

$ git checkout master
$ git merge new_feature

This will create a new commit, containing your new_feature. Now is also the time to push your changes to the remote repository.

$ git push origin master

What’s next?

More often than not, you’ll encounter conflicts when running rebase. That’s okay, but you’ll need to know how to approach a situation like that. I’ll spend another blog post on that topic later.

Need Git training?

I’m currently available to provide on-location Git training. Please contact me for more info.

ActiveRecord: Skipping callbacks like after_save or after_update

Sun, 07 Jun 2009 00:00:00 +0000

Active Records provides callbacks, which is great is you want to perform extra business logic after (or before) saving, creating or destroying an instance of that model.

However, there are situations where you can easily fall into the trap of creating an infinite loop.

class Beer < ActiveRecord::Base
 def after_save
 x = some_magic_method(self)
 update_attribute(:my_attribute, x)
 end
end

The above will give you a nice infinite loop (which doesn’t scale). It’s possible to update your model, without calling the callbacks and without resorting to SQL.

class Beer < ActiveRecord::Base
 def after_save
 x = some_magic_method(self)
 Beer.update_all("my_attribute = #{x}", { :id => self.id })
 end
end

This is a bit unconventional, but it works nicely. You can use all the following ActiveRecord methods to update your model without calling callbacks:

decrement
decrement_counter
delete
delete_all
find_by_sql
increment
increment_counter
toggle
update_all
update_counters

An important warning: These methods don’t do all the nice SQL injection protection stuff you’re used to. In the example, the value of x will be inserted straight into the SQL. I recommend you only use these methods if you’re absolutely sure you’ve cleaned the values you’re inserting.

Check out the rails documentation on how to use these methods.

has_one - find all that have no associated object

Sun, 07 Jun 2009 00:00:00 +0000

Let me pose a typical Rails situation:

class Person < ActiveRecord::Base
 has_one :fancy_hat
end

class FancyHat < ActiveRecord::Base
 belongs_to :person
end

Now, how can you get all the people that don’t have a fancy hat?

class Person < ActiveRecord::Base
 has_one :fancy_hat

 named_scope :hatless, :joins => 'LEFT JOIN fancy_hats ON fancy_hats.person_id = people.id', :conditions => 'fancy_hats.person_id IS NULL'
end

Now you can find all the hatless people you want.

FYI: Finding fancy hats that have no one to wear them is a lot easier, because the foreign key is stored in the fancy_hats table.

class FancyHat < ActiveRecord::Base
 belongs_to :person

 named_scope :wearerless, :conditions => { :person_id => nil }
end

IMDB 0.3.0 now including console utility - query IMDB from your console

Sun, 07 Jun 2009 00:00:00 +0000

With the release of IMDB 0.3.0, a command-line utility is included!

Why is this awesome for you? Basically, because you can now query IMDB and process the results with any of the great GNU tools available to you like grep.

Let me show you:

Search IMDB

$ imdb Star Trek
>> Searching for "Star Trek"
 > 0060028 | Star Trek (1966) (TV series)
 > 0796366 | Star Trek (2009)
 > 0092455 | Star Trek: The Next Generation (1987) (TV series)
 > 0112178 | Star Trek: Voyager (1995) (TV series)
 > 0106145 | Star Trek: Deep Space Nine (1993) (TV series)
 > 0117731 | Star Trek: First Contact (1996)
 > 0084726 | Star Trek: The Wrath of Khan (1982)
 > 0092007 | Star Trek IV: The Voyage Home (1986)
 > 0079945 | Star Trek: The Motion Picture (1979)
 > 0244365 | Enterprise (2001) (TV series)

For clarity, only the ten first search results are shown. I’m thinking of including an option to set the number of returned titles. Let me know what you think about that.

Getting movie details

So, let’s pick a movie we want to get details about.

$ imdb 0796366
>> Fetching movie 0796366

Star Trek (2009)
========================================================================
Rating: 8.4
Duration: 127 minutes
Directed by: J.J. Abrams
Cast: Chris Pine, Zachary Quinto, Leonard Nimoy, Eric Bana, Bruce Greenwood
Genre: Action, Adventure, Sci-Fi
A chronicle of the early days of James T. Kirk and his fellow USS Enterprise crew members. |
========================================================================

Combine with GNU commands

Or, as mentioned earlier, you can combine it with other console commands. So to get the rating of a movie, do this:

$ imdb 0796366 | grep "^Rating" | cut -d" " -f 2
8.4

Installation, source and issues

I’m still working on perfecting the console output, so if you have any tips, please report it in the form of an issue.

You can get this as a Ruby Gem with

sudo gem install imdb

The source is also available at http://github.com/ariejan/imdb.

Install Hpricot on Ubuntu

Fri, 05 Jun 2009 00:00:00 +0000

It’s quite easy. Make sure you have RubyGems and Ruby installed first, of course.

The problem:

$ sudo gem install hpricot
Building native extensions. This could take a while...
ERROR: Error installing hpricot:
 ERROR: Failed to build gem native extension.

/usr/bin/ruby1.8 extconf.rb
extconf.rb:1:in `require': no such file to load -- mkmf (LoadError)
 from extconf.rb:1

The solution:

sudo apt-get install ruby1.8-dev build-essential
sudo gem install hpricot

Ruby Gem: IMDB

Wed, 03 Jun 2009 00:00:00 +0000

I just released version 0.1.0 of my IMDB gem which allows your app to search IMDB for IMDB movie ID’s and access most data that’s publicly available.

Installation

sudo gem install imdb

This will also install the dependencies Hpricot and HTTParty.

Usage

In your project, include the gem (and possibly rubygems as well).

require 'rubygems'
require 'imdb'

search = Imdb::Search.new('Star Trek')
=> #<Imdb::Search:0x18289e8 @query="Star Trek">

puts search.movies[0..3].collect{ |m| [m.id, m.title].join(" - ") }.join("\n")
=> 0060028 - "Star Trek" (1966) (TV series)
 0796366 - Star Trek (2009)
 0092455 - "Star Trek: The Next Generation" (1987) (TV series)
 0112178 - "Star Trek: Voyager" (1995) (TV series)

st = Imdb::Movie.new("0796366")
=> #<Imdb::Movie:0x16ff904 @url="http://www.imdb.com/title/tt0796366/", @id="0796366", @title=nil>

st.title
=> "Star Trek"
st.year
=> 2009
st.rating
=> 8.4
st.cast_members[0..2].join(", ")
=> "Chris Pine, Zachary Quinto, Leonard Nimoy"

As you can see, both Imdb::Search and Imdb::Movie are lazy loading, only doing a HTTP request when you actually request data. Also, the remote HTTP data is cached trhough-out the life span of your Imdb::Movie object.

Documentation

Generated RDoc documentation can be found at http://ariejan.github.com/imdb/.

Issues, feature requests, patches, the works

Please use https://github.com/ariejan/imdb to supply patches (preferably through a pull-request) and to report issues.

Speak louder! I can't hear you over the sound of how awesome I am!

Fri, 22 May 2009 00:00:00 +0000

Get the t-shirt!

Second RubyFest Speaker: Geoffrey Grosenbach

Wed, 06 May 2009 00:00:00 +0000

Geoffrey Grosenbach is going to deliver a talk about MacRuby at RubyFest on may 14th! He’ll be joining us over a live video feed. Afterward Geoffrey will be available for a short Q&A session.

More details about Geoffrey’s talk are expected shortly.

If you didn’t get your tickets for RubyFest yet, be sure to head over to http://rubyfest.nl and get them right now. We’ve got limited seating for this first edition of RubyFest. (Oh, drinks are included)

Available for iPhone Development

Tue, 28 Apr 2009 00:00:00 +0000

Just to put it out there: I’m available for iPhone development (preferably in the Netherlands).

If you’re interested in having an iPhone app developed, feel free to contact me to discuss your options.

I’m developing iPhone apps a an employee at Kabisa.

Compacting a SQLite3 DB file

Thu, 23 Apr 2009 00:00:00 +0000

If you have a lot of mutations in your SQLite3 database the file size of the db file will increase a lot over time.

This can be annoying, so you’ll need to clean up old indices and other cruft that’s making your db heavy.

The solution is über-easy:

$ sqlite3 mystuff.db
SQLite version 3.6.6.2
sqlite> VACUUM;
sqlite> .quit

Note that you need to use capitals here. This will clean your db file and reduce its file size dramatically (depending on the amount of cruft you have, of course).

How to create a DSA OpenSSL certificate

Sun, 19 Apr 2009 00:00:00 +0000

I just needed an OpenSSL DSA public key. This is not really difficult, you just need to know the right commands. On my Mac I ran the following commands to obtain both private dsa_priv.pem and public dsa_pub.pem keys.


openssl dsaparam 1024 < /dev/random > dsaparam.pem
openssl gendsa dsaparam.pem -out dsa_priv.pem
openssl dsa -in dsa_priv.pem -pubout -out dsa_pub.pem

Needless to say, keep your private key in secure location and make sure you have a backup of it!

MacBook Pro: Black Screen of Death (or is it just faking?)

Sun, 05 Apr 2009 00:00:00 +0000

Disclaimer: If you are reading this, chances are there is a hardware problem with your Mac. In my case it was a faulty logic board, which had to be replaced.

Use this guide to get your Mac up and running again and create a full backup of you system as soon as possible. If the problem repeats itself, I recommend you take your Mac back to Apple for a check-up.

Today I was happily working on some Java code, when I decided to relocate to a sunny spot in the backyard. I closed my MacBook Pro, walked outside and opened my MBP again to continue work: a black screen!

The screen of my Mac stayed black, although it did not indicate to be in a sleeping state. WTF? I restarted the Mac, removed the battery, reset PRAM/NVRAM and the PMU, but nothing worked! ARGH! Then, I found a solution that worked.

If you find yourself in a similar situation, do the following:

Shutdown your Mac entirely by holding the power button for 5 seconds.
Hook up your Mac to another one with a FireWire cable. (You have one, right?)
Hold the T key on your mac's internal keyboard and press the power button. When your MBP's disk will shows up as an external firewire disk on your HelperMac, release the T button.
Open finder and go to the /private/var/vm folder on your MacBook's hard drive. You may need to use Command-Shift-G in Finder, because this folder is hidden by default. Once in /private/var/vm remove the sleepfile you find there.
Next, go to /System/Library on you MBP's disk and remove Extensions.mkext.
Now, eject your MBP drive and shutdown your MBP by hitting its power button.

After these steps, you should be able to boot your MBP as normal again.

Although I’m not sure of the problem at hand, it appears that the sleepfile gets corrupted in some way.

Oh, didn’t have that FireWire cable ready? Here’s the one I used.

Update: There are some common questions in the comments I’d like to answer here. You cannot use an USB cable to connect your Mac to another one, only Firewire 400 and 800 are supported.

In my case, the MacBook Pro was returned to a Apple Dealer and repaired under warranty, because Apple apparently has acknowledged the problems with the video card on these machines. Your mileage may vary with your local Apple store.

May 14th: RubyFest!

Tue, 31 Mar 2009 00:00:00 +0000

Well, the time has come to announce RubyFest! RubyFest is an official meetup of Ruby developers and enthusiast which will be held on May 14th in Eindhoven, the Netherlands.

Besides organising RubyFest, I’ll also be one of two speakers at RubyFest. I’ll be giving a talk about using git with your (open source) project effectively. The other speaker is yet to be announced, and we’re still accepting proposals!

So, head over to RubyFest and register today, because we only have limited seating!

Follow me or RubyFest on Twitter to get the latest and greatest quickly.

warcraft-armory 0.1.0 Released

Sat, 07 Feb 2009 00:00:00 +0000

Yay! warcraft-armory version 0.1.0 has been released!

The warcraft-armory gem allows your application to easily access information from the World of Warcraft Armory site.

This is an early version that allows the retrieval of character information from EU and US armories. But, more is in the making!

Feel free to check-out the code, read the docs or just install the gem:

sudo gem install warcraft-armory

require 'warcraft-armory'
adries = WarcraftArmory::Character.find(:eu, :aszune, :adries)
adries.description
# => "Level 48 Human Warrior"

It’s my first gem, so useful comments are appreciated! Keep an eye out for updates!

Pagerank 3, 19k Hits/Month

Fri, 30 Jan 2009 00:00:00 +0000

I just found out Ariejan.net has a PageRank™ of 3! If you don’t know what PageRank™ means, let Google explain:

PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important". - source

To make a long story short, Ariejan.net is getting a lot of attention from around the web. Personally I think of it as a sign of appreciation for my articles. I’m also quite impressed with the number of page views my site is getting. Page views average out to 19k a month over 2008. Thanks all!

I found out about all this because I updated my listing at BuySellAds.com. I suggest you check them out some time.

How To Start A Rails Edge App The Easy Way

Sun, 04 Jan 2009 00:00:00 +0000

There’s a lot of cool stuff pooring in about what’s new in Rails Edge (which will become Rails 2.3 and/or Rails 3).

Most likely you can’t wait to get started with these new features, especially when you’re about to start a new project, which doesn’t have to be stable yet, but will be by the time 2.3/3.0 come out. This post shows you the way to create a new Rails app based on the most current Rails code, also called Edge Rails.

Let’s go…


mkdir -p myapp/vendor
cd myapp
git init
git submodule add git://github.com/rails/rails.git vendor/rails
git commit -m "Frozen Rails Edge as submodule"
ruby vendor/rails/railties/bin/rails .
# Add generated files to git, and code on...

First, you create a new directory for your app, including the vendor directory. Easy, right?

Next, you initialize a Git repository for your empty project. We’ll be using Git to track the remote Rails Edge code. Stay with me.

By adding a Git submodule we tell git to clone the code from git://github.com/rails/rails.git into the vendor/rails directory. Nice! If you check the current git status with git status you see git has already staged two files for you, .gitmodules and vendor/rails. Commit them now to attach the submodule to your local git repository.

Git will not automatically update your submodule, you’ll have to do that by hand. I’ll show you this in a minute.

With vendor/rails containing Rails Edge, you can now generate your Rails Edge application. In you project directory (myapp/), you call ruby vendor/rails/railties/bin/rails .. This will generate a new Rails Edge application in the current directory.

Now it’s up to you to create a fitting .gitignore file and commit the files to your repository.

That’s all, you now have a new Rails Edge application. Try ruby script/server to see it all in action. Enjoy!

Cloning your project

At some point you’ll push your myapp project to a remote git server. When you clone a fresh copy, you’ll have to initialize the git submodules. This is quite easy:


git submodule init
git submodule update

Updating Rails Edge

As I said earlier, Git will not keep your submodules up-to-date for you, but will stick with the revision you added. To keep track of Rails Edge’s progress, you’ll need to update the submodule. This is done like this:

cd myapp/vendor/rails
git remote update
git merge origin/master

This will update your Rails Edge code. Make a commit, stating you updated the code!

After updating Rails Edge, you may want to update your rails application (like javascript files, config files etc).

rake rails:update

Good luck! And happy coding!

Twitterlicious!

Sat, 27 Dec 2008 00:00:00 +0000

A bit late, but “Merry Christmas” everyone!

I’m on Twitter lately, so feel free to check and follow. It’s a also a nice way to keep up-to-date about new articles on Ariejan.net.

Hey, that’s all for now. And if you don’t come back here this year: “Happy New Year!” - Cheers!

Google FriendConnect now on Ariejan.net

Wed, 03 Dec 2008 00:00:00 +0000

Well, I just got invited for Google FriendConnect! Let me tell you a bit about FriendConnect.

With FriendConnect you can easily join Ariejan.net. See of it as a pool of Ariejan.net visitors or fans if you like.

Once you’ve joined up (with your google account, for example) you’ll be able to interact more closely with Ariejan.net. In the case of Ariejan.net you can share your thoughts with other Ariejan.net visitors. Just vent your frustration about Linux or let others know what Rails project you’re working on.

Another benefit for you is that you can see what sites other people have signed up for as well. This way you’ll be able to get a list of sites that Ariejan.net visitors like as well. You might compare it with Amazon’s “other people also bought..” feature.

The big benefit for me is that’s it’s nice to see the list of fans/visitors growing. After all, that’s what I’m keeping a blog for after all.

In the future I’ll be able to add other widgets to Ariejan.net. This allows for other features to appear on Ariejan.net and a better over-all user experience.

You may want to read more about Google FriendConnect and OpenSocial.

Export CSV directly from MySQL

Thu, 27 Nov 2008 00:00:00 +0000

How ofter were you asked by a client for a CSV (or excel) file with data from their app? I get asked that question quite often, so I wanted make the process as easy as possible. And guess what? You can create CSV files directly from MySQL with just one query!

Let’s say you want to export the id, name and email fields from your users table to a CSV file. Here is your code:

SELECT id, name, email INTO OUTFILE '/tmp/result.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
ESCAPED BY ‘\\’
LINES TERMINATED BY '\n'
FROM users WHERE 1

Well, if you know MySQL, you’ll know how to customize this query to spit out the the right data. Your csv file can be found in /tmp/result.csv

Make sure your MySQL server has write permissions to the location where you want to store your results file.

SQL: Ordering with NULL values

Fri, 14 Nov 2008 00:00:00 +0000

First seen at Kabisa Blog: SQL: Ordering with NULL values

This post tells you how to sort NULL values in a column to the bottom and sort the remaining non-NULL values.

This is really great in combination with LEFT JOIN queries that may yield NULL values.

RSpec'ing with Time.now

Wed, 05 Nov 2008 00:00:00 +0000

I’m currently writing some RSpec tests that use Time.now.

I want my model to calculate a duration and store the future time in the database. I’ve already specced the calculation of the duration, but I also want to spec that everything gets saved correctly. Here’s my first spec:

it "should do stuff" do
 m = Model.create()
 m.expires_at.should eql(Time.now + some_value)
end

This fails.

It fails because Time.now is quite accurate and some milliseconds have passed between the two calls.

So how do you test this kind of behaviour? I was not going to let this one beat me. Get out your gloves, because we’re going to start stubbing!

What you need to do is stub out Time#now to return a constant value within this test. This way, both calls will use the same Time.now value and thus yield the same result. This in turn makes your test pass (if the saving goes well, of course).

it "should do stuff" do
 @time_now = Time.parse("Feb 24 1981")
 Time.stub!(:now).and_return(@time_now)

 m = Model.create()
 m.expires_at.should eql(Time.now + some_value)
end

BaseApp: a quick start for your Rails App

Sun, 28 Sep 2008 00:00:00 +0000

BaseApp is no longer maintained. There is a very good alternative called bort.

For the impatient: http://github.com/ariejan/baseapp

Got issues? Feature requests or patches? http://baseapp.lighthouseapp.com/

Every Rails developer has at least once developed an application that needed user authentication and some basic UI features like tabs and a sidebar. Ask yourself now: “how often have you installed and extended the restful_authentication plugin?”.

Yes, I have done it quite a few times and everytime I find myself writing the same code over and over again. User login, password reset, ‘forgot password’ functionality. I’ve build the same basic UI over and over again. Added administrator users and roles.

Are you tired of doing the same old things over and over again? I was! So, I created BaseApp.

BaseApp is a Ruby on Rails application which contains a lot of code you want in your project by default. To give you an idea of what is does out of the box:

User Authentication including password recovery, account activation and account suspensio.
Admin interface where the admin user can easily manage users and tweak app settings
Default CSS-based UI with tabs and a sidebar. Very acceptable by default and easy to customise.

BaseApp is currently based on Rails 2.1.1. And although it’s a pretty complete package and ready to be used for your next project, it still needs a bit of work. Check out the README for features that should be in BaseApp.

Of course, BaseApp is open source so fork a copy at the GitHub and send me those patches (of pull requests)!

There are tons of feaures that can be included into BaseApp, so the next big thing is to include some sort of configuration that allows you to disable/enable certain BaseApp features.

So, go right ahead! Use it! Fork it! Send me those pull requests!

JRuby with Thomas Enebo

Wed, 03 Sep 2008 00:00:00 +0000

This morning I attended another JRuby talk, this time with Thomas Enebo. It turned out to be, almost default, Sun JRuby talk.

There was one interesting difference, though. Normally we are shown how to run Ruby on Java and how to use Java components in our Ruby apps. Thomas took JMonkeyEngine (a java 3d scenegraph/game engine) and showed that he coded a simple game in Ruby, steering clear of the rather complex Java code.

This is, of course, another great feature of JRuby, you can code a “Java” app, and in certain places actually use Ruby to write cleaner or “easier” code.

Well worth the time!

Panel Discussion with DHH and Rails Core Members

Wed, 03 Sep 2008 00:00:00 +0000

Yesterday evening we attended a panel discussion with DHH and Rails Core Members Jeremy Kemper and Michael Koziarski.

DHH elaborated on default choices (like database, templating system and test suite) after being asked if Rails would switch over to RSpec instead of TestUnit. The answer was that Rails offers several defaults, which should suffice for new and basic users, who don’t know about all options and just want to get started. More experienced users will generate a taste for different components and Rails should provide for easy integration of these components if needed.

All in all there wasn’t any really new stuff to be heard, but a nice gathering non the less.

JRuby with Nick Sieger

Tue, 02 Sep 2008 00:00:00 +0000

A tutorial on JRuby with Nick Sieger holding your hand is just great. This guy knows JRuby inside-out and he has an answer to even the most difficult of questions.

After a short 15 minute introduction, Nick left us with our Rails app to start it with JRuby on a mongrel. I’ve been toying around with JRuby a bit before, and this was rather easy. Some people, however, encounter some issues with their apps because of incompatible gems, like OpenSSL or RMagic which are not available for the Java platform. Luckily, there are some nice Java alternatives like JRuby OpenSSL and ImageVooDoo. My app didn’t suffer from these issues, but from what I’ve heard, these alternatives work great.

Next for the big work: creating a WAR file and deploying it. I’ve done this before, and it’s dead easy. In stall the Warble gem, warble your Rails app and deploy the resulting WAR file.

Nick also talked about the differences between using the JVM and Unix. The biggest differences are memory management and the creation of new threads/processes.

There are quite a few things that Rails and a Java app server have in common, like logging and session handling. In his sheets (I’ll link to those later), you can see how you can tune your Rails app to make use of these features and improve your apps performance (by not logging and keeping sessions in two different places).

Continuing on the performance of Glassfish/JRuby, Nick shows several graphs on how well JRuby performs. A lot of attention goes to Rails 2.2, which will be thread-safe and what it means for JRuby.

JRuby can run with 1 runtime and fill your CPU with threads to handle requests using native Java threads. To handle database connections, Nick Sieger has written a connection pooling component for ActiveRecord (to prevent every thread having it’s own connection to your database). An unreleased version of warbler was distributed that will allow you to set max_instances to 1. This will create 1 Rails instance and use threading in Glassfish. Of course, this is only useful with Rails 2.2.

This was altogether a great tutorial afternoon. Besides the basic warble-your-app-and-deploy-it, Nick provided a lot of interesting facts and tips about deploying to Glassfish. I already was enthusiastic about JRuby, but now I really have to get my hands on a decent server that will run Glassfish comfortably. This means at least at least 1Gb of RAM, so I think I have found at least one drawback of using JRuby.

RailsConfEurope: The first tutorial

Tue, 02 Sep 2008 00:00:00 +0000

Today, RailsConfEurope 2008 started! Well, at least if you signed up for Tutorial Day.

The first tutorial I’m attending is about Hacking Rails Internals. At first I thought about this as a great way to enhance my own apps and be able to easily integrate other apps like Radiant or Mephisto. However, the more I saw of the demo (it really is more a demo than a tutorial really), the more I disliked hacking Rails Internals.

Basically, you hook your own application code directly into the Rails code to make it jump through hoops. But, this comes at a big price. First of all, your app code is very dependent on a specific version of Rails, so upgrading Rails will most likely break your app and cost a lot of time (and money) to upgrade.

Next, you are overriding internal Rails code. You just don’t know if any other code (either yours, Rails or a plugin) depends on the behaviour of that code. So you should probably expect strange stack traces.

So, although hacking Rails to include your own or others code directly is great, it has some major drawbacks you should consider. To quote the presenter “It’s brutish, but it’s how to do it”.

I’m really looking forward to Nick Sieger and his JRuby tutorial this afternoon. I’m all set with Glassfish installed and my Rails app ready.

Leaving for RailsConf Europe 2008

Mon, 01 Sep 2008 00:00:00 +0000

Today a delegation of Kabisa is leaving for RailsConf Europe 2008! We’re going by train this year and we hope to arrive this afternoon around 5.

I’ll keep regular updates here on Ariejan.net about RailsConf and all the things I’ve seen and done.

If you’re going to visit RailsConf Europe yourself, feel free to seek us out and have a chat!

See you at RailsConf!

ActiveRecord Read Only Model

Sun, 17 Aug 2008 00:00:00 +0000

ActiveRecord is great in providing CRUD for your data models. In some cases, however, it’s necessary to prevent write access to these models. The data may be provided by an external source and should only be used as a reference in your application, for example.

I’m going to show you how you can easily mark a Model as read only all the time. In this example I have a Item model like this:

class Item < ActiveRecord::Base
end

ActiveRecord::Base provides two methods that may be of interest here:

def readonly!
 @readonly = true
end

def readonly?
 defined?(@readonly) && @readonly == true
end

The first method sets the record to read only. This is great, but we don’t want to set the read only property every time we load a model. The second, readonly?, return true if the object is read only or false if it isn’t.

So, if we return true on the readonly? method, our object is marked as read only. Great!

class Item < ActiveRecord::Base
 def readonly?
 true
 end
end

That is all! All Item objects are now marked as read only all the time. If you try to write to the model, you’ll receive an error.

item = Item.find(:first)
item.update_attributes(:name => 'Some item name')
=> ActiveRecord::RecordReadOnly

Skinny Controllers and Overweight Models

Sun, 17 Aug 2008 00:00:00 +0000

All Rails developers know the slogan “Skinny Controllers, Fat Models” and I heartily agree with it. Every conference you go to, you hear it. But there’s a problem! My Fat models got overweight!

What happened? By stuffing all applications logic in the Models, they become fat, very fat. Although this is supposed to be a good thing, I don’t like it. My models get so fat that it takes me forever to scroll through it and find the method I’m working on. There must be a better way!

Well, yes there is: create modules! Normally you’d write a module to reuse your code in different places, but there’s no rule that says you may not use a module only once.

So, I package all related code (e.g. Authentication, state management, managing associated objects, etc) into different modules and place them in the /lib directory. Let’s say you have a a bunch of methods to handle keep a counter on your User model

class User < ActiveRecord::Base
 attr_accessor :counter

 def up
 counter += 1
 end

 def down
 counter -= 1
 end

 def reset
 counter = 0
 end
end

You could create a new file lib/counter.rb and include that module in your User model.

class User < ActiveRecord::Base
 attr_accessor :counter
 include Counter
end

module Counter
 def up
 counter += 1
 end

 def down
 counter -= 1
 end

 def reset
 counter = 0
 end
end

As you can see, this keeps your fat User model clean and makes it easier for you to find code that applies to a certain function.

Useless Ruby Gems for your pleasure

Thu, 14 Aug 2008 00:00:00 +0000

The past few days I’v taken some time to find out how to create a Ruby Gem. This has been on my to-do list for quite a while, but now I’m able to tick it off.

Well, what did I make?

The first Gem can also be used as a Ruby on Rails plugin and is called ActsAsGold. If you’ve ever played World of Warcraft, you’ll know how the money system works. You have Copper. A 100 Copper is worth 1 Silver. And 100 Silver is worth 1 Gold.

The ActsAsGold Gem allows you to extend your ActiveRecord model with this money system. All you need on your model is an attribute that stores a single integer value.

Let me give you a small tour.

class Player < ActiveRecord::Base
 acts_as_gold :column => :money
end

# This will be store like @player.money => 2003652
@player.gold => 200
@player.silver => 36
@player.copper => 52

# You can also easily earn or spend money
@player.earn(10.gold + 75.silver)
@player.spend(1.gold + 10.silver + 95.copper)

Read more about the Gem, or install the gem right now:

sudo gem install ariejan-acts_as_gold --source http://gems.github.com

The other gem is WarcraftArmory, which is still in early development, so new stuff can and will be added in the future.

WA (WarcraftArmory) allows you to easily retrieve character information from the World of Warcraft Armory. Currently it can retrieve:

Name of the character
Name of the characters guild
Level
Race
Class

It works for both EU and US warcraft servers.

require 'warcraft_armory'

character = WarcraftArmory.find(:eu, 'Aszune', 'Nosius')

character.race => "Human"
character.level => 15

Again, simply install the plugin and use it like any other gem or read the README first.

sudo gem install ariejan-warcraft_armory --source http://gems.github.com

Of course, these gems are released under the MIT license. The code is on Gitub (acts_as_gold, warcraft_armory) and patches with fixes and new features are gladly accepted.

Please let me know if you find these gems useful or if you use them in one of your projects.

Ruby on Rails: UUID as your ActiveRecord primary key

Tue, 12 Aug 2008 00:00:00 +0000

Sometimes, using the good old ‘auto increment’ from your database just isn’t good enough. If you really require that all your objects have unique ID, even across systems and different databases there’s only one way go: UUID or Universally Unique IDentifier.

A UUID is generated in such a way that every generated UUID in the world is unique. For example: 12f186e6-687e-11ad-843e-001b632783f1. This string is randomly generated based on several factors that guarantee it’s uniqueness.

Anyway, you want to replace the default integer-based primary keys in your model with a UUID. This is quite easy, but there are some caveats.

First off, you should have a column in your database table that holds the UUID. You may be tempted to just change the column definition for id from integer to string and be done with it. But this won’t work as expected. For your development, and maybe even your production system, this may work fine, but you might be in for some unexpected surprises.

The best example of such a surprise is RSpec. RSpec uses ‘rake db:schema:dump’ to create a sql dump to quickly load the database with. However, the ‘schema:dump’ does not look at the id column in your database, but instead adds the default primary key definition from the ActiveRecord adapter.

The solution is to disable the id column and create a primary key column named uuid instead.

create_table :posts, :id => false do |t|
 t.string :uuid, :limit => 36, :primary => true
end

In your Post model you should then set the name of this new primary key column.

class Post < ActiveRecord::Base
 set_primary_key "uuid"
end

The next step is to create the UUID itself. We’ll have to do this the Rails app, because most databases don’t support UUID out of the box.

First install the uuidtools gem

sudo gem install uuidtools

Create a file like lib/uuid_helper.rb and add the following content.

require 'rubygems'
require 'uuidtools'

module UUIDHelper
 def before_create()
 self.uuid = UUID.timestamp_create().to_s
 end
end

Then, include this module in all UUID-enabled models, like Post in this example.

class Post < ActiveRecord::Base
 set_primary_key "uuid"
 include UUIDHelper
end

Now, when you save a new Post object, the uuid field is automatically filled with a Universally Unique Identifier. What else could you wish for?

Ariejan.net Link Party 07/21/2008

Mon, 21 Jul 2008 00:00:00 +0000

Creating JavaScript widgets in Rails - igvita.com
How to offer your users a way to 'widgetize' their content and use it elsewhere as well.
tags: rails, ruby, ror, widget, widgets, share
Free Photoshop Brushes at Brusheezy!
A great resource for (photoshop) brushes.
tags: photoshop, brushes, design, resources, graphics, patterns, webdesign, share

Ariejan.net Link Party 07/17/2008

Thu, 17 Jul 2008 00:00:00 +0000

gearsonrails - Google Code
Gears on Rails helps developers to write fully offline functional web applications based on Gears without learning a bit of Gears.
tags: gearsonrails, ruby, rails, ror, share
Sexy Forms in Rails
Easily create very sexy form in Rails with minimum effort.
tags: rails, forms, sexy, share

Photography Heaven

Fri, 11 Jul 2008 00:00:00 +0000

For those who didn’t know yet, I’m on Flickr - and I just reorganized my entire photo collection.

Back in the day when I signed up with Flickr, my idea was to store all my photo’s there and be done with it. Now, I realize that I only want to upload the good stuff, and keep the rest for myself.

Please, hop by at http://flickr.com/photos/ariejan. Feel free to add me to your contacts or to leave a comment.

For those interested, I’m currently shooting with a Canon EOS 400D Digital in combination with the EF-S 18-55mm 1:3.5-5.6 II kit lens, an old and battered EF 75-300mm 1:4-5.6 and very new and shiny EF 50mm 1:1.8 II Prime lens.

I am looking to buy an EF 17-40mm L or EF 16-35mm L lens, so if you have one that’s just sitting there, please let me know.

That’s all. Hope to see from you on Flicrk!

Ariejan.net Link Party 07/10/2008

Thu, 10 Jul 2008 00:00:00 +0000

The Pirate Bay Wants to Encrypt the Entire Internet « NewTeeVee
tags: pirate bay, privacy, encryption, internets, share
Favicon Generator
Create a favicon based on text and colours selected by you. Very nice.
tags: favicon, design, generator, web, webdesign, icons, tool, graphics, share
Git for the lazy - Spheriki
tags: spheriki, git, usage, tutorial, tricks, tips, share
10 Ways to Take Stunning Portraits
tags: portraits, photography, share

How to digg-proof your WordPress blog

Wed, 09 Jul 2008 00:00:00 +0000

Every blogger hopes to be able to say some day to his friends: “I got Dugg!”.

It means you’ve written something special that a lot of other people are interested in and it gives you a big ego boost. But there won’t be much to enjoy when your server can’t handle the extra work.

In this post I’ll try to explain some measures you can take to be ready for when “The Big Digg” arrives at your blog’s door step. I’ll focus on WordPress in particular here, because that’s what I’m using. However, the concepts I’ll show you can be applied to any web site.

There are a few things you can do to survive when you get Dugg. The key to my tips here is that you can implement them at (almost) no cost in advance of getting dugg.

It’s important to understand that once you’re blog get’s dugg, you don’t have time to install a plugin or make any other sort of change. The only thing you can do when you’re not prepared is take down your web server (including any other apps than your blog).

Trust me, you don’t want to take down your blog when you got dugg.

Another obvious constraint is upgrading hardware. You simply can’t move to a new server or upgrade your RAM when you’re getting dugg.

The key is to be prepared. Here are some tips that will help you get ready for when the day comes…

Caching

WordPress, by default, is most likely not able to survive the Digg-effect. This is mainly because every page served requires WP to query your database a few dozen times and compile it all together in that nice looking template of yours.

So, the most straight forward measure you can take to improve performance is caching. By caching your blog, you basically create static HTML files that will be served to visitors instead of the generated page mentioned earlier.

Web servers are very good at serving static files. It’s ultra-fast. Also, it reduces the load on your server because you’re database doesn’t have to be woken on every request and there’s no PHP code that needs to be executed.

Of course, your blog is dynamic (at least if you post new things occasionally). This means that you’ll need to update those static HTML files regularly to keep them up-to-date.

To implement caching in your WordPress blog you don’t have to do all that much work, really. All you need is the WP Super Cache plugin.

The plugin takes care of generating static HTML files for all your blog pages and make sure Apache (which is the most commonly used webserver for WordPress) serves these pages very fast.

Compression

Another measure you can take, in combination with caching, is compressing the static HTML files. Because compressing is a one-time action when the page gets cached, you get a very small file that can be served by your web server even faster with minimal bandwidth usage.

As you already suspect, WP Super Cache has this feature included. Nice ;-)

As an added bonus, WP Super Cache will not perform caching for logged in users, users who’ve left a comment or users who’ve accessed password protected pages. This means that if a user leaves a comment, he will not see the ‘static’ cached file (which does not include the comment yet), but the generated version of your blog with the comment.

Hardware and platform

Another, very obvious, factor to take into account is your hardware and platform.

If you’re on a shared hosting account you won’t survive a Digg. Your provider will shut down your site quite quickly. They do this because there are quite a few users (think hundreds) on the same server as your blog. When dugg, your site will take up a very large portion of the available server capacity, leaving all other users with an usable blog as well.

My advice is, if you’re serious about your blog, to host it on a VPS or dedicated server. These forms of hosting guarantee you the resources you paid for and will only cause trouble to your own web sites.

For most people a dedicated server is not an option, simply because it’s too expensive for running a sole blog.

A VPS (Virtual Private Server) can be a good compromise between shared hosting and a dedicated server. You are guaranteed a certain amount of memory, disk space and processing power and your VPS won’t interfere with other VPS’es on the same server (unless your hosting provider uses crappy VPS software. Always look for Xen VPS hosting).

The most important factor in choosing a VPS is the available, or guaranteed, memory you get. You can get ‘slices’ ranging from 128Mb up to 4Gb of RAM. For an average blog I think 256 - 512Mb should be sufficient for running Apache smoothly and handling large amounts of traffic on the static files.

I can’t help to plug SliceHost here. I recently moved Ariejan.net to a slice (a 512Mb one, I also run some Ruby on Rails apps) and I’ve been very happy with them ever since. So go buy a slice now! I’d make me happy ;-)

Anyway, it’s important to make sure you run your blog on a platform that can handle a peak in traffic and run your blog smoothly all the time.

What else? Remove all the cruft

What else can you do to improve performance on your WordPress blog. Well, it would be to remove everything that’s not really needed there. Dynamic JavaScripts that retrieve and parse a remote RSS Feed with every request are a classic example of cruft.

There are a lot of nice plugins available for WordPress and there are even more ‘widgets’, ‘badges’ and ‘buttons’ you can put on your blog to show off. But every one of these will slow down the loading off your site. Especially the ones that come in the form of WordPress plugins.

Almost every plugin that adds stuff to your sidebar will include its own stylesheets, javascript files and images that need to be downloaded from your server. Even with a cached, static HTML file these need to be downloaded.

The point is, add a few of these nice plugins, and you’re web server has to handle up to 15x the amount of files per page view.

My advise is to keep plugins and usage of javascript, stylesheet and most importantly images to an absolute minimum. I don’t want to encourage you to write a plain text file as a blog, but keep things functional.

Do you have any other tips?

Sure, lots.

Besides keeping your site up-and-running while getting dugg, it’s important to keep records. I personally like Goole Analytics a lot, because it gives a lot of information I haven’t seen from any other log or traffic analyzer. By gathering these statistics you can analyze later what page(s) on your blog got dugg, where all those people came from and what else they did on your site.

Another tip is: make backups! This may sound as a stupid tip to most, but you won’t believe how many people run a blog without ever making a backup. This hasn’t much to do with getting dugg, but when terror strikes, you’ll be happy to have a spare copy of all your posts and comments.

Famous last words…

Well, to test out all my tips, please digg this story to see if we can get Ariejan.net dugg for once ;-)

Thank you for reading. Cheers.

Ariejan.net Link Party 07/08/2008

Tue, 08 Jul 2008 00:00:00 +0000

StarCraft 2 Gameplay
tags: starcraft, blizzard, gaming, share
Cheat Sheets for Front-end Web Developers
tags: frontend, cheat, cheatsheet, html, css, share
40 Photoshop Tutorials for Lighting and Abstract Effects | Vandelay Website Design
tags: vandelay, photoshop, tutorials, effects, share
Trollop: Simple Yet Powerful Command Line Option Processor
tags: trollop, ruby, cli, option processor, share
Typetester – Compare fonts for the screen
tags: fonts, design, typography, webdesign, css, tools, web, font, share
Unwakeable Theme at T. Longren
lso added some more functionality to this theme. For example
tags: unwakeable, longren, wordpress, theme, share

Zoek jij 'n uitdagende baan??

Fri, 06 Jun 2008 00:00:00 +0000

First off, sorry to all the English-reading people, but this post is intended for my Dutch audience.

Even wat updates voor mijn Nederlandstalige publiek. Ik heb wat nieuwtjes voor jullie, dus lees snel verder!

Allereerst wil ik even melden dat ik a.s. dinsdag (10 juni) te vinden zal zijn op RubyEnRails 2008. Dus, ben jij er ook, laat ’t me even weten! Laat even ’n commentaartje achter, of nog beter: bel/sms me even dinsdag op 06-17103624, dan spreken we elkaar zeker!

Dan nog even een oproep voor alle Nederlandse Ruby, Java en PHP developers. Als jij echt coole webapps kan (of wilt gaan) maken, dan moeten we praten! Kabisa zoekt namelijk op korte termijn Ruby en Java developers. Ik kan er zelf over meepraten: bij Kabisa werken is leuk, afwisselend en het biedt je alles wat je van een goede ICT job mag verwachten.

Spreek me dinsdag even aan (zie m’n nummer hierboven) als je meer wilt weten over werken bij Kabisa! Of maak gewoon voor de lol even ’n praatje!

Ben jij ’n Ruby, Java of PHP developer - en ben jij toe aan een leuke ICT baan? Neem dan ’ns contact met me op, want ik ben op zoek naar collega’s!

Tot dinsdag bij Ruby en Rails 2008!!!

The best IT books hand-picked for you!

Fri, 30 May 2008 00:00:00 +0000

Are you ready to dive into Rails? Want to familiarize yourself with the deeper dungeons of Ruby? Are you an aspiring game developer? Or maybe you just want to learn how to use Git or Subversion effectively?

In any case, I’ve opened up a little book shop with a hand-picked selection of books on a variety of subjects, included Ruby and Rails, Game Development (with Java) and Version Control.

Please, feel free to take a look around. All books are priced the same as Amazon.com, and you’d be supporting Ariejan.net in the process.

I’m also open to suggestions for other categories with some hand-picked books. I’m already planning on Java and Databases categories. Feel free to post a comment with your tips.

Thanks for shopping a Ariejan’s Book Shop!

The migration that cannot be undone: Irreversible Migration

Tue, 06 May 2008 00:00:00 +0000

Migrations have up and down methods, as we all know. But in some cases, your up method does things you can’t undo in your down method.

For example:

def self.up
 # Change the zipcode from the current :integer to a :string type.
 change_column :address, :zipcode, :string
end

Now, converting integers to strings will always work. But, you feel it coming, converting a string into an integer will not always be possible. In other words, we can’t reverse this migration.

That’s why we should raise ActiveRecord::IrreversibleMigration in the down method.

def self.down
 raise ActiveRecord::IrreversibleMigration
end

Now, if you run your migration (upwards), you’ll see it being applied like it shoud. However, if you try to go back, you’ll see rake aborting with ActiveRecord::IrreversibleMigration.

$ rake db:migrate VERSION=4
-- Database is migrated
$ rake db:migrate VERSION=3
-- Rake aborted!
-- ActiveRecord::IrreversibleMigration

So, if you have things you can’t undo, raise ActiveRecord::IrreversibleMigration in your migration’s down method.

How to: Compile packages on Debian/Ubuntu by hand

Sun, 04 May 2008 00:00:00 +0000

In some very rare situations you may find yourself in the need to recompile a Debian (or Ubuntu) package. Luckily for all of use, the great Debian packaging system makes this a piece of cake.

Let’s say we want to recompile mod_python for apache 2 to hook in to python 2.5, instead of the default 2.4.

First, get every thing installed you may need to build the libapache2-mod-python package.

$ apt-get build-dep libapache2-mod-python

Okay, next let’s grab the source for the package. The source will be unpacked to your current working directory, so it may be a good idea to create a seperate directory for this.

$ mkdir src
$ cd src
$ apt-get source libapache2-mod-python

In the case of this example, you don’t need to do anything special to get python 2.5 linked. Just install the python2.5 and python2.5-dev packages.

If you need to apply patches to the source or need to make any other modifications, this is the time!

Okay, now go to the source directory and build the package. This is the tricky command here:

$ cd libapache2-mod-python
$ dpkg-buildpackage -rfakeroot -b

This will build the package. You will get some warnings and errors about missing GPG keys. This is okay. You are not the package maintainer, so your packages should not be marked as ‘original’.

You’re now ready to install your compiled package.

dpkg -i ../libapache2-mod-python-3.3.1-3-i386.deb

That’s all! You compiled and installed a package from source!

GIT: Using the stash

Wed, 23 Apr 2008 00:00:00 +0000

I bet the following has happened to you: you are happily working on a project and are in the middle of something. You are not ready to commit your changes, because you your tests don’t pass yet. Then your client calls with a bug report that needs to be fixed right now. (You know how clients can be.)

So, what do you do? Throw away your current changes to make the patch? Checkout a clean copy of your project to make the changes? No! You just stash your changes away, and make the patch! Afterward you grab your changes back and continue work.

Git features The Stash, which is as much as a good place to store uncommitted changes. When you stash you changes, the will be stored, and your working copy will be reverted to HEAD (the last commit revision) of your code.

When you restore your stash, you changes are reapplied and you continue working on your code.

Stash your current changes

$ git stash save <optional message for later reference>
Saved "WIP on master: e71813e..."</pre>

List current stashes

Yes, you can have more than one!! The stash works like a stack. Every time you save a new stash, it’s put on top of the stack.

$ git stash list
stash@{0}: WIP on master: e71813e...

Note the stash@{0} part? That’s your stash ID, you’ll need it to restore it later on. Let’s do that right now. The stash ID changes with every stash you make. stash@{0} refers to the last stash you made.

Apply a stash

git stash apply stash@{0}

You may notice the stash is still there after you have applied it. You can drop it if you don’t need it any more.

git stash drop stash@{0}

Or, because the stash acts like a stack, you can pop off the last stash you saved:

git stash pop

If you want to wipe all your stashes away, run the ‘clear’ command:

git stash clear

It may very well be that you don’t use stashes that often. If you just want to quickly stash your changes to restore them later, you can leave out the stash ID.

$ git stash
# ...
$ git stash pop

Feel free to experiment with the stash before using it on some really important work.

Please leave a comment if you like this article, and would like to see more Git related stuff here on Ariejan.net.

Permanently redirect WordPress pages

Tue, 15 Apr 2008 00:00:00 +0000

After my trip to Mephisto some time back, I noticed that some pages were accessible from different URLs. After moving back to WordPress, these permalinks no longer work.

I’m running WordPress with Apache2, so it shouldn’t be too hard redirect those old permalinks to their new locations. (That’s what rewriting is all about anyway).

Here is the default .htaccess file generated by WordPress:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

As I said, it’s generated by WordPress. It would be very unwise to edit this, because our changes might get lost in the future.

The solution is simple. Add another block above the generated one that does permanent redirects. In this example I rewrite pages/svnsheet to http://ariejan.net/svncheatsheet.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^pages/svnsheet$ http://ariejan.net/svncheatsheet [R=301,L]
</IfModule>

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

FYI, the R=301 means the browser (or search bot) receives a “Moved Permanently” message. L means this is the last rewrite rule to read.

Of course, you can go nuts and add all sorts of funky regular expressions to rewrite URLs.

The nice thing is that WordPress will only replaces the block between the BEGIN and END tags, keeping your rewrites in tact.

Note: don’t forget to restart apache if your rewrites don’t seem to work.

Debian Etch: RMagick LoadError

Thu, 10 Apr 2008 00:00:00 +0000

If you’re on Debian Etch, you may encounter the following error

libMagickWand.so.1: cannot open shared object file: No such file or directory - /usr/lib/ruby/gems/1.8/gems/rmagick-2.3.0/lib/RMagick2.so

This basically means that the libMagickWand.so.1 file cannot be found. However, it is available on your system. All you need to do to fix it, is tell your box to look in the right place for the file.

To fix this issue once and for all, open up /etc/ld.so.conf.d/whatever_file_is_here. The whatever_file_is_here is named after the kernel you have installed.

In this file, add the following line at the bottom

/usr/local/lib

Save the file and next run the ’ldconfig’ command. This will reread the configuration file you just edited. Now, restart your Rails app and you’ll notice the error is gone and all is good again.

ldconfig

This change will be kept after you reboot, so you won’t encounter this error any time soon again.

Enabling Trac Email notifications

Wed, 09 Apr 2008 00:00:00 +0000

If you’ve ever reported a but to Ruby on Rails, you’ll have noticed that their Trac has nice email notification feature. And I bet you want that in your Trac as well!

Now, email notification are nothing exotic. You don’t need any plugins, just an outgoing SMTP server and access to your trac.ini file.

In your trac directory open up conf/trac.ini and look for the [notification] header. Make sure you have at least the following settings. Of course, make sure you enter valid values for your situation.

always_notify_owner = true
always_notify_reporter = true
always_notify_updater = true
smtp_always_cc = trac-updates@mailinglist # Optional if you want to archive outgoing emails
smtp_enabled = true
smtp_from = trac@ariejan.net
smtp_from_name = Ariejan.net Trac
smtp_replyto = noreply@ariejan.net
smtp_user =
smtp_password =
smtp_server = localhost
smtp_port = 25

There are a few more options like ‘ignore_domains’ if you don’t want to send emails to certain domains.

Update: good news for you GMail users! There is a ‘use_tls’ attribute. I think that if you set it to true, you can then specify GMail’s SMTP server to send out mails. Of course, a better solution would be to setup Postfix to relay mail to GMail, and just leave Trac to it’s default ’localhost’ settings.

Here we go again: WordPress 2.5

Wed, 09 Apr 2008 00:00:00 +0000

Okay, here we go then. I’ve managed to drop Mephisto after only a few weeks of service.

As a Rails developer, I liked the idea of running my own blog on something Rails. However, Mephisto was a big disappointment. Especially compared to WordPress 2.5. Mephisto has been on life support for quite a while now, and it’s just too complex to be easy to hack. Now, don’t get me wrong. I love hacking Ruby, but not too much on my blog. It “should just work”.

The reason I skipped to Mephisto in the first place was that my Wordpress 2.3 blog was very slow. I managed to track down the problem to the Twitter plugin, which was next to useless anyway.

Now, with all the new funky stuff in 2.5, I switched back. I haven’t managed to write anything substantial in the past weeks anyway, so it wasn’t a problem to loose one post.

There are many reasons for my not posting substantial stuff lately. One of them being my new hobby: photography (link). Check out my Flickr page and let me know what you think of my more recent photos.

So long for now. Expect more from me soon.

Rails Snippet: Caching expensive calls

Wed, 09 Apr 2008 00:00:00 +0000

In Rails, from time to time, you may encounter you have a method you call several times, but which returns always the same result. For example, have the following:

class Person < ActiveRecord::Base
 has_many :articles

 def get_approved_articles
 self.articles.find(:all, :conditions => {:approved => true}, :order => 'approved_on DESC')
 end
end

A query is fired every time you call Person#get_approved_articles. To cache the result of the query during this request, just add a bit of magic

class Person < ActiveRecord::Base
 has_many :articles

 def get_approved_articles
 @approved_articles ||= self.articles.find(:all, :conditions => {:approved => true}, :order => 'approved_on DESC')
 end
end

This will return the @approved_articles value if it exists. If it doesn’t, which is the first time you access the method, the query is run and stored in @approved_articles for later use.

Note: I know it’s much easier to define this kind of behaviour, but it’s just an illustration.

class Person < ActiveRecord::Base
 has_many :articles
 has_many :approved_articles, :class_name => "Article", :conditions => {:approved => true}, :order => 'approved_on DESC'
end

Ruby on Rails plugin: Throttler

Thu, 07 Feb 2008 00:00:00 +0000

For those of you who have missed it: I’ve released a plugin yesterday that allows you to throttle your Rails app.

Read the original announcement and installation/usage instructions Read how you can put Throttler to good use in your app

The Presidents of the United States of America Live

Fri, 25 Jan 2008 00:00:00 +0000

Yeah! I got myself some tickets to The Presidents of the United States of America, live at Effenaar, Eindhoven.

To be honest with you, I’ve never been to a real concert before, so this is going to be very much fun for me! Among the few albums I bought back in the year 1995 was PUSA.

Your help is needed! - Railsjobs.nl

Tue, 22 Jan 2008 00:00:00 +0000

Since I started Railsjobs.nl some time ago, I’d like to ask you a simple question. Thanks for putting in your $0.02 :)

{democracy:2}

Roles: Admins pretending to be users!

Sat, 19 Jan 2008 00:00:00 +0000

Most applications have some sort of role system. You have administrators, who can do basically everything, semi-administrators and “regular Joe” users.

Now, sometimes you want, as an administrator, be able to login as a user to solve some problem, or manage settings for a user, or whatever.

Harm did a very nice write-up on how we did this in a Rails project of ours. You can read the article here.

Attack of the Killer Bunnies

Tue, 15 Jan 2008 00:00:00 +0000

Bunny killing the garden Originally uploaded by ariejan

Well, here they go then! Just swooping in, eating all the green stuff from your garden!! Someone stop them!

This was taken in the back-yard at Laura's parents house. The neighbours have about a dozen rabbits, but they are allowed to walk around as they wish, occasionally finding their way into other people's gardens.

Kabisa Blog

Sun, 13 Jan 2008 00:00:00 +0000

Well, we’ve got our blog up and running at Kabisa now! Our entire team will be posting interesting Ruby, Rails, Java and Subversion articles that may be of very good use to you.

Check out the blog now on http://blog.kabisa.nl or grab the RSS Feed to stay up-to-date.

Review: Parking London

Thu, 03 Jan 2008 00:00:00 +0000

This is a paid review

I’ve been asked to review a service named Parking London. What this service does is two things, really. Firstly, it allows owners of garages and parking spaces in London to offer their spots to others. A location can be specified, with a small description. Of course, the price for the monthly tenancy or sale is added.

The service doesn’t look very special at a first glance. Just a listing of available parking spots in London. When I took a closer look, I noticed a link “Map view”. When I clicked it, a Google map popped up of the UK, and I was able to zoom in and see where all the available parking spaces where. Hovering over the pin points shows the monthly cost for them. Great!

Being a good reviewer, I setup a scenario where I was a potential buyer who’s looking for garage space in London. I have a budget of 250 pounds a month, and I want the garage to be in the north of the river Thames.

This proved very difficult indeed. With a service like this, I expect to be able to specify some search criteria. I can enter a ‘filter’ parameter, but this didn’t seem to do anything useful.

There is an option, however to search for locations and postcodes. I gave it a try, but again, I found the results next to useless to me.

I left the location issues for what they were, and focussed on the pricing. I’m not sure what a reasonable price for a garage is, but I set myself an imaginary budget of 250 pounds. I was not able to find any way to select items by price, or sort the listing by price to simplify my search.

In my opinion, this service is really basic. A lot of time went into making the Google map work correctly, and it really does. It’s easy to locate a parking space near a location you want.

Finding items with a certain pricing, or any other form of search was pretty much impossible. With a little work, this site might be a very useful tool for garage-seeking Londoners.

Write a DVD-Video from the Linux console

Mon, 31 Dec 2007 00:00:00 +0000

This is probably my last post for this year, and it’s not about Ruby on Rails! Or web development! It’s about how to burn a DVD-Video from the Linux console.

I know, there are tons of fancy apps (with a decent GUI) that allow you to create DVD-Video disks from Gnome or KDE. But, my Linux machine has no monitor attached and I don’t care for hogging up resources by running X.

All you really need to burn a DVD-Video is one packages: dvd+rw-tools. If you’re on Debian or Ubuntu, just run this:

apt-get install dvd+rw-tools

So, I have this VIDEO_TS folder and a blank DVD in my DVD driver (which is known to my system as /dev/sr0). To make a correct DVD, you’ll also need an AUDIO_TS folder. It doesn’t matter if it’s empty, it should be there.

I’m assuming you’re all doing this as root. If not, just add ‘sudo’ in front of every command you perform

mkdir AUDIO_TS

Next, we need to set the correct permissions on the VIDEO_TS, AUDIO_TS and the files contained in these directories.

chown -R root:root VIDEO_TS AUDIO_TS
chmod 500 VIDEO_TS AUDIO_TS
chmod 400 VIDEO_TS/*

Well, you already have all your files prepped, and you’re good to go.

The following command will burn you a nice video DVD. Make sure a blank DVD is inserted into your drive. As I said before, my drive is located at /dev/sr0. You system is probably different. Check your boot log (dmesg) for messages of a dvd driver.

growisofs -dvd-compat -Z /dev/sr0 -dvd-video .

It’s very important not to forget the tailing dot, it tells growisofs that you want to use the current directory as the source for your DVD. Depending on the speed of your burner, you’ll have a new DVD to watch in a few minutes. Enjoy! And a happy 2008!

For You: Merry Christmas and a Happy 2008!!!

Thu, 20 Dec 2007 00:00:00 +0000

We (Laura and I) wish you a very merry Christmas and a happy 2008!

I hope to welcome you back on Ariejan.net in 2008. Happy Holidays!

How to install MySQL on Ubuntu/Debian

Wed, 12 Dec 2007 00:00:00 +0000

It may seem easy for some, but for others, installing MySQL on Ubuntu or Debian Linux is not an easy task. This article explains to you how to install the MySQL Server and Client packages on a Ubuntu/Debian system.

First of all, make sure your package management tools are up-to-date. Also make sure you install all the latest software available.

sudo apt-get update
sudo apt-get dist-upgrade

After a few moments (or minutes, depending on the state of your system), you’re ready to install MySQL.

By default, recent Ubuntu/Debian systems install a MySQL Server from the 5-branch. This is a good thing, so don’t worry.

First, install the MySQL server and client packages:

sudo apt-get install mysql-server mysql-client

When done, you have a MySQL database read to rock ’n roll. However, there’s more to do.

You need to set a root password, for starters. MySQL has it’s own user accounts, which are not related to the user accounts on your Linux machine. By default, the root account of the MySQL Server is empty. You need to set it. Please replace ‘mypassword’ with your actual password and myhostname with your actual hostname.

sudo mysqladmin -u root -h localhost password 'mypassword'
sudo mysqladmin -u root -h myhostname password 'mypassword'

Now, you probably don’t want just the MySQL Server. Most likely you have Apache+PHP already installed, and want MySQL to go with that. Here are some libraries you need to install to make MySQL available to PHP:

sudo apt-get install php5-mysql

Or for Ruby:

sudo apt-get install libmysql-ruby

You can now access your MySQL server like this:

mysql -u root -p

Have fun using MySQL Server.

Run Internet Explorer 5, 5.5, 6 and 7 natively on Mac OS X Leopard or Tiger

Wed, 12 Dec 2007 00:00:00 +0000

As a web developer, you probably know all about browsers. They suck. Well, some more than others. But, if you develop apps for Windows users, you’ll have to test your app with Internet Explorer.

Now, as a good Rails developer, I’m using a Mac. I can test apps with FireFox, Safari and Opera without problems. But Internet Explorer is always a problem. I used Parallels for a while to run an instance of Windows XP to test with IE. But this in itself presented me with a problem: you can’t, without nasty hacks and a lot of trouble, run IE6 en IE7 side-by-side.

My only option seemed to run two Parallels sessions, one with IE6, the other with IE7. Since every images takes about 5 Gb of disk space, I just wasted 10 Gb of disk space to test web apps with Internet Explorer. This is kind of ridiculous.

But, now there is a very, very nice solution to this problem that every Mac-oriented web developer should know about: ies4osx This solution uses darwine (±70Mb), a wine version for OS X, using X11 and allows you to install IE 5, 5.5, 6 and even IE7. All at once, right in Mac OS X. You can run any of these browsers, from the same system, all at once in parallel!

Although this allows me to use Internet Explorer the way I want it to, I’m still not a fan…

Wil jij mijn collega zijn? - Kabisa is hiring!

Fri, 07 Dec 2007 00:00:00 +0000

Zoals je waarschijnlijk al wel weet, werk ik al een tijdje bij Kabisa, en ik ben op zoek naar nieuwe collega’s!

Ben jij een een ICT’er (liefst met Java of Ruby ervaring, maar dat is niet vereist), laat dan even je gegevens achter op jobs.kabisa.nl.

Ook als je iemand kent die ICT’er is, en toe is aan een nieuwe uitdaging, kun je deze persoon opgeven op jobs.kabisa.nl.

De eerste 100 aanmeldingen ontvangen een Nationale Bioscoopbon. Bovendien wordt tussen alle aanmeldingen op 29 februari 2008 een Apple iPod Touch verloot!!

Heb je vragen over Kabisa? Neem gerust even contact op met me op.

English version, for the non-Dutch folk

As you may already know, I’ve been working at Kabisa for quite some time. Now, I’m looking for new colleagues.

Are you a programmer (with Java or Ruby experience, but it’s not required), please leave your credentials at jobs.kabisa.nl.

If you know a programmer who is ready for a new challenge, drop his or her credentials (and your own) at jobs.kabisa.nl.

The first 100 applications will be rewarded with a national cinema voucher. At February 29th, 2008 we will also pick one of the applications and reward it with an Apple iPod Touch!

Do you have any questions about working at Kabisa? Feel free to contact me now.

Rails: calculated column caching

Thu, 06 Dec 2007 00:00:00 +0000

Sometimes you’re working on a Rails project and you think: “hey! This should be easy!”. Well, most of the time it is. I’m working on a project that allows people to rate objects (what they really are doesn’t matter at all).

I’m using the acts_as_rateable plugin which creates an extra database table containing all ratings. I also have a table with my objects. Using the plugin I’m now able to do the following:

obj = Object.find(:first)
obj.add_rating Rating.new(:rating => 4)
obj.add_rating Rating.new(:rating => 5)
obj.rating
=> 4.5

This works all perfectly, until you want to sort objects by rating. You could construct a huge SQL query to join the two tables, but that’s not really efficient, especially when your database gets bigger.

The solution is very easy and even more elegant. Use a cache! For this, you’ll first have to add a new field to the objects table. Do this in a migration:

add_column :objects, :rating_cache, :float

Now, in the Object model, add the following method:

def rate_with(rating)
 add_rating(rating)
 update_attribute('rating_cache', self.rating)
end

You’ll need to change your controller from using #add_rating to #rate_with. The syntax is exactly the same. Now, when you add a rating, we also store the average rating in the rating_cache column.

To get back to the sorting problem, you can now use the rating_cache column to sort Objects.

Object.find(:all, :order => 'rating_cache DESC')

Of course, you can use this trick on all sorts of relations. Have fun with it.

MySQL: (Re)set the auto-increment value of a table

Fri, 30 Nov 2007 00:00:00 +0000

Sometimes it’s necessary to set the starting point of a MySQL auto-increment value.

Normally, MySQL starts auto-incrementing at 1. But let’s say you want to start at 10.000, because you want at least a five figure number. You can use the following query to set the MySQL auto-index:

ALTER TABLE some_table AUTO_INCREMENT=10000

If you want to delete all records from your table and restart auto-index at 1, you might be tempted to run a DELETE query, followed by the above example, setting the auto increment value to 1. There is a shortcut, however:

TRUNCATE TABLE some_table

This will basically reset the table, deleting all data and resetting the auto increment index. Do not that the truncate command is a hard-reset option. For instance, any triggers “ON DELETE” will not be fired when using truncate.

RailsJobs.nl - Ruby on Rails Jobs in The Netherlands

Tue, 27 Nov 2007 00:00:00 +0000

The time is now ripe to announce RailsJobs.nl - A shiny new jobboard for Ruby on Rails developers who are seeking a job in the Netherlands or Belgium. I started the site a few days ago and business and job seekers have started to find RailsJobs.nl.

If you’re a business looking for Rails developers, feel free to add your job now. Use the code ‘RJINTRO’ to get a free 30-day trial period.

RailsJobs.nl is a private site, which I developed during the weekend. At first I planned the site to be free for all, but I’ve decided to charge businesses for posting jobs. This will ensure no spam, and only quality job offerings. It will also help me cover hosting bills and I can invest some of the money in advertising the site on quality websites.

Well, I’m not sure what to say more. Just hop over to RailsJobs.nl and see for yourself.

Oh, I have implemented a handy RSS Feed as well, so you can easily stay up to date on new job offerings.

Bash it! - Number of messages in Postfix queue

Thu, 15 Nov 2007 00:00:00 +0000

Got bash? Here’s a nice snippet that will return the number of messages currently in the postfix queue.

postqueue -p | tail -n 1 | cut -d' ' -f5

Feel free to post any updates or improvements.

Party time!

Wed, 17 Oct 2007 00:00:00 +0000

It’s that time of the year again… my birthday. I’m turning 27 today and I’ll be celebrating this today with friends and family.

As to let you know what I’ve been up to the past few weeks: Flickr Phototset, FilmBASE, Linked-in and some “classified” stuff I can’t link to.

Feel free to congratulate me with a comment, or drop me an e-mail.

And oh, me and Kabisa are accepting Rails projects, if you have any.

Google increases storage!

Mon, 08 Oct 2007 00:00:00 +0000

Google today increased the storage available to Google App users.

Before free app users (like me) has 2Gb available, and premium users could store up to 10Gb of mail and documents.

Today, Google upgraded the premium accounts to 25Gb! Store for the free version of Google Apps has been leveled with the current Gmail service (2910Mb at the moment).

Flash not clearing after a request?

Wed, 26 Sep 2007 00:00:00 +0000

We all know “The Flash” to be a very useful tool in almost every application we write. What does “The Flash” actually do?

The flash provides a way to pass temporary objects between actions. Anything you place in the flash will be exposed to the very next action and then cleared out.

Well, that's all nice, but what if you notice that your flash is not cleared, and is shown in one or more subsequent requests as well?

The reason your flash is not cleared is that it will only be cleared if you render the current action, or if you redirect to another action. If you render another action or template, the flash will not be cleared out.

Solving this is quite easy, actually. You only need to know how to do it. Use the following code in combination with render to show it only for the current action:

flash.now[:notice] = "Render, render! This will only show up once."

That’s all there is to it really.

The Glorious Canon EOS 400D Digital

Tue, 25 Sep 2007 00:00:00 +0000

Well, I finally made the big leap into (semi-)professional photography. I bought a Canon EOS 400D Digital SLR Camera. (That’s the XTi Rebel for you American folk) This nifty machine allows me to take very beautiful pictures, and I already had a little practice yesterday.

With it’s 18-55mm lens, I can make quite nice close-up photo’s, as you can see in the example below.

You can check out my personal Flickr page to see what new pictures I've taken lately. (No, I'm not currently available for photo shoots, thank you very much.)

Wordpress 2.3 released!

Tue, 25 Sep 2007 00:00:00 +0000

Today Wordpress 2.3 was released (see the release notes). I’ve already updated Ariejan.net, of course, and I’m now able to use all of the following neat new features:

Tagging (you can convert your categories to tags if you like)
Update Notifications when new versions of your plugins or WordPress are released
Improved post and draft management
Mark articles as 'pending review'
A new and improved visual editor (which doesn't suck! Wow!)

I highly recommend you check out the the current WordPress release from SVN. This will make future updates very easy. Read more on that here.

So, go ahead, grab that new WordPress!

Rails 2.0 New Features

Mon, 24 Sep 2007 00:00:00 +0000

As David Heinemeier Hansson already told us all during his RailsConfEurope 2007 keynote, it’s time to take off the party hats. It’s no longer at time to celebrate all the new stuff we get. It’s time to celebrate what we have already.

With this statement DHH ends the revolution of Rails. During the past three years a lot of new and exiting features were added to Rails. However, now the time has come to evolve Rails further. No more new and exiting stuff, but fine tuning. Making things even better than they already are.

So, Rails 2.0 will not contain any major new features. But don’t despair, there are quite a few nifty changes that you’ll like to know about.

1. HTTP Authentication

HTTP Authentication is a great way to limit access to specific areas of your application that regular users won’t access. It might even be a good second layer of security for your administrator interface. Rails 2.0 will include easy methods to get HTTP authentication working for you.

2. HTTP Performance

The performance of a web application goes down the drain when you add too much JavaScript and stylesheets. Each and every file must be downloaded separately. Rails 2.0 will be able to take all the javascript files, stuff ’em together, compress that one file and sent that to the client, where it will be cached.

<%= javascript_include_tag :all, :cache => true %>
<%= stylesheet_link_tag :all, :cache => true %>

3. Asset Server

If you have a large (and busy) site, serving static files can be quite a performance issue. Rails 2.0 adds the notion of an “asset server”. An asset server (in combination with item #2) will serve static content quickly, allowing your app to respond even faster to a user’s request. To enable asset hosts, add the following line to your configuration:

config.action_controller.asset_host = 'assets%d.example.com'

You can even cycle through multiple asset servers by simply creating the appropriate CNAME records in DNS. Neat, eh?

3. The Debugger

“The Debugger” is here! Simple add ‘debugger’ to your app, and it will stop just there for you to step in and debug your application. This is a very powerful way debugging your apps quickly and with ease.

4. Query Cache

Query Caching is something that normally happens in a database server, not now. ActiveRecord keeps a cache of queries and uses it when the same query is made again. You don’t have to worry about anything, not even expiring your cache, because this is done automatically when you insert, update or delete any data that’s related to the cache.

5. Environment Configuration

If you have ever developed a serious Rails application, you know that your environment.rb can become a mess very quickly. Rails 2.0 allows you to split up your configuration into multiple files to keep things clean. (And you can also reuse your common configuration files for other apps as well.)

6. ActionTemplate Renderer

Say good bye to the trusty old rhtml and rxml files! There are new conventions for templates that will tell you more about what type of file the generate and which parser is used to render the templates.

Normal HTML views will get the .html.erb extension from now on (formerly .rhtml). If you want to build an XML file with “Builder”, you would use: template.xml.builder.

7. Sexy Migrations

Sexy Migrations have been around for some time as a plugin. Some people love them, others don’t. I still have my doubts about it, but at least it’s less code to write, and it keeps things more readable, which is good, I guess.

Let’s take an old migration:

create_table :people do |t|
 t.column :first_name, :string, :null => false
 t.column :last_name, :string, :null => false
 t.column :group_id, :integer
 t.column :description, :text
 t.column :created_at, :datetime
 t.column :updated_at, :datetime
end

We now have this:

create_table :people do |t|
 t.integer :group_id
 t.string :first_name, :last_name, :null => false
 t.text :description
 t.timestamps
end

8. Plugin Mania

A lot of methods/features have been deprecated in Rails 2.0. Some non-core components like in-place editing have been moved to separate plugins. You will also find plugins like resource_feeder and open_id_authentication. Let’s hope that OpenID finally takes off now.

9. Thou shalt use MIT

This one I hadn’t heard at the keynote, but had to read it on the internet.

“Due to licensing confusion for people using plugins, the new plugin generator now creates a license file by default. The assumption is that you are distributing under the MIT license otherwise you have to modify the file to meet your needs."

I think Rails 2.0 will be quite a nice. Apps will be a bit more manageable and performance will be better also. Now, it’s just waiting for DHH to release the “2.0 Preview Release” he promised.

Whooop - here it is! The new Ariejan.net!

Mon, 24 Sep 2007 00:00:00 +0000

Wow! Here it is then! The fully restyled, revamped and repimped Ariejan.net! (I hope you like it). So, “what’s changed?”, you may ask.

I've upgraded to WordPress 2.3, which has very sexy tagging support.
A new template with a bit more of my personal taste included (blue!)
A major division of articles: Blog posts (which are regular messages from me to the world) and featured articles (which are the good stuff you come for).
I've finally written the about page (although a photo is still missing)
I've got Flickr support lined up - ready to go when I can afford myself a Canon 400D SLR Camera (or Rebel XTi for you American folk). (help me out?)

My goal is to publish a bit more on the advanced Rails topics I encounter these days at work. Also, I’m still working on my Subversion Kickstart e-book, which will be published before the end of this year. (Thanks to Josh Blair for pointing out I was still working on it.)

If you have any Rails related questions (especially if you live in Europe) please let me know. We at Kabisa are always happy to free up some of our time to help you out.

Thanks for reading! Please grab my RSS Feed to stay up to date auotma

RailsConf Europe 2007!

Mon, 17 Sep 2007 00:00:00 +0000

Well, after a long trip, I arrived at my hotel yesterday in Berlin. Today I, and my mates from Kabisa, have joined RailsConf Europe 2007. I don’t think I’ll be giving a full, in-detail report of everything that happens, but I will let you know anything worth your (and my) time.

If you happen to be in Berlin these days, feel free to drop me an e-mail at ariejan at ariejan.net and maybe we can have chat or something. I hope to hear from you!

10 reasons why Microsoft's 10 reasons not to use Google Apps suck

Tue, 11 Sep 2007 00:00:00 +0000

You may have already read the 10 reasons why Microsoft thinks the enterprise should not use Google Apps. Well, here’s my response:

1. Google touts having enterprise level customers but how many “USERS” of their applications truly exist within the enterprise?

Does this matter? There are a lot of private users on Google Apps for sure, but there are quite a few large companies (with 10.000+ users) using Google Apps - and they are happy with it! How many enterprise users can say the same about Microsoft Windows and Office in the enterprise?

2. Google has a history of releasing incomplete products, calling them beta software, and issuing updates on a “known only to Google” schedule – this flies in the face of what enterprises want and need in their technology partners – what is Google doing that indicates they are in lock step with customer needs?

This one makes me laugh. Google is being honest about their software not being finished. And they don’t get hopes up by setting a release date that they can’t meet. I don’t think I have to go into a discussion about how Windows Vista doesn’t include all the promised features and the several delays before it’s final release.

3. Google touts the low cost of their apps –not only price but the absence of need for hardware, storage or maintenance for Google Apps. BUT if GAPE is indeed a complement to MSFT Office, the costs actually become greater for a company as they now have two IT systems to run and manage and maintain. Doesn’t this result in increased complexity and increased costs?

Very funny, again. There is no longer the need for Microsoft Office or the services that provide e-mail services like Exchange. There may even not be a need for Microsoft Windows anymore, because Google Apps works perfectly on Macs and Linux. And I think that managing Google Apps is a lot easier than managing a cluster of Microsoft servers.

4. Google’s primary focus is on ad funded search. Their enterprise focus and now apps exist on the very fringe and in combination with other fringe services only account for 1% of the company’s revenue. What happens if Google executes poorly? Do they shut down given it will them in a minimal and short term way? Should customers trust that this won’t happen?

This reason is actually the only one that might be valid. But, isn’t this true for any company? Google does generate a lot of revenue from it’s ad system. What Google tends to do with it’s Google Apps suite is not only providing an alternative for Microsoft’s office, but also binding users to Google. A Google Apps users will be more likely to use Google Search than a non-apps users.

5. Google’s apps only work if an enterprise has no power users, employees are always online, enterprises haven’t built custom Office apps – doesn’t this equal a very small % of global information workers today? –On a feature comparison basis, it’s not surprising that Microsoft has a huge lead.

Microsoft is still ignoring that “less is more”. It’s not about who has the most features anymore. It’s about ease of use and service. Most users -are- always online with broadband internet connections and WiFi. Office does have a lot of nifty features, but only a small % of power users use these features (if they even know the features exist and know how to use them properly).

6. Google apps don’t have essential document creation features like support for headers, footers, tables of content, footnotes, etc. Additionally, while customers can collaborate on basic docs without the above noted features, to collaborate on detailed docs, a company must implement a two part process – work together on the basic doc, save it to Word or Excel and then send via email for final edits. Yes they have a $50 price tag, but with the inefficiencies created by just this one cycle, how much do GAPE really cost – and can you afford the fidelity loss?

Again, Microsoft thinks we can’t do without it. Google Apps is not about creating huge reports and stuff. It’s about collaboration. Users collaborate, and when the document is cleared for release (or whenever you need the fancy stuff), you can use any tool (Pages? Adobe InDesign?) to make it all look sexy. Just a simple question: how many of the documents and spreadsheets you make are for pubishing and which are just quick doc for gathering and sharing thoughts with your co-workers? Personally, I use Google Docs for writing all my documents. When it needs to go to press, I export it and format it properly with Pages. No problem.

7. Enterprise companies have to constantly think about government regulations and standards – while Google can store a lot of data for enterprises on Google servers, there is no easy to use, automated way for enterprises to regularly delete data, issue a legal hold for specific docs or bring copies into the corp. What happens if a company needs to respond to government regulations bodies? Google touts 99.9% uptime for their apps but what few people realize that promise is for Gmail only. Equally alarming is the definition Google has for “downtime” – ten consecutive minutes of downtime. What happens if throughout the day Google is down 7 minutes each hour? What does 7 minutes each hour for a full work day that cost an enterprise?

Again, this makes me laugh! How many minutes a day do you spend restarting, scanning for viruses and spyware on Windows? How often do you need to make backups because you need to restore/reinstall Windows? What is the up-time guarantee of an Exchange server (yes, including possible hardware failure)? There are government rules everywhere, but in my opinion this is a policy thing.

8. In the world of business, it is always on and always connected. As such, having access to technical support 24/7 is essential. If a company deploys Google Apps and there is a technical issue at 8pm PST, Sorry. Google’s tech support is open M-F 1AM-6PM PST – are these the new hours of global business? And if a customer’s “designated administrator” is not available (a requirement) does business just stop?

“Always on and always connected”. Nice. In reason #5 Microsoft points out that users are not always online. What about that? How about the companies internet provider? Are they on call 24 hours a day? Just a note, when I want telephone support from Microsoft I can call them from 6am - 6pm (Pacific) and it’ll cost me $245. If you call after-hours, you’re set back $490. How much did you say Google’s support cost?

By the way, Google does monitor its services/servers very well. If there’s a problem, it will be fixed rather quickly. With Microsoft, you’ll have to go and fix it yourself (even when you pay $490).

(link to Micorsoft’s support page)

9. Google says that enterprise customers use only 10% of the features in today’s productivity applications which implies that EVERYONE needs the SAME 10% of the feature when in fact it is very clear that in each company there are specific roles people play that demands access to specific information – how does Google’s generic strategy address role specific needs?

Google offers a flexible service. In stead of customizing the crap out of the application for every possible role a user could have, they offer a generic application. It’s the ‘one size - fits all’ principle.

10. With Google apps in perpetual beta and Google controlling when and if they rollout specific features and functionality, customers have minimal if any control over the timing of product rollouts and features – how do 1) I know how to strategically plan and train and 2) get the features and functionality I have specifically requested? How much money does not knowing cost?

This just point 2 rephrased. To be more specific: do you have to plan? No - Google does the planning and testing for you. Training? Google keeps it’s apps simple and easy to understand. In terms of planning, you probably don’t know when something is released, but the same can be said for Microsoft. People expected WinFS in Vista, where is it? How many delays did Windows Vista see? How much did knowing when it was released (and when it got delayed, again) cost?

I have written this post not to bash Microsoft, but to illustrate how single minded some companies can be. It’s not just 10 questions enterprises need to ask themselves, it’s indeed 10 reasons not to use Google Apps. Although for some of these reasons may be valid for some companies, the majority of enterprises and users can do their daily work with Google Apps, and in my opinion, they should.

Currently I use Google Apps (free edition) to manage Ariejan.net. At Kabisa ICT we use Google Apps premier with great pleasure. We have not encountered any issues at all, and we really like the collaboration features included.

Content_for, yield and making sure something gets displayed

Sat, 01 Sep 2007 00:00:00 +0000

You may have heard of a very nice Rails technique that used content_for and yield to stuff custom blocks of content into a layout. For example, in a view you could add a block like this:

<% content_for :sidebar do %>
 This goes into the sidebar when viewing this action!
<% end %>

It doesn’t matter where you put it in the view, because, as you may notice, the content within the content_for block is not shown in the view. The layout can pull the content for :sidebar and show it in a custom place, mostly the sidebar:

<%= yield :sidebar %>

Nice, you now don’t have to hard-code the sidebar (and it’s layout elements) in your views everytime. You can even move the sidebar to another place, without breaking any of your views. How great is that?

What does break your views, however, is not assigning anything to :sidebar. If you don’t assign anything to the :sidebar, nothing will be shown, which might break your layout. (Empty div’s tend to do that.)

So, how can you solve this? Quite easily, actually. What you want, probably, is display a default sidebar when no custom sidebar has been assigned. you can do this with the following line of über sexy Ruby code:

<%= (sidebar = yield :sidebar) ? sidebar : render(:partial => 'shared/sidebar') %>

First, you grab the :sidebar content and stuff it into a variable named, you guessed it, ‘sidebar’. If this sidebar containts anything, show it. If it doesn’t, you render a partial in the app/views/shared directory named _sidebar.rhtml.

Well, there you are. Go enjoy yourself now.

Blueprint 0.5 Rails Plugin released

Fri, 31 Aug 2007 00:00:00 +0000

A few days ago BlueprintCSS 0.5 was released (read the Olav’s posts here). I’ve updated the plugin accordingly. The most important change is the use of 24 (!) instead of 14 columns.

Installation and usage of the plugin have not changed. See my original announcement for more information.

BlueprintCSS Rails Generator

Mon, 27 Aug 2007 00:00:00 +0000

This plugin is no longer available. Blueprint nowadays ships with a very good ‘compress’ script that allows you to generate all kinds of nice BluePrint layouts. Having a plugin to just copy some files seems a bit excessive.

I think that, if you’re a web developer, you’ve seen the BlueprintCSS framework. BlueprintCSS offers quite a bit of CSS code that allows you to quickly and easily build a grid-based layout, using pure CSS.

That’s, of course, all very nice, but you should be able to plug it in into your Rails app. And now you can!

Install my plugin:

./script/plugin install http://svn.ariejan.net/plugins/blueprint

And then generate as many BlueprintCSS layouts as you’d like. To create a layout for your posts controller, simply run the following command:

./script/generate blueprint posts

This will create a posts.rhtml template in app/views/layouts, and add the proper CSS and images to your application. That’s all!

Note 1: You may remove a few lines of inline CSS from your layout to remove the supporting background images.

Note 2: Bugs and feature requests go into Trac.

Super Simple Authentication Plugin and Generator

Fri, 24 Aug 2007 00:00:00 +0000

I hereby proudly announce my Super Simple Authentication plugin and generator.

All right, what does it do? Sometimes you need to protect your actions and controllers, but you don’t want to go about installing restful_authentication or anything like that. Adding a simple password for certain actions would suffice. So, I wrote a little plugin that can generate some code for you that allows you to easily protect your app with a simple password.

To get started, you must first install the plugin in your rails application:

script/plugin install http://svn.ariejan.net/plugins/super_simple_authentication

When the plugin is installed, you may generate your SSA controller. This controller verifies your password and makes sure you stay authenticated for the duration of your visit.

script/generate super_simple_authentication sessions

Your password is located in config/super_simple_authentication.yml. Change it.

In the SessionsController, you’ll find an include statement. Move this include to your application controller:

include SuperSimpleAuthenticationSystem

The generator automatically added routes to your config/routes.rb file. If you want easy access to login and logout functionality, add these two lines to your config/routes.rb file as well:

map.login '/login', :controller => 'sessions', :action => 'new'
map.logout '/logout', :controller => 'sessions', :action => 'destroy', :method => :delete

You can now protect you actions and controllers with a before_filter:

# Protect all actions in the controller
before_filter :authorization_required

# Protect all actions, except :index and :recent
before_filter :authorization_required, :except => [:index, :recent]

# Protect only :destroy
before_filter :authorization_required, :only => :destroy

In your views, you can check if you are authorized or not with authorized? E.g.

<% if authorized? %>
<!-- do secret admin stuff -->
<% end %>

Please visit http://trac.ariejan.net to report bugs. Ariejan.net will keep you updated on new major version. Please subscribe to the RSS Feed.

I hope you enjoy this plugin. Please post a comment if you use it in your project, or if you just like it. Bugs, feature requests and support requests should go into Trac

Using Iconv to convert UTF-8 to ASCII (on Linux)

Tue, 21 Aug 2007 00:00:00 +0000

There are situations where you want to remove all the UTF-8 goodness from a string (mostly because of legacy systems you’re working with). Now, this is rather easy to do. I’ll give you an example: çéß

Should be converted to cess. On my mac, I can simply use the following snippet to convert the string:

s = "çéß"
s = Iconv.iconv('ascii//translit', 'utf-8', s).to_s # returns "c'ess"
s.gsub(/\W/, '') # return "cess"

Very nice and all, but when I deploy to my Debian 4.0 linux system, the I get an error that tells me that invalid characters were present. Why? Because the Mac has unicode goodness built-in. Linux does not (in most cases).

So, how do you go about solving this? Easy! Get unicode support!

sudo apt-get install unicode

Now, try again.

Bonus

If you want to convert a sentence (or anything else with spaces in it), you’ll notice that spaces are removed by the gsub command. I solve this by splitting up the string first into words. Convert the words and then joining the words together again.

words = s.split(" ")
words = words.collect do |word|
 word = Iconv.iconv('ascii//translit', 'utf-8', word).to_s
 word = word.gsub(/\W/,'')
end
words.join(" ")

Like this? Why not write a mix-in for String?

I’m back in business!

Mon, 20 Aug 2007 00:00:00 +0000

Ariejan.net is (almost) back in business. With some minor cosmetic improvements to the theme, I’m ready for today.

Tomorrow I will reorganise my tags/categories a bit and add some new fancy plugins that will help searching visitors a hand in finding what they need. (More about that later, I promise).

I hope you like the improved style of Ariejan.net. (If you don’t, feel free to make a redesign and mail it to me.)

That’s all for now. Bye bye.

Kabisa, RailsConf Europe and Ariejan.net

Mon, 20 Aug 2007 00:00:00 +0000

Hello there! Just a quick post to let you know I’m still alive and kicking! Two weeks ago I started my job at Kabisa and I’ve been very busy with that. I really like working at Kabisa and we have some great things planned for the near future. (We’re currently working on a few client projects that need our attention.)

I’d also like to let you know that I’ll be attending RailsConf Europe 2007 this year! I’m really looking forward to the conference and the tutorial day on monday. If you’re attending, please let me know, so we can meet in person.

Then I have a small announcement regarding Ariejan.net I’m going to reorganise my VPS tonight which may cause Ariejan.net to be off the air for a few moments. I’ve been considering moving Ariejan.net to Mephisto, but Wordpress is just a way better solution at the moment. (I do have to migrate 125+ posts, comments and reformat an reorganise everything when moving to Mephisto.)

Suffice it to say that I’ve been a very busy bee the past few weeks. I hope to show you some of my Rails work in the near future when we’ve rounded up some of the client projects were working on. For now – stay tuned!

I’m Back!

Wed, 01 Aug 2007 00:00:00 +0000

All right, I’m back!

Yes, my holiday was nice. Thanks for asking. Now, I’ve been looking at the poll in my previous post and I’ve made up my mind.

I’m going to look into writing a blogging system in Ruby on Rails. I don’t want to copy typo or Mephisto or anything, but just put my own thoughts together, add a bit of spice you guys (and girls?) have to offer and release it some time.

Just to be clear (and prevent myself from getting a lot of mail about this): I’m going to see if I can make time to work on a project like this. That’s all. If I can, I’ll let you know. If I can’t.. well too bad, I guess :)

I’m going to, as I did before, write a lot of articles about Subversion and Ruby on Rails. Maybe I’ll add some new topics too, but those will pop up when they do. Just keep an eye out for news here at Ariejan.net (tip: grab my RSS Feed).

Also, I’m now available to do Ruby on Rails projects for real! Just contact me and we can discuss details at your leisure.

Ariejan.net - What’s next?

Mon, 16 Jul 2007 00:00:00 +0000

Vote now! What’s Next for Ariejan.net? Your opinion counts!

After a few days (or were it weeks?) of silence, an update on Ariejan.net. What’s been happening?

First of all, I recently graduated and am now officially a Bachelor of ICT. Since my current employer didn’t have the same views regarding my future as I did, I decided to go “shopping” for a new job.

I found that new job with a small business named Kabisa ICT. The people at Kabisa and I found common ground very quickly with Ruby on Rails and Agile Development. This, to my surprise, resulted in a job offer very quickly… and I accepted.

The next two weeks I’ve got a holiday planned to the French Alps with Laura. We hope to get some rest there while enjoying the beautiful scenery.

When I get back from France, I start my new job at Kabisa. I hope to learn a lot of new stuff while working there. Hopefully I’ll be able to share that experience with your here, at Ariejan.net, which leads me to the following question.

What’s Next?

What’s next for Ariejan.net? Since all the Web 2.0 buzz is all about communities and all that, I want your opinion on what to do with Ariejan.net.

Please vote in the poll below and let me know what your thoughts are. When I get back from my (well earned) vacation, I’ll check back here and see what you, my guests, really want.

You have the option of adding new stuff. Use it wisely, please!

{democracy:1}

Thanks for voting! It’s greatly appreciated!

Clear DNS Cache on your router

Sat, 07 Jul 2007 00:00:00 +0000

I currently have a Linksys router at home that has the DD-WRT Firmware on it. I’ve been using it for quite some time now, and I’m very happy with it.

In my previous post I mentioned there was some trouble with DNS for Ariejan.net. I’ve changed nameservers and there’s always something that goes wrong.

Anyway, my router runs DNSMasq, a caching nameserver for my local network. (What this does is, it stores DNS queries and when the same request is made later on, the response is already here (on my network), and my ISP’s nameservers don’t have to be queried. This makes for a great speed optimalization!)

The problem was, that my router was caching parts from the old and parts from the new nameservers for Ariejan.net. I could have waited 24-48 hours to let DNSMasq figure it all out by itself, but I’m not that patient. What I needed to was reset the cache DNSMasq had built.

What you need to do is login to the admin panel of your router and enable SSH access (for your own network, not for the outside world!).

You SSH to your route, probably as the ‘root’ or ‘admin’ user. In this case the IP is your routers IP address.

$ ssh -l admin 10.0.0.1

Once you’re in, you can issue the killall command to issue the ‘HUP’ signal to DNSMasq. This does not kill your DNSMasq, but tells it to flush the cache an reread its configuration, thus starting over with a clean slate.

The ‘HUP’ signal has the number ‘1’ so you can run the following command once logged in:

$ killall -1 dnsmasq

Now you can logout, and notice that your cache was indeed cleared.

This method should work on every Linux based router that uses DNSMasq and support SSH logins.

How to write a Rails Plugin (for controllers)

Fri, 06 Jul 2007 00:00:00 +0000

A few days back I posted my very first Rails plugin, Acts As Exportable. Although writing a plugin is rather easy, you must know a few tricks on how to get things going.

This article will show you how to develop a plugin that adds functionality to a controller (other plugins, e.g. for models) will follow later. In fact, I’ll explain to you how I developed my Acts As Exportable plugin.

Let’s take a basic Rails application for starters. You have setup a model with some attributes and a scaffolded controller that allows you to CRUD your items. In this tutorial I’ll be working with books. The model is named ‘Book’ and the controller ‘BooksController’. Start your web server now and add some random data to play with.

Before you dive into writing a plugin for the controller to export data to XML you should have some basic functionality in your controller first. I’ve found it easier to develop my code in the controller first, and then port it to a plugin.

So, add a new method to your BooksController that’ll export books to XML. This looks quite easy:

def export_to_xml
 books = Book.find(:all, :order => 'title')
 send_data books.to_xml,
 :type => 'text/xml; charset=UTF-8;',
 :disposition => "attachment; filename=books.xml"
end

Now, call /books/export_to_xml and you download a real XML file containing all your books! To make things a bit more complicated, we want to be able to feed this method some conditions to select books. A nice solution is to add a special method for this that defines these conditions. (You could also use them in listing books, for example.) I add a new method to the BooksController:

def conditions_for_collection
 ['title = ?', 'some title!']
end

The condition is of the same format you can feed to find. Here you could, for example, select only the books belonging to the currently logged in user.

Next, update the export_to_xml method to use these conditions

def export_to_xml
 books = Book.find(:all, :order => 'title', :conditions => conditions_for_collection)
 send_data books.to_xml,
 :type => 'text/xml; charset=UTF-8;',
 :disposition => "attachment; filename=books.xml"
end

Nice that’s it. Now, you like what you’ve made so far, and want to stuff it into a plugin and put it on your weblog. Here’s how to go about that.

Creating the plugin

First, generate the basic code for a plugin:

./script/generate plugin acts_as_exportable

This will create a new directory in vendor/plugins containing all the basic files you need. First, we’ll take a look at vendor/plugins/acts_as_exportable/lib/acts_as_exportable.rb. This is where all the magic happens.

What we want is to is add a method to ActionControllerBase that allows you to easily enable the plugin in a certain controller. So, how do you want to activate the plugin? Right, you just call ‘acts_as_exportable’ from the controller, or optionally, you add the name of the model you want to use.

acts_as_exportable
acts_as_exportable :book

The vendor/plugins/acts_as_exportable/lib/acts_as_exportable.rb contains a module that’s named after our plugin:

module ActsAsExportable
end

Next, we add a module named ‘ClassMethods’. These class methods will be added to ActionController::Base when the plugin is loaded (we’ll take care of that in a moment), and enable the functionality described above.

module ActsAsExportable
 def self.included(base)
 base.extend(ClassMethods)
 end

 class Config
 attr_reader :model
 attr_reader :model_id

 def initialize(model_id)
 @model_id = model_id
 @model = model_id.to_s.camelize.constantize
 end

 def model_name
 @model_id.to_s
 end
 end

 module ClassMethods
 def acts_as_exportable(model_id = nil)
 # converts Foo::BarController to 'bar' and FooBarsController
 # to 'foo_bar' and AddressController to 'address'
 model_id = self.to_s.split('::').last.sub(/Controller$/, '').\
 pluralize.singularize.underscore unless model_id

 @acts_as_exportable_config = ActsAsExportable::Config.\
 new(model_id)
 include ActsAsExportable::InstanceMethods
 end

 # Make the @acts_as_exportable_config class variable easily
 # accessable from the instance methods.
 def acts_as_exportable_config
 @acts_as_exportable_config || self.superclass.\
 instance_variable_get('@acts_as_exportable_config')
 end
 end
end

So, what happened? The first method you see extends the current class (that’s every one of your controllers with the methods from the ClassMethods module).

Every class now has the ‘acts_as_exportable’ method available. What does it do? The plugin automatically grabs the name of the model associated (by convention) with the controller you use, unless you specify something else.

Next, we create a new configuration object that contains information about the model we’re working with. Later on this can contain more detailed information like what attributes to include or exclude from the export.

Finally we include the module InstanceMethods, which we still have to define. The instance methods are only included when we enable the plugin. In our case, the instance methods include the ’export_to_xml’ and ‘conditions_for_collection’ methods. We can simply copy/paste them into your plugin.

module InstanceMethods
 def export_to_xml
 data = Book.find(:all, :order => 'title', :conditions => conditions_for_collection)
 send_data data.to_xml,
 :type => 'text/xml; charset=UTF-8;',
 :disposition => "attachment; filename=books.xml"
 end

 # Empty conditions. You can override this in your controller
 def conditions_for_collection
 end
end

Take note that we don’t want to define any default conditions, because we don’t know what model we’re using here. By adding an empty method, the method is available and no conditions are used. Another developer can define ‘conditions_for_collection’ in his controller to override the one we write here.

In the ’export_to_xml’ there are a few changes as well. First of all, I generalized ‘books’ to ‘data’.

The most important step is yet to come. We have still application specific code in your plugin, namely the Book model. This is where the Config class and @acts_as_exportable_config come in.

We have added a class variable to the controller named @acts_as_exportable_config. By default, this variable is not accessable by instance methods, so we need a little work around:

self.class.acts_as_exportable_config

This will call the class method ‘acts_as_exportable_config’ we defined in ClassMethods and return the value of @acts_as_exportable_config.

Note that we store the configuration in each seperate controller. This allows acts_as_exportable to be used with more than one controller at the same time.

With the model name made application independent, the whole plugin code looks like:

module ActsAsExportable
 def self.included(base)
 base.extend(ClassMethods)
 end

 class Config
 attr_reader :model
 attr_reader :model_id

 def initialize(model_id)
 @model_id = model_id
 @model = model_id.to_s.camelize.constantize
 end

 def model_name
 @model_id.to_s
 end
 end

 module ClassMethods
 def acts_as_exportable(model_id = nil)
 # converts Foo::BarController to 'bar' and FooBarsController to 'foo_bar'
 # and AddressController to 'address'
 model_id = self.to_s.split('::').last.sub(/Controller$/, '').\
 pluralize.singularize.underscore unless model_id

 @acts_as_exportable_config = ActsAsExportable::Config.new(model_id)
 include ActsAsExportable::InstanceMethods
 end

 # Make the @acts_as_exportable_config class variable easily
 # accessable from the instance methods.
 def acts_as_exportable_config
 @acts_as_exportable_config || self.superclass.\
 instance_variable_get('@acts_as_exportable_config')
 end
 end

 module InstanceMethods
 def export_to_xml
 data = self.class.acts_as_exportable_config.model.find(:all,
 :order => 'title',
 :conditions => conditions_for_collection)
 send_data data.to_xml,
 :type => 'text/xml; charset=UTF-8;',
 :disposition => "attachment; filename=\
 #{self.class.acts_as_exportable_config.model_name.pluralize}.xml"
 end

 # Empty conditions. You can override this in your controller
 def conditions_for_collection
 end
 end
end

Add the following line to your BooksController and restart your web server. (Oh, and make sure to remove the export_to_xml method from the controller as well)

acts_as_exportable

Done! – Or not?

Enabling the plugin by default

We have a very nice plugin now, but it is not loaded by default! If you take a look at your plugin directory, you’ll find a file named ‘init.rb’. This file is executed when you (re)start your web server. This is the perfect place to add our class methods to the ActionController::Base. Just add the following three lines of code to ‘init.rb’:

ActionController::Base.class_eval do
 include ActsAsExportable
end

When we include our module, the ‘self.included’ method is called, and the ClassMethods module is added, thus enabling the acts_as_exportable method.

That’s all! Happy plugin writing!

Feel free to comment on this post and write about any of your own plugin (for controllers) experiences.

Some DNS problems with Ariejan.net

Fri, 06 Jul 2007 00:00:00 +0000

This is just a quick note to let you know there are currently some issues with DNS for Ariejan.net. This basically means that Trac and SVN are currently not available. I’ve made the appropriate changes and things should be working again in a few hours.

I’m very sorry for the inconvenience.

Update: It appears that things are working again. If you have any issues with Trac, please let me know.

How to resolve Subversion Conflicts

Wed, 04 Jul 2007 00:00:00 +0000

If there’s more than one person working on a project, chances are (although slim) that at some point two developers work on the same piece of code and check it in. To clarify, let me give you an example.

The repository is currently at revision 5 and contains a file named ‘README’. Revision 5 of that file contains a single line: ‘This is a README file’.

Now, both you and your colleague check out r5 and edit README. Your colleague changes the line to ‘This is a documentation file’ and commits it back to the repository, which is bumped to revision 6.

You’re an island, and have no clue about the new revision being created. You just happily write away and change the README file to: ‘This is fun stuff!’.

When you commit your changes, you’ll get an error message:

$ svn commit -m "Updated README"
Sending README
Transmitting file data .svn: Commit failed (details follow):
svn: Out of date: '/myproject/README'

This is good. Subversion has detected that the file you want to commit has changed since you last updated it. Update the file to get it up-to-date again.

$ svn update
C README
Updated to revision 6.

The ‘C’ indicates there is a conflict with the README file, and Subversion does not know how to solve this. You are called in to help.

If you now take a look at README, you’ll notice that there are several markers that indicate what parts of the code are conflicting. You can easily see what you changed, and what has changed in the repository:

<<<<<<< .mine
This is fun stuff!
=======
This is a documentation file
>>>>>>> .r6

What are your options?

You have three options for resolving the conflict. Whatever you choose, make sure you confer with your colleague on the matter.

1. Scrap your changes, and go with the current work from your colleague.

This is the easiest solution. All you have to do is revert the changes you made, and update your working copy:

$ svn revert README
Reverted 'README'
$ svn update README
At revision 6.

2. Keep your changes, and dump whatever your colleague did.

Performing a simple ’ls’ will show you that there are four files related to this conflict:

README – the original with markers
README.mine – your version
README.r5 – the original your worked with
README.r6 – the most update version from your colleague

To check in your changes, copy your version over the original and tell Subversion you have resolved the conflict.

$ cp README.mine README
$ svn resolved README
Resolved conflicted state of 'README'

The ‘resolved’ command will clean up all the special files that were generated.

3. Merge both version to a new version

If you choose this option, you will have to manually edit README. Remove the markers and add whatever you need to add here.

Subversion won’t let you commit this file, so you’ll have to mark it as ‘resolved’ as we saw during option 2:

$ svn resolved README
Resolved conflicted state of 'README'

Before you rush ahead

With option 1, you’re done. With options 2 and 3 there is some more work to do, because you didn’t commit your changes yet. Because we’re dealing with conflicts here, I recommend you don’t just commit to your repository, but follow a slightly different route.

First, update your working copy (again) to make sure you have all the latest, and are not trying to check in any more conflicting code. If any conflicts pop-up, fix these first.

Now, run your tests to make sure everything is working as it should.

When all is clear, commit your changes to your repository as you normally would.

Done. Problem solved.

Got updates?

Tue, 03 Jul 2007 00:00:00 +0000

Do you want to stay up-to-date about what’s happening at Ariejan.net? That’s real easy!

You can subscribe to my RSS Feed and receive updates as they happen in your favourite RSS reader.

If you don’t have an RSS reader (or don’t want one), you can subscribe to my e-mail service. This service will gather up all the posts of the previous day (if any) and send them to you between 7am and 9am (CET).

Interested? Sign-up now!

Or grab the RSS Feed.

How to create and apply a patch with Subversion

Tue, 03 Jul 2007 00:00:00 +0000

It’s been a while since I posted something new on the use of Subversion. I’ve been working with the tool a lot, and I’ve found that patches are a great way to communicate code changes.

For those of you who are still learning, let me first explain what a patch is. A patch is a text file that contains the alteration that were made to a specific file. It includes the lines that have been removed and the lines that have been added. In short, if you have a ruby script and edited it, you could create a patch file, containing the changes you’ve made.

Why is this useful? You could check in your changes to your repository directly. True, but there are cases that you don’t have write access to the repository. For example, if you wanted to contribute code changes to Acts As Exportable, you should create a new ticket and attach a patch file. I will then review your changes before I apply them to the code and commit them to the repository.

So, how do you go about creating a patch file and how do you later apply it to your source? ~

Creating a patch file

Creating a patch file is really easy. First, check out the most recent version of the code from Subversion using the ‘checkout’ command.

Make your changes.

Then, in the root the project run the following command. It will store the patch file in your home directory. Make sure to give it meaningful filename.

svn diff > ~/fix_ugly_bug.diff

The file has the .diff extention, which stands for differences. This extension is recognized by many text editors and enables ‘syntax highlighting’ automatically. (Give it a try with TextMate and you’ll know what I mean.)

You can send the diff-file to the author of the project by email, or you can create a ticket in Trac and add it as an attachment. The author will review the changes you made and possibly apply them to the source.

Applying a patch

You should never apply patches from any person other than your development team without first reading through the changes, apply them locally and test your application and then commit them. Patches can not only include bug fixes, but also alterations to create back doors or add other exploits to your code.

Always read through a patch before applying it!

When you are sure the patch will bring no harm to you, your application or your customers, go ahead an apply it to your working copy. Here, I assume that you downloaded the patch file we previously generated, and placed it in your home directory. In the root of your application now run:

patch -p0 -i ~/fix_ugly_bug.diff

This will apply all the changes in the patch to your source. The -p0 option makes sure that all files can be found correctly (this has to do with something called ‘zero directories’, I won’t get into that right now). The -i option tells ‘patch’ what to use as input, in this case the ‘fix_ugly_bug.diff’ file in your home directory.

With the code changes in place, run your tests and make sure everything works as expected. If it does, commit your changes and celebrate with a cup of coffee.

How to force data to be downloaded as a file from your Rails app

Mon, 02 Jul 2007 00:00:00 +0000

In the essence of every application is data. One way or another your application manages data and at some point, you need to get that data out. Either you want to synchronize the data with another application or device. Or you want to move your data to another system all together. Either way, you’ll need to gather your data and send it from your application to the client… as a file.

Downloading files is not the hardest thing around. But the problem is that some formats, like XML, are automatically parsed by the browser and this makes it harder for users to download files like that.

So, what you want to do is, ignore the browser and offer your data (in XML or whatever format you want) as a file that can be downloaded directly. The solution is rather easy, as always with Rails.

Okay, in this example I have an action that renders current ’entries’ as XML and offers this to the user:

def export_to_xml
 @entries = Entry.find(:all)
 render :xml => @entries.to_xml
end

This works as you’d expect it to. When calling this action, the user receives a XML file containing all entries. But really, do you want that in your browser? Especially when the XML file is rather large, this can be very annoying, because your browser will want to load it all in!

What you want here is offer the users a file named ’entries.xml’ for download. In this case we use Rails’ send_data method. The previous action now looks like this:

def export_to_xml
 @entries = Entry.find(:all)
 send_data @entries.to_xml,
 :type => 'text/xml; charset=UTF-8;',
 :disposition => "attachment; filename=entries.xml"
end

It’s clear that we send the XML data to the client. I specify the type and charset of the data with the ’type’ paramater. This way the browser knows what is being send and allows the user to choose an application that can use the data. In this case an XML reader, for example.

The disposition parameter tells the browser this should be downloaded as a file (or attachment). It also specifies what the name of the attachment is, ’entries.xml'.

Now, link to this method and the user will be presented with a downloadable XML file.

ActiveScaffold + acts_as_taggable + Auto Complete

Sun, 01 Jul 2007 00:00:00 +0000

I’ve talked before on how to use ActiveScaffold with acts_as_taggable_on_steroids.

The problem with that solution was that, although the checkboxes for every tag are very nice, you couldn’t easily add new tags. For some people, this may be fine, for others, it is not.

Together with a colleague (who wishes not to be named), I found a solution that is quite elegant. Instead of using check boxes, and creating all kinds of subforms in ActiveScaffold, we opted for an auto_completing, comma-separated list of tags.

This article descripes the solution we found. I think you’ll like it very much!

When you try to use acts_as_taggable with ActiveScaffold, you might use something like this in your BooksController.

active_scaffold :books do |config|
 config.columns = [:title, :body, :tags]
 config.list.columns = [:title, :tag_list]
 config.columns[:tags].ui_type = :select
 # ...
end

This is not so useful when you want the flexibility of creating new tags instantly. Therefore, it’s better to use the tag_list:

active_scaffold :books do |config|
 config.columns = [:title, :body, :tag_list]
 config.list.columns = [:title, :tag_list]
 # ...
end

You get a text_field for writing down the tags (comma-separated). The problem of this is that the user has to keep all the tags in mind and is not allowed to make any typos in the tag. To help our users out, I use Rails’ auto_complete feature.

In your BooksController:

auto_complete_for :book, :tag_list

def autocomplete_tag_list
 @all_tags = Tag.find(:all, :order => 'name ASC')

 re = Regexp.new("^#{params[:record][:tag_list]}", "i")
 @tags = @all_tags.find_all do |t|
 t.name.match re
 end

 render :layout => false
end

We can now create the template for the results which are found. In app/views/books/autocomplete_tag_list.rhtml:

<ul class="autocomplete_list">
<% @tags.each do |t| %>
<li class="autocomplete_item"><%= t %></li>
<% end %></ul>

Now comes the difficult part, integration of the auto_complete widget within ActiveScaffold.

ActiveScaffold has the possibility to change the way every attribute is displayed on the create and edit page. I want to change the form for the attribute ’tag_list’. To do this, I create create a file named app/views/books/_tag_list_form_column.rhtml:

<dl>
<dt>
<label for="record_tag_list">AutoCompleted Tag List</label>
</dt>
<dd>
<%= text_field_tag 'record[tag_list]', @record.tag_list %>
<p class="auto_complete" id="record_tag_list_<%=@record[:id]%>_auto_complete">
style="{height: 80px;}">
<script type="text/javascript">
//<![CDATA[
var record_tag_list_<%= @record[:id].to_s %>_auto_completer =
new Ajax.Autocompleter(
\'record[tag_list]\',
\'record_tag_list_<%=@record[:id]%>_auto_complete\',
\'/articles/autocomplete_tag_list\', {tokens: \',\'});
//]]>
</script>
</dd>
</dl>

This shows a text field and generates a div that contains the available tags that we can show to the user. To populate the list of tags we use Ajax.Autocompler, which requires three arguments: the id of the text_field; the id of the div where you want to show possible tags to the user; and third, the URL of the action we created before, that returns the proper tags.

The ’tokens’ part of the last argument indicates that the user can seperate multiple tags with a comma. So, if you’ve entered one tag, added a comma and start typing a new tag, the auto complete feature will only lookup that second tag you’re typing!

That’s it. Just spice things up a bit with some Style, and you’re done. Enjoy!

AJAX Rules! 80 JavaScript Solutions for professional coding

Thu, 21 Jun 2007 00:00:00 +0000

Smashing Magazine has put together a very comprehensive list of 80 AJAX JavaScript solutions for professional coding. How great is that?!

The list includes solutions for auto-completion, inline editing, menus, tabs, calendars, all sorts of interactive stuff and tables, charts, graphs, forms, grids, lightboxes, galleries, showcases, visual effects and also, some basic JavaScripts that everybody should know about.

Check out the list now!

Action Mailer: All mail comes from MAILER DAEMON

Wed, 20 Jun 2007 00:00:00 +0000

Today I was trying to send mail from my Rails application through Action Mailer. This is quite simple, but I wanted to use a custom from-address. So, I create a setup_email method in my UserNotifier class that sets some defaults for every email sent out:

class UserNotifier < ActionMailer::Base
 protected
 def setup_email(user)
 @recipients = "#{user.email}"
 @from = "My Application <no-reply@example.com>">
 end
end

May you spotted the problem already, but I didn’t. All the mail sent came from “MAILER DAEMON”.

From: MAILER DAEMON

The problem was that @from didn’t contain a properly formated from-address. It is missing the closing >, and so my email server ignores it.

If you have this issue, double check the from address, and make sure it’s valid! Cheers.

Rails production server setup and deployment on Ubuntu/Debian

Wed, 20 Jun 2007 00:00:00 +0000

Please digg this story to spread the word! Thanks!

Okay, this is a big one! This article will show you (and explain to you) how to setup a Ruby on Rails production server with Ubuntu 7.04 or Debian 4.0 and how to deploy your Rails application there.

First, what’s getting installed:

Ruby 1.8.5
Ruby on Rails 1.2.3
Subversion 1.4
MySQL 5.x Server
Apache 2.2.x
Mongrel Cluster

I assume that you have just installed a fresh system with Ubuntu Linux 7.04 or Debian 4.0. If you haven't, do so now! You don't need to install the "DNS" or "LAMP" server in Ubuntu. Just a minimal system is enough for this tutorial.

I’ll be deploy an imaginary Rails application named “myapp” which uses MySQL and is stored in Subversion. More on that later on.

Well, let’s get going and get that Ruby on Rails server ready.

Update your system

Before you do anything, use apt-get to update your system to the latest possible version.

sudo apt-get update
sudo apt-get dist-upgrade

This probably installs a new kernel (linux-kernel) so a reboot may be in order for optimal performance.

Enable SSH

Most people will want to have SSH on their server to login remotely. In this case I install both the server and client so you can SSH out if you need to:

sudo apt-get install openssh-server openssh-client

You will now be able to login remotely wiht SSH.

You’ll need SSH later if you want to use Capistrano to deploy your Ruby on Rails application. In any case, SSH is a good thing to have around.

Subversion

If you are serious about your Ruby on Rails server, you want to have Subversion around. Most deployment scripts pull the latest revision of your code from subversion. No configuration needed here.

sudo apt-get install subversion

We only need the client on the production server. We’re not going to host Subversion repositories here.

Install MySQL Server

This is the first serious step you’ll have to take. Both Ubuntu 7.04 and Debian 4.0 come with MySQL 5.0.x.

sudo apt-get install mysql-server mysql-client libmysqlclient15-dev

Be sure to set a password for the root MySQL user. Failing to do so will leave your database open for anyone who wishes to see.

mysqladmin -u root -h localhost password 'secret'
mysqladmin -u root -h myhostname password 'secret'

Make sure to replace secret with your actual password.

Try logging in to MySQL with your new password to make sure everything works okay.

mysql -u root -p
Enter password:
mysql>

MySQL is in place now, so let’s get cracking at Ruby and Rails now.

Ruby, Gems, Rails

Installing Ruby is quite easy:

sudo apt-get install ruby

You’ll now have Ruby 1.8.5. You will also need to install some other develoment package to help you build native Ruby Gems.

sudo apt-get install make autoconf gcc ruby1.8-dev build-essentials

I’ll install ruby gems the conventional way (so I’m not going to use Ubuntu’s packages here). Download the latest Gems .tgz here.

wget http://rubyforge.org/frs/download.php/17190/rubygems-0.9.2.tgz
$ tar xvf rubygems-0.9.2.tgz
$ cd rubygems-0.9.2/
$ sudo ruby setup.rb
$ gem -v

That’s it for the gems. Now, install Rails an all it’s dependencies:

sudo gem install rails --include-dependencies

If you get an error message complaining that the ‘rails’ gem cannot be found include the –remote option.

Oh no! More gems!

Next, let’s install some essential Ruby Gems that will make your life quite a bit easier. Here we’ll install the following gems:

mysql - For good MySQL connectivity
capistrano - Just to have it handy when needed
mongrel - Rails server
mongrel-cluster - To operate mongrel clusters

sudo gem install mysql capistrano mongrel mongrel-cluster --include-dependencies

You’ll be asked several times to choose a version for different gems. Always choose the latest available version for ‘ruby’. (Don’t choose win32. I don’t need to explain why.)

Test Rails and MySQL operability

Before you continue you may want to take some time to test Rails and MySQL. It’s not essential, but I recommend it because it will save you a lot of trouble later on.

Create a new Rails application in your homedir and a create the corresponding MySQL database. Also edit config/database.yml to reflect your root password!

mysqladmin -u root -p create testapp_development
$ mkdir testapp
$ rails testapp
$ cd testapp
$ vi config/database.yml
$ rake db:migrate

If you get any errors from that last command, check the previous steps. Normally, all should be fine and you can continue safely.

Most people configure a ‘socket’ in their config/database.yml for MySQL. Note that this socket is in different places for different distribution. Ubuntu and Debian keep it in: /var/run/mysqld/mysqld.sock. You may need to update your configuration in order to connect to the database.

Apache 2.2

The good thing about Ubuntu/Debian is that they both include Apache 2.2.x now. This branch of Apache includes the a balancing proxy, which allows you to distribute your workload over several Mongrel servers (in your mongrel cluster). I’ll come back to that later.

sudo apt-get install apache2

Before you continue, enable several modules which we’ll be using later on.

sudo a2enmod proxy_balancer
sudo a2enmod proxy_ftp
sudo a2enmod proxy_http
sudo a2enmod proxy_connect
sudo a2enmod rewrite

That’s it for now on Apache. Let’s move along.

Prepping the Rails application

Okay, let’s prepare the Rails application ‘myapp’ for deployment now, shall we?

First, create a production database on your server, and configure it in config/database.yml:

mysql -u root -p

create database myapp_production;
grant all privileges on myapp_production.* to 'myapp'@'localhost'
identified by 'secret_password';

The next step is to install capistrano. You can install it on your development machine as a gem, as demonstrated above. Next, apply capistrano to your application:

sudo gem install capistrano
cap --apply-to myapp

Take a look at myapp/config/deploy.rb. This file contains the configuration for the deployment of your application. Take special care of the following, here’s an example for ‘myapp’:

require 'mongrel_cluster/recipes'
set :application, "myapp"
set :repository, "http://svn.myhost.com/svn/#{application}/trunk"
# We only have one host
role :web, "myapp.com"
role :app, "myapp.com"
role :db, "myapp.com", :primary => true
# Don't forget to change this
set :deploy_to, "/home/ariejan/apps/#{application}"
set :mongrel_conf, "#{current_path}/config/mongrel_cluster.yml"

As you can see, I’ve already included several lines that enable the use of mongrel, we’ll get to that next.

Make sure you adapt this file to your own needs. myapp.com is the address of the server you’re going to deploy your application to. the mongrel_cluster.yml file will be created in a moment.

On the server, make sure you create the ‘apps’ directory. You can now setup a basic file structure for the deployment:

cd myapp
cap setup

On your server, you’ll notice that the /home/ariejan/apps/myapp directory was created, including some subdirectories.

If you are annoyed with entering your SSH password every time, create and upload your public SSH key to automate this. (I’ll write something up about that later on.)

Now, configure mongrel. For a normal setup, with moderate traffic, you can handle all traffic with two mongrel instances. The mongrel servers will only be accessable through ’localhost’ on the server on non-default ports. Apache will do the rest later.

In your rails app, run the following command:

mongrel_rails cluster::configure -e production -p 9000 \
-a 127.0.0.1 -N 2 -c /home/ariejan/apps/myapp/current

The configuration file we saw earlier has been created. Check-in all new files in to subversion now, and cold deloy your application!

cap cold_deploy

The deployment will checkout the most recent code from Subversion and start the mongrel servers.

After the deployment, migrate your production database and restart the mongrel cluster:

cap migrate
cap restart

To check that your application is running, issue the following command on your server. It should return you the HTML code from your app:

curl -L http://127.0.0.1:9000

Configure Apache and the Balacing Proxy

You have two mongrel servers running, ready to handle incoming requests. But, you want your visitors to use ‘myapp.com’ and not an IP address with different port numbers. This is where apache comes in.

Create a new file in /etc/apache2/sites-availbale named ‘myapp’ and add the following:

<proxy>
 BalancerMember http://127.0.0.1:9000
 BalancerMember http://127.0.0.1:9001
</proxy>

<virtualhost>
 ServerName myapp.com
 ServerAlias www.myapp.com

 DocumentRoot /home/ariejan/apps/myapp/current/public

 <directory>
 Options FollowSymLinks
 AllowOverride None
 Order allow,deny
 Allow from all
 </directory>

 RewriteEngine On

 RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
 RewriteCond %{SCRIPT_FILENAME} !maintenance.html
 RewriteRule ^.*$ /system/maintenance.html [L]

 RewriteRule ^/$ /index.html [QSA]

 RewriteRule ^([^.]+)$ $1.html [QSA]

 RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
 RewriteRule ^/(.*)$ balancer://myapps_mongrel_cluster%{REQUEST_URI} [P,QSA,L]

 ErrorLog /home/ariejan/apps/myapps/shared/log/tankfactions_errors_log
 CustomLog /home/ariejan/apps/myapps/shared/log/tankfactions_log combined
</virtualhost>

Now enable this new site in apache:

sudo a2ensite myapp
/etc/init.d/apache2 force-reload

In some cases you may need to make a small change to /etc/apache2/mods-enabled/proxy.conf and swap

Order deny,allow
Deny from all

for

Order allow,deny
Allow from all

That’s all, you can now access your app on myapp.com!

Maintaining your application

Now, happily develop your application and make update (you check them in to Subversion). To update your web server run:

cap deploy

If you made changes in the database, you may want to run a long_deploy:

cap long_deploy

And if for some reason, your mongrel cluster dies, just restart it.

cap restart

That’s it! Happy hacking :)

Please digg this story to help spread the word! Thanks a lot!

Comments can be posted here or in my new and shiny Google Group!

Geslaagd! / Passed my final exams!

Tue, 19 Jun 2007 00:00:00 +0000

Please scroll down for the English version.

Bij wijze van hoge uitzondering, een post in het Nederlands op mijn weblog! Ik ben vandaag geslaagd voor mijn opleiding Hogere Informatica (aan Fontys Hogeschool ICT, Eindhoven) en mag mezelf nu gediplomeerd Software Engineer noemen!

Ik ben op drie punten beoordeeld:

Eindrapport: 7,5 – Ik ben er gewoon niet goed in, maar het is toch nog wat redelijks geworden.
Eindpresentatie: 8 – Het ging gewoon lekker
Resultaat: 9 – Dit slaat op het werk dat ik heb afgeleverd, mijn inzet, zelfstandigheid etc.

Samen komt dit uit op een eindcijfer 8. Natuurlijk ben ik hier heel trots op, en ga het in ieder geval de rest van de dag vieren!

Feliciteren kan in ’n commentaartje :P

English version below:

Today I passed my final exams for my degree in Software Engineering. I’ve been working on a Ruby on Rails project called “Course Builder” for the past few months. Today, I had to defend my work for a jury.In the Netherlands students are judged on 1 to 10 scale, where 10 is ’total perfection’, 6 is ‘seems okay’ and 1 is total nonsense.

I’ve been judged on three subjects:

Final report: 7.5 – I'm not very good at writing reports, but it's above average, still.
Presentation: 8 – It went very well. I'm so happy with my MacBook and KeyNote
Results: 9 – The results of my work and how I approached the project as a whole

The final grade is an 8. I’m very happy with the over-all result and I’ll be celebrating the rest of the day.

You may congratulate me in a comment :P

Buy it now! Mac OS X 10.5 Leopard

Thu, 14 Jun 2007 00:00:00 +0000

It appears that Amazon is accepting pre-orders for Apple Mac OS X Version 10.5 Leopard, which is scheduled for release in October 2007.

There are, as usual two version available, Apple Mac OS X Version 10.5 Leopard at $129 for a single license and the Apple Mac OS X Version 10.5 Leopard Family Pack for $199, which includes licenses for up to five Macs in your home.

If you want to be sure you have the latest version of when it arrives: pre-order now!

BASH your SVN and Trac installation!

Tue, 12 Jun 2007 00:00:00 +0000

I’ve already discussed how to install Subversion and Trac on your Ubuntu server. In my case I have a server that manages different SVN and Trac installations for a group of developers.

Creating a new SVN repository and Trac installation every time is quite boring and “if you need to do it more than once, you should automate it”. So, that’s what I did.

The result is the following BASH script. It takes one argument, the name of the project you want to create. E.g if you wanted to create a SVN repository and trac installation for “My Project” you would run the following command:

$ ./create_dev_env my_project

The script it self looks like this:

#!/bin/sh
echo == Creating Subversion and Trac installation for $1
echo = Creating SVN Repository...

# Subversion
cd /var/lib/svn
mkdir -p /var/lib/svn/$1
svnadmin create /var/lib/svn/$1
sed s/EXAMPLE/$1/g /usr/share/trac/contrib/post-commit > /var/lib/svn/$1/hooks/post-commit
chmod +x /var/lib/svn/$1/hooks/post-commit
chown -R www-data:www-data /var/lib/svn/$1

# Trac
echo = Creating Trac install...
cd /var/lib/trac
mkdir -p /var/lib/trac/$1

echo - Creating files
trac-admin /var/lib/trac/$1 initenv $1 sqlite:db/trac.db svn \
/var/lib/svn/$1 /usr/share/trac/templates

echo - Removing anonymous permissions
trac-admin /var/lib/trac/$1 permission remove anonymous BROWSER_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous CHANGESET_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous FILE_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous LOG_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous MILESTONE_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous REPORT_SQL_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous REPORT_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous ROADMAP_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous SEARCH_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous TICKET_CREATE
trac-admin /var/lib/trac/$1 permission remove anonymous TICKET_MODIFY
trac-admin /var/lib/trac/$1 permission remove anonymous TICKET_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous TIMELINE_VIEW
trac-admin /var/lib/trac/$1 permission remove anonymous WIKI_CREATE
trac-admin /var/lib/trac/$1 permission remove anonymous WIKI_MODIFY
trac-admin /var/lib/trac/$1 permission remove anonymous WIKI_VIEW

echo - Creating Trac admins
trac-admin /var/lib/trac/$1 permission add ariejan TRAC_ADMIN

chown -R www-data:www-data /var/lib/trac/$1

echo
echo == Done.

First it creates the SVN directory in /var/lib/svn/my_project and creates repository and adds the trac post-commit hook for trac integration.

Next, it creates the trac installation in /var/lib/trac/my_project and removes all the persmission the anonymous users has. (You may want to remove these lines for open source or public projects.)

And, finally, I’m added as an administrator to the project. Make sure to replace this with you own username.

Hope you find this script useful. Any improvements are welcome, please let me know.

ActiveScaffold, Acts_as_taggable_on_steroids

Mon, 11 Jun 2007 00:00:00 +0000

Update: also read Active Scaffold + Acts_as_taggable + Auto Completion.

This is kind of an advanced topic, but I think it may be useful to a lot of people.

ActiveScaffold is a great plugin to start building a user interface. The great thing about AS is, that is automatically recognizes associated models. When editing a model, you can easily add or select another model that you want to associate with is.

The best example of this is an Article, where you can select the author (the associated User model) with a drop down box.

There is only one point where I ran into trouble with ActiveScaffold: acts_as_taggable_on_steroids.

Acts_as_taggable_on_steroids allows you to easily attach tags to models and do all kinds of crazy stuff with them. But, if you want to integrate in into AcitveScaffold, you’re in for a tough ride.

ActiveScaffold supports has_many :through associations, but not in a way that is compatible with acts_as_taggable_on_steroids. Let me show you.

In your ArticlesController you specify which columns to show. “tag_list” is a stringified version of the tags associated with the Article, which is great for showing to a user.

However, if you want to edit it an article (or create one), I don’t want a text field where I have to enter tags manually, all I want are a bunch of check boxes, so I can check which tags apply to this article.

Showing the check boxes is easy with AS. By default I show ’tags’, only in the list view do I use ’tag_list’ instead. Also, make sure to set the ui_type for the tags column to :select. This will show you check boxes, instead of a sub form that allows you to create tags manually.

active_scaffold :article do |config|
 config.columns = [:title, :body, :tags, :author, :created_at]
 config.list.columns = [:title, :author, :tag_list, :created_at]
 config.columns[:tags].ui_type = :select
end

Well, very nice, right. You can now happily select the tags you want, and save your article. Not.

As you may have noticed, the tags are not saved. Why? Acts_as_taggable adds a ’tags’ attribute to the model, however, when the Article model is saved, the tags attribute is overwritten by the tags specified in the “tags_list” attribute.

The only way to solve this is to convert the tags selected in AS and store them as the tags_list attribute for the Article.

First, let’s add a private method in the ArticleController class:

private

def new_tag_list(tag_ids)
 tag_ids.map {|k,h| h['id']}.collect {|i| Tag.find(i)}.map do |tag|
 tag.name.include?(Tag.delimiter) ? "\"#{tag.name}\"" : tag.name
 end.join(Tag.delimiter.ends_with?(" ") ? Tag.delimiter : "#{Tag.delimiter} ")
end

And add two protected methods that extend the functionality of ActiveScaffold:

protected

def before_create_save(record)
 record.tag_list = new_tag_list(params[:record][:tags])
end

def before_update_save(record)
 record.tag_list = new_tag_list(params[:record][:tags])
end

This will take the actual form values from AS and create a tags_list. This new tags_list is then assigned to the article (named ‘record’ here). The two protected methods process the tags every time an Article is created or updated.

With this in place, you can happily assign tags to your articles! Please let me know if it worked for you, or if you have made any improvements to this solution.

Find and Replace with a MySQL Query

Sun, 10 Jun 2007 00:00:00 +0000

There are times when you have a lot of data in a database (let’s say wp_posts for a Wordpress blog like Ariejan.net). When you need to find and replace certain strings, this can be a very tedious task. Find all posts containing the “needle” string and manually replace all these occurrences with “chocolate”. With about 200 posts, you can imagine how long this would take to do manually.

But, as I always say: “You’re a programmer! You should script the hell out of everything!"

So, I found this: MySQL has built-in support to find and replace! Just a simple query will do:

UPDATE wp_posts set post_body = replace(post_body, 'needle', 'chocolate');

That’s it. The entire table ‘wp_posts’ is searched and all occurrences of “needle” are replaced with “chocolate”. The query only took about a split second.

Send mail with a BASH Shell Script

Sun, 10 Jun 2007 00:00:00 +0000

Like any good programmer, I try to automate the crap out of everything. If you have to do it more than once, I try to write a script for it.

This time I want to show you how you can easily send an e-mail from a BASH script. The idea is that you want the script to send out an email to notify a user that something has happened.

We’re going to use the GNU Mail utility here. The basic syntax to send an email is like this:

/usr/bin/mail -s "Subject" someone@example.org < message.txt

The trick when using this in a shell script is creating and using the message.txt file correctly.

Let’s setup the basis first:

#!/bin/bash
SUBJECT="Automated Security Alert"
TO="alarms@ariejan.net"
MESSAGE="/tmp/message.txt"

/usr/bin/mail -s "$SUBJECT" "$TO" < $MESSAGE

All we need to do now is create the message. In this example we’re going to notify the receiver that something happened at a certain time. We can use the append (») operator to add text to the message file. Afterwards, we must remove the temporary message file, of course. The complete script now becomes:

#!/bin/bash
SUBJECT="Automated Security Alert"
TO="alarms@ariejan.net"
MESSAGE="/tmp/message.txt"

echo "Security breached!" >> $MESSAGE
echo "Time: `date`" >> $MESSAGE

/usr/bin/mail -s "$SUBJECT" "$TO" < $MESSAGE

rm $MESSAGE

The email will contain the a timestamp from when the mail was sent.

This method is great for letting an administrator now if something happened. Maybe you need to check if your webserver is up and running. This script can an administrator about the issue.

Ultimate List of Ruby Resources

Sun, 10 Jun 2007 00:00:00 +0000

This is the first post, named “Ruby”, in a series of “Ultimate List of … Resources”. I’m going to compose several lists for different topics I encounter during my development work. To start, I begin with Ruby. Later, I will add “Ultimate Lists” about Ruby on Rails, Subversion, AJAX and some other topics.

Feel free to let me know if I missed an important resource. I’m also open to suggestions about other “Ultimate Lists”.

For now, you’ll have to settle for the “Ultimate List of Ruby Resources”.

Books

"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition" by Dave Thomas, Chad Fowler and Andy Hunt
"The Ruby Way, Second Edition: Solutions and Techniques in Ruby Programming" by Hal Fulton
"Ruby Cookbook" by Lucas Carlson, Leonard Richardson
"Everyday Scripting with Ruby: For Teams, Testers, and You" by Brian Marick
"Beginning Ruby: From Novice to Professional" by Peter Cooper
"Ruby In A Nutshell" by Yukihiro Matsumoto
"Enterprise Integration with Ruby" by Maik Schmidt

Online manuals and API's

Noobkit — Easily searchable Ruby API
Core API documentation
Ruby Standard Library

Groups, chat, forums, wiki and other community websites

Ruby Blogs (is yours listed yet? — Let me know!)

Coming up: Ubuntu Development Server Guide

Thu, 07 Jun 2007 00:00:00 +0000

My articles about setting up a Ubuntu Development Server (part 1 and part 2) have been very successful.

I’m considering writing a new guide with more up-to-date information on how to setup a development server that allows you (and your team) to develop software, manage source code, track tickets and all that stuff.

What should, according to you, be included in this guide? Please let me know! Just leave a comment and I’ll pick it up.

Thank you for your input!

Update: Estimated release of the “Ubuntu Develpment Server Guide is mid july

Trac, WebAdmin plugin and global configuration

Wed, 30 May 2007 00:00:00 +0000

As you may know I manage quite a few trac installations. A few days ago I upgrade my server from Ubuntu “Dapper Drake” 6.06 to Ubuntu “Feisty Fawn” 7.04. This also upgrade trac 0.9.x to 0.10.3.

I was happy, since trac 0.10.3 has many improvements over 0.9.x, but there was one thing I was not so happy about. After the upgrade, I upgraded all my trac installations and everything seemed to be okay, except for the WebAdmin plugin. Apparently it was not installed anymore.

What happened? After upgrading the trac package, the plugins directory was emptied. Well, just re-install the WebAdmin plugin for 0.10.x.

cd /usr/share/trac/plugins
sudo svn co http://svn.edgewall.org/repos/trac/sandbox/webadmin/
cd webadmin
sudo python setup.py bdist_egg
cd dist
sudo easy_install-2.4 TracWebAdmin-0.1.2dev_r4429-py2.4.egg

That was easy, next I wanted to enable the plugin for all my trac installations by adding the proper configuration to /usr/share/trac/conf/trac.ini, the global trac configuration file that is used by all trac installs.

[components]
webadmin.* = enabled

After restarting Apache (this is needed for some reason to get trac to read the new configuration file), no admin button showed up in any of the projects.

What went wrong is that Ubuntu (or Debian?) maintainers have changed the location of the global configuration file for trac. There are three solutions to this, all of them work fine, although I recommend you use the first one.

1. Move your global configuration

The best way to tackle this problem is to move your global configuration file to the new location: /etc/trac/trac.ini

sudo mv /usr/share/trac/conf/trac.ini /etc/trac/trac.ini

This way you’re configuration is safe from new upgrades and confirms to the defaults the package maintainer has set.

2. Symlink the configuration

If for some reason you don’t want to actually move your /usr/share/trac/conf/trac.ini file, you can create a symlink to the new location:

sudo ln -sf /usr/share/trac/conf/trac.ini /etc/trac/trac.ini

This leaves your original configuration file in tact, but it may be removed by new upgrades.
3. Change Trac
You may also change the location where trac looks for the configuration file. Open up /var/lib/python-support/python2.5/trac/siteconfig.py and change the following:

< __default_conf_dir__ = '/etc/trac'
> __default_conf_dir__ = '/usr/share/trac/conf'[/pre]

(Note: the > and < symbols mark what is removed and what is added to the file.)

In any case, reboot your web server and you should be good to go again.

Installing RMagick Ruby Gem on Mac OS X 10.4.9

Tue, 29 May 2007 00:00:00 +0000

When you want to manipulate images with Ruby (or your Rails application) you’ll probably want RMagick installed. This is no easy feat on Mac OS X.

The official guide suggests installing X11 and using darwinports to install everything. This guide shows you how to easily install RMagick on you Mac OS X system. In this case I use Mac OS X 10.4.9.

Before you jump in, make sure you have Xcode installed. You can get it for free from Apple.

I’ll also assume you have Ruby and rubygems installed and working already.

You will need to download, compile and install several graphics libraries that RMagick needs. Let’s do this now.

curl -O http://download.savannah.gnu.org/releases/freetype/freetype-2.1.10.tar.gz
tar xzvf freetype-2.1.10.tar.gz
cd freetype-2.1.10
./configure --prefix=/usr/local
make
sudo make install
cd ..

curl -O http://superb-west.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.10.tar.bz2
bzip2 -dc libpng-1.2.10.tar.bz2 | tar xv
cd libpng-1.2.10
./configure --prefix=/usr/local
make
sudo make install
cd ..

curl -O ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar xzvf jpegsrc.v6b.tar.gz
cd jpeg-6b
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.4
./configure --enable-shared --prefix=/usr/local
make
sudo make install
cd ..

curl -O ftp://ftp.remotesensing.org/libtiff/tiff-3.8.2.tar.gz
tar xzvf tiff-3.8.2.tar.gz
cd tiff-3.8.2
./configure --prefix=/usr/local
make
sudo make install
cd ..

Next we install ImageMagick:

curl -O http://easynews.dl.sourceforge.net/sourceforge/imagemagick/ImageMagick-6.3.0-0.tar.gz
tar xzvf ImageMagick-6.3.0-0.tar.gz
cd ImageMagick-6.3.0
./configure --prefix=/usr/local
make
sudo make install
cd ..

And now, ladies and gentlemen, what you’ve all been waiting for: RMagick:

sudo gem install --no-rdoc --no-ri RMagick

In my case the generation of the documentation fails, so I tell rubygems not to compile the docs.

You now have RMagick installed on you Mac OS X 10.4.9 machine! Congratulations!

Slow connections with ProFTPD

Tue, 29 May 2007 00:00:00 +0000

My shiny new VPS, which is running Ubuntu Linux, uses ProFTPD for FTP access. Today I noticed that setting up the connection takes about 5 to 10 seconds. This is really annoying when editing files through FTP.

So, I investigated and found that by default ProFTPD tries to revolve the hostname of the client in order to put that in the logs instead of a plain IP address. This lookup can take quite some time, let’s say 5 to 10 seconds, especially when the look up fails and you have to wait on a time-out.

It’s easy to stop ProFTPD from behaving like this by adding the following line to your proftpd.conf in /etc/proftpd:

IdentLookups off

Restart ProFTPD and you’ll have a fast FTP connection to enjoy!

BAT - TER - Y

Mon, 28 May 2007 00:00:00 +0000

You all know the word: “Battery”. Like most words, it has several meanings. This site greatly explains the meaning of the word battery, and adds a little lyric-sugar to it!

Come on boys and girls! Sing along to BAT • TER • Y.

(Please digg the site or post a link to it. Thanks!)

MERGE request failed on ‘/path/to/file’

Mon, 21 May 2007 00:00:00 +0000

After upgrading my Subversion server to Ubuntu Feisty, I noticed that when committing I got the following error:

svn: MERGE request failed on '/svn/repository/trunk'
svn: MERGE of '/svn/repository/trunk': 200 OK (http://svn.myserver.com)

Although the messages says that the commit failed, it has not. A simple ‘svn update’ will merge the changes you made to the repository to your working copy again and you’re good to go.

So, what is the problem here? Are you by any chance running Trac? Did you install the post-commit hook to integrate Subversion with Trac? Right, so did I.

The problem here is that the trac-post-commit-hook script needs a few updates in order to include the proper modules. Take a look at the most recent 0.10.x version.

After updating the trac-post-commit-hook script, commits worked fine again. Such an easy fix for such a nasty problem.

FeedBurner acquired by Google!

Fri, 18 May 2007 00:00:00 +0000

It looks like Google is about to acquire FeedBurner. Just a note to all of you to let you know. If you have a blog or RSS enabled website, burn your feed now and get your AdSense account ready! Soon you’ll be able to put targeted ads in your RSS feed and add another source of income to your list! Great news!

Please digg and share! Thank you!

Rails Snippet: Write like Orwell with to_sentence

Wed, 09 May 2007 00:00:00 +0000

A few weeks ago I posted an article that explained how to create a comma separated list from a hash of objects. When I was browsing the Rails API documentation I came across a method named to_sentence.

What this little bugger does is create a human readable, comma separated list of items in an array or hash. But the big difference here is that you can specify what the last separator must be. By default this is set to ‘and’. See the following example.

@users = User.find(:all)
@users.collect {|u| u.firstname}.to_sentence
=> "Tom, Dick, and Harry"

Of you course, you can specify the last separator, called the connector. Also it’s possible to not show the last comma.

@users.collect {|u| u.firstname}.to_sentence(:connector => "and of course,", :skip_last_comma => true)
=> "tom, Dick and of course, Harry"

I bet this will greatly simplify the way you list names, tags, categories or whatever else you want summed up in a comma separated list with a human touch.

Ariejan.net server move

Sun, 29 Apr 2007 00:00:00 +0000

Just a note to let you know that Ariejan.net has just been moved to a new server! Well, I’ve moved from shared hosting with Delta9 Internet to a VPS solution.

Why on earth would I do such a thing? I’m very happy with the service I’ve recieved from Delta9 in the past. However, I wanted more control over my hosting. I’m a web developer and I have special needs. Such needs include Subversion and Ruby on Rails hosting, state-of-the art PHP and MySQL installations. PostgreSQL support and some other things. Concluding that for Ariejan.net, shared hosting was no longer an option.

I could have taken a big step and buy myself a shiny new server all for my self, but that just doesn’t fit into my budget at this time. When the time comes to host a dedicated server I will come back to Delta9 Internet.

So, Ariejan.net has moved to a VPS solution for the time being. Other sites I run are still hosted at Delta9, so I don’t abandon them all together.

If you are looking for quality hosting with a personal touch, give Delta9 Internet a try! Tell ’em I send you :)

I want to thank Delta9 Internet for hosting Ariejan.net for such a long time, and when the time’s right… I’ll be back!

Rails, Resources and Permalinks

Thu, 12 Apr 2007 00:00:00 +0000

There has been quite a bit of discussion about creating permalinks with a rails resource. In this article I will show you how to create permalinks for a resource named ‘pages’ without giving up on any of the resource goodness!

Before I start I’ll presume you have a page scaffold_resource setup in your rails application. Make sure you have at least the following fields in your page model:

t.column :title, :string
t.column :permalink, :string
t.column :content, :text

Okay, what you want is the permalink_fu plugin. This plugin greatly simplifies the act of generating a permalink from a title. Install it first:

cd railsapp
./script/plugin install http://svn.techno-weenie.net/projects/plugins/permalink_fu/

In your Page model you may now add the following line. This line will generate a permalink in the permalink attribute automatically, so you don’t have to show the permalink field in any forms.

has_permalink :title

That’s it for generating the appropriate permalink string in your database.

Rails goodness has already provided you with the basic RESTful routes:

/pages
/pages/123
/pages/new
/pages/123;edit

But what you really want, is something like:

/pages/perma-link-here

Notice that the permalink url is only a GET request and should not be used for editing or updating the page in question.

Since using any other identifier than :id in a resource is madness, I create two new routes that will allow me to access permalinked pages. Not only that, but I do maintain the format option. Basically this means that you get three routes:

/page/perma-link-here
/page/perma-link-here.html
/page/perma-link-here.xml

Notice that I removed the ’s’ from ‘pages’ here. This is to avoid confusion with the resource ‘pages’. But more on that later.

Now in config/routes.rb add the following two lines:

map.permalink 'page/:permalink', :controller => 'pages', :action => 'permalink'
map.connect 'page/:permalink.:format', :controller => 'pages', :action => 'permalink', :format => nil

The first line adds a named route to an action named ‘permalink’ in your PagesController. This gives you the ability to add peralink links easily:

permalink_url(@page.permalink)

The second link is unnamed, and allows you to specify a format like HTML or XML.

The permalink action looks like this:

# GET /page/perma-link
# GET /page/permal-link.xml
def permalink
 @page = Page.find_by_permalink(params[:permalink])

 respond_to do |format|
 format.html { render :action => 'show' }
 format.xml { render :xml => @page.to_xml }
 end
end

This special permalink action uses the same ‘show’ view as your resource.

If you want to maintain the ‘pages’ part of the URL, that’s possible. You’ll have to write a condition that makes sure that the :permalink parameter is a string an not an integer (ID). This article does not cover this.

You may now use permalinks for your pages! Congratulations.

TipSnippet: Create a RSS feed

Tue, 03 Apr 2007 00:00:00 +0000

RSS is hot! So, you want to fit your new Rails app with one too! That’s easy, of course, but you just need to know what to do.

This snippet will show you how to create an RSS feed form your RESTful articles. I’ll assume you know how to generate a resource named ‘article’ with a title, body and the default created_at and updated_at attributes.

You’ll first need to add a new collection to your resource in config/routes.rb

map.resources :articles, :collections => {:rss => :get}

This will expose your RSS feed as http://localhost:3000/articles;rss

Create a corresponding action in the articles controller in app/controllers/articles_controller.rb. This method fetches the ten latest articles.

def rss
 @articles = Article.find(:all, :limit => 10, :order => 'created_at DESC')
 render :layout => false
end

I assume you render your articles in a layout. The render method here prevents your layout from rendering to create a plain XML file (which is what an RSS feed is).

Next we create a view. This is not the regular RHTML you’re used to but RXML. This enables the XML generator which we’ll use to generate the RSS feed. Create app/views/articles/rss.rxml

xml.instruct! :xml, :version=>"1.0"
xml.rss(:version=>"2.0"){
 xml.channel{
 xml.title("My Great Blog")
 xml.link("http://www.example.com/")
 xml.description("My great blog about very interesting stuff!")
 xml.language('en-uk')
 for article in @articles
 xml.item do
 xml.title(article.title)
 xml.description(article.body)
 xml.author("support@example.com")
 xml.pubDate(article.created_at.strftime("%a, %d %b %Y %H:%M:%S %z"))
 xml.link(article_url(article))
 xml.guid(article_url(article))
 end
 end
 }
}

Well, that’s it. You now have a working RSS feed!

If you want to enable auto discovery, you should add the following line to the header of your layout. (Auto discovery enables that little RSS icon in the address bar of your browser.)

<%= auto_discovery_link_tag(:rss, :controller => 'articles', :action => 'rss') %>

Share and enjoy! Thank you.

What do you want for AutoFlickr?

Mon, 02 Apr 2007 00:00:00 +0000

Almost a month ago I published my WordPress plugin AutoFlickr, which shows one or more related photos from the popular Flickr site in your post.

Now I’m wondering what new features AutoFlickr users want. What parts of the current plugin are superfluous. Also, I’d like to know how is using AutoFlickr?

Please post a comment with your wishes for AutoFlickr 1.1. People using AutoFlickr may contact me to let me know about their site. I will include a list of the ten most beautiful (my pick) sites that use AutoFlickr with the announcement of AutoFlickr 1.1.

I hope for a lot of feedback to make this plugin even better. Thank you.

5 Reasons why PC OEMs should offer Linux

Thu, 29 Mar 2007 00:00:00 +0000

DELL currently offers Linux as an option for certain laptop and desktop models. Talk is that this service may be expanded to all models. This is a good thing, of course. Results of a recent survey by DELL show that more than 70% of over 100.000 respondents want to use Linux for home and office! What has been keeping companies like DELL away from this?

I’ve found five good reasons why OEMs should offer Linux as at least an option:

1. Freedom of Choice You pay for your hardware. Its’ yours. You should be able to use that hardware any way you want. The OEM should not force you to use anything else than what you want.

2. It saves customers money. If a customer doesn’t want to use Microsoft Windows? Why charge him for a license anyway?

3. It’s an unserved market. There are no mayor OEMs out there offering Linux as a default option. You get Microsoft Windows. In some cases you don’t have to buy Windows and get a clean hard drive instead. People want to use Linux, but they don’t want to go through the download-install-configure process. This is a big oppertunity, if you ask me.

4. Your hardware gets more popular. We’ve seen this with Nvidia already. May Linux users buy Nvidia cards because nvidia offers superb Linux drivers. Plug in the cards, install the drivers and off you go. If a player like DELL would support their hardware like that, people would be willing to buy DELL instead of Brand X. Once the word gets out that Linux ‘just works’ on a DELL, you’ll know that more customers will come your way.

5. Current support options are sufficient. A majority of respondents to the Linux survey at DELL indicated that the current community support forums would suffice their needs. Besides that, there’s a big Linux community out there willing to help new Linux users.

This space is reserved for linking to a list of why Linux should be the default OS for OEMs and Windows an option. (Contact me if you want to post that list a a guest blogger to Ariejan.net).

Rails Tip Snippet: Create a comma-seperate list

Tue, 27 Mar 2007 00:00:00 +0000

Do you have the need to create a list of roles a certain user belongs to? Enumerate the users attached to a company? All you want is a simple list with the names seperated by commas.

Users: John, Dick, Harry

With Ruby on Rails this is really easy. You probably have a collection of user objects. All you want is a list of names:

@users.collect{|u| u.name}.join(', ')

Subversion: How to revert to a previous revision

Tue, 27 Mar 2007 00:00:00 +0000

You’ve been there. You have been developing in your trunk for a while and at revision 127 you get the feeling you’ve done it all wrong! The production server is humming away at revision 123 and that’s where you want to start out again. But how can you start again from revision 123? Easy as this with Subversion:

svn merge -rHEAD:123 .

This will see what changes you’ve made since r123 up until now (r127 in your case) and ‘undo’ them. Next you check in the code and you’ve go a sweet r128 that is exactly the same as r123. You can start over now!

Speedlinking: Top 28 startup resources

Sun, 25 Mar 2007 00:00:00 +0000

I’ve been reading a lot about startups this weekend. Here is a quick post with all the articles I came across this weekend and found worth mentioning:

Got any articles that are not on the list? Please add a comment to let me know!

Be productive! 37 things you can do on the toilet

Fri, 23 Mar 2007 00:00:00 +0000

We all have to go. Most of us once a day. Others have to do it more often. However, we can be much more productive while doing or daily duty at the toilet.

Besides the old “joke book” and “newspaper” what can you do on the toilet to kill the time? I’ve found 37 things that might be a good option for you next number 2.

1. Newspaper - The age old classic of reading the newspaper. Nothing strage about that, right?

2. Joke book - Some loos feature a book with jokes for you to read. Nice, but mostly the jokes are of a very low quality.

3. Puzzle - Bring your sudoku book (any other type of puzzle will do) and start puzzling.

4. Novels - If you find it hard make time to read that novel everyone has been talking about, read it on the toilet. It’ll take a bit longer than usual, but you’ll get there.

5. Brainstorm - It’s quiet out there. Use your time to brainstorm and get some new ideas for whatever you need new ideas for. A pen and paper are handy to write it all down. Otherwise you may flush your ideas with the rest.

6. Redo your makeup - This one is for women only, but this is a nice place to redo your makeup with that travel-makeup-kit-thingy you always carry around.

7. Poke your nose - This is also a classic one. Nobody pokes his nose in public (well, most of us don’t) but if you like to clean it out, the toilet a good place to do it. No one sees you and you’ll get to wash your hands afterwards.

8. Listen to music - iPods are portable. Carry them with you at all times and listen to some tunes while sitting there. Other music devices such as mobile phones or mp3 players can be used as well.

9. Listen to podcasts - If you subscribe to podcasts, this is the time to listen to a few of them.

10. Watch some video - Have an iPod video or another portable video device such as the Playstation Portable? Make sure to set a clock, some movies run longer than 2 hours.

11. Play a game - You can play games on mobile phones, PDAs, iPods and portable gaming devices such as the Playstation Portable or a GameBoy. This is the time to finally beat the machine or set a new high score.

12. Learn stuff - Want to learn how to play Texas Hold’em poker? How to build your own kite? How to … You get it. Print out an article or two, bring a book or even your laptop and read more on these subjects.

13. Watch movie trailers - Don’t have time to watch trailers? Put ’em on your iPod and watch while waiting.

14. Chat - A lot of mobile devices have chat services. A laptop is the easiest way of chatting on the loo. If you want to talk, go ahead.

15. Read your mail - Use you mobile phone, PDA or laptop to check your (personal) email.

16. Check your stock - If you have stock feel free to check how much it’s worth now.

17. Blog - Want to write a blog post? Write it on the loo. If you don’t have wireless access (you should get that right now!) you can submit the post when you come back.

18. Tune your phone - This is more of a one time thing, but you may want to change your phones theme, background or ringtone every so often.

19. Read news - If you don’t have a newspaper feel free to read the news now on your laptop or PDA.

20. Organise your schedule - While you have the time, plan all the activities you need to plan and mark any problems in your schedule so you can fix them later (or fix them right now!).

21. Check Adsense - It’s always a good time to check up on Adsense and todays earnings.

22. Check your site statistics - Login to Analytics or view your logfiles and see if anything new is developing.

23. Decide on dinner - Don’t know what to eat after work? This is a time to decide on that.

24. Install/update software - Are new updates for your favourite programs waiting, take a bit of time and install them now.

25. Organise your to-do list - Check off items you have finished and add new ones you just thought off.

26. See what movies are running - Feel like going to a movie? Check online what’s running tonight.

27. Ping your coffee machine - If you’re a real geek, ping your coffee machine and brew a fresh pot for when you’re finished here.

28. Browse YouTube - Find a few movies you like and watch them. Laptop required.

29. Uninstall software - Uninstall all applications you don’t use to free up more space for your iTunes library.

30. Organise your documents - Organise all your files. Put them in neat folders and rename them properly.

31. SEO your site - Apply some quick patches to your site to improve SEO.

32. Link! - Post a speedlinking post or add links to your blogroll for sites your recently visisted.

33. Check free sites - Check those sites that collect free offers. Maybe something interesting has come up.

34. Hack the wireless routes - Try hack the wireless router your currently using. Make sure you don’t get fired over it!

35. Plan your trip home - Use an online planner to plan your trip home.

36. Work! - Need to finish a document fast? Continue working!

37. Do nothing - Old fashioned Number 2. Just sit there, do your thing and go!

Do you know other tasks you may perform while doing a number 2? Please continue the list in the comments. Thanksk for sharing!

Rails Tip Snippet: Logging informational messages to your log

Thu, 22 Mar 2007 00:00:00 +0000

This “Rails Tip Snippet” is one in a series of small blocks of code that will make your life developing Rails applications a bit easier.

This first snippet shows you how you can log informational message to your log file so you can track any important actions that happened. As an example you may want to log all user accounts that get deleted:

def destroy
 @user = User.find(params[:id])
 @user.destroy!
 logger.info("User account '#{@user.login}' deleted by user #{current_user.login})
 ...
end

Line 4 will put a string like “User account ‘johnlocke” delete by user ariejan" in your log file. Use that information any way you want.

You may also use the logger to inspect variables:

logger.info("article inspection: #{@article.inspect}")

Until next time! Any tips are welcome at tips@ariejan.net.

26 Things you can do with an old PC

Wed, 21 Mar 2007 00:00:00 +0000

We’ve all been there. Your old trusty PC dies. First thing you do is buy a new one. After you’re done playing with all the new bells and whistles your old PC didn’t have, it’s time to thing what to do with the old fella. Here are some tips!

The whole system can be used like this:

Give it away to friends or family who can use the "upgrade"
Take it apart and see what comes out
Create route/firewall for your network
Creata a file server
Install and try Linux
Create a jukebox / media player
Sell it
Trash it / Recycle it
Ceremonially set it on fire
See what happens when you shoot it
Symbolically throw it out of the window

If you only have the casing, this can be put to great use!

Take everthing out and turn it into a barbeque
See what happens when you don't cool your processor
Strip it and sell the working parts
Blow it up

You an also strip the insides of your case and put another device in it:

Coffee maker
Aquarium (with plexi glass plating, of course)
Webcam
Paper shredder
Use it as a trash can
Make it a mailbox
Build a can dispenser (hit 'reset' to obtain a beverage of your choice)

The monitor (CRT) can be used for various purposes as well:

This can also be converted into an aquarium
Remove the tube and create a cage for small animals
Ash tray
CD / DVD Rack

Got your own personal favourite? Please post a comment and share it with us all!

8 Great ways to use Google for your start-up

Tue, 20 Mar 2007 00:00:00 +0000

Google is more than a search engine or an advertising company. Of course, it’s their core business, but Google has more to offer. In this article I’ll describe how you can put Google to work for your start-up (or existing business!). Oh, did I mention you can do all of this for free?

The most commonly known service Google provides besides search and advertising is Gmail. Nowadays Gmail is a common concept in the field of e-mail services. When the services started you got a 1Gb inbox. Today you can store up to around 2.8Gb of emails and Google Talk chat transcripts! Wouldn’t it be great to use this for you company for free?

Well you can! And a lot more!

Most people may have heard about Google Apps.

So, what is Google Apps and how can I put it to use for my start-up?

When you visit Google Apps you can sign up for the “free” edition which will give you the following

Gmail with 2Gb inbox per user - You know how it works. Just for @yourdomain.com
Google Talk - Allow your users to chat easily with Google Talk
Google Calendar - Shared and personal calendars for free!
Docs & Spreadsheets - Write documents and spreadsheets easily online and collaborate with co-workers.
Page Creator - Creating a web site has never been easier!
Start Page - Know Google Personal Homepage? Same stuff but for yourdomain.com!
Easy administration panel to manager services and users
Unlimited number of users and online support

So, if you already own a domain name, or register one for a few bucks (or for $10 at Google), you can run email, instant messenging, calendar services, online office (docs & spreadsheets), page creator and a custom start page for your users for free!

This will cost you $10 bucks a year (for the domain name when registered with Google). All you need is a computer, Internet connecton and a decent browser.

How’s that for an initial investment in your start-up company? Oh, don’t forget that you can access all the above everywhere as long as you have an Internet connection! You can always (and easily) access your documents and e-mail or check your calendar. You can even do that with your mobile phone!

Personally I think Google Apps are a great investment in your start-up company. I’ll give you the pro’s and cons one more time:

Pro

It's free (except for the domainname)
Quality service you expect from Google
You have it all under one roof ready to go.
Access to your stuff everywhere in the world!

Con

Nothing really.

Let me know if you have any experience with Google Apps. Did I miss some feature? Please let me know!

How I made 6 figures with Google Adsense in 10 days

Tue, 20 Mar 2007 00:00:00 +0000

There has been a lot of talk about making money with Google Adsense. There are claims of people earning over $10.000 every month by just putting a few ads up on their site.

I say they are amateurs! I’ll show you how to make a 6-figure earning within 10 days after you put up ads on your site! Just follow these easy steps and you can quit your day job after just ten days of work. It’ what I did.

Decide on a topic for your site

First you must decide on what the topic of your site is. Try a keyword bid searching tool to find out which keywords and topics will generate the most.

It’s not important if you don’t have any knowledge on the topic you will be writing about. It’s just about making money.

Register a domain

Next you must register a domain and get your hosting setup. You can do this for a few bucks. This is the only investment you have to do here! For the $10 investment in hosting and domain registration you’ll get a ROI of more than 100.000%.

When choosing a domain, make sure it’s catchy. Something like “earn-money-with-adsense-for-free.com” or “imearningsixfigureswithadsensesucker.com”. It doesn’t need to be readable it just needs to contain a lot of keywords for you site.

Don’t bother with quality hosting. Just find the cheapest reseller you can find. Make sure you get at least 5Tb of bandwidth, though. You’ll need it.

Setup your site

Open up Frontpage and write an article. It only needs to be a few paragraphs. Make sure to include lots of keywords so Google can attach the right, high paying, ads.

Add the ads!

With your site in order now, add the maximum amount of ads allowed. For optimal results use the biggest ad format available and make sure a visitor sees the ads even before they reach the content of your site. This ensure the maximum clich-thru rate possible.

Also include a link-block. This will lure visitors to a page with only ads from Google! This is a real gold mine!

The third thing Google offers are referrals. Plaster some big images at the bottom. Claim how much you make with your site and have users sign-up for adsense too. This will make you $200 almost instantly!

Don’t forget the obligatory “Use Firefox” referral!

Publish your site

Now it’s time to publish your site and start making the big bucks! It’s almost time to quit your day job.

Surf around and post a link to your site everywhere you can. Comment on any blog you see and put your link there. Roam forums and randomly insert links to your site there too.

If you’re feeling adventurous you may also setup an Adwords campaign yourself and pay for visitors. Since you have many high-paying ads on your site you can make money at a 1:10 ratio ($1 spend on adwords is $10 earned in Adsense)

And now... you wait

Now you wait ten days and then quit your job (you have a two weeks notice anyway). After those ten days you’ll have made a nice 6 figures in hard currency. Just wait for the check to arrive and you’re done!

Oh, don’t forget to ***** my *** since I gave you this very valuable information and I still need to reach my 6 figures for this month. Thank you.

Wordpress Plugin: AutoFlickr 1.0

Wed, 07 Mar 2007 00:00:00 +0000

This post announces my first Wordpress Plugin: AutoFlickr.

What does it do?

When enabled, AutoFlickr will automatically find and insert one ore more photos that are related to the content of your post. Photos are found and hosted on Flickr.

Features

Automatically insert one or more photos
Customizable (photo size, number of photos and type of search).
Option to select what pages to automatically show photos.

Installation

Download the latest version here: AutoFlickr 1.0.

<li>Unzip the downloaded file and put the entire 'autoflickr' directory in /wp-content/plugins</li>
<li>Activate the plugin under Plugins</li>
<li>Go to Options -> AutoFlickr to customize the settings. </li>

That’s it. Optionally you may want to adapt autoflickr.css to change the way the images are aligned and styled.

If you want to include images in another place than the default, you may disable AutoFlickr for all pages and include the following PHP code in your template. Note that this code must be used within “The Loop”.

<?php echo autoflickr_html(); ?>

Demo

This plugin was developed for CityTrippers.net. You can see the plugin in action when reading a single story.

Support

Please leave a comment or contact me directly if you have any questions regarding this plugin.

New versions?

Please subscribe to my RSS Feed (or use e-mail subscription) to get automatic updates about Ariejan.net and this plugin.

Subversion Cheat Sheet Update: 1.0.1

Tue, 06 Mar 2007 00:00:00 +0000

I’ve just uploaded version 1.0.1 of the Subversion Cheat Sheet to Ariejan.net.

Please download this new version and get the following change:

Fixed typo. Thanks to Gregory Gerard.

Head to the Subversion Cheat Sheet page now and download the new version.

4 Unusual uses for Subversion

Mon, 26 Feb 2007 00:00:00 +0000

The most common use of Subversion is to keep source code of applications versioned and secure. However, there are quite a few other options that are not so common at all.

Quickly read on and find out if maybe you can put Subversion to use in quite a few ways you didn’t expect.

1. CMS

Maybe CMS is a bit strong, but it’s very easy to develop a static website locally and store it in Subversion. You just checkout your website on your production server and you’re good to go. You can easily jump back to previous content and you always have a backup around.

2. Todo list / Notes

You can version a list of text files as todo items or notes. You can check it out anywhere you want and keep track of all your past notes and items. You will have to establish some sort of formatting convention for yourself as how you name files and all that.

3. Backups

Put your entire home directory in Subversion. You’ll always have a backup handy and you can easily jump back in time! Maybe it’s not as sophisticated as Mac OS X Leopard’s Time Machine, but it serves its purpose.

4. Community Story Writing

It’s a bit like coding. You could use Subversion to write a story (or manuscript or book or …) as a community. It’s a bit like a wiki, but you can all use your favourite editors. You’ll have to agree on an ASCII based file format, though.

Maybe you know some very cool way to use Subversion as well? Please comment below!

Subversion Cheat Sheet 1.0!

Fri, 23 Feb 2007 00:00:00 +0000

I’ve noticed a huge interest in my Subversion articles lately and I thought to create a nice cheat sheet for all you.

The cheat sheets includes common commands that you’ll use when using Subversion on a daily basis. I didn’t include every option or command that subversion supports, nor did I include any administration stuff. This sheet is aimed at developers who use Subversion on a daily basis.

You can find the Subversion Cheat Sheet and future updates at:

http://ariejan.net/svncheatsheet

Please link to the url above from your blog to spread the word! Thanks.

Topics covered in this cheat sheet are:

Subversion Status codes
Daily usage commands
Bug fixing
Branching and tagging
File Magic
Locking
Properties
Experimenting

Please let me know if anything’s missing.

Ruby: Sort an array of objects by an attribute

Sun, 28 Jan 2007 00:00:00 +0000

In this example I’ll show you how easy it is to sort an array of (the same kind of) objects by an attribute. Let’s say you have an array of User objects that have the attributes ’name’ and ’login_count’. First, find all users.

@users = User.find(:all)

Now, we have to sort this array by ’name’. Since we don’t know if any user used capitals in his name or not, we use ‘downcase’ to sort without case sensitivity.

A small not. ‘sort’ returns a new array and leaves the original unchanged. You may want to just reorder the @users array, so use the ‘sort!’ method. The ‘!’ indicates it’s a destructive method. It will overwrite the current @users array with the new sorting.

@users.sort! { |a,b| a.name.downcase <=> b.name.downcase }

That’s all! Since strings are comparable, this will sort you user objects alphabetically by name. Want to sort on login_count instead?

@users.sort! { |a,b| a.login_count <=> b.login_count }

So, now you can easily sort any object in an array just like you want it too!

New in Rails: Resource Scaffold Generator

Tue, 23 Jan 2007 00:00:00 +0000

Oh boy! Rails 1.2 is all about resources. A product entry in your application is not just a rendered HTML page, but it “is” data. Rails 1.2 allows you to add a .xml extension to your url to retrieve the same product information in XML format!

Now, this all sounds hard. Two ways of rendering the same action depending on an extension. Generating XML code for every model in your database. No way you want to spend time developing all that stuff.

Well, Rails wouldn’t be Rails if there weren’t a generator for it!

This example is really easy. I want to create products in my database and expose them as HTML and XML to my users. Just do the following:

$ ./script/generate scaffold_resource Product title:string description:text

A migration is generated automatically so update that if you want to before running

$ rake db:migrate

As you can see in config/routes.rb, Rails has automatically generated routes for this resource!

map.resource :products

Now you can start up your Rails app and access http://localhost:8000/products. Add some products if you like. Now if you got to http://localhost:8000/products/1 you a nice HTML scaffold. If you add .xml: http://localhost:8000/products/1.xml you’ll get nice XML!

Well, writing a seperate desktop application that retrieves informatin from your web application just got 100x easier!

Rails: Nested resource scaffold

Tue, 23 Jan 2007 00:00:00 +0000

In my previous post I told you about the resource scaffold. What you’ll be doing a lot is nesting these resources. Ingredients in recipes, comments on posts, options for products. You name it, you nest it!

Since Rails does not automatically nest resources for you, you should do this yourself. This is, with some minor tweaks, really easy to accomplish. In this example I’ll create recipes that have multiple ingredients.

I assume you have Rails 1.2.1 installed for this tutorial to work properly.

First, I create an new rails project named ‘cookbook’. I use an SQLite3 database because it’s easy to do so. You may use any Rails compatible database for this example.

$ mkdir cookbook
rails --database sqlite3 cookbook
cd cookbook

First I create resource scaffolds for both the Recipe and Ingredient models:

$ ./script/generate scaffold_resource Recipe title:string instructions:text
./script/generate scaffold_resource Ingredient name:string quantity:string

As you can see I did not add a recipe_id to the ingredient model because of the has_many relationship. Add this column to the migration file. You should now be able to migrate your database:

$ rake db:migrate

If you add the recipe_id to the generate script the view for your ingredients will include a field for the recipe_id and that’s not what you want.

Next, make the has_many relationship in your models.

app/models/recipe.rb:

class Recipe < ActiveRecord::Base
 has_many :ingredients
end

app/models/ingredient.rb ```ruby class Ingredient < ActiveRecord::Base belongs_to :recipe end


So far, nothing new. Next we check out config/routes.rb:
```ruby
map.resources :ingredients
map.resources :recipes

What we want is to map ingredients as a resource to recipes. Replace these two lines with:

map.resources :recipes do |recipes|
 recipes.resources :ingredients
end

This will give you urls like /recipes/123/ingredients/321

Now we need to make some changes to the ingredients controller. Every ingredient belongs to a recipe. First add the filter:

before_filter(:get_recipe)

private
def get_recipe
 @recipe = Recipe.find(params[:recipe_id])
end

This will make sure that every ingredient knows what recipe it belongs to.

In the index method of the ingredient controller, make sure you have this:

@ingredients = @recipe.ingredients.find(:all)

This makes sure you only show ingredients for this recipe, and not all ingredients in the database.

Because we changed the route for ingredients, we need to update all ingredient_url() and ingredient_path() calls in our controller and views. Change all occurrences of

ingredient_url(@ingredient)

and

ingredient_path(@ingredient)

ingredient_url(@recipe, @ingredient)

and

ingredient_path(@recipe, @ingredient)

Note: Make sure that you don’t replace ‘ingredient’ with ‘@ingredient’ in your views!

Add a link to the ingredients to your recipe’s index.rhtml view.

link_to 'Ingredients', ingredients_path(recipe)
And, at last, make sure the create method of your ingredients controller attaches the ingredient to the right recipe. Make sure the first two lines look like this:

def create @ingredient = @recipe.ingredients.new(params[:ingredient])
You may now start your webserver and check out http://localhost:8000/recipes . Create some recipes and click ‘ingredients’. You can now add ingredients for this recipe!

The next step will be customizing each method to suit your own needs.

Updates: Wordpress 2.1, Themes and Social

Tue, 23 Jan 2007 00:00:00 +0000

You can’t really see it, but Ariejan.net has been upgraded to Wordpress 2.1. I’ve been running 2.1 beta’s on a private server for some time now, so there weren’t any surprises during the upgrade.

I’ve also updated the theme to something more stylish and sober. Google Ads are less annoying now and merge nicely with the content. I’ve also re-enabled the social bookmark links so you can quickly bookmark articles on Del.icio.us or Digg.

Hope you like the new style. Some minor tweaks will be applied the following days where needed. Please let me know your thoughts on ariejan.net!

“Print this page” with Ruby on Rails

Fri, 19 Jan 2007 00:00:00 +0000

You have put a lot of effort into creating a sexy overview of whatever data your application stores and allow your users to manipulate that data through AJAX controls. But, some people just want to print their data.

How to go about that? Just printing the page with data is generally not a good idea because it has been optimized for display on a screen. The first step we need to take is adapting our page for printing. Stylesheets are very handy tools for this. Check the following part of the header of my layout:

stylesheet_link_tag('default', :media => :screen)
stylesheet_link_tag('print', :media => :print)

This will generate HTML code that includes two stylesheets. However, only the ‘default’ stylesheet is used on screen. When the users prints a particular page, the print stylesheet is used instead. So, what do you want to change in the print stylesheet?

Fonts should be serif (not sans-serif) for printing
Hide images as much as possible
Hide ads
Display printable ads that are hidden on screen
Hide navigational elements
Use a black-on-white colour scheme
Underline links and colour them blue for easy recognition
Add the actual URL to your links (see below)

To add the actual URL in the href-part of your link to the name of your link add the following to you print stylesheet:

a:after { content:' [' attr(href) '] '}

Now, with all this in place, load your page with data. Print your page or watch a preview and be amazed at what stylesheets can do for you!

The final touch is adding a “Print this page” link to your navigation on screen. Here you can use link_to_function:

link_to_function("Print this Page", "javascript:print()")

That’s it. Just hit the Print this Page button and your browsers print dialog will pop up and use your fancy new print stylesheet.

Why Ruby Rocks - Convince your fellow developers

Fri, 19 Jan 2007 00:00:00 +0000

I often hear questions from my Java and PHP oriented friends about what makes Ruby so great and easy to use. Until today I’ve shown them some of my Rails feats (AJAX Scaffold always amazes people). Now, I came across this 20 minute Ruby introduction. Starting with the basic “Hello World” item, this article show step by step improvements to end up with blocks, objects and all that makes Ruby really worth while.

So, if you need to show developers what Ruby (and not so much Rails) can do, show ’em this! It impresses the shit out of ’em.

Rails: Group results by week (using group_by)

Fri, 12 Jan 2007 00:00:00 +0000

The Enumerable class in Rails contains a method named ‘group_by’. This method is pure magic for a developer’s point of view. I’ll give you a simple example that shows the power of group_by.

Let’s say you have a table ‘posts’ containing blog posts. Now, you normally show these chronologically a few at a time. Nothing special there. For some special overview page, you want to group your posts by week.

With normal ActiveRecord operations this would be quite an elaborate task. But with group_by from Enumerable, it becomes child’s play.

First of all, in the controller, just get all the posts you need. In this case, all of them:

Controller:

def list
 @posts = Post.find :all
end

As you can see, I perform no ordering or whatsoever here.

Now, in your view you normally would iterate over all posts like this:

<%= render :partial => 'post', :collection => @posts %>

But, as I said, we want to group the posts by week. To make life easy, I add a method to the Post class that returns the week number in which a post was written:

Model Post:

def week
 self.created_at.strftime('%W')
end

Now, the magic will happen in our view:

<% @posts.group_by(&:week).each do |week, posts| %>
<div id="week">
<h2>Week < %= week %></h2>
< %= render :partial => 'post', :collection => @posts %>
</div>
<% end %>

Let me explain the above. We specify that we want to call group_by for @posts. But we need to say how we want to group these posts. By specifying &:week we tell group_by that we want to group by the result of the week attribute of every post. This is the attribute we specified earlier in the model.

Well, when the grouping is done we create a block that will handle every group of items. We extract ‘week’ and ‘posts’ here. ‘week’ contains the week number and ‘posts’ all the posts for that week.

As normal, we can now show the week number and iterate over the posts.

Sorting groups

The result of group_by is not guaranteed to be ordered in any way. Simply call ‘sort’ before each and you’re set:

@posts.group_by(&:week).sort.each do |week, posts|

Mostly, you’ll find that the posts for every group are not sorted either. With the example above I think it’s easy to figure out how to do that now. (hint: .sort)

Textmate+Rails: Easy partials for better code

Fri, 22 Dec 2006 00:00:00 +0000

As you may know, I use TextMate for editing Rails code.

I’ve just been browsing the Rails bundle today and I came across some very interesting things. Today I’ll tell you about partials.

Partials are ERb templates. They are mostly HTML (or RJS or XML or whatever output format you use) and include some embedded Ruby to show actual content. Partials are not linked to a method in a controller, but instead they can be easily rendered through-out your application.

TextMate allows you to refactor your application to use partials with almost no effort!

First something about partials for those who don’t know about it.

Partials are stored in files that start with an _ (underscore) and end in .rthml (or .rjs or .rxml) and you may include them in your ERb like this:

< %= render :partial => 'userinfo' %>

This will include the partial _userinfo.rhtml for you. This way you can avoid duplication by including the same ERb template (partial) over and over again.

Partials are also useful to keep your views (templates) clean and easy to understand. In large projects this will allow you later to easily change a view without having to spit through 500 lines of ERb.

Let’s say you have a profile page where you want to show a users’s profile and some options for when the user is viewing his own profile. Also, you want to show a simple message when the users has not yet created a profile, which is a possibility in my application. The show.rhtml template for the profile may look like this

<h2>Profile for < %=h @profile.user.login %></h2>

< % if @profile.exists? -%>

 <div id="membersince">< %= @profile.firstname %> has been a member for < %= distance_of_time_in_words_to_now @profile.user.created_at %></div>

 <h3>General information</h3>
 <div id="name">Name: < %= @profile.fullname %></div>
 <div id="country">Country: < %= @profile.country %></div>
 <div id="birthday">Birthday: < %= fmt_date @profile.birthday %></div>

 <h3>About < %= @profile.firstname %></h3>
 <div id="about">< %= markdown @profile.about %></div>

< % else -%>

 <div id="membersince">< %= @profile.user.login %> has been a member for < %= distance_of_time_in_words_to_now @profile.user.created_at %></div>

 <h3>Oh my!</h3>
 <p>This user has not yet created a profile.</p>

< % end  -%>

< % if @profile.user.id == current_user.id -%>
<h3>Options</h3>
<p>This is your profile, so you have some options:</p>
<ul>
 <li>< %= link_to "Change password", :controller => 'account', :action => "change_password" %></li>
 <li>< %= link_to "Edit your profile", profile_edit_url(:login => current_user.login) %>
</li></ul>
< % end -%>

As you may see, this is chaos. It’s not easy to read and you can’t easily change something when you need to. Wouldn’t it be great to store all three seperate blocks (profile, no profile message and options) in partials?

This is where TextMate does it’s magic. Here’s a before snaphot:

Now, select the part between the if and else statements. This is where a normal profile is shown. Next, use TextMate: SHIFT-CTRL-H. You’ll get a pop-up asking you how you’d like to name the partial. In this case I choose ‘profile’.

After you click OK, the partial will be created containing the code you selected and an appropriate render method is placed instead. This is what the code looks like after:

After all this the same ERb template looks like this:

<h2>Profile for < %=h @profile.user.login %></h2>

< % if @profile.exists? -%>
 < %= render :partial => 'profile' %>
< % else -%>
 < %= render :partial => 'noprofile' %>
< % end -%>

< % if  @profile.user.id == current_user.id -%>
 < %= render :partial => 'options' %>
< % end -%>

Now, isn’t that more readable? Even, if you want to change a profile, you can just edit _profile.rhtml and you’re done.

SVN: How often should you commit?

Wed, 20 Dec 2006 00:00:00 +0000

I often hear discussion about how often developers should commit their work to the central repository. Some say that you should only commit when you’re next ‘release’ is ready. Others say that you should commit every change you make in your code. There are even people who say you should commit your changes only at the end of the day.

All wrong! There is no such thing as ’the way’, but there is a thing called best practice and that’s what I want to talk to you about. How often and what should you commit to your Subversion repository.

Actually, it’s quite simple. If you use a tool like Trac or any other ticketing tracking system you have already got a great starting point (given, of course, that you add ’things you need to do’ to your tracker).

I split my development activities into small atomic actions. Normally these actions take up a few minutes to an hour of your time. Besides the fact that these atomic actions are managable, they also give you a sense of satisfaction because you can complete quite a few of these smaller actions every day.

All right. You have these atomic actions. I grab my code, perform the atomic actions and when it succeeds, I comment my changes back to Subversion. This always closes a ticket in my Trac system and most of the time refences the ‘big picture’ ticket.

Here’s what I mean.

Big picture ticket: “Create an accounting system for users”

Atomic Action ticket: “Create and customize the user model and unit tests” Atomic Action ticket: “Install acts_as_authenticated plugin”

So, I start with my code and first I install the acts_as_authenticated plugin. Great. Commit that. This takes a few minutes at the most.

Next, I create and customize the user model and unit tests. When that’s done, I commit my changes again. As you can see, I’ve already closed two tickets (and got that warm satisfactory feeling).

Okay, you get the picture. Why would you work like this? Subversion allows you to keep track of changes in your code, but it doesn’t force you on how ofter or when you track changes.

The most logical way of keeping track of changes is atomic actions. If you decide later that you don’t want the installed version of the acts_as_authenticated plugin, but a later release. You can see what files were added when you installed the plugin.

By using atomic actions you create atomic changesets in Subversion. This way you can easily jump between different states (revisions) of you code without breaking things. It keeps your code working and it allows you to easily track changes you made and to what purpose.

SVN: Merge a branch with your trunk

Wed, 20 Dec 2006 00:00:00 +0000

When created a TRY-branch a few days back to try some fancy new AJAX technology in my application. Not problems there, so now I want to merge the code in the branch with my trunk.

Since I’m a lone hacker, the trunk has not been touched since I created the branch. I have a checked-out working copy of the branch available.

Here’s a handy-dandy guide on how to merge your branch code with your trunk.

Firstly, make sure you have a working copy of your trunk. I choose to switch my working copy back: (oh, make sure you have all your changes checked in in your branch before you switch!)

$ svn switch http://example.com/svn/myproject/trunk

This removes, adds and updates all files you have worked on in your branch and creates a working copy of the code in the trunk.

Now, with my trunk in place, I can call ‘merge’ and apply the changes.

$ svn merge http://example.com/svn/myproject/trunk http://example.com/svn/myproject/branches/TRY-AJAX

Since the files from the trunk and the beginning of the TRY-branch are still exact copies, I won’t get in any trouble. If you do (and you did change your code in your trunk), make sure to resolve merging problems before checking in. When ready, check in your new code!

$ svn ci -m "Merge TRY-AJAX branch with trunk"

That’s it. You have now merged the branch with your trunk. Enjoy!

Show the current SVN revision in your Rails app

Wed, 13 Dec 2006 00:00:00 +0000

I’m current developing a Rails application. I deploy this application to a demonstration server using capistrano.

To streamline feedback and bug reporting I want to show the current revision number of the code that’s published on the demo server to show in the footer of every page.

First I looked into Subversion keyword expansion, but this is marked as ’evil’ and it doesn’t meet my requirements. I want to show the latest revision number of the entire repository and not just that of the current file.

Luckily for me, I use capistrano. Here’s how I fixed the problem.

First of all, I created a partial that contains the revision number and render this in my layout.

app/views/layouts/_revision.rhtml:


CURRENT

This shows CURRENT always when I work with my working copy.

app/views/layouts/mylayout.rhtml


 < %= render :partial => 'layout/revision' %>

Now, I assume you have already setup capistrano for your project and that you have a config/deploy.rb file.

I’ve added the following helper to my config/deploy.rb file:

desc "Write current revision to app/layouts/_revision.rhtml"
task :publish_revision do
 run "svn info #{release_path} | grep ^Revision > #{release_path}/app/views/layouts/_revision.rhtml"
end

Next, I added the following hook ‘after_update_code’. This will automatically be run after update_code which is called in ‘deploy’.

desc "Run this after update_code"
task :after_update_code do
 publish_revision
end

That’s it. When I deploy the application, the current code is checked out and and the layouts/_revision.rhtml file is overwritten with the current revision information.

Bonus

You could also leave the layouts/_revision.rhtml files empty and update it for your demonstration server, but not for your production box. This way there won’t be a revision added.

Of course, you could also create a deploy_demonstration method in deploy.rb and call publish_revision manually from there.

Install ruby-mysql on Mac OS X 10.4 Tiger

Wed, 06 Dec 2006 00:00:00 +0000

You probably know that the built-in mysql code in Rails sucks. To rephrase that, the ruby-mysql gem contains better code, so you want that. Rails automatically detects if you have ruby-mysql installed or not, and uses it if you have it.

Most notably, you want to install this gem if you get dropped MySQL connections running your Rails application.

So we do:

$ sudo gem install mysql
...
ERROR: Failed to build gem native extension.

Normall this would install fine, but not on Mac OS X. This is because Mac OS X keeps its code, headers and libraries in odd places (compared to Linux). But don’t panic. There’s an easy solution to all this!

Just compile the gem yourself, and add some special ingredients. Well, not even that. Just compile it again like this:

$ cd /usr/local/lib/ruby/gems/1.8/gems/mysql-2.7
$ sudo ruby extconf.rb --with-mysql-lib=/usr/local/mysql/lib --with-mysql-include=/usr/local/mysql/include
$ sudo make
$ sudo make install

Now, you have the ruby-mysql gem installed and ready to go! No more dropped MySQL connections for Rails!

Installing Rails on Ubuntu Dapper / Edgy

Sun, 03 Dec 2006 00:00:00 +0000

Update 2010-03-25: Bumped to RubyGems version 1.3.6.

Update 2009-02-19: Bumped to RubyGems version 1.3.1 and MySQL 5 libraries. This guide now works for all recent version of Ubuntu and Debian. Enjoy!

Installing Ruby on Rails on your Ubuntu box is not always as easy as it seems. Here’s a comprehensive overview of the steps you need to take. Mostly you’ll be using apt-get and gems, so it’s not all that hard after all.

This method was tested on both Dapper and Edgy systems. It may work on other Ubuntu releases as well. It’s also possible that it works on Debian.

Besides Rails, I’ll also be install mysql and sqlite3 support.

1. Install Ruby

Before putting anything on Rails, install Ruby.

sudo apt-get install irb1.8 libreadline-ruby1.8 libruby libruby1.8 rdoc1.8 ruby ruby1.8 ruby1.8-dev

You may now check what version of Ruby you have by running ruby -v. It’s just for your information.

2. Install Ruby Gems

Surf to http://rubyforge.org/frs/?group_id=126 and download the latest available gems pacakge in tgz format. (You may also use the zip if you feel comfortable.)

wget http://rubyforge.org/frs/download.php/69365/rubygems-1.3.6.tgz
tar zxf rubygems-1.3.6.tgz
cd rubygems-1.3.6
sudo ruby setup.rb
sudo ln -sf /usr/bin/gem1.8 /usr/bin/gem

3. Install Rails!

You may now install Rails now!

sudo gem install rails

That’s it! Well, almost. You probably want some other things as well.

4. Install development tools

Before you continue, stop a moment to install some development tools. These tools are probably needed to compile and install the gems we are going to install next.

sudo apt-get install build-essential

MySQL Support

You probably want MySQL Ruby support. The MySQL code in Rails sucks (no offence), but the Ruby code is much better. You should install it. Rails will notice that it’s available and use it.

First, install MySQL.

sudo apt-get install mysql-server mysql-client

Next, install development files for MySQL. We’ll need these in order to make Ruby understand.

sudo apt-get install libmysqlclient15-dev

Then, you may install the gem

sudo gem install mysql

You have to choose what version you want to install. Enter the number corresponding with the latest version that is tagged ‘ruby’. Installing win32 stuff on linux is generally not a good thing.

SQLite 3

For SQLite 3 you need to install some packages before installing the gem.

sudo apt-get install sqlite3 libsqlite3-dev
sudo gem install sqlite3-ruby

If things go wrong

If things go wrong, make sure you installed all recommended packages mentioned above. Also , check any log files that the error message refers to.

How to setup a Ubuntu development server - Part 2

Sat, 02 Dec 2006 00:00:00 +0000

Install Trac

First thing to do is install trac. Here I will also install mod_python for your apache webserver and python-setuptools that we’ll need later with the webadmin plugin.

sudo apt-get install trac libapache2-mod-python python-setuptools

Now, I create a directory where all Trac information will be stored.

sudo mkdir -p /var/lib/trac

Common sense dictates that you use the same name here for the trac environment as for the subversion repository.

Change to the trac directory and intitialize the project:

cd /var/lib/trac
sudo trac-admin colt initenv

You’ll need to name the project, choose a database file (default is okay), specify where the subversion repository resides ( /var/lib/svn/colt, in this case) and a template (the default is okay here too).

I recommend you also create an administrator user right now. Make sure you add a user who’s already in your /etc/apache2/dav_svn.passwd file.

sudo trac-admin colt permission add ariejan TRAC_ADMIN

Well, that’s it. Trac has been installed. Now let’s make sure we can access trac through the web.

Configuring Apache

Configuring apache is rather easy when you know what to do. Add the following code to /etc/apache2/sites-available/default (at the bottom before the end of the virtualhost tag) or put it in a seperate virtual host file if you want to dedicate a special domain to this.

<location /projects>
 SetHandler mod_python
 PythonHandler trac.web.modpython_frontend
 PythonOption TracEnvParentDir /var/lib/trac
 PythonOption TracUriRoot /projects
</location>

<locationmatch "/projects/[^/]+/login">
 AuthType Basic
 AuthName "Trac Authentication"
 AuthUserFile /etc/apache2/dav_svn.passwd
 Require valid-user
</locationmatch>

Notice here, again that we use TracEnvParentDir to show we host multiple instances of Trac. You may change the TracUriRoot to something different.

Again, make sure to chown your Trac installation to www-data:

sudo chown -R www-data.www-data /var/lib/trac

Now, access your trac over the web: http://example.com/projects for a complete listing of hosted projects or http://example.com/projects/colt for the COLT project.

You may also login now! As you can see, we use the dav_svn.passwd file here so everyone with subversion access also has access to trac.

Webadmin

Normally you would administrate a Trac installation through the command-line interface we used to initialize the environment and add the administrator user. Nowadays there is a webadmin plugin for Trac, which will be included in Trac from version 0.11. Since Ubuntu ships with Trac 0.9.3 we need to add this webadmin ourselves.

First, download the following file to your server: http://trac.edgewall.org/attachment/wiki/WebAdmin/TracWebAdmin-0.1.2dev_r4240-py2.4.egg.zip.

Don’t unzip this file, just remove the .zip extension.

Because we installed setuptools earlier, we can now use easy_install to install this plugin system-wide, enabling it for all our trac installations.

sudo easy_install TracWebAdmin-0.1.2dev_r4240-py2.4.egg

Next we enable webadmin in the global configuration file of track. You may need to create the ‘conf’ directory in this case:

cd /usr/share/trac
sudo mkdir conf
sudo vi conf/trac.ini[/conf]

Next enter the following in trac.ini

[components]
 webadmin.* = enabled

Save the file and off you go. Login as the administrator user you specified earlier and you can make use of the ‘admin’ button that has appeared in the menu of Trac.

Enjoy your trac! Next time (in Part 3) I will talk about setting up a commit-hook so Trac tickets are updated by posting subversion commit messages (throught commit-hooks).

Also read Part 1 - Subversion on how to install Subversion over WebDAV.

How to setup a Ubuntu development server - Part 1

Fri, 01 Dec 2006 00:00:00 +0000

Since I’m starting some real work on my final school project, I want to install a Ubuntu development server here at home. I have a Pentium 4 box here that will perform that task.

In this first part I will show you how to install Subversion over WebDAV. All of this will be done in such a way that it’s easy to serve multiple projects at once.

In future parts I will tell you more about installing Trac, FastCGI (with Apache) to host Rails applications and how to use Capistrano to deploy your app properly.

For now, let’s get cracking at Subversion.

First off, I installed Ubuntu 6.10 on my server. Because I don’t need a graphical user interface, I have installed Ubuntu in text-only mode.

Open up to the universe

The first thing I always do when I install a Ubuntu box is to enable the universe package repositories. This is rather easy.

Edit /etc/apt/sources.list and uncomment all the Universe related lines. Also, comment out your install disk. Here’s what my /etc/apt/sources.list looks like:

# deb cdrom:[Ubuntu 6.10 _Edgy Eft_ - Release i386 (20061025)]/ edgy main restricted

deb http://nl.archive.ubuntu.com/ubuntu/ edgy main restricted
deb-src http://nl.archive.ubuntu.com/ubuntu/ edgy main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://nl.archive.ubuntu.com/ubuntu/ edgy-updates main restricted
deb-src http://nl.archive.ubuntu.com/ubuntu/ edgy-updates main restricted

## Uncomment the following two lines to add software from the 'universe'
## repository.
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://nl.archive.ubuntu.com/ubuntu/ edgy universe
deb-src http://nl.archive.ubuntu.com/ubuntu/ edgy universe

## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://nl.archive.ubuntu.com/ubuntu/ edgy-backports main restricted universe multiverse
# deb-src http://nl.archive.ubuntu.com/ubuntu/ edgy-backports main restricted universe multiverse


deb http://security.ubuntu.com/ubuntu edgy-security main restricted
deb-src http://security.ubuntu.com/ubuntu edgy-security main restricted
deb http://security.ubuntu.com/ubuntu edgy-security universe
deb-src http://security.ubuntu.com/ubuntu edgy-security universe

The next step is to make sure all software present is up-to-date. There are already a few updates available so run these two commands:

sudo apt-get update
sudo apt-get dist-upgrade

That’s it.

Getting SSH up and running

Because I have a MacBook, I’d like to use it to do my work. To do so I need to install the OpenSSH server on my server so I can access it over the network.

sudo apt-get install ssh

This will install ssh and the OpenSSH server. It also generates everything you need automatically like RSA keys and all that.

Now, try to login wiht SSH from your desktop machine.

Apache

Since I want to use Subversion of WebDAV I will need to install Apache first. I’ll grab a vanilla copy of Apache from Ubuntu here.

sudo apt-get install apache2

If that’s finished you should see a placeholder when you access your server with a browser. Check this now.

Getting Subversion

Subversion is also easily installed.

sudo apt-get install subversion subversion-tools

Next we should setup a location for our Subversion repositories. I choose to put them in /var/lib/svn. Create this directory now:

sudo mkdir -p /var/lib/svn

I also create a repository now and create a basic Subversion structure there. In my case the project is called ‘colt’.

sudo mkdir -p /var/lib/svn/colt
sudo svnadmin create /var/lib/svn/colt
sudo svn mkdir file:///var/lib/svn/colt/trunk -m "Trunk"
sudo svn mkdir file:///var/lib/svn/colt/tags -m "Tags"
sudo svn mkdir file:///var/lib/svn/colt/branches -m "Branches"

Note that you need to sudo the svn commands because only root has write access to your repository currently.

WebDAV for SVN

Okay. You are already at revision 3 on your repository. Good work! Now let’s make sure that you repositories are accessable over the web. First, we install libapache2-svn. This packages includes WebDAV support for SVN.

sudo apt-get install libapache2-svn

Next I open up /etc/apache2/mods-available/dav_svn.conf. This file contains configuration for the WebDAV and SubVersion modules we just installed.

Here I enable basic HTTP authentication, which is good enough for my local network. I also enable DAV by uncommenting “DAV svn”.

You need to take special care of the “SVNPath” variable here. We don’t host just one repository. We host several and /var/lib/svn is the parent directory of all our repositories. E.g. ‘colt’ is a sub directory that resides in /var/lib/svn. To make Apache understand this we need to change “SVNPath” to “SVNParentPath”. This enables all sub directories to be independent repositories.

Note: The authentication file we use here can be recycled later when we install Trac! Handy-Dandy, isn’t it?

I made my configuration look like this:

# dav_svn.conf - Example Subversion/Apache configuration
#
# For details and further options see the Apache user manual and
# the Subversion book.

# <location URL> ... </location>
# URL controls how the repository appears to the outside world.
# In this example clients access the repository as http://hostname/svn/
<location /svn>

 # Uncomment this to enable the repository,
 DAV svn

 # Set this to the path to your repository
 SVNParentPath /var/lib/svn

 # The following allows for basic http authentication. Basic authentication
 # should not be considered secure for any particularly rigorous definition of
 # secure.

 # to create a passwd file
 # # rm -f /etc/apache2/dav_svn.passwd
 # # htpasswd2 -c /etc/apache2/dav_svn.passwd dwhedon
 # New password:
 # Re-type new password:
 # Adding password for user dwhedon
 # #

 # Uncomment the following 3 lines to enable Basic Authentication
 AuthType Basic
 AuthName "Subversion Repository Access"
 AuthUserFile /etc/apache2/dav_svn.passwd

 # Uncomment the following line to enable Authz Authentication
 # AuthzSVNAccessFile /etc/apache2/dav_svn.authz

 # The following three lines allow anonymous read, but make
 # committers authenticate themselves.

 <limitexcept GET PROPFIND OPTIONS REPORT>
 Require valid-user
 </limitexcept>

</location>

Now kick Apache to reload and you should be able to access your repository over the web! Try http://example.com/svn/colt.

Authentication

Reading of the repository is okay without authentication. But writing needs to be protected. We need to create a password file for this. This is easy and is already explained in /etc/apache2/mods-available/dav_svn.conf:

sudo htpasswd2 -c /etc/apache2/dav_svn.passwd ariejan

Go ahead, add as many users as you need.

Apache, Subversion and the world

Before you start using Subversion, make sure you make the repositories owned by Apache. Apache is the one who wil access the repositories physically. This is really easy:

sudo chown -R www-data.www-data /var/lib/svn

When you access your repository for write actions now, you will recieve the following message:

Authentication realm: <http ://example.com:80> Subversion Repository Access
Password for 'ariejan':

Alright sparky! Subversion access is ready for you now! Next time I’ll tell you how to integrate Trac with your hot new Subversion repositories.

Notes

All users in you passwd file can write to all repositories. I’ve not yet found a way to prevent this, since I don’t need that functionality right now. If you know more about this, please let me know.

Subversion repositories are always readable by anonymous people. You should remove the LimitExcept block from dav_svn.conf to make sure all users authenticated themselves before accessing your repository.

If you add more repositories, you always have to chown them to your apache’s user and group. After that you can use them through WebDAV.

Rails: Security Check-up

Thu, 30 Nov 2006 00:00:00 +0000

Is your Rails app secure? Really? Maybe you need to perform a major check-up of your Rails application to make sure.

Here’s a comprehensive list that will take you through the most common mistakes and forgotten security risks in your Rails application.

It’s a great post. Print it, hang it on your wall and create secure Rails apps from now on.

The Three Corner Stones of Developerhood

Sat, 25 Nov 2006 00:00:00 +0000

As a developer, there are just some things you can’t do without. I have found that there are three things that I need. Really need. If one of them is missing, I have trouble doing my job properly.

I call them “The Three Corner stones of Developerhood”: ~

1. Coffee

Most importantly coffee. I just can’t do without it. I don’t drink it to stay awake or anything, but it’s just a warm beverage that keeps me focused and gives me the energy I need to keep working. I prefer coffee over all those horrible caffeine drinks.

2. A good text editor

No matter how good Integrated Developement Environments are, I always prefer my trusty TextMate to get things done for me.

Syntax highlighting and the huge amount of macro’s is just a pleasure to work with. I support PHP, HTML, CSS and most importantly, Ruby on Rails.

If you have a Mac. Buy this now!

3. Google

I can’t tell you how many problems I’ve solved by using Google to find the answer.

Of course, you can just insert any search engine here, but Google is really my favourite. With Google Groups I can find stuff in news groups easily, and as a developer, news groups are a very valuable resource of information.

It’s very well possible that you have three other corner stones. Feel free to comment them below and let me know.

SVN: How to structure your repository

Fri, 24 Nov 2006 00:00:00 +0000

You are reading an article about Subversion. That’s great, because it means you’re thinking about the benefits of version control for your project. However, I have long since moved from Subversion to Git. I strongly recommend you read up on my Best Practice - The Git Developmenet Cycle. Git is much faster an flexible than Subversion, go check it out now!

Most people know what Subversion is and that there’s something called “The Trunk” with code in it. Well, there’s more to your SubVersion repository than you think! This article will discuss how to structure your repository in order for you to take full advantage of Subversion’s possibilities.

As you may have read in my previous Subversion articles the base of your Subversion repository are three directories: branches, tags and trunk.

Each directory in subversion can be checked out seperately. See the examples for more information.

Trunk

The trunk contains the most current development code at all times. This is where you work up to your next major release of code.

I see, almost too often, that people only use the trunk to store their code. Releases are pulled from a certain revision and then development continues. This is no good. Not for you, not for your product.

The trunk should only be used to develop code that will be your next major release. Don’t brand the trunk with version numbers or release names. Just keep the trunk in “development mode” at all times.

Example: https://svn.example.com/svnroot/project/trunk

Branches

There are different types of branches. I’ll tell you about the most common ones here.

With the branches directory you can create paths for you code to travel to more specific goals like an upcoming release. As I discussed in my article on releasing software from Subversion the branches directory contains copies of your trunk at various stages of development.

Release Branches

We have already seen the RB-x.x release branches. When the trunk reaches the stage that it’s ready to be released (or when you want to freeze the addition of new features) you create a release branch. This release branch is just a copy of your current trunk code.

The branch can be checked out seperately and you can start branding and versioning the project. You can also use the release branch to fix bugs that pop up during (beta) testing (See my article on that too). The idea of this is to keep development in the trunk going without being bothered by release specific stuff. So it’s perfectly fine to add new features to your trunk while you (or others) prepare the release.

Of course, you can address a release branch directly to check it out:

https://svn.example.com/svnroot/project/branches/RB-1.0

Bug fix branches

Branches may also be used to address the more serious bugs found in the trunk or a release branch. The bugs are of such a magnitute that you can’t fix them by yourself in a single commit. So, in order to focus on the problem of fixing this bug you should create a new branch for this purpose. This allows development in the trunk or your release branch to continue, and you won’t disturb them with new bugs or tests that break the current code.

Bug fix branches are named after the ID they are assigned in your bugtracking tool. Mostly this is a number: BUG-3391

Of course, you can access your bugfix branch like any other.

https://svn.example.com/svnroot/project/branches/BUG-3391

Read my how to fix bugs properly article for more specific bug fixing information. Also read on in this article to the tags section.

Experimental branches

Something that also happens a lot is the introduction of new technologies. This is fine, of course, but you don’t want to bet your entire project on it.

Imagine that you want to change from PHP 4 to PHP 5 for your web application. How long would it take you to convert your entire project? Do you want your entire code base (trunk) to be useless until you have converted all of your code? Probably not!

These experiments, maybe PHP 5 is a bridge too far for your app, should be given their own branch. You can hack your way to PHP 5-ism there and if you fail, you still have your current PHP 4 code in the branch.

Experimental branches may be abandonned when the experiment fails. If they succeed you can easily merge that branch with the trunk and deliver your big new technology. These branches are named after what you’re experimenting with. I always prefix them with ‘TRY-’:

https://svn.example.com/svnroot/project/branches/TRY-new-technology

Google Project Hosting: SourceForge Competitor

Wed, 22 Nov 2006 00:00:00 +0000

I just found out that Google Code is now offering Project Hosting! It’s in the same style as SourceForge, but the Google way!

As a Google user you can create a project with a built-in issue tracker and Subversion repository. And if that’s not all, you can tie in a blog (at Blogger.com is you like) and adiscussion group (on Google Groups, of course).

This is really a nice package Google is offering and it looks like a serious competitor for SourceForge.

It’s just that I didn’t find any possibility to release files yet, but other than that, I’d like to move there!

SVN: How to fix bugs properly

Wed, 22 Nov 2006 00:00:00 +0000

I’ve already told you about releasing your project with help from Subversion. Now I want to talk to you about using Subversion to fix bugs in your application.

Fixing bugs can be as easy as fixing a few lines of code or as hard as rewriting a significant portion of your application. Both situations need a different approach from us. Let’s talk about the easy stuff first.

For this example let’s say we have a project. It has a release branch named RB-1.0 and current development is going on in the trunk.

A user has submitted a bug report (numbered #3391) against your 1.0 release. Here’s what to do:

Easy bug fixes

Let’s say bug #3391 is an easy fix. First, check out a working copy of the release branch.

svn co https://example.com/branches/RB-1.0 rb-1.0

Now, go in there, write tests that expose the bug and fix it. As I said, it’s an easy fix so you can commit all your changes at once to the release branch. When you do this, remember the new revision number.

Note: it’s always smart to include the number of the bug (in this case #3391) in your commit message. This will make sure other developers (and later on, yourself) know what bug was fixed here.

$ svn commit -m "Bug fixed #3391"
...
Committed revision 183.

As I said, remember the revision number: 183.

Now you have the bug fixed in the release branch for version 1.0, you are probably wondering if the bug exists in your current development code as well. Since branches are plain copies of your development code, the big probably is there too.

Don’t start editing your working copy of the trunk and start fixing the bug all over again. Let Subversion do the work.

Go into your trunk working copy and update it to the latest revision, this is revision 183. But, we only made changes to the release branch, and not to the trunk, so we need to merge those changes. We can do this by running the following command:

svn merge -r182:183 https://example.com/branches/RB-1.0 rb-1.0

You’ll now see the fix you applied in the release branch getting merged with your current development code. Great, isn’t it?

Before you leave to party, don’t forget to commit the changes to the trunk. Again, name the bug number you fixed here and also which revision you userd to merge it.

svn commit -m "Merge r183 (bug fixed #3391)"

You can apply this merge process with any other release branch you have if that’ necessary.

Difficult bug fixes

Another scenario is that the bug is not easy to fix. You need to rewrite a lot of tests to expose the bug in the first place, and even then you’re not sure how to go about fixing it.

If that’s the case, you are better of creating a seperate bug fix branch. This allows you to fix the bug in the background, possibly with the help of others, while current development can still go on. Also, you want to create snapshots of before (pre) and after (post) the bug fix. These tags will help you merge the fix into other branches later on.

First, create a bug fix branc. By convention Release Branches were called RB-1.0, Bug fix branches are called BUG-###. Of course, ### corresponds to the bug report number. In this case we create a branch named BUG-3391. We also need to create a snapshot of the code before we start fixing the bug. We call this tag PRE-3391.

svn copy -m "Create bugfix branch" https://example.com/branches/RB-1.0 https://example.com/branches/BUG-3391
svn copy -m "Tag start of bug fix" https://example.com/branches/BUG-3391 https://example.com/tags/PRE-3391

Now, you can checkout the bug fix branch and start work. You may call in the help of others if you need to. It’s okay to make multiple commits to this branch.

When you have reached the point were the bug is fixed, you’ll need to mark the end of it. We create a new tag named POST-3391 to mark the end of the bugfix:

svn copy -m "Tag end of bug fix" https://example.com/branches/BUG-3391 https://example.com/tags/POST-3391

Well. You’re done! Your bug has been fixed! But wait a minute. The fix is not present in the release branch yet! Here, again, we need to merge the bug fix into the release branch (and possibly into the trunk also).

First, update your current working of the release branch and merge the changes between the PRE-3391 and POST-3391 tags with the release branch. When done, run your tests to make sure everything works as expected and commit your changes.

svn update
svn merge https://example.com/tags/PRE-3391 https://example.com/tags/POST-3391
svn commit -m "Merged bug fix for bug #3391"

Final note

It’s clear that fixing bugs the easy way is the preferred method. Try to use this method as much as possible. But it’s okay to use the difficult way to solve the more difficult bugs.

Putting the bug report number into Subversion comments is a good way to keep track of what code changes have been made based on what bug report. Most bug tracking tools can make use of this and link bug reports to the appropriate revisitons and visa-versa. If your bug tracking tool doesn’t have this ability, it’s a good idea to comment the appropriate revision number or tags on the bug so that others know where to look for the fix.

Happy bug fixing!

WordPress: Author comment highlighting

Wed, 22 Nov 2006 00:00:00 +0000

I’ve seen it lots of times before, but I just added it to Ariejan.net (and the next release of the iAriejan theme). Sometimes there are lots of comments and it’s nice for visitors to see what the official reaction of the blog author is.

Since I haven’t really looked into a plugin or anything, this is just a very simple theme hack.

You can apply it to your current theme with almost no effort at all.

Open up your comments.php file in your themes directory. And look for the following code:

<li class="<?php echo $oddcomment; ?>" id="comment-< ?php comment_ID() ?>" id="comment-< ?php comment_ID() ?>">

and replace it with

<li class="<?php if ( $comment->comment_author_email == get_the_author_email() ) echo 'authorcomment'; else echo $oddcomment; ?>" id="comment-< ?php comment_ID() ?>" id="comment-< ?php comment_ID() ?>">

What this will do is match the e-mail address of the poster with the e-mail address of the post author. This is in some way spoofable, as users may be able to post a comment with your e-mail address on it.

If you posted the comment an extra CSS class named ‘authorcomment’ is added. So add the following to your style.css file. (You may change this to suit your own taste of course):

.authorcomment {
 background-color: #363636;
 border: 1px solid #969696;
}

To prevent this you can add your e-mail address (the one you use with your WP account) to Options -> Discussion -> Comment Moderation. This will keep any post that contains your email address back for moderation by you. This is the only fool-proof method I know right now to keep people from spoofing. There might be some other hacks for this, but I haven’t had time to think about that yet.

CSE-Tool 1.1.0 Released

Tue, 21 Nov 2006 00:00:00 +0000

Hot of the press! CSE-Tool 1.1.0 has been released just a few minutes ago! Grab the code now or check a live demonstration!

Please report any feature, support or bug requests back at the project’s SourceForge page.

Also, if you like CSE-Tool (and want your feature added sooner rather than later) please consider making a small donation to the project to encourage further development and support.

Donations are handled by SourceForge.

Please consider making a donation to the project. Thank you!

To see how I manage a the creation of a release through SubVersion, check this post over here.

SVN: How to release software properly

Tue, 21 Nov 2006 00:00:00 +0000

Many projects use SubVersion nowadays to store their project code. I do this also at work, and for my personal projects like CSE-Tool.

The question, however, is how to release your current code properly to the public. You probably don’t want your users to check out your current development code. Either you want them to check out a certain version (release) or you want to present them with a download archive containing the code.

I’m going to show you how to release a simple PHP application from SubVersion as an archive file to my users.

The base layout of my svn repository is like this. I have directory named ’trunk’ that always contains the most recent version of the software. This is the development branch, so to say. I also have a ‘branches’ and a ’tags’ directory. If you don’t have these, you’ll need to create them now:

$ svn mkdir -m "Creating branches directory" svn://yourrepository/branches
Commited revision 123.
$ svn mkdir -m "Creating tags directory" svn://yourrepository/tags
Commited revision 124.

In this case my current development code, in the trunk of the svn repository is at revision 10. All files in the trunk are marked to be develoment quality. This means that I don’t display version numbers, but simply show ‘HEAD’ to indicate you’re working with a development quality product. Before I release this code to the public, I want to tweak a few things. Since this is not general development, I create a Release Branch. This release branch is basically a copy of the current code in the trunk. Changes to that branch are stored seperately from the development code, so I can easily tweak it to release quality.

Creating the release branch is really easy. Let’s say I want to release version 1.1.0 of CSE-Tool. I just copy the code in subversion. Note that I don’t download and store the code on my local machine. SubVersion is smart and duplicates the code on the server. This can save you valueable bandwidth, especially when you’re working on a large project.

Well, create the Release Branch which is named, by convention, RB-1.1.0.

$ svn copy -m "Creating release branch 1.1.0" https://svn.sourceforge.net/svnroot/cse-tool/trunk https://svn.sourceforge.net/svnroot/cse-tool/branches/RB-1.1.0
Committed revision 11.

You can see at http://cse-tool.svn.sourceforge.net/viewvc/cse-tool/branches/ that the new release branch (RB-1.1.0) was created as a directory containing a copy of the current develoment code.

I can now do two things. Either I checkout a seperate working copy of the release branch or I change my current working copy (which is using the trunk) to use the release branch. Checking out is easy, but I’ll show it first to you anyway. Next I’ll show you how to switch your repository.

Just check out your code as you’d normally do, but make sure you specify the release branch. I’ve also specified to store this code in a directory named cse-tool-1.1.0 so I don’t confuse it with the trunk code, which is stored in a directory named ‘cse-tool’.

$ svn co https://svn.sourceforge.net/svnroot/cse-tool/branches/RB-1.0.0 cse-tool-1.1.0

I could also swich my current working copy to the release branch. This may be useful if your project is very huge and you don’t want to download the whole thing again. Switching between a release branch and the trunk is usually more efficient because you only need to download the differences between the two.

$ svn switch https://svn.sourceforge.net/svnroot/cse-tool/branches/RB-1.0.0

Okay, now I can work on the release branch. Branding it with the right version number among other things. In your case it might be a good place to two SQL files to install or update a database you are using. You might want to update the changelog and other documentation.

When you commit changes, they will be applied to the release branch only, not not to the current development code.

When you’re done you may switch back to the current development code:

$ svn switch https://svn.sourceforge.net/svnroot/cse-tool/trunk

The code in the release branch is now ready to be shipped out. We want to mark this code as being Relese 1.1.0. This is called tagging. A tag is nothing more than a copy of the repository on a give moment. Technically, a branch and tag are the same. However, the conventions I use dictate that you don’t change the code in a tag because it represents a certain state of your code, in this case the state the code was in at the time of Release 1.1.0.

Now, to actually create a release tag, named REL-1.1.0, we use the same procedure as with the creation of the release branch. Just note the differences in the source and destination repositories.

$ svn copy -m "Tag release 1.1.0" https://svn.sourceforge.net/svnroot/cse-tool/branches/RB-1.1.0 https://svn.sourceforge.net/svnroot/cse-tool/tags/REL-1.1.0
Committed revision 13.

With the REL-1.1.0 tag we can create an archive that we can distribute to our users. Because we don’t want to include svn metadata in our release we can’t use checkout for this. SubVersion allows us to export our code, which is basically a check out, but without all the svn metadata. This is ideal to ship to our customers.

$ svn export https://svn.sourceforge.net/svnroot/cse-tool/tags/REL-1.1.0 cse-tool-1.1.0

Next I can tar up the cse-tool-1.1.0 directory and put the files on SourceForge. (Download them here :))

Announcing CSE-Tool: Deploy you Google CSE with ease

Fri, 17 Nov 2006 00:00:00 +0000

This is just a short note to let you know that CSE-Tool 1.0 has been released.

The CSE-Tool allows you to easily (just copy-paste a few items presented to you by Google) deploy your Custom Search Engine. All you need is a CSE (of course) and optionally a Google Analytics account to track who’s coming by.

System requirements are pretty low. You’ll only need PHP 4.3.x or better. Check out the SourceForge Project Page or jump directly to the downloads section.

Ubuntu 6.10 Live DVD on the Apple MacBook

Wed, 15 Nov 2006 00:00:00 +0000

Since I teach various Linux courses at Fontys Centrum IT, I want to run a live CD or DVD on my MacBook. First off, here are my specs:

Apple MacBook (White)
Intel Core Duo 2.0Ghz
1.0Gb RAM
13.3" 1280x800 TFT
60 Gb Internal Harddisk
300 Gb External FireWire harddisk

As my Live medium I chose the Ubuntu Linux 6.10 Live DVD. This DVD has several nice options (install server, for example) that I like. It also can boot up in a live desktop environment. ~ As a matter of fact, everything I need works out of the box. Wireless works fine, my mouse (USB Logitech) works, the FireWire harddisk (which has a FAT32 filesystem, so I can hook it to my girl friend’s Windows PC) works perfectly.

There are only two things that didn’t work.

Xvid codecs are missing, so I can’t watch my Prison Break episodes from the Live environment, but that’s perfectly fine. That’s not what I need the live environment for.
The screen resolution can only be set to a maximum of 1024x768. As I mentioned before, the TFT has a native resolution of 1280x800. Because of the scaling, my screen doesn’t look as sharp and sexy as it should.

In order to solve this problem I had to take a few, rather easy, steps.

To get started, let us assign a password to the default ubuntu user.

sudo passwd ubuntu

Now enter something that you’ll remember easily, twice.

In order to get Ubuntu to recognize the native screen resolution automatically, we’ll need a special package: 915resolution. Since the MacBook has an Intel 945 graphics chipset, we can use this tool. But, this tool is not in the default package respository. It’s in the Universe.

So, we now need to change /etc/apt/sources.list and add the universe repository. This is rather easy, because these repositories already exist, but are commented out. Just open up /etc/apt/sources.list and uncomment the two universe lines. Make sure your sources.list looks like this:

deb http://archive.ubuntu.com/ubuntu edgy main restricted
deb-src http://archive.ubuntu.com/ubuntu edgy main restricted

## Uncomment the following two lines to add software from the 'universe'
## repository.
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://archive.ubuntu.com/ubuntu edgy universe
deb-src http://archive.ubuntu.com/ubuntu edgy universe

deb http://security.ubuntu.com/ubuntu edgy-security main restricted
deb-src http://security.ubuntu.com/ubuntu edgy-security main restricted

Now, we can update our system and install the 915resolution package.

apt-get update
apt-get install 915resolution

You’ll notice that 915resolution spits out a lot of information on your chipset and native resolution. You should check here that the 1280x800 resolution has been detected.

Now you’re all set. Well, you’ll need to restart the X Server in order for it to recognize the newly discovered resolution. I basically pressed CTRL-ALT-BACKSPACE to kick the X Server. I’m presented with a login screen. Here I login as ‘ubuntu’ with the password I specified earlier.

If everything went fine, you’ll now get a clear and sharp display of your Ubuntu life desktop.

I also turned on sub-pixel smoothing in System -> Preferences -> Fonts to get some sexier and smoother looking fonts.

That’s all. I can now boot-up the Ubuntu Live DVD and get the native resolution for my MacBook.

For those who want to keep the native resolution on their installed Ubuntu, just install the 915resolution package mentioned above and you’re set. Easy as that.

CUPS: 426 - Upgrade Required

Mon, 13 Nov 2006 00:00:00 +0000

As I was installing my printer on my Ubuntu 6.06 Dapper LTS server with CUPS I noticed the following error:

426 Upgrade Required

After some research I came to the conclusion that CUPS, by default, tries to use SSL whenever possible. So, with this 426 error, you are redirected to the SSL domain. Chances are, you haven’t configured SSL properly, if at all.

In my case, I didn’t want to configure SSL. To get rid of this problem, the key lies in editing your configuration files ( /etc/cups/cupsd.conf ) and adding the following line:

DefaultEncryption Never

There are several options, Never, IfRequired and Required. By setting this to Never, SSL will never be enforced. Just restart your CUPS server with

/etc/init.d/cupsys restart

and you’re good to go.

WordpressMu: Don’t allow new blogs

Tue, 31 Oct 2006 00:00:00 +0000

If you’re using WordpressMu, the blog hosting tool used on Wordpress.com, you may want to disable the creation of blogs by your visitors.

Whatever your reasons for this are, I wanted to prevent this, because I (and my team of editors) want to maintain several blogs on different topics. Users are free to register and post comments, but creating new blogs is reserved for the administrator.

So, how do you implement this in WordpressMu? There is no checkbox (yet) that disables this feature. So, I had to hack the WordpressMu code a bit.

First, open up wp-signup.php. If you access a blog that does not exist, you’ll be redirected to the signup page and be presented a signup form for that particular blog.

Open up wp-singup.php en just above the get_header(); call, place the following code:

if (!is_user_logged_in() || $user_identity != 'admin') {
 header("Location: http://example.com/gofishatthispage/");
 exit();
}

What this does is make sure that only a logged in user named ‘admin’ is allowed to proceed to the blog creation form. Others will be redirected to a location of your choice. A good idea is to send people to a page that explains why they can’t’ create a blog or what they have to do to get an administrator to create one for them.

Adsense Resource Inventory

Mon, 30 Oct 2006 00:00:00 +0000

In addition to my previous post I have gathered some popular resources from around the web that talk about making money off Google Adsense. I hope this sums up all you need to start making money yourself.

Please, let me know if I missed something.

Things to know about making money with Google Adsense

There are three things you need in order to make money off your site or blog.

Visitors
Content
Ads

This post is about the last part, Ads. I’ll talk about getting visitors and creating content in a later post, so stay tuned!

So, how to get those ads setup right so you generate the most revenue possible. It’s not as easy as it sounds and mostly it’s a trial-and-error process. But, there are some general guidelines that will help you get started.

I highly recommend you do NOT buy any software that claims to track Google Ads or create content or sites for you! If you need to track your ads, use Google Analytics, it’s free. Content and site generators give you the same site and same content as others. Google knows all and they don’t like copy-cats whos only goal is to make money.

Google Adsense Tips and Tools Collection
This should be in your bookmarks! Great tips, tools and free software that will get you started immediately!
Adsense Calculator
This nifty device tries to predict how much money you can make of your site according to your current statistics. Let me know if this actually works or not!
How to boost your Adsense revenue
Great article that introduces new users to Adsense and gives tips on how to get started. See my remark about buying software, though.
15 Common mistakes
15 Common mistakes made by Adsense publishers. Read this, and don't make the same mistake.
SEOBook
Great site. It contains many links and resources to Adsense related articles.
More Adsense Tips and Secrets
Another site with great tips and Adsense secrets.
Adsense Wisdom
10 Easy steps in making your Adsense experience successful.
Explorer Destroyer
This site has some code that will kick Internet Explorer users to use Firefox instead. Oh, you use Google referrals for that.
Inside Adsense
The official AdSense blog from Google.

Please post a comment if I have missed anything important.

Cheat sheets? Look here!

Mon, 30 Oct 2006 00:00:00 +0000

Getting confused of all the tools you use on a daily basis? Don’t remember all the exact method names and possibilities of SubVersion, Apache, HTML, CSS, MySQL, Ruby on Rails, CVS, AJAX, JavaScript, FireFox, Google, etc. etc.? There is a solution! Cheat sheets make your daily work easier by providing you all you need to know on a single paper!

Here’s a very comprehensive list of cheat sheets.

Cheat sheets are simple put a collection of all important (if not just all) functions and methods you can use for a given product, framework or language. They’re great to have on your desk if you just can’t quite remember what function to use. Check ’em out, print ’em out and start loving ’em!

Confused about CSS Columns?

Mon, 30 Oct 2006 00:00:00 +0000

I’m mainly confused on how to make different layouts with pure CSS. Back in the time when tables were okay, it was rather easy. But with CSS it’s gotten rather tricky.

Dynamic Drive has some great examples (with CSS code!) of different kinds of layouts. It’s great to use as a starting point for your design!

How does your site look on …?

Mon, 30 Oct 2006 00:00:00 +0000

When designing a web site you always check how the site looks in different browsers, even maybe at different resolutions. But how many browsers do you really use? Firefox, Safari when on Mac, Internet Explorer when on Windows and that’s mostly it.

Browser Shots allows you to select different browsers and set several options like colour depth, screen resolution, java etc. and make screenshots of your site (by entering the URL).

I don’t need to tell you this is a great resource. It’s great to check if your design works and it’s even better to get a screenshot generated from your new designs. Only problem is that you’ll have to put them online first.

Check out some screenshots here.

Do your ads pay your (blogging) bills?

Sun, 29 Oct 2006 00:00:00 +0000

I’ve read many articles on earning money with Google’s AdSense. Some guru’s claim to recieve five figure checks from Google every month.

The trick with these people is that they have a lot of content. More content, means more visitors, which means more clicks and thus more money. The content is written to attract certain keywords which are known earn a lot of money. Of course, professionals don’t just have one site, they have several. In order to cope with all this content they have an almost full-time job. That’s not bad if you earn a five figure amount every month. ~ But, how about us, regular bloggers who just want to share our bit of knowledge with the rest of the world. Most of us have Google Ads on our site, including me. My site has been refurbished quite a few times in the past few months and the content I had is gone. So, I’m not really hot in this business.

Most bloggers post a few items every few days. Some of us are really cool and post several items a day. What I want to know is: can Google Ads pay your (blogging) bills?

I mean, is there really anyone, except the kick-ass professional bloggers, who can make a bit of money of their blog? And if so, do they have any tips they’d like to share with the world?

Please let me know or post any relevant articles here. I’m looking forward to your reponse!

Ruby On Rails for PHP: CakePHP

Mon, 23 Oct 2006 00:00:00 +0000

The framework has been around for some time, but I found out about it a few days ago: CakePHP.

I’ve been using Ruby on Rails for quite some time now. It’s a very cool framework and it’s fun to work with. However, there’s one big problem I have with it: I can’t host it anywhere!

Well, there are several hosting companies that offer Rails hosting, but the price is pretty steep when compared to my current hosting plan. I could get a VPS or dedicated server, but that just be overkill for the applications I want to run.

So, I started thinking. What can be done in Ruby could be done in PHP as well. Maybe not in such an elegant and intuitive way, but it should be possible. And I was right. ~ After some googling I came across CakePHP. CakePHP aims to implement the MVC (Model-View-Controller) structure in PHP much in the same way Rails does it for Ruby.

After giving it a try I found that CakePHP is a pretty good alternative to Rails. Most importantly, it runs with both PHP4 and PHP5 and it can handle MySQL, SQLite and other standard databases.

If you’re an agile developer but you don’t want to nail yourself down on hosting with Ruby or if you really want to use PHP, CakePHP is a great framework to use!

http://www.cakephp.org

Having fun with SPAM!

Fri, 13 Oct 2006 00:00:00 +0000

I was just wading through my SPAM at Gmail to see if it flagged anything important. Then I came across this message:

Is your website www.gmail.com offline, or why can’t I find it on Yahoo?

Your website (www.gmail.com) on top positions on Google, Yahoo and MSN search! We will get your website (www.gmail.com) to the top positions on all major search engines.
Use our great value offer: We will submit your website (www.gmail.com) to 890 Search Engines, including Google, Yahoo and MSN.

Just keep an eye out for funny SPAM mails. Feel free to let me know if you ever read a funny or really stupid SPAM message.

Migrate SQLite3 to MySQL easily

Fri, 13 Oct 2006 00:00:00 +0000

I’ve been using a simple Rails application locally with a SQlite 3 database for some time. Now I want to move to another host and use MySQL instead. But guess what? You can’t just migrate your data!

Here are some easy steps on how to migrate your data to MySQL. First of all you need to dump your SQLite3 database. This includes transaction statements and create commands. That’s fine. Since we also migrate the schema information, our RoR app will not know any difference after we change config/database.yml.

The biggest probem I encoutered was that the SQLite3 dump places table names in double quotes, which MySQL won’t accept.

First, make sure you create your MySQL database and create a user to access that database. Then run the following command. (It’s a long one, so where you see a , just continue on the same line.)

sqlite3 db/production.sqlite3 .dump | \
grep -v "BEGIN TRANSACTION;" | \
grep -v "COMMIT;" | \
perl -pe 's/INSERT INTO \"(.*)\" VALUES/INSERT INTO `\1` VALUES/' | \
mysql -u YOURUSERNAME -p YOURPROJECT_production[/source]

This will take the SQLite 3 dump, remote the transaction commands. Next I use perl to replace all INSERT commands containing double quotes with something MySQL will understand.

That’s it. You MySQL database will be populated with your data.

Don’t forget to change your config/database.yml file after this!

Note. You may also migrate your MySQL database using Rails. If you do this I recommend that you dump the SQLite3 database to a file first before you commit it directly to MySQL. You’ll have to remove the CREATE TABLE statements as well as any reference to the schema_info table.

Tagging in ajax_scaffold

Fri, 13 Oct 2006 00:00:00 +0000

I’ve been using the Ajax Scaffold for quite some time now. It’s a great piece of software by Mr. Richard White for Ruby on Rails. It seems that the plugin version of AS is getting quite a bit more attention than the generator. I started out with the generator but quickly reverted to the plugin since it’s way more flexible and easier to use.

Since I wanted to create a quick app to inventory my CD/DVD collection (which is now in a very sexy alu DJ case) I used Ajax Scaffold to get me started. In the spirit of Web 2.0 I wanted to add tags to every CD so it would be easier to find certain kinds of disks later on. So, I added acts_as_taggable.

Acts_as_taggable basically allows you to tag any model in your app. So, I made my Disk model taggable. Great. Now I could do this:

d = Disk.new(:number => 1, :name => "Mac OS X 10.4.6 Install DVD 1")
d.tag_with("macoxs apple macbook install")
d.save

The real problem was, how to get this functionality easily integerated in Ajax Scaffold?

First of all I had to show a column in AS that included the tags attached to a given disk. I specify all rows manually in the Manager controller. Manager is scaffolded using AS. Here’s what my Manager controller looks like:

class ManagerController < ApplicationController
 ajax_scaffold :disk

 @@scaffold_columns = [
 AjaxScaffold::ScaffoldColumn.new(Disk, { :name => "number" }),
 AjaxScaffold::ScaffoldColumn.new(Disk, { :name => "name" }),
 AjaxScaffold::ScaffoldColumn.new(Disk, { :name => "tags",
 :eval => "row.tag_list", :sort => "tag_list"}),
 ]
end

This will show three columns, including a column named ’tags’. Every model that acts_as_taggable has some extra methods. tag_list is a single string containing all tags seperated by spaces. So, the tags column shows the tag_list for that row. With :sort I specify that AS can just sort keywords alphabetically.

Great! I now can see tags on disks! But, we also need to add those tags and that’s where it got tricky.

Adding tags

Adding tags is not done by assignment but by calling a method with your tags, as shown before: tag_with (string). I could create a custom create and update method for the Disks, but there’s a prettier solution available.

tag_list returns a string with the current tags. How about using that same name to assign tags? It’s rather easy. Here’s my Disk model:

class Disk < ActiveRecord::Base
 acts_as_taggable

 validates_presence_of :name, :number

 def tag_list=(new_tags)
 tag_with new_tags
 end
end

Now we can assign tags to tag_list as well as read the tags out. Now the only step is to add a special textfield to the form partial for AS.

<label class="required">Tags</label>
<%= text_field 'disk', 'tag_list' %>

Now when a new disk is created or when one is updated, the tag_list will automagically be updated in a correct fashion.

Enjoy!

Easily create a FavIcon online!

Tue, 10 Oct 2006 00:00:00 +0000

You know those little icons you see in your browsers address bar? Yes, the ones in you bookmarks! Well, you want such an icon for your site?

These icons are called FavIcons, short for Favourites Icon. Most modern browsers make use of them.

To create such an icon you’ll need some image first. Normally the process would be rather difficult involving several graphics manipulation programs and conversions. Don’t fall for that! Use the followin FavIcon from Pics site!

Just upload a gif, jpeg or png image (with transparency if you like) and get a FavIcon for free! There are also some other goodies included!

Of course, there’s information available on how to upload your FavIcon and how to let your browser know about it.

Generate a SQlite-based Rails app

Tue, 10 Oct 2006 00:00:00 +0000

When you create a Rails application a database.yml files is included with some default configuration for your database. Unfortunately these are defaults for MySQL. If you want to use another database, like SQlite, you’d have to rewrite the entire configuration file. And that’s not what you want!

Luckily, rails is very adapative and we can make it do all the work for us.

When you create a rails application, firstr check out the help message from rails itself.

$ rails –help

When you look closely you’ll see that there’s an option available called ‘–database’. It has the following message attached to it:

Preconfigure for selected database (options: mysql/oracle/postgresql/sqlite2/sqlite3).

Basically, you can specify, when create your rails project, what database type you want to use, and Rails’ll create a matching configuration file for you. When you leave this option out, it defaults to MySQL.

As you can see there are several databases supported, including SQLite. In order to create a new rails project with SQlite3 configured by default, run this:

$ rails –database=sqlite3 myproject

Check out config/database.yml and you’ll see that there are sensible defaults for you there.

Hi-Res Wallpapers for your MacBook!

Tue, 10 Oct 2006 00:00:00 +0000

For all of you who have a flashy Apple MacBook (or any other device with a 1280x800 resolution) here’s a nice site with lot’s of hi-res wallpapers. Of course, they’re all for free!

Oh, they have other sizes as well.

Go check out InterfaceLIFT.

Top 5 DVDs you must have

Tue, 10 Oct 2006 00:00:00 +0000

Everybody at this point in time at least owns one or more DVD - Digital Versatile Disk - Videos. Some people try to collect as many DVD’s as possible, others just buy the ones they like (or can afford).

I’m not sure how many DVD’s I have at this time, but there are quite a few DVD’s that you just have to have in your collection.

If you feel wronged because your favourite movie is not listed, feel free to add a comment stating your favourites. (Oh, let me know why it’s such a great movie too!)

5. Gladiator

Gladiator, by Ridley Scott, is tells the story off a Roman general who became a slave and then a gladiator. It sounds pretty boring when you put it like that, but Scott really makes it into a very gripping movie.

Gladiator is more than a few man in shorts fighting in a big bowl of sand. You should really check this one out and add it to your collection.

4. The Lord of the Rings Trilogy

Well, The Lord of the Rings is not really a trilogy since it is one big story. Still, this movie is great.

LOTR, by Peter Jackson, attracts old and young alike. It features humour, heroism, love, massive battles between good and evil and everything else needed to make a successful movie.

Many complain that the Fellowship (part 1) does not have much action in it and that Return of the king (part 2) is just one big fat ending. Well, that’s correct. In my opinion you should watch the three movies in sequence all at once. That will give you the full sensation the way Jackson intended it.

Because this movie appeals to a very broad audience, it’s placed at position 4.

3. Pulp Ficton

The combining of several story lines into one movie with a lot of violence by Quentin Tarantino. The result is Pulp Fiction.

Every story line has it’s own characters and Tarantino was able to get a lot of big names attached to this film including Bruce Willis, John Travolta, Samuel L. Jackson and Uma Thurman.

This movie is not just entertainment, it’s art. If you don’t have it, you should probably get is a.s.a.p.

2. Star Wars (all six of 'em)

Star Wars is the work of George Lucas. George has done many innovative things during the creation of his Star Wars movies. Not only was he the first to introduce (and claim the rights to) merchandising, he has also brought special effects to a new level. Of course, I don’t have to remind you that George Lucas is also the genius behind the THX sound system.

If I have to tell you what Star Wars is about, you should not be allowed to have a DVD collection at all.

1. The Godfather Trilogy

I think everybody knows the Godfather. Directed by Francis Ford Coppola, this epic trilogy gives us an inside look in the Corleone family.

Part one is really the best movie of all three. It’s based on the original novel by Mario Puzo. However, parts two and three really complement the whole story and that’s what makes this such a great trilogy.

There are some very nice Trilogy DVD Boxes available. If you say you like movies, you cannot be taken seriously unless you have this trilogy in your collection.

Welcome to Ariejan.net

Mon, 09 Oct 2006 00:00:00 +0000

Well, it has happened to me. Although I make regular backups of my site, this time I was screwed!

Since I’m very interested in Ruby on Rails, I gave a RoR weblog, TypoSphere, a try. I exported my entire Wordpress blog to typo and I was a happy man. Until disaster struck…

After a while I noticed that typo’s search functionality was not working. I recieved a MySQL error messages that the table containing my artilces did not exists! To make a long story short, all database files containing data were gone. Luckily I make regular backups every day automatically.

Well, I make regular backups and keep the seven most recent backup files at hand. But, you guessed it, apparently the disaster happened more than a week ago. So, all my backup files contained the corrupted database files. Bugger!

So, after some flirting with Dreamweaver, Drupal and Joomla, I’m back at Wordpress which really is the best, easy to use and customize publishing tool I know.

Since all my previous tutorials are gone, I’m going to start with a clean slate here. Please comment on my articles if you like and feel free to link anything you find here.

For now, I’ll have to re-customize my site and then I’m ready to start posting useful things here.

devroom.io

Why Asahi Linux is not for me

The good

The bad

The ugly

Conclusion

MacbookPro M1 with Asahi Linux: Fractional Display Scaling in GNOME

What is scaling, anyway?

Enabling fractional scaling in GNOME

Fix for Chromium

Saving for automation

MacbookPro M1 with Asahi Linux: Fixing the touchpad

Accidental tapping

Fixing trackpad scroll speed

Peel before use: mistakes were made

Import Google Photos Takeout into Immich

Prepare your takeout

Getting immich-go

Getting your Immich API key

Dry-run

Importing for realz

Result?

How to extract multiple .tgz Google takeout archives

Immich

Google Takeout

cat | tar

pv | tar

Importing into Immich

ZFS: Upgrade single disk to mirror

Prepare the new disk

Upgrading the ZFS pool

Arch Linux: Improve boot time performance

Slllooooowwww…

Fixed it!

Was it worth it?

Troubleshooting zfs online not working

Prepare new harddisk for ZFS/NAS

Current state of affairs

It’s all about trust

S.M.A.R.T.

Badblocks

What’s next?

Volkswagen Golf Cabrio - Update

Big jobs

Looks

What’s next?

Base2Tone Cave Dark for Slack

Preview

Installation

10GB Ethernet with Proxmox and Ryzentosh 3700X with macos Catalina

Proxmox setup

Hackintosh / Open Core setup

Testing speed

Samba

Accessing 10,000mbps Samba shares

That’s all

The Big DIY NAS Update

A brief history

Update November 2020

Are you done now?! What more could you want?!

Norco 4U Rack with hot swap bays

Motherboard + CPU + Memory

Proxmox

10GB Ethernet

Repairing a Xiaomi Roborock S5 Max

Why I will not repair your amplifier

Test Flutter on Drone CI

A Drone Hugo plugin that works

Hugo

Drone + Hugo

Can I do better?

Using ariejan/drone-hugo

The git submodule cheat sheet

Add a submodule

Cloning a project with submodules

Update your submodule

Track a specific branch of version

Remove a submodule

Bonus: see submodule status in git status

Building a DIY Home Server with FreeNAS

Getting `immich-go`

Bonus: see submodule status in `git status`