ls /etc/ | more (Dave Reisner)

Don't Mock Me

Dave Reisner — 2012-12-25T00:00:00+00:00

Due to some (favorable) changes in my day job, I’ve been writing less code outside of work. Without going too much into the reasons, I’ve become accustomed to having tests available for any code I write. Tests give you a nice baseline for making changes and refactoring. It’s easy to know that you’ve broken something right away. Knowing that you need to write tests also makes you structure your code in such a way that it’s reasonable to actually write tests for it. Conversely, making changes to longstanding code that doesn’t have tests becomes a chore, especially for more invasive changes. Suddenly, what used to be a weekend hack because I loved getting into the guts of a program starts to look like a real pain in the ass.

It’s not that I have an aversion to writing tests, but finding a framework in the open source world that doesn’t suck and embracing it can be difficult. This is especially true given the languages I tend to deal with outside of work, namely shell (Bash, in particular). I’m aware of projects like “shunit”:http://sourceforge.net/projects/shunit/ and some of the “derivatives”:http://code.google.com/p/shunit2/, but they don’t solve some of the real problems that one generally encounters with testing shell.

Let’s look at what mkinitcpio does. When you boil it down, it builds CPIO archives. That’s really it. You’re talking about a lot of raw calls to the filesystem to read/write modules, binaries, and configuration files. Some of the code in its current state is relatively simple to test. Some of it, with a little bit of massaging, would be a lot easier to test. Some of the functions – namely, some of the important ones like add_module, are currently not at all possible to test due to calls to external binaries like modinfo or modprobe. So what’s the solution?

In “real” languages, you have libraries, such as “gmock”:http://code.google.com/p/googlemock/, to mock side effects. For shell, I’ve started writing something I’m calling “Apron”:https://github.com/falconindy/apron/.

I’ll say it up front: Apron is a hack. But, I’ve found that everything “good” in shell feels that way. Apron leverages two notable features of Bash:

The lookup order for command execution favors functions over external binaries.
The existance of the @command_not_found_handle@ function.

With these 2 things in mind, you can intercept any call which isn’t a function, but which also depends on a PATH lookup. By setting @PATH@ to a bogus value (due to a bug going back at least to bash 3.2 you can’t simply unset it), you force all these external calls to trigger your @command_not_found_handle@ function. Now, everything you call which isn’t a function can do your bidding. You’re in a relatively safer environment than you used to be, but of course you can still call a binary by its absolute path and avoid the PATH lookup. I’ll ignore the fact that Bash will let you define a function called /bin/rm, but you can certainly leverage that, if you know to expect the call. This might tie in to writing your code in a way that it facilitates being tested.

This still isn’t very interesting, since causing all your external commands to do nothing will probably also cause your actual code to fail. So, simply define functions with the same name as the binaries you call. Now you can set the behavior for those functions which really matter and simply leave Apron to automatically stub out the rest.

With a little extra bookkeeping, Apron tracks what functions you’ve defined as mocks and lets you suspend and restore the mocked environment. I’ve written a simple “example”:https://github.com/falconindy/apron/blob/master/mock-test as a test of the framework in action.

It’s likely going to take a fair bit of effort, but I’m already quite pleased with how Apron is working (even at under 100 lines of code), and I’m fairly excited about making mkinitcpio more testable on a unit level. Perhaps, in the process, someone else finds this useful too.

Update: Apron supports expectations now, too. The latest “README”:https://github.com/falconindy/apron/blob/master/README.md documents how to use it, and there’s of course also an “example”:https://github.com/falconindy/apron/blob/master/expect-test to go along with it.

Back to Basics With X and systemd

Dave Reisner — 2012-08-25T00:00:00+00:00

2012/11/02 Update: Arch Linux now provides this by default. /etc/X11/xinit/xserverrc will start your X session on XDG_VTNR, if the environment variable is set. Therefore, if you’re using Arch, stop reading here and update to at least xorg-xinit 1.3.2-3.

This is for everyone who says that systemd doesn’t work well with startx as far as maintaining an authenticated session. There’s no “ck-session-launch” equivalent, sure, but it isn’t needed. When you login on at a getty, you already have an authenticated session. However, if you run startx without parameters, you’ll find that a new terminal is allocated – one where you aren’t authenticated. According to upstream, it’s just a matter of convincing xinit (the underlying mechanism in startx) to not allocate a new TTY. How do we do that? By merely passing the VT we’re current on to xinit.


xinit -- vt01

And this just works – you’ll keep your authenticated session, and X is started up according to the contents of $HOME/.xinitrc.

…But I’m lazy. What if I’m on tty2? Passing vt01 is going to break this – we should pass vt02 in this case. Add some shell voodoo:

#!/bin/bash

TTY=${TTY:-$(tty)}
TTY=${TTY#/dev/}

if [[ $TTY != tty* ]]; then
  printf '==> ERROR: invalid TTY\n' >&2
  exit 1
fi

printf -v vt 'vt%02d' "${TTY#tty}"

xinit -- "$vt" "$@"

Now it’s even easier. All we have to run is our little helper script.

…But that’s not quite enough. startx itself is a wrapper around xinit and performs some modicum of useful setup. So how can convince startx to do this for us?

xinit is a really stupid program. It’s a glorified compiled shell script with some extra signal handling. It launches 2 programs – a server (generally X), and a client (or many clients, as described by your .xinitrc). Just as .xinitrc exists to describe the client side setup, there’s another file, .xserverrc, which a user can drop in $HOME to describe the server startup behavior. You’ll find the default file in /etc/X11/xinit/xserverrc. It’s a one liner which just runs /usr/bin/X with a couple of flags. You can probably already guess where this is going. Add $HOME/.xserverrc as the following:

#!/bin/bash

TTY=${TTY:-$(tty)}
TTY=${TTY#/dev/}

if [[ $TTY != tty* ]]; then
  printf '==> ERROR: invalid TTY\n' >&2
  exit 1
fi

printf -v vt 'vt%02d' "${TTY#tty}"

exec /usr/bin/X -nolisten tcp "$vt" "$@"

Now when you run startx, it’ll start the server on the appropriate VT to keep your authenticated session, and .xinitrc continues to describe your client side behavior.

Ten Years of Arch Linux

Dave Reisner — 2012-03-11T00:00:00+00:00

“Arch Linux turns 10 today”:https://www.archlinux.org/news/arch-linux-01-homer-released/. It’s pretty spunky for a pre-teen! Inspired by some fellow developers, I figured I’d post my (alebit short) involvement with Arch over the past 2 1/2 years:

2009-10-22: Wiped Ubuntu, installed Arch and joined the forums
2010-03-21: First release of cower
2010-04-25: Applied for junior dev position, and was rightfully turned down!
2010-07-11: “First commit”:http://projects.archlinux.org/mkinitcpio.git/commit/?id=9ef825ced to Arch’s codebase (Thomas later reverts it!)
2010-07-24: “First commit”:http://projects.archlinux.org/pacman.git/commit/?id=7f5c486666c to pacman
2010-11-04: “My first breakage”:http://projects.archlinux.org/pacman.git/commit/?id=4fb3cfc48f6 – later “fixed”:http://projects.archlinux.org/pacman.git/commit/?id=40a6c5c5e
2010-11-30: “Applied to be a TU”:http://mailman.archlinux.org/pipermail/aur-general/2010-November/012054.html – thanks Ionuț for sponsoring me
2010-12-11: Accepted as TU! (Yes: 29, No: 0, Abstain: 1)
2011-03-01: Applied for initscripts maintainership (I passed this role to Tom Gundersen, who’s done an awesome job)
2011-03-19: libfetch is removed from pacman, and my curl branch is merged
2011-03-20: systemd joins [community]
2011-06-13: Dan McGee (aka toofishes) sponsors me to be a developer (thanks, Dan!)
2011-10-04: My first release of mkinitcpio (0.7.3) as maintainer
2011-10-12: pacman 4.0.0 is released into the wild with curl
2011-12-20: kmod enters the repos
2012-02-12: systemd moves to [extra]

I consider several of the subsequent mkinitcpio and pacman releases fairly memorable as well, but I’ll leave those off the list. I can honestly say that working with Arch has been an amazing experience. Ignoring the 1s and 0s, the people backing Arch Linux are an amazingly intelligent group of individuals, all with differing strengths working towards the same goal.

Having only picked up Linux as my daily operating system in the summer of 2009, I can honeslty say that without the flexibility and transparency that Arch offers, there’s no way I could have learned as much as I have in the past few years, elsewhere. Thanks to what I’ve learned I’ve become involved, to varying degrees, with other upstream projects. I’ve made numerous contributions to systemd, util-linux, and kmod.

I’m truly honored to be a user and developer of this awesome distro. Here’s to 10 more years!

Optimizing Bootup With mkinitcpio

Dave Reisner — 2012-02-04T00:00:00+00:00

Recently, I’ve seen a bunch of questions to the tune of “how do I cut back on the number of modules in my initramfs?” To be brutally honest, this is sort of an annoying question. In general, the type of person who asks this question doesn’t understand what the autodetect hook is doing and fails to realize that it’s doing a 90% effective job of exactly this. lsinitcpio would have happily shown you exactly what’s on the image. In addition, this sort of mindless pruning doesn’t really cut back on boot time (stop using xz compression!) and only serves to remove functionality from your initramfs. In case you’re still bent on doing this the manual way, I’ll outline what’s involved.

h2. What’s it take to boot?

Perhaps surprisingly little. In the simplest case, mounting your root partition requires drivers for:

storage bus (PATA, SATA, SCSI, etc)
block device
filesystem

For purposes of simplicity, I’ll ignore things like mdadm, lvm, and crypto stacks. With only 3 modules (plus dependencies), the kernel is able to discover your storage bus, create a block device for the drives attached, and understand the underlying filesystem. It’s really that simple. In my case, if I didn’t have these boiled into my kernel, it’d be simply: ahci, sd_mod, and ext4.

h2. How do I know what I need?

There’s a variety of tools at your disposal to figure this out. For starters, ‘mkinitcpio -M’ will scan your PCI bus and probe your root filesystem, returning a tidy list of modules, sans dependencies. You can be almost sure that you’ll find your answers in this list. This isn’t very useful though. You might not recognize these modules by name. So, take a look at something like ‘lspci -vk’. In particular, search for something like SATA or PATA in the output.


00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 66
        I/O ports at 9c00 [size=8]
        I/O ports at 9880 [size=4]
        I/O ports at 9800 [size=8]
        I/O ports at 9480 [size=4]
        I/O ports at 9400 [size=32]
        Memory at f7dfc000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ahci

This looks promising! It even tells you ‘ahci’ is in use. Add that to the pile. You might have a motherboard with both PATA and SATA connectors. Make sure you grab the driver for both. We can confirm which one is in use soon enough.

Figuring out your needed block device driver is actually not guaranteed to be straightforward. We can employ the use of udevadm to knowledgeably walk around sysfs, and in particular, walk up the chain from our root device to the PCI backplane. Assuming /dev/sda1 is your root device, check out the following:


$ udevadm info --attribute-walk -n /dev/sda1 | grep 'DRIVERS=="[^"]'
    DRIVERS=="sd"
    DRIVERS=="ahci"

This gives us confirmation that we chose wisely with ahci, and it points out ‘sd’. Based on what ‘mkinitcpio -M’ tells us, we can make an educated guess that this is the ‘sd_mod’ driver. Add that to the pile.

For your filesystem driver, it should literally be the name of the filesystem in use. There’s a few exceptions – reiser4 being one of them. Chances are if it’s not a 1:1 match, you’re already familiar with this particular gotcha.

h2. Alternative brute force method

There’s another way of doing this with a bit less guessing involved. Simply make sure that your initramfs image is properly setup to boot (make sure it has udev and drivers for your keyboard), and reboot, appending ‘break=postmount’ to your kernel commandline. When you arrive at the rootfs shell prompt, run ‘lsmod’. Ignoring anything like usb input drivers, that’s what you need to boot. You’ll, of course, also see module dependencies listed. Modules like ahci use libahci for functionality shared elsewhere, and ext4 uses mbcache and jbd2.

h2. Trim the fat

With your module list ready to go, it’s time to tear apart mkinitcpio.conf. Since you’re explicitly finding and loading modules, you’re going to be very light on hooks. Based on the above, you could put together the following config:


#
# /etc/mkinitcpio.conf
#

MODULES="ahci sd_mod ext4"
BINARIES="fsck fsck.ext4"
HOOKS="base"

And that’s it. We don’t need udev, since anything in the MODULES variable will be explicitly loaded. mkinitcpio is also kind enough to do dependency resolution for us. I still advise you to keep (or add?) fsck to your image, as checking your filesystem before it’s even mounted is greatly beneficial. I’ll leave it as an exercise to the reader to figure out additional modules for things like: usb keyboards or raid/lvm root devices.

For your maiden voyage, I highly recommend creating a separate image in case you’ve forgotten something.


# mkinitcpio -g /boot/initramfs-linux-tiny.img

Either add another entry to your bootloader, or feel free to modify it on the fly at bootup. If this image isn’t sufficient and init won’t mount your root, go back through sysfs another time and check your work. Pick through the output of ‘mkinitcpio -M’ and check over what the modules do with ‘modinfo’. The description may not be very useful, but the path within the module directory can be very telling.

That’s pretty much all there is to it. A little bit of understanding about your hardware and some familiarity with the common kernel modules can go a long way.

Google Bought Me! The First Two Days

Dave Reisner — 2011-09-09T00:00:00+00:00

For the past several months, I’ve been playing Operations Developer for “Zagat Survey”:http://www.zagat.com, taking care of the production website infrastructure, expanding our internal development environments, acting as liason to our hosting provider, and performing some light scripting tasks to glue it all together. If you’ve talked to me in the past 2 days, you already know this. If you don’t, perhaps you read the news. If you live in a hole, or maybe you just ignore what Google is doing, you might be interested to know that Google has “purchased”:http://googleblog.blogspot.com/2011/09/google-just-got-zagat-rated.html my current (former?) employer.

Rewind about 2 weeks. I’m tasked with ensuring that our homepage, and only our homepage can withstand five times our normal traffic. Why? Apparently, it’s the imminent release of the next New York restaurant guide. Ok, I’ll believe that. Sure enough, the site doesn’t fall over. I report the good news. Life continues on.

Back to this week. On Wednesday, things start to get a little suspicious. Unfamiliar people start to infiltrate the office and walk around as if they’re scoping out the employees, and the office space. Wednesday night, my project manager texts me and reassures me that I should be ignoring notifications from our hosting provider that our staging environment is down. Weird, but I let it slide.

Thursday morning, one of our architects has beaten me into the office. I usually arrive first in the office at 8AM, and he’s usually in by 10. This morning, he says he’s been in since 7AM. Very odd. At 10:55, Nina Zagat sends a company wide email informing us in large capital letters that there’s a meeting in 5 minutes in the IT area. At this point, the announcement of “the biggest news of the 32 year history of Zagat” comes as no shock that Zagat has been purchased… but by Google? Wow. My nerdy dream come true.

Marissa Mayer, the Vice President of Location and Local Services gives us all a hurray, a thank you, and a short welcome speech, followed by similar adulation from Bernardo Hernández and a few footnotes by an HR rep. Everyone from Google is thrilled, as are of course Tim and Nina, who almost certainly pocketed large sums of money as signing bonuses to become Googlers.

The important takeaway from this meeting: Google culture is being “imposed” upon us. That means at the least, breakfast and lunch is served daily. Let’s not mess around. This was good food, catered from a nearby shop.

!(center)/img/content/google_lunch.jpg(lunch)!

Marissa impresses upon us the fact that they’re interested in leaving as much of our current processes in place and merely bolstering our efforts with Google’s resources. That’s all well and good, but let’s see what the future brings in terms of integration with the Googleplex.

The afternoon is filled with single department meetings, one after another, to allay everyone’s fears of downsizing and to answer some other menial questions to the best of their knowledge. They’re clearly more prepared than we are. They’ve had almost a month to mull over this decision after roughly 3 months of internal planning.

Friday is seemingly back to normal with a hint of “wtf just happened?” still in the air. Wait, where’s my breakfast? Did Google stiff me? Is it really too much to ask for your employer to provide two square meals a day? Come on now… Okay, okay, it was a goof on Bernardo’s part. That all starts on Monday. Being the sole individual with proper knowledge of the production infrastructure, my inbox slowly fills with integration meeting invites over the course of the afternoon. Next week should be interesting.

At 4pm, Tim and Nina show up with the Google crew, and with them comes food. Oh, and booze. This is a typical Friday at Google.

!(center)/img/content/google_tgif.jpg(tgif)!

Cheers!

In addition, there’s a bit of trivia and lava lamps, USB Google desk lamps, and plush androids are awarded.

!(center)/img/content/google_lamp.jpg(googlelamp)!

Everyone gets a t-shirt with Google from the front, and ZAGAT across the back. Awesomeness. The mood in the room is upbeat, with lots of excitement.

While I can’t officially call Google my employer until the 6-8 week transition is through, I’m rather optimistic. My presence at the upcoming integration meetings with the remainder of the crowd being senior management is reassuring. Google is of course interested in new our vehicle for survey dissemination – the 5.0 website we redesigned and launched in February of this year. All this seems to hint towards IT being safe.

It’ll be hard to say that we’re back to “business as usual” on Monday, but I’ll do my best. I wonder what they’ll serve for breakfast…

Building a Virtual Army

Dave Reisner — 2011-05-17T00:00:00+00:00

Recently, in testing my latest new toy (geninit), I’ve needed to create a variety of different root device setups to put geninit through the proverbial ringer. Up until a few weeks ago, this would have been done through VirtualBox. However, I’ve never really been a huge fan of VirtualBox, increasingly due to my opposition of Oracle. The management is fairly straight forward, but the machines themselves feel fairly limited, and don’t take full advantage of processor extensions like VT-x. A few years back, it was even the case that VirtualBox was recommending not to enable VT-x at all, as it would actually hinder performance.

The other obvious option is “QEMU”:http://www.linux-kvm.org/, with KVM support. On a few occasions, I’ve poked around with QEMU, but was never really satisfied. Turns out, I really just didn’t give it enough time.

QEMU has a lovely feature that allows it to emulate a serial console – effectively giving you a VM in a terminal. In addition, it supports the “virtio”:http://wiki.libvirt.org/page/Virtio family of devices, which allows for much better performance, particularly in the realm of I/O, where the typical bottlenecks lie. Now things start to get more appealing. With a little bit of bash, and a fair bit of time with some documentation, things were starting to come together. I figured I’d share what I came up with, in case anyone else finds themselves in a similar situation.

h2. Initial Setup

You’ll need a few packages to get started: qemu-kvm, vde2, and iptables for now. You’ll also, of course, want a liveCD for your favorite distro. Start by creating a qcow2 image, which will serve as the disk for your VM:

qemu-img create -f qcow2 imagename.qcow2 5G

qcow2 is the QEMU image format of choice, which supports compression, encryption, dynamic sizing, copy on write, and snapshots. There are other formats, but this is by far the winner for versatility. Creation should be instant.

modprobe -a kvm-intel tun virtio

Note that I’m using intel, but there also exists kvm-amd for you other folks. Make sure your processor actually supports this – you can grep for ‘vmx’ in /proc/cpuinfo, which will hopefully return results in your processor flags. You’ll also want to make sure that /dev/kvm is created with ‘kvm’ as group. Arch Linux provides a udev rule to do this by default. Your mileage may vary. Make sure that you add yourself to the kvm group and log out for the changes to take effect.

h2. Networking

We’re going to be using “VDE”:http://vde.sourceforge.net for networking support which will essentially create an internal VLAN for our guests. Start by creating the gateway for the VLAN:

vde_switch -tap tap0 -mod 660 -group kvm -daemon

This launches vde_switch, which creates a new network device: tap0. It doesn’t yet have an IP, so we’ll need to assign it:

ip addr add 10.0.2.1/24 dev tap0
ip link set dev tap0 up

Note that I could have picked any “RFC 4193 internal address”:http://en.wikipedia.org/wiki/Private_network, just as long as its not the same network as my LAN.

With our gateway created, we need to allow traffic to forwarded properly through it:

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -s 10.0.2.0/24 -o eth0 -j MASQUERADE

A few points of interest here. The first is that you’ll want to add both these commands to a file that will be routinely read on bootup. I’ll leave it up to the reader as an exercise as to find the distro specific and recommended location for these. The second is that the iptables rule should be allowing any traffic whose source is the same as network our gateway device’s IP. The output for the rule, specified by -o, doesn’t necessarily need to be defined. In the case of my laptop which sometimes uses wlan0 and sometimes usb0, I left this undefined and routing rules take care of finding the correct path.

h2. Launch It!

With networking in place and disks ready, we should be all set to launch the VM. I’d suggest making a small script out of this, as it’ll be useful later on:

#!/bin/bash

mem='-m 1024'  # RAM allocation
cpus='-smp 2'  # CPU allocation
net='-net nic,model=virtio -net vde' # networking using virtio & VDE
drive='-drive file=/path/to/qcow,if=virtio' # disk image we created

qemu-kvm $mem $cpus $net $drive -cdrom /path/to/livecd.iso -boot d

Note that you don’t currently have the ability to acquire an address for guest’s network device via DHCP. I’ll cover that later as an optional feature. For now, just assign a static IP from within the guest:

ip addr add 10.0.2.100/24 dev eth0
ip link set dev eth0 up
ip route add default via 10.0.2.1
echo 'nameserver 8.8.8.8' >> /etc/resolv.conf

Note how we’re using the IP of the host’s tap0 device as our default route, and assigning an IP on the same subnet. Install as per usual. Before rebooting, make sure that the serial console is setup. It needs to be defined in your bootloader, on the kernel command line, and possibly as a getty. There’s quite a few flavors for the moving pieces here – some simple googling should quickly lead to results.

Once the serial console is setup, you can boot the VM with the -nographic option, which should happily dump output into your terminal.

h2. DHCP Server

Because I’m lazy, I decided that my little VLAN needs dhcp. The official ISC dhcp server is one option, and requires very little setup, but I was convinced that using dnsmasq was a better solution. It provides a lightweight DHCP server as well as DNS caching, which my desktop can benefit from as well. With dnsmasq installed from your trusty repositories, fire up your favorite editor and open /etc/dnsmasq.conf. We only need to make a few small changes. dnsmasq needs to be set to listen on addresses and/or interfaces, as well as to specify a dhcp range. My chosen settings were:

listen-address=127.0.0.1
interface=tap0
dhcp-range=10.0.2.100,10.0.2.150,12h

Now I can add 127.0.0.1 as the primary nameserver for my desktop, and guests on my VLAN will have dhcp as well as the cached DNS goodness. You could easily take this one step further and add in a DNS server for the VLAN. The addition of dynamic DNS updates would be excellent for keeping track of your soldiers. A more blunt approach would be to simply create DHCP reservations. Note that for this approach, you would need to manually pick out MAC addresses for each of the VMs as they’re always the same 52:54:00:12:34:56 across all guests.

h2. Tying it All Together

And last but not least, I expanded on the bash script I described earlier to start up the VM. It’s posted in my bin-scripts repo on “Github”:https://github.com/falconindy/bin-scripts/blob/master/qinit. Rather straight forward – define a function called vm_GUESTNAME for each VM and add the appropriate options. All that’s required is the drives array.

What Happens in Early Userspace...

Dave Reisner — 2011-04-30T00:00:00+00:00

I know what you’re thinking, but it doesn’t stay in early userspace. If you’re stuck there, then you’re doing it wrong.

A few months ago, the Arch developer crew put up a “Developers Wanted” sign on their door in search of devoted folks to take over the reigns on some of the in house projects: initscripts, netcfg, and ABS. I had actually applied for the initscripts job, but when I was approached by one of the developers and told that it had come down to myself and Tom Gunderson, I joyfully passed the job onto Tom. He’s an extremely smart and capable hacker. I had really only applied to make sure that someone reasonable was given the spot.

I had to wonder though. Why wasn’t mkinitcpio on the list? Sorry Thomas, but I think its a mess. Even today, look at the “open bug reports”:https://bugs.archlinux.org/index.php?string=mkinitcpio it still has. I spoke with the current maintainer directly and he stated that he was going to keep control of the project. I have plenty of respect for that if say you have the time, but I question his availability. Several bug reports which I posted patches for went untouched for months. Needless to say, I felt something needed to be done.

Before I continue, a little bit about mkinitcpio from an architectural standpoint. I consider it vaguely divisible into two pieces of functionality: what happens in every day userland (creation of the initramfs image), and what happens in early userland (on the initramfs itself). /sbin/mkinitcpio heads up the creation half of this, and pulls in files from /lib/initcpio/install and /lib/initcpio/hooks. ‘install’ coincides with files, binaries, and modules pulled in during the creation phase. ‘hooks’ are added to the resultant image as is and run during bootup. The resulting image is championed by a crudely written shell script which doesn’t even take advantage of ash’s features (which it runs under).

On a few occasions, I’ve sat down and tried to contend with the codebase. I understand the process just fine, but I constantly find myself pounding my desk and throwing things when I’m jumping from file tracing functions to other functions with similar names, all of which perform similar tasks. The state of the global variables in the code is a bit disturbing as well. Add in the typical “Bash” style I see in Arch’s projets, mix in odd commenting choices and you’ve got yourself a utility which I find myself surprised to work whenever I roll out a new kernel build.

I can do this. I can make it better. I’ve tried. I really have. But every time I sit down and start to make some small changes, I see myself in store for a full rewrite. So why not just rewrite it? Nuts to that, who has the time…

A few weeks ago, this insane idea floated into my head – the early userspace init can be written in C. There is somewhat of a sparking event, but I’m not going to go into detail. In the course of a few days, I threw together “dinit”:https://github.com/falconindy/dinit which serves as a proof of concept for a compiled pid 1. I hacked up my copy of mkinitcpio and glued together an image. Would it boot? Yes! With the help of some other archers, I ironed out a few bugs. Awesome, we have a starting point.

I took this last week off from work, and decided that rather than take it as a mental break, I was going to write a whole lot of Bash and C. mkinitcpio shall be reborn! No rewriting sections and hoping the maintainer accepts it. Just fucking forge on and write your own. It shall be done. It’s been a long week, and I’m quite pleased to say that “geninit”:https://github.com/falconindy/geninit not only works in theory, but it’s currently booting my desktop and a small army of testing VMs.

My design goals were/are pretty straightforward:

Be mostly compatible with mkinitcpio: keep the options and functionality largely the same. Some naming conventions will change. Your initramfs should not have the word ‘kernel’ in it.
More cleanly written: Lots of safe, best practices and commenting when logic might be unclear.
Faster than mkinitcpio: It’s not easy to profile shell code, but mkinitcpio does a lot of unnecessary forking for things that Bash can and should do in house.
Maintain modular architecture: This goes in hand with the first goal, and pretty much mimics what most other early userspace image creation tools do.

geninit still features things like presets and a very similar config file. mkinitcpio’s hooks all had to be redone as they’re not longer being run in the context of busybox, but being called as a fork/exec from the compiled pid 1. install files, now known as builders, were reworked slightly to use the new cleaner API that I’ve put in place.

Clean Bash is what I do. I write lots of it, and I’m damn good at it. I won’t hang onto any modesty on this point. I’m very knowledgable about what Bash can do in house and refuse to pass off work to an external unless it’s proven necessary.

Speed was important, because I hear a lot of complaints about how long it takes to generate new images. My initial implementation was roughly on par with mkinitcpio. autodetect images were a bit faster, and full images a bit slower. No good. The silver bullet was, of course, thinking outside the box. mkinitcpio uses a utility from the kernel source tree called gen_init_cpio. It requires that you build a list of contets in a specific file format which is passed to this utility which then poops out an image on stdout. The problem is this: inevitably, when tracing dependencies of modules or binaries, you find duplicates. You have to check for these and avoid adding them. Checking an external file for these things requires either iteration or grep. Both are slow. There’s another way. “The Gentoo Wiki”:http://en.gentoo-wiki.com/wiki/Initramfs#Creating_a_Separate_File was a big help here in getting me the speed I was looking for. The tradeoff is that you end up using a few Mb of diskspace in /tmp during the process, but I think its worthwhile. It also means you’re unable to create devices for the initcpio as an unprivileged user. Not an issue for me.

On the modularity front: In addition to keeping the concept of “builders”, I split the code into 2 pieces – a main Bash script which lives at “/usr/sbin/geninit”:https://github.com/falconindy/geninit/blob/master/geninit, and “/usr/share/geninit/geninit.api”:https://github.com/falconindy/geninit/blob/master/geninit.api. The API is a public API for which “builders” (known to mkinitcpio as “install”) can draw from to add files to the resultant image. The main file remains as private API. Since its Bash, I can’t stop someone from using the “private” API calls, but it’s a friendly suggestion.

While I still consider geninit to be a bit of a beta, I’m very excited to see it come to life so quickly. Early adopters are very much appreciatedd to help iron out bugs. It’s, of course, “on the AUR”:http://aur.archlinux.org/packages.php?ID=48542, and available directly from Github.

Happy booting!

New Year, New Toys

Dave Reisner — 2011-01-02T00:00:00+00:00

While it’s likely that in 2012 I’ll be voting “four more years”, I’ll be voting for “four more cores” in 2011. The major plumbing of my current desktop is aging (oh noes, it’s 2 years old!) but those are stuffed into the shell of an older system, dating back 5-6 years. To be quick about the specs:

GigaByte GA-EP45T-DS3R LGA 775
Intel E8400 Wolfdale @ 3.16ghz
2x1GB 1600mhz DDR3 Crucial Ballistix SDRAM
550W Corsair PSU

It runs 4 drives, ranging in size from 80gb to 1TB, totalling 1.8TB of storage. These are all of varying ages, and some are showing their age better than others. The sound card, dvd-drive, floppy (!!), and lian-li case are also of varying ages with the sound card (a snazzy emu10k1 chip with hardware mixing) dating back to 2002.

So, my major goals for the new build are:

Replace everything – my current desktop will only be pillaged for its 1TB drive
Create a system that’s as future-proof as one could possibly make it
Make it pretty. Spend a little extra time/money on cable management.

With that in mind, here’s what I’ve pieced together from the ground up:

Case: “Lian-Li PC-A05NB”:http://www.lian-li.com/v2/en/product/product06.php?pr_index=289&cl_index=1&sc_index=25&ss_index=62&g=q : You can’t beat Lian-Li for build quality, features, or looks. This case is mini-ATX sized but fits a full size ATX motherboard. It has a front, low mounting PSU, and case flow is inverted (intake is in the back). It’s also a little wider than their usual offerings, which makes putting in “sufficient” cooling a lot more feasible.
Power Supply: “SilverStone Strider 80 Plus Gold 750W”:http://www.silverstonetek.com/products/p_contents.php?pno=ST75F-G : This a single rail unit that offers higher than 80% efficiency at idle, peaking at 90%. Efficiency is important. I don’t buy the garbage about dual rail units being better, particularly in a non-SLI setup. High quality components will provide the amperage for components when they request it, without needing to worry about balancing output with a second rail. A single rail also lends to higher efficiency due to less chance for leakage. SilverStone offers a short cable kit with this PSU which I snatched up.
Motherboard: “Asus P6X58D-E LGA”:http://www.asus.com/product.aspx?P_ID=gFBKfNyhppW9tDbB : This was a tough call. My last builds have used Gigabyte boards and for a while I was considering the X58A-UD3R, but consumer reviews seem to imply that GigaByte’s QA has been slipping somewhat. Asus was my initial supplier for my early builds dating back to high school and their quality apparently hasn’t slipped. The board itself is somewhat less featureful than other socket 1366 boards, but it provides what I need and nothing else. I don’t need eSATA, I don’t need 2 onboard NICs, and I certainly don’t need 4 PCI-E slots. What I do want is: reliability, SATA3 and USB3. Asus, you’ve won me back.
Memory: “G.SKILL Ripjaws Series 12GB (3x4GB)”:http://www.gskill.com/products.php?index=265 : G.SKILL is entirely new to me, but they’ve been in the business for a while and have grown popular in the performance segment. My previous builds used either Crucial Ballistix or Corsair XMS, but I can’t say I’m married to either. These chips are spec’d at 1600mhz, and should provide for a nice (sane) OC on the Asus board.
CPU: “Intel Core i7 980x EE”:http://ark.intel.com/Product.aspx?id=47932 : If you’re going to claim your build is something resembling “future-proof”, you can’t skimp here. Hail to the king, baby. Gulftown is going to be a better buy (performance wise) than any of the upcoming Sandy Bridge based offerings clear through Q1 and Q2 of this year. And, it remains to be seen if the Q3 chips will be competitive with the 980x. None of the mainstream Sandy Bridge chips will be hexa-core, nor will they see the same oversized L3 caches as Gulftown. Also, Intel won’t be moving to their 22nm manufacturing process until 2012 at the earliest. Despite the price, I couldn’t say no to this one.
Heatsink/Fan: “Noctua NH-D14”:http://www.noctua.at/main.php?show=productview&products_id=34&lng=en&set=2 : I enjoy the outrageousness of the air cooling segment. This is an enormous 6 heat-pipe copper/aluminum design sandwiched together with an equally large dual fan setup – 140mm and 120mm. Noctua did a great job packing this much heat dissipation into a small enough package, and the fans are quiet – no more than 20dBa.
SSD: “Corsair Force F90”:http://www.corsair.com/solid-state-drives/force-series/cssd-f90gb2-brkt.html : There really only seems to be 2 choices for reasonably priced (I’m looking at you, Z-Disk) reliable SSDs right now: buy Sandforce, or ante up and buy an SLC disk like the intel X-25. Sandforce it is. SATA2 SSDs are nearing the peak of what the interface itself can handle, and SATA3 drives are still too much in their infancy to trust (nor is Sandforce offering antyhing in a SATA3 flavor). I expect that I’ll replace this a year from now with a SATA3 unit.
HDD: “WD Caviar Black SATA2 2TB”:http://www.wdc.com/en/products/products.aspx?id=100 : Not much to say here. 64mb cache, SATA2 performance (as if that matters on a rotational disk), and plenty of space. I’m a little weary of putting all my eggs in one basket compared to my current multi-disk setup, but that’ll be solved a few months down the road when I build a proper RAID-based NAS.
Sound Card: “M-AUDIO Audiophile 2496”:http://www.m-audio.com/products/en_us/Audiophile2496.html : I’ve got one of these cards in the HTPC I build for my parents and it’s a phenomenal sounding card. The RCA outputs are an aural joy, despite being analog. Feeding crap from onboard sound through a high-quality optical line doesn’t make it sounds high-quality. It just sound less bad than if you were to shove it through a 1/8” mini jack. I started hoarding FLAC over mp3 some time ago, so I’d like to hear the difference.
Video Card: “eVGA GeForce GTS 450”:http://www.evga.com/products/moreinfo.asp?pn=01G-P3-1450-TR&pwindow=features : This should have been a BFG Tech card, but sadly BFG decided to go out of business. Those bastards. It also would have been nice to have grabbed a single slot card, but so be it. Dual DVI is a must, HDMI is meh, and the price point is just right for my needs.

And then we throw in some case fans: a “Noctua NF-P12”:http://www.noctua.at/main.php?show=productview&products_id=12&lng=en&set=1 for the exhaust and a “SilverStone AP121”:http://www.silverstonetek.com/products/p_spec.php?pno=AP121&area=usa for the intake.

And now as Tom Petty might say – the waiting is the hardest part. Expect a build log with pretty pictures by the end of the month.

The Cowpac Trifecta

Dave Reisner — 2010-12-10T00:00:00+00:00

Some time around the end of October, I got to the end of my rope with cower’s code. The (now) legacy branch was unmaintainable. Errors were cropping up that were getting difficult to fix because the code path was absurd. The output methodology was somewhat salvaged, but still ugly. Perhaps I fall into the category of the melodramatic artist never satisfied with their own work, but I saw an opportunity to make things better. Around the time I was reading up on libcurl’s multi interface, a feature request came into Github for parallelizing updates. While the multi interface didn’t pan out, multiplexing requests onto threads turned out to be a valid option. I quickly dove into the pthreads library, grok’ed some examples from curl’s website, and wrote a proof of concept. It worked without a hitch.

After applying the logic to cower, I started on a complete rewrite. Within a month, I was using it day to day in place of the old code. Advantages? Plenty, for both me and users:

Almost entirely multi-threaded. Each request is given its own thread and works asynchronously, only pausing for alpm queries. Cower is faster.
Single file. I find it much easier to follow the logic and not have to worry about local headers this way. Cower is more maintainable.
Roughly 500 less SLOC. I owe a lot of this to condensing the half dozen files into a single file. Cower is smaller.
A sane output system. I ditched the config file which greatly simplified the logic. Strings are declared up front and used as is. Ask about color once in initialization, and then never again. Cower is more readable.

Given this, I’ve taken the liberty of working in some new features:

Searching by maintainer. Not a big deal, but it wasn’t at all hard to blend in.
Searching with regex. A little hackish because the AUR doesn’t actually support this, but it works well.

Long story short, I’m really happy with the new cower. Feel free to grab “cower-git”:http://aur.archlinux.org/packages.php?ID=35888 and give it a whirl. I promise it’s mostly stable. A new release won’t be tagged until pacman 3.5 is released, and that’s a few months away. I’d like to be able to get custom output formatting implemented as well. It’s already started, and I have some local commits which borrowed a lot of code from “expac”:http://aur.archlinux.org/packages.php?ID=44048. Wait, what the heck is expac?

Expac is an alpm data extraction tool. Perhaps that doesn’t make sense, so let’s dive into a use case. You need to know the dependencies of a package, and you need it in a parseable format for a script. Okay pacman, I don’t like you, and you don’t like me. Let’s do this.

$ pacman -Qi cower | sed -n '/^Depends/s/.*://p' | while read...

It’s mungeable, but there’s a better way – expac it.

$ expac '%D' cower | while read...

Expac basically rips out all the querying features of pacman and balls them up into a data dumper with an emphasis on flexibility. Output is described by the user, not by pacman. I think the “man page”:https://github.com/falconindy/expac speaks for itself. Because expac can also do searches, you can do fun stuff with mimic’ing pacman’s -Ss or -Qs flags. Ever wanted to see download/install size while searching the syncs? Now you can.

$ expac -Ss '%r/%n %v [%k|%m]\n    %d' foo

Alias it or wrap it, but print it how you want it. That’s how expac rolls.

A few thank you’s are necessary. A fellow Archer, kaitocracy, has been generous enough to host a “binary repo”:http://cowpac.kiwilight.com/ for cower, burp, and expac, complete with frequent builds of the -git packages. It’s somewhat humbling to see my little projects getting some attention. While I generally code for my own interests, I’m always happy to see that other people enjoy what I produce. I was asked to name this repo. The only thing I could come up with was cowpac, which if I may say, is awesome. My only regret is being unable to weave in burp somehow.

I also have to thank another Archer, “Keenerd”:http://www.kmkeen.com for testing the hell out of expac and constantly reminding me that my code sucks.

Systemd on Arch

Dave Reisner — 2010-09-12T00:00:00+00:00

Recently, I’ve been putting a fair bit of effort into smoothing out the Systemd experience on Arch. Wait, what’s Systemd? While I’ll be leaving the “details”:http://0pointer.de/blog/projects/systemd.html to the lead developer, Lennart Poettering, I can summarize some of the major selling points as I see them:

Fully parallelized boot
Daemon monitoring (like Monit)
Socket activation of services (like inetd)

Systemd also expands its control over what PID 1 would normally oversee. Normal day to day systemd config files, such as /etc/fstab, become configfiles for Systemd to parse and generate what its calls Units. Units could be anything – a socket, a service (daemon), a mount point, or even a device. Why would you want this? Well, consider PID 1 to be the insane conductor of the worst symphony ever assembled. Not only is the conductor insane, but he’s very controlling. He wants to know that the walls are of high acoustic quality and that the lighting is effective enough for his performers to see their music. Essentially, despite his OCD, he wants to know that his whole environment is going to be conducive to getting every last bit of talent out of the high school level musicians he has to put up with. That’s a long enough metaphor. What the hell is the point? Well, just like the conductor wants to fix the lighting so that the fat, half asleep tuba player in the back can see his music (and not fall asleep), you want your init to be aware of processes going awry in the background and having a domino effect on your system. Did the network just go down? Well, with it just went all your remote mounts as well. Systemd knows that your remote mounts rely on the network, so the network gets restarted followed by the remote mounts.

If you’re still reading, perhaps you’d be interested in trying Systemd. Perhaps you even have an Arch box you’d like to try it on! There’s a real dearth of info and not much highlighting of the work I’ve done (it’s all buried in the AUR and on GitHub), so I thought I’d outline all my tomfoolery. Maybe I’ll incite one or two people to help out with polishing off the remainder of this project.

There’s a fair number of dependencies we need to knock out from the AUR. I’m probably going to scare a few people off when I mention that this involves a kernel compile. There’s a few knobs that need to be turned from the vanilla Arch kernel, and we need to apply a patch while we’re still on 2.6.35 (it’s merged mainline in 2.6.36). In order, you’ll want to download, build, and install the following:

“kernel26-systemd”:http://aur.archlinux.org/packages.php?ID=37059
“libcgroup-git”:http://aur.archlinux.org/packages.php?ID=36901
“vala-devel”:http://aur.archlinux.org/packages.php?ID=38260
“udev-systemd”:http://aur.archlinux.org/packages.php?ID=38950
“dbus-systemd”:http://aur.archlinux.org/packages.php?ID=40399
“systemd-git”:http://aur.archlinux.org/packages.php?ID=36902

Or really, what you should just do is:

cower -dd systemd-git

And then just build in the order above. But wait! There’s one other thing that needs to be done before systemd builds correctly. You’ll need to fetch the PKGBUILD for extra/docbook-xsl and make a small change, which can be summarized as:

sed -i 's/xhtml/&{,-1_1}/' PKGBUILD

Build and install as normal.

After systemd is built and installed, please for the love of Arch, read Pacman’s output. If you reboot now, you’ll be fairly dissatisfied with the results. You won’t have any daemons running, so you won’t have network, syslog, cron, or any other goodies you normally have with Arch’s stock setup. This means its time to introduce systemctl, the assistant to our mad conductor. As I mentioned before, normal config files become Units for systemd. Thanks to some early patchwork that Lennart accepted from myself, Systemd detects that we’re running Arch at build time and sets some parameters to find our old and trusted daemon init scripts. Using systemctl, you’ll be able to trigger these scripts, for example:

$ sudo systemctl start network.service syslog-ng.service

They should start right up. There’s a junction right here, and you have 2 options. The first is to create the forest of /etc/rcN.d directories, where N is 0-6, and symlink these scripts into the appropriate runlevel. It should work, but you’re not really taking advantage of what Systemd was meant to do. The second option is to use the native unit collection I put together: “systemd-arch-units”:http://aur.archlinux.org/packages.php?ID=40419. Install this, and note Pacman’s output. You can get a list of the included unit files in this package from Pacman via `pacman -Ql systemd-arch-units’. Enable the services that fit your install (and please submit unit files for services I don’t have!).

Whew. If you’ve done everything correctly, you should have a mostly functional system when you reboot. But wait! There’s more! We can do better. We’re still relying on Arch’s stock initscripts to fork and exec a few hundred times to get our system up and running. How do we get Systemd to take over more ownership of this? There’s a package for that! My most recent work has been to tear up the Arch initscripts and trade off shell script for more native units. You can find these butchered scripts in “initscripts-systemd-git”:http://aur.archlinux.org/packages.php?ID=40592. A word of caution here: up until this point, you’ve been able to remove the init=/bin/systemd from your bootloader and fall back gracefully onto Arch’s stock init SysVinit. Installing this package removes this safety net! I’ve only been able to test on my own (live) box and a VM, but it’s been working quite well so far. Bug reports and patches are extremely welcome.

At this point, your Arch box is looking a bit less Arch-like. I propose that perhaps that’s a good thing. SysVinit works, and it certainly fits the Arch philosophy, but I think it’s on the side of too simple at the cost of modern (very desirable) functionality. Systemd is in rapid development and Lennart has been very responsive to concerns that I’ve voiced to him. You’ll soon see Systemd as the default init system in the next big releases of Fedora and OpenSUSE. There’s work being done by the Debian, Gentoo, and Slackware crowds as well. While I’m not expecting this to ever make it to core repo in Arch, I’m hopeful that this gains some attention amongst the curious hackers of Arch because this is definitely worth talking about.

Digging Up the Past

Dave Reisner — 2010-04-16T00:00:00+00:00

My former host, GoDaddy, is gone. I have nothing positive to say about my experience with them as a web host, right up until the moment I hit the cancel button. There is no way to pre-emptively cancel your account aside from conveniently providing a credit card that will expire before the next renewal, effectively making your hosting expire when renewal rolls around. Actively cancelling your account will terminate your hosting immediately. Regardless of whether or not you had time remaining before the renewal rolled around (one month in my case), your site is destroyed, your access is revoked, and no refunds are possible. Not very classy. What’s the new solution? GitHub. Slightly more expensive, but I thoroughly enjoy the services GitHub provides and have no qualms about throwing them $7/month for them to honor a CNAME redirect.

I won’t mention much about the new site, except to say thanks to the “bro”:http://code.alexreisner.com for providing endless nudging to finally do this, and graciously letting me use his template as a basis. Web design isn’t my forte, nor one of my major interests in the world of programming. The site itself is written in “Jekyll”:http://github.com/mojombo/jekyll which means that I can write mostly in textile markup, and GitHub happily renders it into something gorgeous. I recommend reading the article “Blogging Like a Hacker”:http://tom.preston-werner.com/2008/11/17/blogging-like-a-hacker.html if this is something that is appealing to you. It’s written by the author of Jekyll and co-founder of GitHub, Tom Preston-Werner.

On a more exciting topic of history, shell history will make your life easier and more productive. As most references on the web focus on the ability to directly recall numbered items (available via the history command), I’ll skip to more interesting things. Something I use on an insanely frequent basis is the !! operation, which recalls the last command in its entirety. Why is this so useful? Consider the following:

$ pacman -Syu
error: you cannot perform this operation unless you are root.
$ sudo !!
Password:
...

What if the last command isn’t exactly what you want? Suppose you’re bouncing between vim, gdb, valgrind, and gcc. What if you want to recall not the last command, but the last gcc command? Easily done:

$ !gc
gcc myproject.c -o .....

You can add to the command as well.

$ ls -la ~
# Oops, I meant to grep that...
$ !ls | grep foo

This only works, however, for commands starting with the pattern passed to !. If you want to match a comannd with a pattern found anywhere, you can modify the ! with ? wildcards as such:

$ !?pacm?
sudo pacman -Syu

Note that if you’re matching on the end of a command, you can omit the trailing question mark.

History can help you correct spelling mistakes. Suppose you’re cd’ing to a long directory path but make a mistake:

$ cd ~/doc/school/COMP233/hw4
cd: no such file or directory: ~/doc/school/COMP233/hw4
# Hmm, I meant to cd to COMP228, not COMP233
$ !!:s/233/228/

And if you haven’t already guessed, that colon operator would work for modifying a recalled command like !gc as well.

Another way to fix typos if you call the wrong command, for example:

$ echo /var/log/pacman.log
# Well that's clearly not right, we meant to cat it
$ ^echo^cat

One final tip for you ZSH users out there. Adding the following to your .zshrc will greatly improve your experience with history:

export HISTIGNORE="&:ls:[bf]g:exit:reset:clear:cd*"
export HISTSIZE=25000
export HISTFILE=~/.zsh_history
export SAVEHIST=250
setopt APPEND_HISTORY
setopt INC_APPEND_HISTORY
setopt HIST_IGNORE_ALL_DUPS
setopt HIST_IGNORE_SPACE
setopt HIST_REDUCE_BLANKS
setopt HIST_SAVE_NO_DUPS
setopt HIST_VERIFY

Many of the exports apply to Bash as well, but the setopts are ZSH only.

This is just the tip of the iceberg but I find that, for myself, these are my most commonly used history commands. You can, of course, check out mman history for a full onslaught of goodness.

The Anatomy of an Old Friend

Dave Reisner — 2010-03-25T00:00:00+00:00

I recently reignited my interest in C after an epiphany regarding pointers. This lead to a weekend orgy of roughly 1000 SLOC to create an AUR agent based on a “similar Python solution”:http://github.com/rson/slurpy that I had been using for several months. Of course, being who I am and usually shooting from the hip when it comes to coding, I roughly sketched out the program in my head and dove into it. This was, of course, a bad idea. During my second rewrite, I decided that a great way to clean up some portions of my code would be to cut back on the number of printf statements I had due to the option of color in the output. But how to do this? I decided the best route would be to create my own patterns and reimplement printf.

printf is probably one of the most used functions, and exists in nearly every language (I’m looking at you, C++). However, I would bet that most programmers haven’t taken the time to examine exactly how a function like printf works at a lower level. Inquiring minds want to know! To paraphrase Linus, C isn’t really a programming language – it’s more like writing portable assembly. So let’s take a look at a simplified version of printf in C. It should give us a good idea of how the bugger works.

As per the man page, printf(3), the prototype is as follows:

int printf(const char *format, ...);

From this, we see that printf accepts a char pointer to our format string, and an undefined number of additional arguments which are our pattern replacements. The return value is the number of characters printed. If you dig deeper, the printf family of functions are analogous to functions which take a fixed numbers of arguments and do the real work: vprintf, vfprintf, etc. These accept a va_list in place of the elipsis, with “va” being short for variable argument. The front end of printf is, as it turns out, very simple:

int printf(const char *format, ...) {
	va_list argp;
	int result;

	va_start(argp, format);
	result = vprintf(stdout, format, argp);
	va_end(argp);

	return result;
}

We declare a va_list to hold our arguments contained within the elipsis and an int to carry the return value from our vprintf. The calls to va_start and va_end are actually macros provided to us by the preprocessor. The first call to va_start loads our va_list, argp, with all arguments passed after our last defined argument, format. Understand that despite a va_list not being defined as a pointer, it really is just a stack pointer to our passed parameters, thus why we need to give it the name of the last defined parameter. We then pass a file descriptor, our format and va_list to the workhorse vprintf where it’s parsed, and when it returns we free the va_list by calling the va_end macro.

So let’s look at the juicy bits of the operation. Keep in mind that this is a simplified version, which doesn’t take into account things such as field width, precision, or alignment. We’re also going to implement vfprintf instead of vprintf, because vprintf is nothing but a call to vfprintf with our file descriptor defaulted to stdout. Looking at our trusty man page, we see exactly what we expect from vfprintf’s prototype, so let’s dig into the function itself.

int vfprintf(FILE *stream, const char *format, va_list ap) {

	const char *p;
	int count = 0;

	int i;
	char *s;

We declare a few variables here. A char pointer, which will be used to examine our format string, and a character counter initialized to zero, which keeps track of how many characters we write. The two other variables will be used with a 3rd va macro which will be introduced very shortly.

	for (p = fmt; *p != '\0'; p++) {
		if (*p != '%') {
			fputc(*p, stream);
			count++;
			continue;
		}

We establish a for loop, with our pointer initialized to the start of the format string, and ending when we reach the null terminator at the end of it. We compare each character to see if it’s a percent sign, denoting the start of a pattern. When it isn’t, we print the character, increment our counter, and skip on to the next character in the format. Note that unlike other functions in the fput family, fputc does not return the number of characters it wrote – this obviously going to be one. Rather, it returns the character we printed. Because of this, we need to increment our counter separately.

		switch (*++p) {
		case 'c':
			i = va_arg(ap, int);
			fputc(i, stream);
			count++;
			break;

Within the for loop, we create a switch block on the next character after the percent sign. This is where we determine what pattern was specified in the format: was it %c, %d, %f, etc. We call our 3rd va macro, va_arg, which is told to extract the next argument from the va-list ap, and expect it to be of type int. Wait, an int? I thought we were catching a character? Well, it turns out that some promotions occur within printf. A char will always be accepted as an int, and a float will always be accepted as a double. There’s no harm done by either of these promotions, and it saves us from declaring 2 extra variables. A savings of a whole 9 bytes! The rest of this block is fairly dull. It’s a simple character, so pass it to fputc with our file stream, increment the counter and move on.

		case 's':
			s = va_arg(ap, char*);
			count += fputs(s, stream);
			break;
		case 'd':
			i = va_arg(ap, int);
			if (i < 0) {
				i = -i;
				fputc('-', stream);
				count++;
			}
			count += fputs(itoa(i, 10), steram);
			break;

This next bit should be pretty easy to decipher, knowing how the last one worked. A %s pattern is encountered, we extract it from our va_list, and call to fputs to write it out to our stream. Notice that now we’re using fputs, which returns the number of characters written to the stream, so we can easily increment our counter with the return value. A %d pattern requires a little extra effort. We’re going to simplify the process by catching a negative number and converting it to a positive number, prepending the eventual output string with a unary negative. Given C’s low level nature, we can’t just directly print the number to stdout. We have to make it into a string. For that, we call itoa(), which accepts the number and the numerical base that its represented in. I won’t detail the innards of itoa. Consult your white bible (K&R2) for details. Just know that it returns a char pointer to our ascii representation of the number. We pass it to fputs and continue on.

		case '%':
			fputc('%', stream);
			count++;
			break;
		}
	}

	return count;
}

I’ll finish our simple vprintf implementation with one final possibility, which is when you really do want a percent sign in your output. Can’t forget that! We return our character count, and life is good. At this point, you should be able to see how easy it would be to implement other patterns, including your own custom patterns to add color to the output. This would be, of course, implementation specific and certainly not portable. You can see my full implementation at “cower’s GitHub page”:http://github.com/falconindy/cower. My own printf exists in util.c.

I hope this little expedition was helpful in understanding the basic idea behind an every day function that helps to make a programmer’s life infinitely simpler with regard to console output.

squashfu

Dave Reisner — 2010-01-12T00:00:00+00:00

I pulled all my hair out for nothing. A bug with Aufs which was stopping me completing my previously mentioned backup script has been determined to be upstream. Better yet, an Arch dev was kind enough to pass along a fix at least to get the mount stable on Ext4. So, without further ado, I present the very stylish, very sleek, (very beta) SquashFu. It slices! It dices! Well, okay. It just backs up data. The link to Github has full details, but a quick overview:

h2. Requirements:

bash
aufs2 w/ aufs2-utils
squashfs-tools
rsync

h2. Features:

Easy setup: Define a few variables in the config, create your backup “root”, and run sudo squashfu -B!
Very reasonable compression (2.6GB squashes down to 1.7GB on my backup)
Flexible backup options, using rsync’s include/exclude syntax
User defined control of incremental backups – keep 5 or 500 (not tested with 500)
Fine grained control of incrementals: each individual backup is isolated in its own bin
Easy rollback: instant access to any backup point
Reporting: Detailed breakdown of disk usage for the base and each of the incrementals

So far I’ve been running it on my own desktop as well as my server, and it’s doing quite nicely. However, that isn’t to say that it doesn’t still have undiscovered bugs. If anyone is interested in testing it, I’d love to know how it fares, particularly on distros other than Arch. Bug reports and suggestions are welcome. Archers, here’s your PKGBUILD.

Con Kolivas is a Mad Man

Dave Reisner — 2009-12-22T00:00:00+00:00

I’ve recently gotten back into the insane (and mindless) hobby of compiling kernels for fun and purposelessness of:

How fast I can compile another kernel (currently 6 minutes)
How fast I can boot (slightly under 20 seconds)
How fast I can find primes in the first 1,000,000 numbers (there’s 78,498 of them)
Masochistic pleasure

Up until recently, this involved a lot of trial and error, going back and forth with utilities like lsmod and hwdetect in order to strip out unneeded kernel modules and build in various crucial ones. I’ve been running without an init image for over a month now. Recently, I rediscovered the anaesthetist by day, kernel hacker by night Con Kolivas. I first heard about Con a few months back when he released “BFS”:http://ck.kolivas.org/patches/bfs/bfs-faq.txt, a.k.a. the Brain Fuck Scheduler. The name alone inspires some amount of fear – would it set my house on fire if I made one too many typos? BFS is a linear performance, single runqueue scheduler that’s touted as being outrageously efficient for any workload, while remaining stable. Due to its lacking “scalability”:http://xkcd.com/619/, BFS would never be included in the mainline kernel. It will, however, continue to exist and be developed as part of the -ck patchset.

There’s some amount of drama surrounding Con and his involvement (and departure) with kernel development. In a “2007 interview”:http://apcmag.com/interview_with_con_kolivas_part_2_his_effort_to_improve_linux_performance_on_the_desktop.htm, he notes that he initially got into kernel hacking as a hobby. Starting in 2003, he wrote some low impact code, learned from it, and then dug deeper into scheduling and desktop response. Eventually, major developers (Linus included) got wind of his work and asked him to clean it up for mainline submission. He proceeded to churn out some other projects in the realm of desktop performance: the Staircase Scheduler, Plugsched (a hotplug CPU scheduler to integrate the Staircase Scheduler), and the Staircase Deadline Scheduler . Fast forward, and Con is starting to feel like constant requests for bug fixes and resolution for corner cases is more of a job and less of an intellectual hobby. Throw in some more corner cases, a few extremely vocal users, and a medical condition (related to programming) that landed Con immobile for 6 weeks, and he decides to throw in the towel. In addition to the frustration, he also felt that mainline developers weren’t concerned enough with desktop interactivity. Maybe it’s just a “coincidence”, but around the same time Con makes his departure, mainline adopts a linear performance scheduler called CFS (Completely Fair Scheduler), written by Ingo Molnar.

2 years later, in August of 2009, Con makes a reappearance with BFS and the revival of the -ck patchset. About a week ago, I finally had the courage to try not only BFS, but the entire patchset. In addition to BFS, also offers a 10000Hz timer frequency and a few other tweaks to beef up desktop response in the preemption sector. Consider him the ultimate advocate of the desktop user. The best part, of course, is that the patches work beautifully. Multi-tasking is faster, cleaner, and even while compiling in the background (yes, yet another kernel) I can run whatever else I want without a hitch. My “aging” Core2Duo doesn’t even blink.

Welcome back, oh prolific poet of processing precision.

12/24 update: There’s some “noise”:http://doom10.org/index.php?topic=78.0 on the Linux Kernel Mailing List about a comparison of CFS and BFS. “Con chimes in”:http://lkml.org/lkml/2009/12/18/490 a few days later.

Just Call Me One Happy Hacker

Dave Reisner — 2009-12-13T00:00:00+00:00

I recently had the extreme pleasure of coming into possession of one of the finest keyboards ever created: the Happy Hacking Keyboard Lite. Granted, this is the little brother (fitting for me) of the infamous HHKB 2 Professional, but this little guy has me hooked. For those of you completely lost, I present to you a picture:

!(left)/img/content/hhkb.jpg(HHKB Lite)!

At this point, I can only assume you, the reader, falls into 1 of two categories. Either you’re a little intrigued, possibly even jealous, or you’re just more confused. Allow me to try and explain. The HHKB has a few distinct features that set it apart from your normal “pedestrian” keyboard:

It has, by design, only 87 keys. This is sometimes called a ‘tenkeyless’, in reference to its missing number pad. In the lower right hand corner, is a key labelled ‘Fn’, similar to a laptop keyboard which gives you access to F1 through F12 as well as navigational keys such as end, home, pgup, pgdn, and the standard arrow keys.
Rejoice, for there is no caps lock. In place of it is a standard control key, which, after only a few minutes of playing with the layout, is a mind blowingly more comfortable place for it.
The keys to the left and the right of the space bar are not, as even I first thought, the equivalent of the Windows key. Quite to my surprise, xev shows their keysym is “Henkan” and “Henkan-mode”. Upon further investigation, this is for constructing Japanese characters.
4 little DIP switches buried in the back of the unit allow you to slightly alter the key behavior. By default, the backspace key acts as a delete. You’re also able to swap the Alt and Henkan keys, alter the Henken keys to act as a Windows key (bonus: without the logo), and alter the Alt key to be an extra Fn key.

Of course, with this unconventional design comes a learning period. It’s been 24 hours and I’m already feeling more comfortable with it. The DIP switches in particular can completely alter the experience. I presume in a week I’ll forget I ever had another keyboard.

The true joy with the HHKB is the actual mechanical construction behind it. This requires some technical detail to truly appreciate it. The majority of keyboards feature rubber domes, which have metal contacts. When you depress the key, the rubber dome makes contact with a printed circuit sheet, and the keyboard controller recognizes, based on the location on the sheet, that you’ve pressed a button. It’s a flimsy design in that the parts are more prone to wear than other designs. However, this gives the keyboard a nice smooth feel and tends to be fairly quiet.

Older keyboards (nowadays make that more expensive keyboards) use an actual mechanical switch. I won’t go into all the various types, but if you’ve ever come across an IBM keyboard, you know the click clack that these keyboards make. This is a far more solid design, and although it can be on the noisy side, the tactile sensation allows for far more accurate and quicker touch typing. You gradually get a feel for where the switch activates which, over time, can turn into a muscular reflex allowing you depress the key just enough to make contact.

Leaving out a few of the inbetween (and low end) mechanisms, enter the HHKB which combines both these concepts in what is called a Topre switch. The topre design is a capacitive switch which combines a rubber cup on top of a conical spring for maximum tactile feel as well as smoothness and reliability. It’s an amazing difference that you immediately feel. By design, these switches are also essentially indestructable. They will, however, destroy your wallet. Only a few companies offer these $200+ keyboards: Realforce, Das, Filco, and PFU – makers of the HHKB.

Thanks to discovering geekhack.org some time ago and introducing me to this wonder, I now know that a whole other world of keyboards exists. My last purchase several years ago of a Saitek Eclipse II, which served me well, was thusly eclipsed by my findings on the geekhack forums. Not only do these guys know far more than you would ever need (or possibly want) to know about a keyboard, their knowledge sometimes extends into addiction. Check out the signatures where they claim ownership of dozens of keyboards, with some dating back to the early 1980s. They love them all as if they were their children. Now, don’t get me wrong. I’m not putting geekhack down. Quite the contrary. I commend these proud folks for being so committed to their cause. After all, if you spend you days hacking away at a keyboard, why leave untouched the one device that’s the most touched? If you’ve ever been even remotely curious about what else is out there for keyboards, spend a little bit of time reading their wiki. It’s worth it. You may find yourself sliding down the same slippery slope as I did.

I’m very much looking forward to getting acquainted with the HHKB. It’s been a joy to work with it so far. Thanks Alex.

Saving Keystrokes With Bash

Dave Reisner — 2009-12-03T00:00:00+00:00

Bash is a wonderful fully featured shell that provides a multitude of ways to cut back on your keystrokes. One of my favorite features, which I don’t see used often enough, is brace expansion. Simply put, brace expansion lets you specify multiple similar arguments without retyping the commonalities. Let’s take a look at how this is accomplished.

With this handy feature, you can do a multitude of things. You can make a backup copy of a file:

cp /etc/mpd.conf{,~}  #Same as cp /etc/mpd.conf /etc/mpd.conf~

Then you can restore that file:

cp /etc/mpd.conf{~,}  #Same as cp /etc/mpd.conf~ /etc/mpd.conf

Obviously, it doesn’t stop here. You can make a whole directory structure:

mkdir -p /squashed/{usr,opt}/{rw,ro}
#Same as mkdir -p /squashed/usr/rw /squashed/usr/ro /squashed/opt/rw /squashed/opt/ro

Bash will expand ranges as well. You can create a zero padded range like so:

echo {000..100}  #Will print 000 001 002 003 ... 097 098 099 100

Ranges aren’t limited to being numerical. Better yet, expressions can be the preamble and postscript to each other.

echo {0..9}{A..Z}   #Prints 260 strings!

If you really use your noodle you can nest expressions, though I’ve personally not come across a situation where this has been needed (yet).