Odd Bits

Capturing Envoy Data

noreply@blogger.com (Lars) — Wed, 22 Feb 2012 06:26:12 PST

Pursuant to my last post, I've written a simple man-in-the-middle proxy to intercept communication between the Envoy and the Enphase servers. The code is available here.

What it does

As I detailed in my previous post, the Envoy sends data to Enphase via http POST requests. The proxy intercepts these requests, extracts the XML data from the request, and writes it to a local file (by default in /var/spool/envoy). It then forwards the request on to Enphase, and returns the reply to your Envoy.

In addition to extracting the XML data, the proxy also logs the complete contents (headers and message content) of the request and the reply to files.

How it works

Out of the box, your Envoy configures itself automatically using DHCP. Possibly, you've configured it statically. In either case, it will typically be configured to connect via your default gateway -- generally, your home router, cable modem, etc. In order to intercept the communication between the Envoy and Enphase, we insert another server between the Envoy and your network gateway. In the foollowing diagram, the dotted line represents the original communication path, while the solid lines represent the new communication path:

The intermediate system -- which we'll call the interceptor -- will use a few tricks to redirect traffic destined for Enphase to the local proxy (which will log the data locally and then forward it on to Enphase).

Assumptions

For the purposes of this article, we'll assume that your Envoy is at address 192.168.1.100, and the address of the interceptor is 192.168.1.200.

I'm assuming that your interceptor is running Linux. It may be possible to accomplish the same thing with other tools, but I'm relying on the Linux netfilter subsystem (aka "iptables") to perform certain key tasks.

Configuring the Envoy

You will need to congfigure the Envoy to use your interceptor host as its default gateway.

Go to the network connectivity page on your Envoy.
If it's checked, uncheck the "Use DHCP" setting and select the "Updating DHCP setting" button.
Set the "Gateway IP" field to the address of your interceptor (192.168.1.200 in this example).
Select the "Update Interface 0" button.

Configuring the interceptor

Redirecting requests

We need to configure the interceptor to redirect requests to the Enphase servers to a local application. We'll add the following firewall rule:

iptables -t nat -A PREROUTING -s 192.168.1.100 -p tcp \
  --dport 443 -j REDIRECT --to-ports 4430

This rule matches https (port 443) requests from your Envoy (192.168.1.100) and redirects them to port 4430 on the interceptor.
Note that this rule will be lost if you reboot your system. Making firewall rules persistent is beyond the scope of this article; consult the documentation for your distribution of choice.

Handling SSL

My simple Python proxy doesn't speak SSL, so we need to create a plain http request from the https request. Normally this would be difficult, but Enphase has made our life easier by not checking the validity of the SSL certificate. We're going to use stunnel as an https-to-http proxy. Create a file called /etc/stunnel/envoy-ssl.conf with the following contents:

[https_in]
accept = 4430
cert = /etc/pki/tls/certs/localhost.crt
connect = 127.0.0.1:8080

Run stunnel with this configuration:

stunnel /etc/stunnel/envoy-ssl.conf

This assumes you have an SSL certificate in /etc/pki/tls/certs/localhost.crt. You will probably need to generate one, which again is left as an exercise to the reader.

Installing bottle

The proxy relies on the bottle Python web framework, which is probably not installed on your system. The easiest way to get things going is to install a Python "virtual environment" with the appropriate modules. Create a new virtual environment:

virtualenv ~/env/envoy

And install bottle:

~/env/envoy/bin/pip install bottle

Creating directories

By default the proxy will write data to /var/spool/envoy. You'll need to make sure this directory exists and is writable by whatever account you're using to run the proxy.

Running the proxy

Now that you've got all the prerequisites in place, you should be able to start the proxy by running:

~/env/envoy/bin/python proxy.py

You should see something like this:

Bottle server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.

Assuming that everything else went as planned, sometime within the next five minutes you should see the proxy service a request from your Envoy:

localhost.localdomain - - [22/Feb/2012 09:03:57] "POST /emu_reports/
performance_report?webcomm_version=3.0 HTTP/1.1" 200 103

From this request you will end up with three files in /var/spool/envoy:

2012-02-22T09:03:01-0j1FFs.xml
This is the XML data from the Envoy and is probably the most interesting file.
2012-02-22T09:03:01-ZMxw6b.request
This is the raw request from the Envoy.
2012-02-22T09:03:02-NB4DbR.response
This is the response from the Enphase servers.

If you find some bugs, please let me know by creating a new issue here. Note that this is only for bugs in the code; if you need basic networking tutorials and so forth the Google has lots of help for you.

Enphase Envoy XML Data Format

noreply@blogger.com (Lars) — Mon, 13 Feb 2012 18:44:27 PST

We recently installed a (photovoltaic) solar array on our house. The system uses Enphase microinverters, and includes a monitoring device called the "Envoy". The Envoy collects data from the microinverters and sends it back to Enphase. Enphase performs monitoring services for the array and also provides access to the data collected by the Envoy product.

I'm interested in getting direct access to the data provided by the Envoy. In pursuit of that goal, I set up a man-in-the-middle proxy server on my home network to intercept the communication from the Envoy to the Enphase servers. I'm documenting the results of my exploration here in case somebody else finds the information useful.

The Envoy sends deflate-encoded XML data to the Enphase servers. You can decompress the data using Python's zlib module like this:

import zlib
xml_data = zlib.decompress(deflate_compressed_data)

The request is an http POST to https://reports.enphaseenergy.com/emu_reports/performance_report?webcomm_version=3.0. The request headers look like this:

POST /emu_reports/performance_report?webcomm_version=3.0 HTTP/1.1
Accept: */*
Connection: close
Content-Type: application/x-deflate
Content-Length: 729
Host: reports.enphaseenergy.com:443

The request body -- after inflating it -- looks something like this:

<?xml version='1.0' encoding='utf-8'?>
<perf_report report_timestamp='1329134202'>
  <envoy ip_addr='192.168.1.166' mac_addr='00:11:22:33:44:55'
  timezone='US/Eastern' part_num='800-00024-r04'
  sw_version='R3.0.0 (9720d7)' serial_num='123456115374' />
  <event correlation_id='161' event_state='1' id='221'
  event_code='10009' eqid='123456102635' serial_num='123456102635'
  event_date='1329134118' />
  <event correlation_id='194' event_state='1' id='222'
  event_code='101' eqid='123456103685.1' serial_num='123456103685'
  event_date='1329134119' />
  <event correlation_id='196' event_state='1' id='223'
  event_code='101' eqid='123456105331.1' serial_num='123456105331'
  event_date='1329134120' />
  <event correlation_id='202' event_state='1' id='224'
  event_code='101' eqid='123456103294.1' serial_num='123456103294'
  event_date='1329134130' />
  <event correlation_id='206' event_state='1' id='225'
  event_code='101' eqid='123456105321.1' serial_num='123456105321'
  event_date='1329134139' />
  <event event_state='2' id='226' event_code='101'
  eqid='123456104335.1' serial_num='123456104335'
  event_date='1329134152' />
  <event event_state='0' id='227' event_code='509'
  eqid='123456104335' serial_num='123456104335'
  event_date='1329134152' />
  <event correlation_id='213' event_state='1' id='228'
  event_code='101' eqid='123456105296.1' serial_num='123456105296'
  event_date='1329134153' />
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105041' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456105041.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456104335' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456104335.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456104246' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456104246.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456104224' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456104224.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456102776' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456102776.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456103271' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456103271.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105190' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456105190.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105321' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456105321.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105178' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456105178.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456104988' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456104988.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456103294' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456103294.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105346' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456105346.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105297' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456105297.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456104896' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456104896.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105331' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456105331.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456103546' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456103546.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456103685' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456103685.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2359296' part_num='800-00103-r05'
  observed_flags='0' eqid='123456103215' control_bits='0'>
    <channel channel_type='1' condition_flags='16'
    observed_flags='0' eqid='123456103215.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456102635' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456102635.1' control_bits='0' />
  </device>
  <device image_bits='131072' device_type='1' admin_state='1'
  condition_flags='2097152' part_num='800-00103-r05'
  observed_flags='0' eqid='123456105296' control_bits='0'>
    <channel channel_type='1' condition_flags='0'
    observed_flags='0' eqid='123456105296.1' control_bits='0' />
  </device>
</perf_report>

The response headers look like this:

HTTP/1.1 200 OK
Date: Mon, 13 Feb 2012 11:56:48 GMT
Server: Apache
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.11
ETag: "95bc2eaddfac613540f469946f28af4b"
X-Runtime: 297
Cache-Control: private, max-age=0, must-revalidate
Content-Length: 142
Status: 200
Cache-Control: max-age=31536000
Expires: Tue, 12 Feb 2013 11:56:48 GMT
Connection: close
Content-Type: application/x-deflate; charset=utf-8

And the response body -- after inflating it -- looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<perf_report_response status="success">
  <events_processed>
    <event id="221"/>
    <event id="222"/>
    <event id="223"/>
    <event id="224"/>
    <event id="225"/>
    <event id="226"/>
    <event id="227"/>
    <event id="228"/>
  </events_processed>
</perf_report_response>

My goal is to set up a simple proxy that in addition to passing the information along to Enphase will also collect it locally. I'll update the blog if I make progress on that front.

Very simple rate limiting

noreply@blogger.com (Lars) — Mon, 26 Dec 2011 20:01:07 PST

I use CrashPlan as a backup service. It works and is very simple to set up, but has limited options for controlling bandwidth. In fact, if you're running it on a headless system (e.g., a fileserver of some sort), your options are effectively "too slow" and "CONSUME EVERYTHING".

There is an open request to add time-based limitations to the application itself, but for now I've solved this using a very simple traffic shaping configuration. Because the learning curve for "tc" and friends is surprisingly high, I'm putting my script here in the hopes that other people might find it useful, and so that I can find it when I need to do this again someday.

#!/bin/sh


# The network device used for backups
dev=p10p1


# The remove address of the CrashPlanserver
crashplan_addr=50.93.246.1


# The port
crashplan_port=443


# The rate limit. See tc(8) for acceptable syntax.
crashplan_limit=2mbit

if [ "$1" = "enable" ]; then
    #
    # This creates and activates the traffic shaper
    # configuration.
    #
    logger -s -t ratelimit -p user.notice "enabling rate limits"
    tc qdisc del dev $dev root > /dev/null 2>&1
    tc qdisc add dev $dev root handle 1: htb
    tc class add dev $dev parent 1: classid 1:10 htb rate $crashplan_limit
    tc filter add dev $dev parent 1: prio 0 protocol ip handle 10 fw flowid 1:10
    iptables -t mangle -A OUTPUT -d $crashplan_addr -p tcp --dport $crashplan_port -j MARK --set-mark 10
elif [ "$1" = "disable" ]; then
    #
    # This removes the traffic shaper 
    # configuration.
    #
    logger -s -t ratelimit -p user.notice "disabling rate limits"
    tc qdisc del dev $dev root > /dev/null 2>&1
    iptables -t mangle -D OUTPUT -d $crashplan_addr -p tcp --dport $crashplan_port -j MARK --set-mark 10
elif [ "$1" = "show" ]; then
    #
    # Shows the current traffic shaper configuration.
    #
    tc qdisc show dev $dev
    tc class show dev $dev
    tc filter show dev $dev
    iptables -t mangle -vnL OUTPUT
fi

Puppet, scope, and inheritance

noreply@blogger.com (Lars) — Tue, 16 Aug 2011 11:19:49 PDT

I note this here because it wasn't apparent to me from the Puppet documentation.

If you have a Puppet class like this:

class foo {
  File {  ensure  => file,
          mode    => 600,
          }
}

And you use it like this:

class bar {
  include foo

  file { '/tmp/myfile': }
}

Then /tmp/myfile will not be created. But if instead you do this:

class bar inherits foo {
  file { '/tmp/myfile': }
}

It will be created with mode 0600. In other words, if you use inherits then definitions in the parent class are available in the scope of your subclass. If you include, then definitions in he included class are "below" the scope of the including class.

Fixing rpmsign with evil magic

noreply@blogger.com (Lars) — Tue, 26 Jul 2011 06:20:50 PDT

At my office we are developing a deployment mechanism for RPM packages. The general workflow looks like this:

You build a source rpm on your own machine.
You sign the rpm with your GPG key.
You submit the source RPM to our buildserver.
The buildserver validates your signature and then builds the package.
The buildserver signs the package using a master signing key.

The last step in that sequence represents a problem, because the rpmsign command will always, always prompt for a password and read the response from /dev/tty. This means that (a) you can't easily provide the password on stdin, and (b) you can't fix the problem using a passwordless key.

Other people have solved this problem using expect, but I've opted for another solution which in some ways seems cleaner and in others seems like a terrible idea: function interposition using LD_PRELOAD.

The rpmsign command prompts for (and reads) a password using the getpass() function call. If you look at the getpass(3) man page, you'll see that the function is defined like this:

       #include 
       char *getpass( const char *prompt);

So we start with the following short block of C code:

#include 
#include 

char *getpass( const char *prompt) {
 printf("I ATE YOUR PASSPHRASE.\n");
 return "";
}

This -- when properly loaded -- will replace the standard C library getpass() function with our own version, which simply returns an empty string. This of course means we'll be using a passwordless key, but you could obviously have our replacement function return an actual password instead of an empty string. I would argue that by doing so you would not substantially increase the security of your solution.

Next we create a shared library:

$ cc -fPIC -g   -c -o getpass.o getpass.c
$ ld -shared -o getpass.so getpass.o

And now we perform our magic:

$ LD_PRELOAD=$(pwd)/getpass.so rpmsign --addsign some.src.rpm
I ATE YOUR PASSPHRASE.
Pass phrase is good.

And voila! A solution for operating rpmsign in batch mode.

Installing CrashPlan under FreeBSD 8

noreply@blogger.com (Lars) — Mon, 23 May 2011 07:25:27 PDT

This articles describes how I got CrashPlan running on my FreeBSD 8(-STABLE) system. These instructions by Kim Scarborough were my starting point, but as these were for FreeBSD 7 there were some additional steps necessary to get things working.

Install Java

I had originally thought that it might be possible to run the CrashPlan client "natively" under FreeBSD. CrashPlan is a Java application, so this seemed like a possible solution. Unfortunately, Java under FreeBSD 8 seems to be a lost cause. I finally gave up and just installed Java under Linux.

Set up your Linux compatability environment

The simplest way to do this is to follow the instructions in the FreeBSD Handbook. This will get you a Fedora 10 based Linux userspace, which should be more than sufficient. I'm using a CentOS 5.6 userspace, but for what we're doing it shouldn't matter, modulo some minor differences in paths.

Note that Linux software running in this environment will have a modified view of your filesystem. In particular, /etc will map to /compat/linux/etc, and ZFS filesystems with non-default mountpoints seem to behave oddly (they are accessible, but not necessarily visible before you access them). This may require some workarounds in CrashPlan, depending on what you're trying to back up.

Install Java JRE

I installed a compatible Java environment from the CentOS package repository:

# chroot /compat/linux bash
bash-3.2# yum install java-1.6.0-openjdk
bash-3.2# exit

Install CrashPlan

Install the CrashPlan software

Download CrashPlan for Linux
Unpack the archive (named something like CrashPlan_3.0.3_Linux.tgz)
Change to the CrashPlan-install directory.

Run the following commands:

# export PATH=/compat/linux/usr/lib/jvm/jre-1.6.0-openjdk/bin:$PATH
# /compat/linux/bin/bash install.sh

Install CrashPlan into /usr/local. When prompted for where to locate init scripts ("What directory contains your SYSV init scripts?" and "What directory contains your runlevel init links?"), enter /tmp (because the installed init scripts aren't ideal for your FreeBSD environment -- we'll install our own later on).

Fix Java

The Linux runtime provided by the FreeBSD Linux compatability layer does not include all of the features of recent Linux kernels. In particular, it is missing the epoll* syscalls, which will cause Java to die with a Function not implemented error. The workaround for this is documented in the linux-kernel page on the FreeBSD wiki:

If you run an application in the linux java which wants to use the linux epoll functions (you should see "not implemented" messages in dmesg), you can start java with the argument -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

Install an rc script

Place the following script into /usr/local/etc/rc.d/crashplan:

#!/bin/sh
#

# PROVIDE: crashplan
# REQUIRE: NETWORKING
# KEYWORD: shutdown

. /etc/rc.subr

name="crashplan"
rcvar=`set_rcvar`
start_cmd=crashplan_start
stop_cmd=crashplan_stop

crashplan_start () {
  /compat/linux/bin/bash /usr/local/crashplan/bin/CrashPlanEngine start
}

crashplan_stop () {
  /compat/linux/bin/bash /usr/local/crashplan/bin/CrashPlanEngine stop
}

load_rc_config $name
run_rc_command "$1"

And then add:

crashplan_enable="YES"

To /etc/rc.conf (or /etc/rc.conf.local).

Start CrashPlan

Run:

/usr/local/etc/rc.d/crashplan start

Wait a moment, then run:

/compat/linux/bin/bash /usr/local/crashplan/bin/CrashPlanEngine status

This should verify that CrashPlan is running.

Connect CrashPlan client

Follow the instructions provided by CrashPlan for connecting to a headless CrashPlan desktop.

Signing data with ssh-agent

noreply@blogger.com (Lars) — Mon, 09 May 2011 13:46:17 PDT

This is follow-up to my previous post, Converting OpenSSH public keys.

OpenSSH allows one to use an agent that acts as a proxy to your private key. When using an agent -- particularly with agent forwarding enabled -- this allows you to authenticate to a remote host without having to (a) repeatedly type in your password or (b) expose an unencrypted private key to remote systems.

If one is temtped to use SSH keys as authentication credentials outside of ssh, one would ideally be able to take advantage of the ssh agent for these same reasons.

This article discusses what is required to programmatically interact with the agent and with the OpenSSL libraries for signing data and verifying signatures.

Signing data with ssh-agent

The SSH agent does not provide clients with direct access to an unencrypted private key. Rather, it will accept data from the client and return the signature of the SHA1 hash of the data.

The agent communicates over a unix socket using the ssh agent protocol defined in authfd.h. The Python Paramiko libary (a pure-python implementation of ssh) includes support for interacting with an ssh agent.

Signing data is very simple:

import hashlib
import paramiko.agent

data = 'something to sign'
data_sha1 = hashlib.sha1(data).digest()
a = paramiko.agent.Agent()
key = a.keys[0]
d = key.sign_ssh_data(None, data_sha1)

Internally, the agent computes the SHA1 digest for the data, signs this using the selected key, and returns a signature_blob that varies depending on the key type in use. For an RSA signature, the result format is a series of (length, data) pairs, where the length is encoded as a four-byte unsigned integer. The response contains the following elements:

algorithm name (ssh-rsa)
rsa signature

For example, after signing some data using a 1024-bit private key, the value returned from sign_ssh_data looked like this:

0000000: 0000 0007 7373 682d 7273 6100 0000 8027  ....ssh-rsa....'
0000010: 953c 771c 5ee4 f4b0 9849 c061 0ac2 2adb  .<w.^....I.a..*.
0000020: b53d 2bcb a545 8dbb d582 05e5 a916 6490  .=+..E........d.
0000030: 1b67 3210 9bfc c74d d0ad 5011 394b a3fe  .g2....M..P.9K..
0000040: 96e2 910b bbfd 19cd 73e5 6720 503a 95e1  ........s.g P:..
0000050: 5b8b 63c4 14a3 ec3d bf57 846e f0b4 e66c  [.c....=.W.n...l
0000060: ce5d 6327 6055 b4e2 3c14 c13f 8303 4b1a  .]c'`U..<..?..K.
0000070: 7ce3 9f33 9e7c 7ca4 a97b 506d fa0b a39e  |..3.||..{Pm....
0000080: cb53 befc d725 9cd1 a8af 6042 5ac8 01    .S...%....`BZ..

The first four bytes (0000 0007) are the length of the algorithm name (ssh-rsa). The next field is the length of the signature (0000 0080, or 128 bytes), followed by the signature data. This means we can extract the signature data like this:

parts = []
while d:
    len = struct.unpack('>I', d[:4])[0]
    bits = d[4:len+4]
    parts.append(bits)
    d = d[len+4:]

sig = parts[1]
open('signature', 'w').write(sig)

Signing the data with OpenSSL

Using M2Crypto

You can accomplish the same thing using the M2Crypto library for Python like this:

import hashlib
import M2Crypto.RSA

data = 'something to sign'
data_sha1 = hashlib.sha1(data).digest()
key = M2Crypto.RSA.load_key('testkey')
sig = key.sign(data_sha1)
open('signature', 'w').write(sig)

This assumes that testkey is the private key file corresponding to the first key loaded into your agent in the previous example.

Using the command line

You can also generate an equivalent signature using the OpenSSL command line tools:

echo -n 'something to sign' |
  openssl sha1  -binary |
  openssl pkeyutl -sign -inkey testkey -pkeyopt digest:sha1 > signature

Note that including -pkeyopt digest:sha1 is necessary to get a signature block that is compatible with the one returned by the ssh agent. The pkeyutl man page has this to say:

In PKCS#1 padding if the message digest is not set then the supplied data is signed or verified directly instead of using a DigestInfo structure. If a digest is set then the a DigestInfo structure is used and its the length must correspond to the digest type.

Veryfying the data

You can verify the signature using the corresponding public key.

Using M2Crypto

This uses the M2Crypto module to verify the signature computed in the previous step:

import hashlib
import M2Crypto.RSA

# let's pretend that you've read my previous blog post and have
# created an "sshkey" module for reading the ssh public key format.
import sshkey

data = 'something to sign'
data_sha1 = hashlib.sha1(data).digest()

# read the signature generated in the previous step
sig = open('signature').read()

e,n = sshkey.load_rsa_pub_key('testkey.pub')
key = M2Crypto.RSA.new_pub_key((
    M2Crypto.m2.bn_to_mpi(M2Crypto.m2.hex_to_bn(hex(e)[2:])),
    M2Crypto.m2.bn_to_mpi(M2Crypto.m2.hex_to_bn(hex(n)[2:])),
    ))

if key.verify(data_sha1, sig):
  print 'Verified!'
else:
  print 'Failed!'

If you have converted the ssh public key into a standard format, you could do this instead:

import hashlib
import M2Crypto.RSA

data = 'something to sign'
data_sha1 = hashlib.sha1(data).digest()

# read the signature generated in the previous step
sig = open('signature').read()

key = M2Crypto.RSA.load_pub_key('testkey.pubssl')

if key.verify(data_sha1, sig):
  print 'Verified!'
else:
  print 'Failed!'

Using OpenSSL

We can do the same thing on the command line, but we'll first need to convert the ssh public key into a format useful to OpenSSL. This is easy if you have the private key handy...which we do:

openssl rsa -in testkey -pubout > testkey.pubssl

And now we can verify the signature:

echo 'something to sign' |
  openssl sha1  -binary |
  openssl pkeyutl -verify -sigfile signature \
    -pubin -inkey testkey.pubssl -pkeyopt digest:sha1

Converting OpenSSH public keys

noreply@blogger.com (Lars) — Mon, 09 May 2011 17:21:05 PDT

I've posted a followup to this article that discusses ssh-agent.

For reasons best left to another post, I wanted to convert an SSH public key into a PKCS#1 PEM-encoded public key. That is, I wanted to go from this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD7EZn/BzP26AWk/Ts2ymjpTXuXRiEWIWn
HFTilOTcuJ/P1HfOwiy4RHC1rv59Yh/E6jbTx623+OGySJWh1IS3dAEaHhcGKnJaikrBn3c
cdoNVkAAuL/YD7FMG1Z0SjtcZS6MoO8Lb9pkq6R+Ok6JQjwCEsB+OaVwP9RnVA+HSYeyCVE
0KakLCbBJcD1U2aHP4+IH4OaXhZacpb9Ueja6NNfGrv558xTgfZ+fLdJ7cpg6wU8UZnVM1B
JiUW5KFasc+2IuZR0+g/oJXaYwvW2T6XsMgipetCEtQoMAJ4zmugzHSQuFRYHw/7S6PUI2U
03glFmULvEV+qIxsVFT1ng3pj lars@tiamat.house

To this:

If you have a recent version of OpenSSH (where recent means 5.6 or later), you can just do this:

ssh-keygen -f key.pub -e -m pem

If you don't have that, read on.

OpenSSH Public Key Format

The OpenSSH public key format is fully documented RFC 4253. Briefly, an OpenSSH public key consists of three fields:

The key type
A chunk of PEM-encoded data
A comment

What, you may ask, is PEM encoding? Privacy Enhanced Mail (PEM) is a specific type of Base64 encoding...which is to say it is a way of representing binary data using only printable ASCII characters.

For an ssh-rsa key, the PEM-encoded data is a series of (length, data) pairs. The length is encoded as four octets (in big-endian order). The values encoded are:

algorithm name (one of (ssh-rsa, ssh-dsa)). This duplicates the key type in the first field of the public key.
RSA exponent
RSA modulus

For more information on how RSA works and what the exponent and modulus are used for, read the Wikipedia article on RSA.

We can read this in with the following Python code:

import sys
import base64
import struct

# get the second field from the public key file.
keydata = base64.b64decode(
  open('key.pub').read().split(None)[1])

parts = []
while keydata:
    # read the length of the data
    dlen = struct.unpack('>I', keydata[:4])[0]

    # read in <length> bytes
    data, keydata = keydata[4:dlen+4], keydata[4+dlen:]

    parts.append(data)

This leaves us with an array that, for an RSA key, will look like:

['ssh-rsa', '...bytes in exponent...', '...bytes in modulus...']

We need to convert the character buffers currently holding e (the exponent) and n (the modulus) into numeric types. There may be better ways to do this, but this works:

e_val = eval('0x' + ''.join(['%02X' % struct.unpack('B', x)[0] for x in
    parts[1]]))
n_val = eval('0x' + ''.join(['%02X' % struct.unpack('B', x)[0] for x in
    parts[2]]))

We now have the RSA public key. The next step is to produce the appropriate output format.

PKCS#1 Public Key Format

Our target format is a PEM-encoded PKCS#1 public key.

PKCS#1 is "the first of a family of standards called Public-Key Cryptography Standards (PKCS), published by RSA Laboratories." (Wikipedia). You can identify a PKCS#1 PEM-encoded public key by the markers used to delimit the base64 encoded data:

-----BEGIN RSA PUBLIC KEY-----
...
-----END RSA PUBLIC KEY-----

This is different from an x.509 public key, which looks like this:

-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----

The x.509 format may be used to store keys generated using algorithms other than RSA.

The data in a PKCS#1 key is encoded using DER, which is a set of rules for serializing ASN.1 data. For more information see:

The WikiPedia entry for Distinguished Encoding Rules
A Layman's Guide to a Subset of ASN.1, BER, and DER

Basically, ASN.1 is a standard for describing abstract data types, and DER is a set of rules for transforming an ASN.1 data type into a series of octets.

According to the ASN module for PKCS#1, a PKCS#1 public key looks like this:

RSAPublicKey ::= SEQUENCE {
    modulus           INTEGER,  -- n
    publicExponent    INTEGER   -- e
}

We can generate this from our key data using Python's PyASN1 module:

from pyasn1.type import univ

pkcs1_seq = univ.Sequence()
pkcs1_seq.setComponentByPosition(0, univ.Integer(n_val))
pkcs1_seq.setComponentByPosition(1, univ.Integer(e_val))

Generating the output

Now that we have the DER encoded key, generating the output is easy:

from pyasn1.codec.der import encoder as der_encoder

print '-----BEGIN RSA PUBLIC KEY-----'
print base64.encodestring(der_encoder.encode(pkcs1_seq))
print '-----END RSA PUBLIC KEY-----'

Appendix: OpenSSH private key format

Whereas the OpenSSH public key format is effectively "proprietary" (that is, the format is used only by OpenSSH), the private key is already stored as a PKCS#1 private key. This means that the private key can be manipulated using the OpenSSL command line tools.

The clever folks among you may be wondering if, assuming we have the private key available, we could have skipped this whole exercise and simply extracted the public key in the correct format using the openssl command. We can come very close...the following demonstrates how to extract the public key from the private key using openssl:

$ openssl rsa -in key -pubout
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA+xGZ/wcz9ugFpP07Nspo
6U17l0YhFiFpxxU4pTk3Lifz9R3zsIsuERwta7+fWIfxOo208ett/jhskiVodSEt
3QBGh4XBipyWopKwZ93HHaDVZAALi/2A+xTBtWdEo7XGUujKDvC2/aZKukfjpOiU
I8AhLAfjmlcD/UZ1QPh0mHsglRNCmpCwmwSXA9VNmhz+PiB+Dml4WWnKW/VHo2uj
TXxq7+efMU4H2fny3Se3KYOsFPFGZ1TNQSYlFuShWrHPtiLmUdPoP6CV2mML1tk+
l7DIIqXrQhLUKDACeM5roMx0kLhUWB8P+0uj1CNlNN4JRZlC7xFfqiMbFRU9Z4N6
YwIDAQAB
-----END PUBLIC KEY-----

So close! But this is in x.509 format, and even though the OpenSSL library supports PKCS#1 encoding, there is no mechanism to make the command line tools cough up this format.

Additionally, I am trying for a solution that does not require the private key to be available, which means that in any case I will still have to parse the OpenSSH public key format.

Python ctypes module

noreply@blogger.com (Lars) — Tue, 10 Aug 2010 08:55:31 PDT

I just learned about the Python ctypes module, which is a Python module for interfacing with C code. Among other things, ctypes lets you call arbitrary functions in shared libraries. This is, from my perspective, some very cool magic. I thought I would provide a short example here, since it took me a little time to get everything working smoothly.

For this example, we'll write a wrapper for the standard statvfs(2) function:

SYNOPSIS
       #include <sys/statvfs.h>

       int statvfs(const char *path, struct statvfs *buf);
       int fstatvfs(int fd, struct statvfs *buf);

DESCRIPTION
       The function statvfs() returns information about a mounted file
       system.  path is the pathname of any file within the mounted file
       system.   buf is a pointer to a statvfs structure defined
       approximately as follows:

Note the wording there: "...defined approximately as follows." Our first job is finding out exactly what the statvfs structure looks like. We can use gcc to show us the contents of the appropriate #include file:

echo '#include <sys/statvfs.h>' | gcc -E | less

Browsing through the results, we find the the following definition:

struct statvfs
  {
    unsigned long int f_bsize;
    unsigned long int f_frsize;

    __fsblkcnt_t f_blocks;
    __fsblkcnt_t f_bfree;
    __fsblkcnt_t f_bavail;
    __fsfilcnt_t f_files;
    __fsfilcnt_t f_ffree;
    __fsfilcnt_t f_favail;
    unsigned long int f_fsid;

    unsigned long int f_flag;
    unsigned long int f_namemax;
    int __f_spare[6];
  };

We need to investigate further to determine what __fsblkcnt_t and __fsfilcnt_t really mean. There are a number of ways to do this (see this question at StackOverflow for some suggestions for automating this process). Here's what I did:

$ cd /usr/include
$ ctags -R
$ ex
Entering Ex mode.  Type "visual" to go to Normal mode.
:tag __fsblkcnt_t
"bits/types.h" [readonly] 197L, 7601C
:p
__STD_TYPE __FSBLKCNT_T_TYPE __fsblkcnt_t;
:tag __FSBLKCNT_T_TYPE
"bits/typesizes.h" [readonly] 66L, 2538C
:p
#define __FSBLKCNT_T_TYPE       __ULONGWORD_TYPE
:tag __ULONGWORD_TYPE
"bits/types.h" [readonly] 197L, 7601C
:p
#define __ULONGWORD_TYPE        unsigned long int

Repeat this for __fsfilcnt_t and we find that they are both unsigned long int.

This means that we need to create a ctypes.Structure object like the following:

from ctypes import *

class struct_statvfs (Structure):
    _fields_ = [
            ('f_bsize', c_ulong),
            ('f_frsize', c_ulong),
            ('f_blocks', c_ulong),
            ('f_bfree', c_ulong),
            ('f_bavail', c_ulong),
            ('f_files', c_ulong),
            ('f_ffree', c_ulong),
            ('f_favail', c_ulong),
            ('f_fsid', c_ulong),
            ('f_flag', c_ulong),
            ('f_namemax', c_ulong),
            ('__f_spare', c_int * 6),
            ]

Failure to create the correct structure (e.g., if you're missing fields) can result in a number of weird errors, including segfaults and warnings from gcc about memory corruption.

Now that we have the appropriate structure defined, we need to load up the appropriate shared library:

libc = CDLL('libc.so.6')

And then tell ctypes about the function arguments expected by statvfs():

libc.statvfs.argtypes = [c_char_p, POINTER(struct_statvfs)]

With all this in place, we can now call the function:

s = struct_statvfs()
res = libc.statvfs('/etc', byref(s))
for k in s._fields_:
    print '%20s: %s' % (k[0], getattr(s, k[0]))

We use byref(s) because statvfs() expects a pointer to a structure. This outputs the following on my local system:

 f_bsize: 4096
f_frsize: 4096
f_blocks: 10079070
 f_bfree: 5043632
f_bavail: 4941270
 f_files: 2564096
 f_ffree: 2419876
f_favail: 2419876
  f_fsid: 18446744071962486827
  f_flag: 4096
f_namemax: 255
__f_spare: <__main__.c_int_Array_6 object at 0x7f718fb6b3b0>

Importing vCard contacts into an LG 420G

noreply@blogger.com (Lars) — Sat, 07 Aug 2010 04:29:49 PDT

Alix recently acquired an LG 420G from TracFone. She was interested in getting all of her contacts onto the phone, which at first seemed like a simple task -- transfer a vCard (.vcf) file to the phone via Bluetooth, and the phone would import all the contacts. This turned out to be a great idea in theory, but in practice there was a fatal flaw -- while the phone did indeed import the contacts, it only imported names and the occasional note or email address. There were no phone numbers.

Thus began the long investigation to find out exactly what the phone expected the contacts to look like.

EOL format

The LG 420G needs DOS end-of-line (CRLF), otherwise it won't recognize more than a single contact in your file. There are a number of ways to convert EOL style in a text document; I used the unix2dos tool.

Phone number labels

In our source vCard files, telephone numbers were typically listed with one qualifier, like this:

TEL;WORK:555-555-5555
TEL;HOME:555-555-5555
TEL;CELL:555-555-5555

I created new contact entry on the phone and sent it back to my computer, and found that the telephone numbers were labelled with two qualifiers (and a character set identifier), like this:

TEL;HOME;CELL;CHARSET=UTF-8:1111111111
TEL;WORK;VOICE;CHARSET=UTF-8:2222222222
TEL;HOME;VOICE;CHARSET=UTF-8:3333333333

So, the first thing we needed to do was to transform the phone number entries in our original list into the format expected by the 420G. I extracted a list of all the unique phone number labels, like this:

grep TEL contacts.vcf | cut -f1 -d: | sort -u

Which resulted in this list:

TEL;CELL
TEL;HOME
TEL;HOME;PREF
TEL;PAGER
TEL;PREF
TEL;PREF;CELL
TEL;PREF;FAX
TEL;WORK
TEL;WORK;PREF

I used this to generate the following sed script:

s/TEL;HOME;PREF/TEL;HOME;VOICE/
t
s/TEL;HOME/TEL;HOME;VOICE/
t
s/TEL;CELL/TEL;HOME;CELL/
t
s/TEL;WORK;PREF/TEL;WORK;VOICE/
t
s/TEL;PREF;CELL/TEL;WORK;CELL/
t
s/TEL;PREF;FAX/TEL;WORK;FAX/
t
s/TEL;WORK/TEL;WORK;VOICE/
t
s/TEL;PREF/TEL;WORK;VOICE/
t
s/TEL;PAGER/TEL;WORK;PAGER/
t

Running this over all the contacts gives us the following list of distinct labels:

TEL;HOME;CELL
TEL;HOME;VOICE
TEL;WORK;CELL
TEL;WORK;FAX
TEL;WORK;PAGER
TEL;WORK;VOICE

No dashes in phone numbers (really, LG?)

It turns out that the 420G does not accept anything other than digits in phone numbers. So, after processing the labels with the above sed script we fix up the data with the following awk script:

# Set input and output field separators to ":".
BEGIN {
  FS=":"
  OFS=":"
}

# Remove anything not a digit from the phone number.
/TEL;/ {
  gsub("[^0-9]", "", $2)
}

{
  print
}

And we're off!

With these transformations in place, the LG 420G was able to successfully import the contacts. I automated the whole process with the following Makefile:

SED = gsed
AWK = awk

FILTERS = \
    fix-tel-labels.sed \
    remove-non-digits.awk

all: all-contacts-filtered.vcf

all-contacts-filtered.vcf: all-contacts-orig.vcf $(FILTERS)
  $(SED) -f fix-tel-labels.sed $< | \
    $(AWK) -f remove-non-digits.awk | \
    unix2dos > $@

Patch to gPXE dhcp command

noreply@blogger.com (Lars) — Sat, 07 Aug 2010 04:31:27 PDT

Update: This patch has been accepted into gPXE.

I just released a patch to gPXE that modifies the dhcp command so that it can iterate over multiple interfaces. The stock dhcp command only accepts a single interface as an argument, which can be a problem if you are trying to boot on a machine with multiple interfaces. The builtin autoboot commands attempts to resolve this, but is only useful if you expect to receive appropriate boot parameters from your dhcp server.

My patch extends the dhcp command in the following ways:

It allows the "dhcp" command to accept a list of interfaces and to try them in order until it succeeds, e.g.:
```
gPXE> dhcp net0 net1 net2
```
In order to preserve the original syntax of the command, this will fail on an unknown interface name:
```
gPXE> dhcp foo net0
No such interface: foo
gPXE>
```
The "-c" flag allows it to continue:
```
gPXE> dhcp -c foo net0
No such interface: foo
DHCP (net0 xx:xx:xx:xx:xx:xx).... ok
gPXE>
```

If given the single parameter "any" as an interface name, iterate over all known interfaces in a manner similar to autoboot():

gPXE> dhcp any
DHCP (net0 xx:xx:xx:xx:xx:xx)........ Connection timed out (...)
Could not configure net0: Connection timed out (...)
DHCP (net1 xx:xx:xx:xx:xx:xx).... ok
gPXE>

I think this manages to preserve the syntax of the existing "dhcp" command while making the magic of autoboot available to gpxe scripts.

Linux and Kerberos authenticated queries to Microsoft Active Directory

noreply@blogger.com (Lars) — Tue, 10 Aug 2010 08:31:36 PDT

There are many guides out there to help you configure your Linux system as an LDAP and Kerberos client to an Active Directory server. Most of these guides solve the problem of authentication by embedding a username and password into a configuration file somewhere on your system. While this works, it presents some problems:

If you use a common account for authentication from all of your Linux systems, a compromise on one system means updating the configuration of all of your systems.
If you don't want to use a common account, you need to provision a new account for each computer...
...which is silly, because if you join the system to Active Directory there is already a computer object associated with the system that can be used for authentication.

This document describes how to configure a Linux system such that queries generated by nss_ldap will use either the current user's Kerberos credentials, or, for the root user, credentials stored in a Kerberos credentials cache.

Prerequisites

Your Linux system must have a valid keytab file.

A keytab is a file containing pairs of Kerberos principals and encrypted keys.

Joining Active Directory using Samba's net ads join will create the necessary keytab. It is also possible to create the keytab on your Windows domain controller and install it on your Linux systems. Instructions for doing this are beyond the scope of this document.
Host objects in Active Directory must have a userPrincipalName attribute.

For example:
```
$ ldapsearch cn=dottiness userPrincipalName
dn: CN=DOTTINESS,CN=Computers,dc=example,dc=com
userPrincipalName: host/dottiness.example.com@EXAMPLE.COM
```
Active Directory does not automatically create a userPrincipalName when a new host object is provisioned. You will either need to provide this value manually or develop an automated process that will populate this field when provisioning new host objects.

Renewing host credentials

Kerberos credentials have a maximum usable lifetime. The cached credentials used for root queries by nss_ldap must be refreshed periodically in order to function.

You will need to install a crontab (e.g., in /etc/cron.d) that looks something like this:

PATH=/bin:/usr/bin:/usr/kerberos/bin
@reboot root kinit -k -c /var/run/ldap_cc > /dev/null 2>&1
@hourly root kinit -k -c /var/run/ldap_cc > /dev/null 2>&1

This periodically reauthenticates to your domain controller used the cached principal in the system keytab (/etc/krb5.keytab) and caches the credentials in /var/run/ldap_cc.

Configuration

You will need something similar to the following in /etc/ldap.conf:

# This is your domain controller.
uri ldap://dc1.example.com
base dc=example,dc=com
scope one
referrals no
timelimit 120
bind_timelimit 120
idle_timelimit 3600
ldap_version 3

# Authenticate using SASL for user and root queries.
use_sasl on
rootuse_sasl on

# Use SASL's gssapi (Kerberos) mechanism.
sasl_mech gssapi

# Use these cached credentials for root.
krb5_ccname /var/run/ldap_cc

nss_base_group ou=groups,dc=example,dc=com
nss_base_passwd ou=people,dc=example,dc=com
nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,radvd,tomcat,radiusd,news,mailman,nscd,gdm,polkituser

# These are common mappings for working with Active Directory.
nss_map_attribute uid sAMAccountName
nss_map_attribute uniqueMember member
nss_map_objectclass posixAccount user
nss_map_objectclass posixGroup group
nss_map_objectclass shadowAccount user

pam_login_attribute sAMAccountName
pam_member_attribute member
pam_password ad
pam_password_prohibit_message Please visit http://password.example.com to change your password.

pam_filter objectclass=User

The use_sasl on directive configures nss_ldap to use the Kerberos credentials for the current user when looking up user/group/etc information. The rootuse_sasl on attribute does the same thing for processes running as root.

Note that this configuration sets scope one, which means that nss_ldap will not recurse down a directory tree. This is a performance optimization, not a requirement.

Validation

As an unprivileged user

Before acquiring Kerberos credentials:

$ getent passwd lars
(times out)

Authenticate to Kerberos:

$ kinit
Password for lars@EXAMPLE.COM:

With valid credentials:

$ getent passwd lars
lars:*:500:500:lars:\\emc00.example.com\staff\l\lars\windows:

As root

Before acquiring Kerberos credentials:

# getent passwd lars
(times out)

Update credentials cache from system keytab:

# kinit -k

With valid credentials:

# getent passwd lars
lars:*:500:500:lars:\\emc00.example.com\staff\l\lars\windows:

Caveats

This configuration makes the operation of nss_ldap dependent on valid Kerberos credentials. If a user remains logged in after her Kerberos credentials have expired, she will experience degraded behavior, since many name lookup operations will timeout. Similarly, local system accounts that do not have valid Kerberos credentials will experience similar behavior (and will thus only be able to see local users and groups).

An alternate solution to this problem would run a local LDAP proxy that (a) uses the system keytab to authenticate to AD, and (b) allows anonymous LDAP queries from localhost. I will discuss this configuration in a future post.

Pushing a Git repository to Subversion

noreply@blogger.com (Lars) — Tue, 11 May 2010 18:12:48 PDT

I recently set up a git repository server (using gitosis and gitweb). Among the required features of the system was the ability to publish the git repository to a read-only Subversion repository. This sounds simple in principle but in practice proved to be a bit tricky.

Git makes an excellent Subversion client. You can use the git svn ... series of commands to pull a remote Subversion repository into a local git working tree and then have all the local advantages of git forcing the central code repository to change version control software. An important aspect of this model is that:

The Subversion repository is the primary source of the code, and
You populate your local git repository by pulling from the remote Subversion repository.

It is possible to push a git change history into an empty Subversion repository. Most instructions for importing a git repository look something like this, and involve replaying your git change history on top of the Subversion change history:

svn mkdir $REPO/{trunk, tags, branches}
git svn init -s $REPO
git svn fetch
git rebase trunk
git svn dcommit

This works, and is fine as long as there are no other clones of your git repository out there. The mechanism outlined here has a fatal flaw: it modifies the change history of the master branch. If you were working in a clone of a remote git repository and you were to run git status after the above steps, you would see something like:

# On branch master
# Your branch and 'origin/master' have diverged,
# and have 3 and 2 different commit(s) each, respectively.

If you were then to try to push this to the remote repository, you would get an error:

$ git push
To .../myrepo:
 ! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to '.../myrepo'

In cases where the git change history is shared with other git repositories, we need a solution that does not modify the master branch. We can get this my modifying the procedure slightly.

The initial sequence is still the same:

svn mkdir $REPO/{trunk, tags, branches}
git svn init -s $REPO
git svn fetch

But instead of rebasing onto the master branch, we create a local branch for managing the synchronization:

git checkout -b svnsync
git rebase trunk
git svn dcommit

At this point we have changed the history of the svnsync branch and we have left the master branch untouched. Subsequent updates look like this:

git checkout master
git pull
git checkout svnsync
git rebase master
git rebase trunk
git svn dcommit

This gives us what we want: we can publish our git repository to a Subversion repository while maintaining the shared change history among our existing git clones.

LDAP redundancy through proxy servers

noreply@blogger.com (Lars) — Wed, 24 Feb 2010 07:57:53 PST

Problem 1: Failover

The problem

Many applications only allow you to configure a single LDAP server. This can lead to unnecessary service outages if your directory service infrastructure is highly available (e.g., you are running Active Directory) and your application cannot take advantage of this fact.

A solution

We can provide a level of redundancy by passing the LDAP connections through a load balancing proxy. While this makes the proxy a single point of failure, it is (a) a very simple tool and thus less prone to complex failure modes, (b) running on the same host as the web application, and (c) is completely under our control.

For this example, I will use Balance, a simple TCP load balancer from Inlab Software GmbH. There are packages available for most major Linux distributions, including Fedora and CentOS.

Balance is configured completely on the command line. To provide round-robin access to a suite of three directory servers running LDAP over SSL, you might use the following command line:

balance -b 127.0.0.1 636 10.1.1.1 10.1.1.2

Using balance's terminology, this creates one group of two channels. Balance will round-robin among the channels in this group. Note that here and in subsequent examples we are binding the proxy to the loopback interface so that it is only available to applications running on the same host.

If you would prefer to preferentially send all the requests to the first server, and only use the second server if the first is unavailable, you could use a configuration like this:

balance -b 127.0.0.1 636 10.1.1.1 \! 10.1.1.2

While you can run balance from a standard init (/etc/rc.d/...) script, I prefer to use a service manager such as runit which takes care of restarting the service if it should exit unexpectedly. You could achieve the same thing in a slightly less flexible fashion by putting your balance command line in /etc/inittab. In either case you need to add the -f option to the command line, which causes balance to stay in the foreground.

Problem 2: Debugging LDAP over SSL

The problem

It is convenient to use a packet tracer such as Wireshark to debug LDAP protocol errors. This is often more informative than the debugging information that will be available to you on the client side, and may be more useful than server side debugging in many cases, even supposing that you have administrative access to the directory servers.

A solution

You can use Stunnel, a general purpose SSL proxy tool, to intercept unencrypted client connections on the local machine and then forward them over an SSL channel to a remote server. This makes the unencrypted LDAP traffic available on the loopback interface while still ensuring that it is encrypted on the wire.

Stunnel can operate both as an SSL server and as an SSL client. In this case, we will be running it in client mode, connecting to a remote SSL server (or to the proxy configured in our previous example). Stunnel is configured by means of a simple INI-style configuration file. To achieve the goals of this example we might put the following configuration in a file (say, stunnel.conf):

[ldap]

accept = 127.0.0.1:389
client = yes
connect = localhost:636

We would run stunnel like this:

stunnel /path/to/stunnel.conf

Again, I would run this under the control of a service supervisor. To keep stunnel in the foreground we would add the following to the global section of the configuration file (i.e., before the [ldap] section marker):

foreground = yes

With both of these solutions in place, we have achieved the following:

High availability.

Our application will transparently make use of multiple directory servers. If a server fails, our application will continue to operate.
Security

Our traffic is encrypted on the wire, regardless of whether the application has support for LDAP over SSL.
Visibility

We are free to examine unencrypted traffic with a packet sniffer running on the local host.

Apache virtual host statistics

noreply@blogger.com (Lars) — Fri, 19 Feb 2010 18:29:20 PST

As part of a project I'm working on I wanted to get a rough idea of the activity of the Apache virtual hosts on the system. I wasn't able to find exactly what I wanted, so I refreshed my memory of curses to bring you vhoststats.

This tools reads an Apache log file (with support for arbitrary formats) and generates a dynamic bar chart showing the activity (in number of requests and bytes transferred) of hosts on the system. The output might look something like this (but with colors):

[2010/02/19 20:21:32] Hosts: 7 [Displayed: 7] Requests: 104

host1.companyA.com   [R:1         ]  #
                     [B:3         ]
devel.internal       [R:1         ]  #
                     [B:208       ]
host2.companyA.com   [R:1         ]  #
                     [B:4499      ]
A-truncated-host-nam [R:10        ]  ############
                     [B:65380     ]  #
host1.companyB.com   [R:21        ]  ##########################
                     [B:166715    ]  ####
www.google.com       [R:32        ]  #################################
                     [B:1566614   ]  ####################################

The tool keeps running totals over a five minute window, but you can change the window size on the command line. You can tail your active access log to see live results, or for a more exciting display you can just pipe in an existing log.

It's not pong, but I've found it useful.

You can download the code from the project page on GitHub.

Group filters for Gmail

noreply@blogger.com (Lars) — Fri, 19 Feb 2010 18:05:22 PST

I've recently spent some time trying to make my Gmail filters more useful. I was frustrated (as are some other people) that it's not possible to refer to contact groups in email filters.

Rather than laboriously duplicate my contact groups as hand-written filter expressions, I have written a short script that will query Google Contacts for all of your contact groups and then emit an XML document suitable for import using Gmail's "import filters" feature.

The code is available in the gmail-contact-filters project on GitHub.

If I am sufficiently motiviated I may turn this into a web service.

Merging directories with OpenLDAP's Meta backend

noreply@blogger.com (Lars) — Wed, 17 Feb 2010 12:56:23 PST

This document provides an example of using OpenLDAP's meta backend to provide a unified view of two distinct LDAP directory trees. I was frustrated by the lack of simple examples available when I went looking for information on this topic, so this is my attempt to make life easier for the next person looking to do the same thing.

The particular use case that motiviated my interest in this topic was the need to configure web applications to (a) authenticate against an existing Active Directory server while (b) also allowing new accounts to be provisioned quickly and without granting any access in the AD environment. A complicating factor is that the group managing the AD server(s) was not the group implementing the web applications.

Assumptions

I'm making several assumptions while writing this document:

You have root access on your system and are able to modify files in /etc/openldap and elsewhere on the filesystem.
You are somewhat familiar with LDAP.
You are somewhat familiar with OpenLDAP.

Set up backend directories

Configure slapd

We'll first create two "backend" LDAP directories. These will represent the directories you're trying to merge. For the purposes of this example we'll use the ldif backend, which stores data in LDIF format on the filesystem. This is great for testing (it's simple and easy to understand), but not so great for performance.

We define one backend like this in /etc/openldap/slapd-be-1.conf:

database        ldif
suffix          "ou=backend1"
directory       "/var/lib/ldap/backend1"
rootdn          "cn=ldif-admin,ou=backend1"
rootpw          "LDIF"

And a second backend like this in /etc/openldap/slapd-be-2.conf:

database        ldif
suffix          "ou=backend2"
directory       "/var/lib/ldap/backend2"
rootdn          "cn=ldif-admin,ou=backend2"
rootpw          "LDIF"

Now, we need to load these configs into the main slapd configuration file. Open slapd.conf, and look for the following comment:

#######################################################################
# ldbm and/or bdb database definitions
#######################################################################

Remove anything below this comment and then add the following lines:

include /etc/openldap/slapd-be-1.conf
include /etc/openldap/slapd-be-2.conf

Start up slapd

Start up your LDAP service:

# slapd -f slapd.conf -h ldap://localhost/

And check to make sure it's running:

# ps -fe | grep slapd
root 15087 1 0 22:48 ? 00:00:00 slapd -f slapd.conf -h ldap://localhost/

Populate backends with sample data

We need to populate the directories with something to query.

Put this in backend1.ldif:

dn: ou=backend1
objectClass: top
objectClass: organizationalUnit
ou: backend1

dn: ou=people,ou=backend1
objectClass: top
objectClass: organizationalUnit
ou: people

dn: cn=user1,ou=people,ou=backend1
objectClass: inetOrgPerson
cn: user1
givenName: user1
sn: Somebodyson
mail: user1@example.com

And this in backend2.ldif:

dn: ou=backend2
objectClass: top
objectClass: organizationalUnit
ou: backend2

dn: ou=people,ou=backend2
objectClass: top
objectClass: organizationalUnit
ou: people

dn: cn=user2,ou=people,ou=backend2
objectClass: inetOrgPerson
cn: user2
givenName: user2
sn: Somebodyson
mail: user2@example.com

And then load the data into the backends:

ldapadd -x -H ldap://localhost -D cn=ldif-admin,ou=backend1 \
  -w LDIF -f backend1.ldif

And:

ldapadd -x -H ldap://localhost -D cn=ldif-admin,ou=backend2 \
  -w LDIF -f backend2.ldif

You can verify that the data loaded correctly by issuing a query to the backends. E.g.:

ldapsearch -x -H ldap://localhost -b ou=backend1 -LLL

This should give you something that looks very much like the contents of backend1.ldif. You can do the same thing for backend2.

Set up meta database

We're now going to configure OpenLDAP's meta backend to merge the two directory trees. Complete documentation for the meta backend can be found in the slapd-meta man page.

Put the following into a file called slapd-frontend.conf (we'll discuss the details in moment):

database        meta
suffix          "dc=example,dc=com"

uri             "ldap://localhost/ou=backend1,dc=example,dc=com"
suffixmassage   "ou=backend1,dc=example,dc=com" "ou=backend1"

uri             "ldap://localhost/ou=backend2,dc=example,dc=com"
suffixmassage   "ou=backend2,dc=example,dc=com" "ou=backend2"

And then add to slapd.conf:

include /etc/openldap/slapd-frontend.conf

Restart slapd. Let's do a quick search to see exactly what we've accomplished:

$ ldapsearch -x -H 'ldap://localhost/' \
  -b dc=example,dc=com objectclass=inetOrgPerson -LLL
dn: cn=user1,ou=people,ou=backend1,dc=example,dc=com
objectClass: inetOrgPerson
cn: user1
givenName: user1
sn: Somebodyson
mail: user1@example.com

dn: cn=user2,ou=people,ou=backend2,dc=example,dc=com
objectClass: inetOrgPerson
cn: user2
givenName: user2
sn: Somebodyson
mail: user2@example.com

As you can see from the output above, a single query is now returning results from both backends, merged into the dc=example,dc=com hierarchy.

A closer look

Let's take a closer look at the meta backend configuration.

database        meta
suffix          "dc=example,dc=com"

The database statement begins a new database definition. The suffix statement identifies the namespace that will be served by this particular database.

Here is the proxy for backend1 (the entry for backend2 is virtually identical):

uri             "ldap://localhost/ou=backend1,dc=example,dc=com"
suffixmassage   "ou=backend1,dc=example,dc=com" "ou=backend1"

The uri statement defines the host (and port) serving the target directory tree. The full syntax of the uri statement is described in the slapd-meta man page; what we have here is a very simple example. The naming context of the URI must fall within the namespace defined in the suffix statement at the beginning of the database definition.

The suffixmassage statement performs simple rewriting on distinguished names. It directs slapd to replace ou=backend1,dc=example,dc=com with ou=backend1 when communicating with the backend directory (and vice-versa).

You can perform simple rewriting of attribute and object classes with the map statement. For example, if backend1 used a sAMAccountName attribute and our application was expecting a uid attribute, we could add this after the suffixmassage statement:

map attribute uid sAMAccountName

Conclusion

The sample configuration files, data, and code referenced in this post are available online in a github repository:

http://github.com/larsks/OpenLDAP-Metadirectory-Example

I hope you've found this post useful, or at least informative. If you have any comments or questions regarding this post, please log them as issues on GitHub. This will make them easier for me to track.

Review: AppBrain Market Sync

noreply@blogger.com (Lars) — Sun, 14 Feb 2010 18:36:44 PST

AppBrain is a web site that lets you browse applications available from the Android Market. AppBrain Market Sync is a companion application that synchronizes your Android phone with your activities on the web site. This means you can:

Select mutliple applications to install on the website, then use AppBrain Market Sync on your phone to install them in one operation.
Select multiple applications to uninstall on the website, then use AppBrain Market Sync to uninstall them.

I like the model -- I find it easier browse the market from my laptop than on my phone, so this is generally a more convenient way to find applications. The fact that applications are actually installed from the Market avoids the security/reliability questions involved when installing applications from "alternative" sources.

Review: ShapeWriter Input Method

noreply@blogger.com (Lars) — Sun, 14 Feb 2010 18:28:01 PST

ShapeWriter is a novel input mechanism for mobile devices that uses the shape you create while sliding your fingers from letter to letter to accurately recognize words. Instead of pecking out letters on a virtual keyboard, you slide your finger (or thumb, more likely) from letter to letter in a smooth, continuous motion. While on the iPhone it is packaged as a note-taking application, on Android devices it is implemented as an input method, which means it is available pretty much anywhere you can enter text.

I've been using ShapeWriter on my Droid for several days now, and I've come to like it quite a bit. It's advantages become increasingly apparent on longer words: while mistakes can add up while "thumb typing" with a regularly keyboard, the unique shape formed as you slide along the letters in longer words greatly increases ShapeWriter's accuracy. ShapeWriter doesn't attempt to automatically capitalize words; a "cycle" button on the right of the keyboard cycles between all lower case, first letter capital, or all caps.

Since our fingers are seldom perfectly accurate, ShapeWriter uses a dictionary of words to help disambiguate among multiple possibilities. This is usually useful, although occasionally it makes inexplicably bad choices.

I've found that, overall, ShapeWriter offers increased speed and accuracy over the default virtual keyboard.

Ksplice for RedHat

noreply@blogger.com (Lars) — Wed, 10 Feb 2010 06:54:50 PST

Ksplice, which offers "rebootless" kernel upgrades by performing in-memory patches to your running kernel, now offers their services for RHEL 4 & 5 and the equivalent CentOS variants.

Ksplice was originally developed at MIT, and was later turned into a for-profit venture. Ksplice has raised a small amount of controversy on the linux-kernel mailing list because it is, after all, patching your running kernel. In memory. This makes people a little nervous, but the author, Jeff Arnold, has tried to address many of the concerns.

I think the technology looks interesting. Certainly in the case of known kernel exploits it seems like it offers a happy medium between downtime for maintenance vs. not patching the system at all.

Filtering Blogger feeds

noreply@blogger.com (Lars) — Tue, 09 Feb 2010 22:12:55 PST

After encountering a number of problems trying to filter Blogger feeds by tag (using services like Feedrinse and Yahoo Pipes), I've finally put together a solution that works:

Shadow the feed with Feedburner.
Enable the Convert Format Burner, and convert your feed to RSS 2.0.
Use Yahoo Pipes to filter the feed (because Feedrinse seems to be broken).

This let me create a feed that excluded all my posts containing the fbpost tag, thus allowing me to avoid yet another postgasm in Facebook when adding new import URL to notes.

While fiddling with this I came across this article that discusses a number of tools (some no longer available) for processing RSS feeds.

Funny usage message

noreply@blogger.com (Lars) — Tue, 09 Feb 2010 17:57:00 PST

I was poking around in a command shell on my Droid to see what was available. While it's a pretty restricted environment, there's a number of commands available in /system/bin, including dexopt.

Apparently dexopt isn't something I'm supposed to poke at:

$ dexopt
Usage: don't use this

Hah.

Review: RealCalc calculator for Android

noreply@blogger.com (Lars) — Tue, 09 Feb 2010 17:56:40 PST

RealCalc is a scientific calculator for your Android powered phone. It offers a well designed interface that looks pretty much like what you would expect from a scientific calculator. The keys are large enough to not feel cramped.

I spent my college years with an HP-48 calculator, which means I learned to love RPN notation. RealCalc has an RPN mode that you can turn on from the settings screen:

It's a substantial improvement on the native Calculator application. My only complaint is that, once installed, I have two applications named "Calculator" and no easy way to differentiate them.

RealCalc is available for free from the Market.

Review: Bedside clock for Android

noreply@blogger.com (Lars) — Tue, 09 Feb 2010 17:56:40 PST

Bedside is a simple nighttime clock for your Android based phone. What differentiates it from other similar applications are its elegeant implementation and one or two particularly useful features that really make it stand apart.

My favorite feature is the simple fact that it can override your phone's default lock screen. With this option enabled, I can turn off my phone, and turn it on again later and still see the pleasantly dim clock, rather than the full brightness of the normal "slide the lock to use the phone" screen. This behavior is exactly what I want in something I'm using as a nighttime clock.

Bedtime can also make sure your phone is silent when the clock is activated. Again, this seems like a no-brainer, at least as an option.

There are number of other useful features:

Bedtime works great in both portrait and landscape mode.
You can select different options for font and color.
There is a text-to-speech option; when enabled, a long press on the screen will speak the time.

Bedtime costs $1.79 from the Market.

MBTA realtime XML feed

noreply@blogger.com (Lars) — Tue, 09 Feb 2010 17:56:40 PST

The MBTA has a trial web service interface that provides access to realtime location information for select MBTA buses, as well as access to route information, arrival prediction, and other features. More information can be found here:

http://www.eot.state.ma.us/developers/realtime/

The service is provided by NextBus, which specializes in real-time location information for public transit organizations. The API (sorry, PDF) is very simple and does not require any sort of advance registration.

At the moment, the service only provides coverage for a small number of routes (39, 111, 114, 116, 117). I hope they expand the coverage of this service in the near future!