MySQL shebang

2011-09-08T17:45:00.000+02:00

The mysql command line client cannot be used in a script's shebang line like many other languages such as bash and perl and etc allow. Patches have been submitted, but so far none of them have been accepted.

A quick work-around is to create a simple wrapper script like this:

If you make that executable and put it in your PATH, then you can do something simple like this:

Note that this also allows you pass any additional command line options and they will be passed through to mysql:

connections.sql -u mysqluser -p

Threshold-based clustering in Perl

2011-06-28T15:45:00.000+02:00

There are many scenarios where you need to cluster similar data into groups. I'm working with protein sequences and protein structures and have been using Algorithm::Cluster which is a small Perl wrapper (written in Perl XS) around the C Clustering Library which was derived from "the famous Cluster program originally written by Michael Eisen while at Stanford University", the manual to which can be found here (PDF). It supports many types of clustering, "hierarchical clustering (pairwise simple, complete, average, and centroid linkage), along with k-means and k-medians clustering, and 2D self-organizing maps."

What it lacks, however, is a way to set a distance (or similarity) threshold between objects from different clusters. For example, I have a set of protein sequences (which are character strings with some additional attributes) that I'm reading with BioPerl. Algorithm::Cluster will do hierarchical clustering and give me a tree.

I'm using Bio::Tools::Run::Alignment::Clustalw to measure the distance between protein sequences. Actually, I'm measuring the similarity, but that's simply the additive inverse. I create a small convenience module Algorithm::DistanceMatrix that will quickly create the lower diagonal distance matrix required by Algorithm::Cluster just by defining a distance function.

Now to get clusters out of the tree, we simply do something like:

But how do you know how many clusters you want? Determining the number of clusters in a data set is a hard problem. If the tree looks like this

you can see that the cut line required to produce three clusters is quite crowded by other nodes in the tree, whereas there is a more natural break in the middle of the tree. Moreover, in general, the type of data that you are clustering will determine how you want to partition your data into clusters. In this case, I wanted to define clusters such that proteins in one cluster were less than 75% similar (distance > 25) to those in another cluster (according the average linkage metric).

I first asked this question on Biostar (StackOverflow for computational biology) before creating a patch to Algorithm::Cluster to implement a cutthresh() method. It's implemented in XS and should be reasonably efficient, but it has not been reviewed yet. In the meantime, I've created a pure Perl implementation in Algorithm::Cluster::Thresh that works simply like

Whether with cut() or with cutthresh() you can finally have a look at what's in each cluster with something like

Issues can be reported on the GitHub issue tracker.

CyanogenMod 6 on the Motorola Milestone

2011-03-08T23:45:00.002+01:00

I was happy to hear that others have gotten a workable Android 2.2 on the Motorola Milestone, from CyanogenMod. There were a couple little tricks, so I wanted to document those here.

I've had this for a little over a day now, and I'm very happy with the improved performance. The home screen is more responsive, the keyboard is much more responsive that it was with Android 2.1. It's running at 900 Mhz with JIT (just-in-time compilation enabled). (I think Milestone was originally running at around 550 Mhz or so, if I'm not mistaken). I installed "Quandrant" from the market to benchmark it, and its just better than the average Samsung Galaxy S, which is much better than the performance I had on the official 2.1update1 (which was around that of the average "Droid", as you might expect).

CyanogenMod is very customizable. It uses the ADW home app and comes with some very useful utilities, including a clever "Toggle2G" app that allows you to dynamically switch from 3G to 2G when th phone is idle to save battery. I have also installed "Set CPU" from the market, which allows the CPU to be clocked down to about 300 Mhz when it's idle to save battery. I have not done exhaustive battery testing, but I get more than 24 hrs from a single charge when I use the phone occasionally throughout the day (a bit of browsing, phoning, GPS, calendar, etc).

Root it

The excellent OpenRecovery (adapted for Milestone) is the central tool. Once you've downloaded the zip file to your computer, extract the contents and copy them to your phone (the root directory). This will include the OpenRecovery directory itself, an update.zip, and an SBF image, titled vulnerable_recovery.sbf.

Before you can start putting custom firmware (i.e CyanogenMod) on your phone, you need root access. If you already have Android 2.2, or probably even if you have the 2.1update1, your phone is probably not able to be rooted. So, first you have to flash an SBF image that is vulnerable, in order to allow the phone to be rooted. This is that SBF file included in OpenRecovery. Unfortunately, OpenRecovery cannot apply the image. So, I downloaded RSD Lite (from Motorola) for Windows. Start the phone in bootloader mode (Hold the D-pad up button while pressing power, then release power, until "OK to program" appears). Then connect the phone to the computer and start RSD Lite, select the vulnerable_recovery.sbf that was included in OpenRecovery and flash it to the phone. But, now this is the tricky part, do not let it reboot into Android. RSD Lite will report that the process is only about 40% complete when the phone is already rebooting by itself (and RSD Lite just keeps counting even when the phone is disconnected). Once the phone has powered itself off and back on, disconnect it from the computer and pull the battery immediately. Do not let it reboot.

The reason is that during boot, the Milestone will erase the back door that you've created. So, you need to be sure to boot into recovery mode instead (hold 'x' when powering on, release power and wait for the recovery logo. Then to display the menu, press the volume-up and camera buttons simultaneously). Select the menu option (using the D-pad) to apply the update.zip (which was from OpenRecovery). This will then load and now you are in the OpenRecovery menu. Take a short breath.

First make a backup of your existing system. Select nandroid from the OpenRecovery menu and make a full backup. You can always restore this backup by booting into OpenRecovery any time in the future. This will get you back to the beginning if you install some new firmware and decide you don't like it.

Install CyanogenMod

Go back to the OpenRecovery main menu, select the menu option to root your phone. Now your phone is rooted. It is safe to reboot. You can continue to use whatever firmware was on your phone before. All of your settings are retained at this point, but now you also have root access. If you need to use OpenRecovery again (e.g. to install CyanogenMod), you will again need to boot into recovery mode, do all of the button combinations, apply the update.zip before you can reach OpenRecovery.

I downloaded the Milestone port of CyanogenMod6 (version 7 is apparently not yet recommended for general consumption). This is a zip file, which you should not unzip, but you can rename it to something shorter like update-cm-6.zip. This file is a backup (made by Nandroid), which you can effectively restore to your phone, thereby installing CyanogenMod6. The boot.img included does not work with my Milestone kernel, however. So, I had to get the older version 0.04. Now unzip that bootimg.zip ; you will need to replace the boot.img that is in the CyanogenMod6 zip file. (I simply unzipped, overwrote boot.img and rezipped, but the zip format that I created was not usable.) So, instead, you should simply add the version 0.04 boot.img to the zip archive (which will overwrite the original boot.img that is in the archive). In Linux (and probably Mac), that's just:
zip update-cm-6.zip boot.img

Now you're ready to install CyanogedMod6. Copy the now-updated (or actually down-graded) update-cm-6.zip to your phone into the /OpenRecovery/updates directory on your SD card. Reboot the phone into recovery mode (as above) and start OpenRecovery again. Select Nandroid to restore a backup. You will see there your original backup, as well as the update-cm-6.zip, which you can now select to restore, thereby overwriting your old system with the CyanogenMod firmware.

Now CyanogenMod6 is installed, but before rebooting, you should cleanup the phone. OpenRecovery has options on the main menu to clear all the caches on the phone, as well as all of your personal settings, which is recommended (this is the same as Android's "Factory reset"). It's not that much work to reconfigure your applications and will probably make for a smoother experience. Of course, things like bookmarks can easily be restored. Note, it was not necessary for me (in Germany) to backup any of the APN (i.e. mobile network provider) settings.

Now you can restart. You will notice that you have no Google account. Due to copyright restrictions, that is a separate download. Once you've got that zip file, just put it into the same /OpenRecovery/updates folder (do you sense a pattern yet?) and use OpenRecovery to apply it. In fact, I didn't, but you could probably install CyanogenMod6 and the Google apps without rebooting in between, if you simply copy both of the zip files to /OpenRecovery/updates before you even flash the phone with the SBF.

Issues

I haven't tested many things yet: camera, camcorder, bluetooth headphones, but will update if issues come up. I haven't tried flash (it's in the market), but I don't even use that on my laptop, as it kills the battery there already.

The only issue I've discovered so far is the auto-suggest on the keyboard does not correctly replace words. If you are still typing the word, the auto-suggest list shows alternatives, you can select one, and the word will be correctly replaced. However, if you put the cursor on word farther back in the sentence, alternatives will also be shown. However, selecting an alternative at this point simply inserts the word, in addition to the original word that is still there and does not get deleted.

I have not, but some have reported that Facebook sync can get stuck and drain the battery. But, you can check it.

Update: bluetooth headphones are working, at least for music, fine. The new camera app has some nice new features. The camera app itself is responsive. However, starting and leaving the camera app can be slow; I notice this back on Android 2.1 already, though. On one occasion the camera froze the phone and I had to pull the battery, but the problem did not reoccur. This also happened with Android 2.1, though only a couple times in the 1.5 years that I was on Android 2.1. I also had one case where the battery drained (90% to 0%) in less than eight hours in airplane mode. This should not happen, as this mode doesn't send or receive any 3G/2G/wifi/GPS/bluetooth or other signals. This may have been related to an app trying to sync without having a network connection (e.g. the Facebook issue mentioned above). However, this was a one-time incident and I was not able to reproduce the problem. On a side note, the Facebook app in the market was updated today, but that may be unrelated.

Still haven't tested: Phone calls on the bluetooth headset.

vorcidel

MySQL shebang

Threshold-based clustering in Perl

CyanogenMod 6 on the Motorola Milestone