Nilvec

Nginx and git via HTTP

2012-01-15T10:41:35Z

Nginx is a high-performance HTTP server and proxy that has been gaining a lot of traction recently. It’s easy to set up, can handle a large number of clients with minimal resources, modular and supports SSL/TLS. Dynamic web content can be served via FastCGI to Nginx.

Git uses WebDAV for pushing via HTTP(S). Even though Nginx handles WebDAV requests, not all DAV commands are implemented in its DAV module. Locking and PROPFIND are missing in particular, and since these commands are used by Git when pushing via HTTP, pushing to a plain Nginx server is not possible at the moment.

One solution is to run Apache on a different port and proxy Git requests to it. However, Apache is such a resource hog that if a lightweight solution is desired it’s out of question. Here comes PyWebDAV, a standalone WebDAV server written in Python, into the picture.

PyWedDAV is available in both Ubuntu and Debian, so just install it via apt-get:
$ sudo apt-get install python-webdav

The actual binary for running the server will be at /usr/bin/davserver. The packages have no init script, but it’s easy to create one, or you can just shove it at the end of /etc/rc.local:
/usr/bin/sudo -u www-data /usr/bin/davserver -D /var/www/git -n -P 8008 -i 0 -J -d start exit 0

This will start davserver on port 8008 running as user www-data.

It’s now only a matter of making Nginx proxy git requests to PyWebDAV:

server {

...

    location /git {
        proxy_pass       http://localhost:8008/;
        proxy_redirect   http://localhost:8008 http://git.mysite.com;
    }

}

This assumes you store your git repositories under /var/www/git.

However, trying to git push to this server reveals something is still not working as expected:
$ git push error: cannot lock existing info/refs error: failed to push to 'http://git.mysite.com/git/foobar.git'

Even though davserver returns 200 for a LOCK operation, Git checks that the lock timeout is indeed correct in the reply. PyWebDAV by default uses ‘Seconds-Infinite’ for timeouts, but Git expects a numerical timeout value. This can be fixed in /usr/share/pyshared/DAV/locks.py, just change ‘Infinite’ at its two occurences to some sane default value. Here’s a diff to make it easier for the lazy:

--- /usr/lib/python2.6/dist-packages/DAV/locks.py       2010-04-15 11:25:35.000000000 +0200
+++ locks.py    2012-01-15 11:05:18.406041553 +0100
@@ -169,7 +169,7 @@
                     token = tokenFinder(listitem)
                     if token and self._l_hasLock(token):
                         lock = self._l_getLock(token)
-                        timeout = self.headers.get('Timeout', 'Infinite')
+                        timeout = self.headers.get('Timeout', '3600')
                         lock.setTimeout(timeout) # automatically refreshes
                         found = 1

@@ -188,7 +188,7 @@
     """ Lock with support for exclusive write locks. Some code taken from
     webdav.LockItem from the Zope project. """

-    def __init__(self, uri, creator, lockowner, depth=0, timeout='Infinite',
+    def __init__(self, uri, creator, lockowner, depth=0, timeout='3600',
                     locktype='write', lockscope='exclusive', token=None, **kw):

         self.uri = uri

Just save it as locks.diff, apply the patch with patch -p0 < locks.diff, restart davserver and now git push should be working fine.

Python-iptables

2011-11-02T23:34:21Z

Python-iptables is a Python project that provides bindings to the iptables C libraries in Linux. Interoperability with iptables is achieved using the iptables C libraries (libiptc, libxtables, and iptables extensions), not calling the iptables executable and parsing its output as most other iptables wrapper libraries do; this makes python-iptables faster and not prone to parsing errors, at the same time leveraging all available iptables match and target extensions without further work. If you need further information on how iptables and netfilter work, this is a great summary.

Iptables is organized into a hierarchy: the basic building blocks are tables which contain chains, and chains consist of the actual rules that tell netfilter what to do with network packets. A rule contains at least one (possibly more) match(es) and a target. There are a few default tables and chains, but users can create further chains to organize their rules into blocks. A rule can also jump into another chain (kind of like calling a subroutine).

Tables, chains and rules (image from thegeekstuff.com)

Python-iptables makes it very easy to programatically manipulate or alter chains and rules. E.g. to traverse all rules in the INPUT chain in the FILTER table:

>>> import iptc >>> table = iptc.TABLE_FILTER >>> inp = iptc.Chain(table, "INPUT") >>> for r in inp.rules: ... print r.src, "->", r.dst, "L4:", r.protocol, ... for m in r.matches: ... print "match:", m.name, ... print "target:", r.target.name ... 192.168.1.0/255.255.255.0 -> 192.168.2.0/255.255.255.0 L4: udp match: udp target: ACCEPT 192.168.1.0/255.255.255.0 -> 192.168.2.0/255.255.255.0 L4: tcp match: tcp target: ACCEPT >>>

You are of course free to create new rules:

>>> import iptc >>> rule = iptc.Rule() >>> rule.dst = "192.168.100.1" >>> rule.protocol = "tcp" >>> rule.in_interface = "eth0" >>> match = iptc.Match(rule, "tcp") >>> match.dport = "8000" >>> rule.add_match(match) >>> target = iptc.Target(rule, "REDIRECT") >>> rule.target = target >>> chain = iptc.Chain(iptc.TABLE_NAT, "iptc_test_redirect") >>> iptc.TABLE_NAT.create_chain(chain) >>> target.reset() >>> target.to_ports = "1234-1235" >>> rule.target = target >>> chain.insert_rule(rule) >>> for r in chain.rules: ... print r.src, "->", r.dst, "L4:", r.protocol, ... for m in r.matches: ... print "match:", m.name, ... print "target:", r.target.name ... 0.0.0.0/0.0.0.0 -> 192.168.100.1/255.255.255.255 L4: tcp match: tcp target: REDIRECT >>> chain.delete_rule(rule) >>> chain.flush() >>> chain.delete() >>>

This is equivalent to the following command line sequence:

# iptables -t nat -N iptc_test_redirect # iptables -t nat -A iptc_test_redirect -d 192.168.100.1 -i eth0 -p tcp --dport 8000 -j REDIRECT --to-ports 1234-1235 # iptables -t nat -F iptc_test_redirect # iptables -t nat -X iptc_test_redirect

You can find detailed API documentation and further examples here. The source code is available on Github.

New Android Market countries

2011-11-02T23:37:03Z

On 11th May 2011 Google announced the availability of paid apps in 99 new countries.

The list of countries where now you can purchase Android applications from the Market:

Albania
Algeria
Angola
Antigua and Barbuda
Armenia
Aruba
Azerbaijan
Bahamas
Bahrain
Bangladesh
Belarus
Belize
Benin
Bolivia
Bosnia and Herzegovina
Botswana
Bulgaria
Burkina Faso
Cambodia
Cameroon
Cape Verde
Chile
Colombia
Costa Rica
Cote d’ Ivore
Croatia
Cyprus
Dominican Republic
Ecuador
El Salvador
Estonia
Fiji
Gabon
Ghana
Greece
Guatemala
Guinea-Bissau
Haiti
Honduras
Hungary
Iceland
Indonesia
Jamaica
Jordan
Kazakhstan
Kenya
Kuwait
Kyrgyzstan
Laos
Latvia
Lebanon
Lithuania
Luxembourg
Macedonia
Malaysia
Mali
Malta
Mauritius
Moldova
Morocco
Mozambique
Namibia
Nepal
Netherlands Antilles
Nicaragua
Niger
Nigeria
Oman
Pakistan
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Qatar
Romania
Rwanda
Senegal
Serbia
Slovakia
Slovenia
South Africa
Sri Lanka
Tajikistan
Tanzania
Thailand
Togo
Trinidad and Tobago
Tunisia
Turkmenistan
Uganda
Ukraine
Uruguay
Uzbekistan
Venezuela
Vietnam
Yemen
Zambia
Zimbabwe

Android emoticons on Ubuntu with Empathy

2011-02-17T11:45:52Z

The default Ubuntu IM client has now been Empathy for a while. Though it supports themes, replacing the stock emoticons (which come from Gnome), unlike in Pidgin, is not a straightforward process. Which is kind of embarrassing, since the default smileys in Gnome are far from being pretty and eye-candy.

Fortunately, you can override the system emoticons for your user: just create the proper directory structure under ~/.local, and applications will use the emoticons found under ~/.local/share/icons/gnome/16×16/emotes, where 16×16 is the resolution they need.

To make it easy for Ubuntu users, we just packed the Android emoticons into a ready to use package. Once you downloaded it, untar the package into your home folder:
$ cd ~ $ tar xvzf ~/Downloads/android-emoticons.tar.gz

It will create all the necessary folders under .local, and place the smileys in different resolutions there.

Implementing SMTP or IMAP XOAUTH authentication in Java

2011-02-10T12:00:06Z

XOAUTH is a SASL authentication mechanism that can be used with the IMAP AUTHENTICATE and SMTP AUTH commands. It is based on OAuth, and uses the same OAuth parameters to authenticate against an SMTP or IMAP server. It is backed by Google, and you can use XOAUTH with Gmail SMTP or IMAP to authenticate without an username and its corresponding password.

You need to go through the usual “OAuth dance” to acquire an access token and its corresponding secret. Once you have that, you can generate the OAuth signature the usual OAuth way, with the following parameters:

The HTTP method is GET,
The URL has the following form: https://mail.google.com/mail/b/[user email address]/[protocol]/, where protocol is either “smtp” or “imap” (without quotes) and
All the usual OAuth parameters except “oauth_signature”.

The actual XOAUTH string will be constructed from the HTTP method, the URL as defined above and the OAuth parameters, the three parts concatenated with space characters, and then base64-encoded. You can find examples for it at Google’s reference page.

We can build the XOAUTH string in a straightforward way with signpost and our Android OAuth helper class (if you’d like to use it in plain Java, just remove the Log lines). Since many people have asked for the source code of the class, here it goes the whole stuff as used in SMSForwarder.


package com.nilvec.oauth;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.util.SortedSet;
import java.util.Map.Entry;

import org.apache.commons.codec.binary.Base64;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;

import android.util.Log;

import oauth.signpost.OAuth;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.OAuthProvider;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthProvider;
import oauth.signpost.commonshttp.HttpRequestAdapter;
import oauth.signpost.exception.OAuthCommunicationException;
import oauth.signpost.exception.OAuthExpectationFailedException;
import oauth.signpost.exception.OAuthMessageSignerException;
import oauth.signpost.exception.OAuthNotAuthorizedException;
import oauth.signpost.http.HttpParameters;
import oauth.signpost.signature.HmacSha1MessageSigner;
import oauth.signpost.signature.OAuthMessageSigner;

public class OAuthHelper {

    private static final String TAG = "OAuthHelper";

    private OAuthConsumer mConsumer;
    private OAuthProvider mProvider;

    private String mCallbackUrl;

    public OAuthHelper(String consumerKey, String consumerSecret,
            String scope, String callbackUrl, String appname)
    throws UnsupportedEncodingException {
        String reqUrl;
        if (appname == null)
            reqUrl = OAuth.addQueryParameters(
                    "https://www.google.com/accounts/OAuthGetRequestToken",
                    "scope", scope);
        else
            reqUrl = OAuth.addQueryParameters(
                    "https://www.google.com/accounts/OAuthGetRequestToken",
                    "scope", scope, "xoauth_displayname", appname);

        mConsumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);

        mProvider = new CommonsHttpOAuthProvider(reqUrl,
                "https://www.google.com/accounts/OAuthGetAccessToken",
                "https://www.google.com/accounts/OAuthAuthorizeToken?hd=default");
        mProvider.setOAuth10a(true);

        mCallbackUrl = (callbackUrl == null ? OAuth.OUT_OF_BAND : callbackUrl);
    }

    public String getRequestToken()
    throws OAuthMessageSignerException, OAuthNotAuthorizedException,
    OAuthExpectationFailedException, OAuthCommunicationException {
        String authUrl =
                mProvider.retrieveRequestToken(mConsumer, mCallbackUrl);
        return authUrl;
    }

    public String[] getAccessToken(String verifier)
    throws OAuthMessageSignerException, OAuthNotAuthorizedException,
    OAuthExpectationFailedException, OAuthCommunicationException {
        mProvider.retrieveAccessToken(mConsumer, verifier);
        return new String[] {
                mConsumer.getToken(), mConsumer.getTokenSecret()
        };
    }

    public String[] getToken() {
        return new String[] {
                mConsumer.getToken(), mConsumer.getTokenSecret()
        };
    }

    public void setToken(String token, String secret) {
        mConsumer.setTokenWithSecret(token, secret);
    }

    public String getUrlContent(String url)
    throws OAuthMessageSignerException, OAuthExpectationFailedException,
    OAuthCommunicationException, IOException {
        HttpGet request = new HttpGet(url);

        // sign the request
        mConsumer.sign(request);

        // send the request
        HttpClient httpClient = new DefaultHttpClient();
        HttpResponse response = httpClient.execute(request);

        // get content
        BufferedReader in = new BufferedReader(
                new InputStreamReader(response.getEntity().getContent()));
        StringBuffer sb = new StringBuffer("");
        String line = "";
        String NL = System.getProperty("line.separator");
        while ((line = in.readLine()) != null)
            sb.append(line + NL);
        in.close();

        return sb.toString();
    }

    public String buildXOAuth(String email) {
        String url =
            String.format("https://mail.google.com/mail/b/%s/smtp/", email);
        HttpRequestAdapter request = new HttpRequestAdapter(new HttpGet(url));

        // Sign the request, the consumer will add any missing parameters
        try {
            mConsumer.sign(request);
        } catch (OAuthMessageSignerException e) {
            Log.e(TAG, "failed to sign xoauth http request " + e);
            return null;
        } catch (OAuthExpectationFailedException e) {
            Log.e(TAG, "failed to sign xoauth http request " + e);
            return null;
        } catch (OAuthCommunicationException e) {
            Log.e(TAG, "failed to sign xoauth http request " + e);
            return null;
        }
        HttpParameters params = mConsumer.getRequestParameters();

        // Since signpost doesn't put the signature into params,
        // we've got to create it again.
        OAuthMessageSigner signer = new HmacSha1MessageSigner();
        signer.setConsumerSecret(mConsumer.getConsumerSecret());
        signer.setTokenSecret(mConsumer.getTokenSecret());
        String signature;
        try {
            signature = signer.sign(request, params);
        } catch (OAuthMessageSignerException e) {
            Log.e(TAG, "invalid oauth request or parameters " + e);
            return null;
        }
        params.put(OAuth.OAUTH_SIGNATURE, OAuth.percentEncode(signature));

        StringBuilder sb = new StringBuilder();
        sb.append("GET ");
        sb.append(url);
        sb.append(" ");
        int i = 0;
        for (Entry<String, SortedSet<String>> entry : params.entrySet()) {
            String key = entry.getKey();
            String value = entry.getValue().first();

            int size = entry.getValue().size();
            if (size != 1)
                Log.d(TAG, "warning: " + key + " has " + size + " values");

            if (i++ != 0)
                sb.append(",");
            sb.append(key);
            sb.append("=\"");
            sb.append(value);
            sb.append("\"");
        }
        Log.d(TAG, "xoauth encoding " + sb);

        Base64 base64 = new Base64();
        try {
            byte[] buf = base64.encode(sb.toString().getBytes("utf-8"));
            return new String(buf, "utf-8");
        } catch (UnsupportedEncodingException e) {
            Log.e(TAG, "invalid string " + sb);
        }

        return null;
    }

}

Don’t forget to provide feedback if you find this class or article useful. Happy hacking!

Implementing client-side OAuth on Android

2011-03-06T17:43:25Z

OAuth is becoming the de-facto protocol to allow access to your data and services without sharing user password. Effectively all the big names such as Twitter, Google, Yahoo or LinkedIn have already implemented it. There are quite a few libraries and code samples in all the popular programming languages out there to implement OAuth in your desktop, mobile or web application as well.

There are guides for Android too, however most of them are not up to date, accurate or just difficult to comprehend if you are in a hurry. Here we provide a few easy to follow steps with some explanation how it can be done in a straightforward way.

First, a short summary how OAuth works. It is based on cryptography, where

a token and a corresponding secret is acquired by a consumer (a desktop or web application) from a provider (a server in the cloud),
this token is authorized by the user as valid and allowed to access their data and then
the token is upgraded, and this can then be used from then on until it is revoked by same user who authorized it.

The token acquired in the first step is called a request token, this is where you usually specify which service you would like to get access to; it is called scope. The second step is called authorization, after which control can be passed back to the consumer application via a callback. The final token that is received in the third step is called access token. This can be used for a long period of time, it won’t expire (but, as mentioned, the user can revoke it any time). It is basically a short string, with a corresponding secret string, and once the application acquired it, it can be used to sign HTTP requests, thus authenticating it for the provider. All three steps have a corresponding URL at the provider, to where an HTTP request is sent to get the token or manipulate it.

If you need further details, there’s a good article with API reference at code.google.com, and another very detailed overview with figures here.

We will use the excellent signpost Java library to implement OAuth access to Gmail. Just download at least the signpost-core and signpost-commonshttp4 jars, copy them to the lib/ folder inside your Android project, right click on the project, and under Properties/Java Build Path you can add them to the build path:

We will implement OAuth support via a helper class called OAuthHelper. The two single most important classes provided by signpost are OAuthConsumer and OAuthProvider; before diving into actual communications, we set these up first:


	private OAuthConsumer mConsumer;
	private OAuthProvider mProvider;

	private String mCallbackUrl;

	public OAuthHelper(String consumerKey, String consumerSecret,
			String scope, String callbackUrl)
	throws UnsupportedEncodingException {
		mConsumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
		mProvider = new CommonsHttpOAuthProvider(
				"https://www.google.com/accounts/OAuthGetRequestToken?scope="
				+ URLEncoder.encode(scope, "utf-8"),
				"https://www.google.com/accounts/OAuthGetAccessToken",
				"https://www.google.com/accounts/OAuthAuthorizeToken?hd=default");
		mProvider.setOAuth10a(true);
		mCallbackUrl = (callbackUrl == null ? OAuth.OUT_OF_BAND : callbackUrl);
	}

The consumerKey and consumerSecret strings depend on your consumer application, you can use anonymous for both. Later you might want to register your application at the provider (Google in this case), which will issue a key and secret for your app. To access an user’s Gmail inbox the scope is “https://mail.google.com/”, the URLs for OAuth are in the constructor of the helper class.

The callbackUrl variable can be used to pass an URL to the provider which will be called once your token is authorized. On Android you can register a special URL scheme to your application, thus the browser will fire up an activity of your app once authorization has been done. E.g. if you would like MyActivity to be called put the following into your app manifest:


<activity android:name="MyActivity">
	<intent-filter>
        <action android:name="android.intent.action.VIEW" />
        <category android:name="android.intent.category.DEFAULT" />
        <category android:name="android.intent.category.BROWSABLE" />
        <data android:scheme="my-activity"/>
	</intent-filter>
</activity>

and pass “my-activity://mywebsite.com/” as the callback URL. This also has the side effect of identifying your application as mywebsite.com at the provider (at least at Google). Your application will get back a verifier code via the callback as a query parameter to the URL, where the query key is “verifier”. You will need this later.

As a next step, retrieve the request token:


	public String getRequestToken()
	throws OAuthMessageSignerException, OAuthNotAuthorizedException,
	OAuthExpectationFailedException, OAuthCommunicationException {
		String authUrl = mProvider.retrieveRequestToken(mConsumer, mCallbackUrl);
		return authUrl;
	}

Once you got back the authentication URL from this method, just start up the browser with it:


try {
	String uri = helper.getRequestToken();
	startActivity(new Intent("android.intent.action.VIEW", Uri.parse(uri)));
} catch (...) {
}

In your OnResume() method in MyActivity you can catch the callback and retrieve the verifier, and upgrade your token with it:


		String[] token = getVerifier();
		if (token != null)
			String accessToken[] = getAccessToken(token[1]);

...

	private String[] getVerifier() {
		// extract the token if it exists
		Uri uri = this.getIntent().getData();
		if (uri == null) {
			return null;
		}

		String token = uri.getQueryParameter("oauth_token");
		String verifier = uri.getQueryParameter("oauth_verifier");
		return new String[] { token, verifier };
	}

In our helper class:


	public String[] getAccessToken(String verifier)
	throws OAuthMessageSignerException, OAuthNotAuthorizedException,
	OAuthExpectationFailedException, OAuthCommunicationException {
		mProvider.retrieveAccessToken(mConsumer, verifier);
		return new String[] {
				mConsumer.getToken(), mConsumer.getTokenSecret()
		};
	}

And that’s it. Just make sure you save the access token and its secret. You can now use signpost to sign your HTTP queries e.g.


	OAuthConsumer consumer = new CommonsHttpOAuthConsumer(accessToken[0], accessToken[1]);

	HttpGet request = new HttpGet(url);

       // sign the request
       consumer.sign(request);

       // send the request
       HttpClient httpClient = new DefaultHttpClient();
       HttpResponse response = httpClient.execute(request);

Happy hacking!

Update: fixed some typos in code snippets.

How to use the latest 802.11 drivers and stack with Linux

2011-01-24T23:12:37Z

There might be several reasons why the wireless drivers from your distro don’t fit the bill. You purchased a new wifi card which doesn’t have a driver in the kernel of your Linux system, or you would like to play with some cool new feature or just to make sure you have the latest and greatest, cutting edge 802.11 software.

Thanks to the work of the Linux wireless developers it is now possible to do that without upgrading your kernel. Using a compatibility layer you can compile and install the latest drivers and the 802.11 stack, and use it with your current kernel (that is, if you have a relatively recent kernel — if you have 2.6.24 or anything more recent, you’re good to go).

To use bleeding edge drivers, first make sure your kernel headers package is installed. Then fetch the necessary git trees:
mkdir compat-wireless && cd compat-wireless git clone git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat.git git clone git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/compat-wireless-2.6.git

The linux-next git tree contains the code for the next kernel release, and 802.11 updates are pulled into it on a regular basis. The compat and compat-wireless-2.6 trees implement the necessary compatibility layer, thus making it possible to use the wireless subsystem from linux-next with your kernel. You can also use your git trees to follow development, and keep your wireless drivers up to date: just pull from the remote repositories when you’d like to update. However, the linux-next repository is rebased on a regular basis, so you may want to check out the daily tag instead of pulling.

You can use the compat-wireless-2.6 directory to actually build the wireless subsystem:
cd compat-wireless-2.6

Specify where your newly fetched kernel tree and the compat tree are:
export GIT_TREE=`pwd`/../linux-next/ export GIT_COMPAT_TREE=`pwd`/../compat/

Then refresh compat-wireless-2.6 with the contents of your git trees:
./scripts/admin-refresh.sh

You can also select just a specific driver or group of drivers, e.g. if you only need the ath5k driver:
./scripts/driver-select ath5k

And that’s it, everything should be ready for the build.
make sudo make install

Your newly built modules will go into the updates folder inside your kernel module directory. Reboot your box (or use the scripts supplied with compat-wireless-2.6 to reload any drivers).

Nexus S details

2010-12-18T03:10:52Z

Is the Nexus S ugly? We don’t think so. The new Google Android phone Nexus S just arrived and looks like some amazing alien toy with a pitch black front. You can’t tell where the screen ends and where the phone starts. Check out these unboxing pictures below.

Google Nexus S box

Google Nexus S box and phone

Nexus S box contents - battery, charger and chord

Nexus S box contents with the pitch black phone

Nexus S, an alien looking phone

Nexus S welcome screen - say hello!

Highlights:

“Gingerbread” operating system, version 2.3.
Keyboard is much better, you can forget Swype.
2 cameras, including 1 in the front (for video chat).
I prefer unmodified Android stock UI more than manufacturer overlay eg. HTC Sense UI.
Near Field Communication ability (so it can become your wallet in the future).
16GB built-in memory (OK, no way to add an SD card, but at this point everything I have is in the cloud anyway).
4.0 inches, WVGA (800 x 480) touch screen.
Awesome, curved, super-black and alien looking gadget.

We are complete Nexus drones, so we don’t buy into all the mixed comments in other Android blogs. But we can be reasoned with – what do you think?

Testing an Android image built from source

2010-11-10T17:02:47Z

So you just built a new android system from source and you would like to test it first in the emulator before flashing it onto your phone. Or perhaps you would just like to know how the bleeding edge version of android straight from the source repositories looks. Fortunately, the emulator used by android SDK is flexible enough, so you can specify what image it should use for the data, system or the other partitions — thus you can build your own flavor of android and test it in the emulator easily.

Once you have checked out the android source tree via repo (either the official one, cyanogenmod or some other mod which published its source via a manifest file) apply your changes if any (add a new component to the system, modify the framework etc). Then just pick a build target (it can be for example full-eng, it will work with the emulator) and build it the usual way. By default the qemu-based emulator is also built, and it will be placed to out/host/linux-x86/bin/. When it’s done the images will be in out/target/product/(your target device name)/. From now on I’ll assume you picked the generic target.

You will also likely need an SD card image that can mounted by the emulator. The SDK contains a tool, mksdcard with which it’s simple enough to create one:

$ mksdcard 1024M out/target/product/generic/sdcard.img

The SDK tools/ folder should be in your $PATH for this to work.

Now the only thing left is to fire up the emulator with your newly built images:

$ ./out/host/linux-x86/bin/emulator \
-sysdir out/target/product/generic \
-kernel prebuilt/android-arm/kernel/kernel-qemu \
-data out/target/product/generic/userdata.img \
-sdcard out/target/product/generic/sdcard.img

That’s all, the emulator should start up, and if you are lucky your new android hack will come to light and work. You can use adb to copy files to/from the emulator image, or just fire up a root shell in the emulator via adb shell and take a look what is going on.

IndentFinder vim plugin

2010-10-28T22:21:04Z

This is a great little python script that finds out what indentation is used in your source files. The author also provides a vim plugin, which is a life saver for anyone working with others on software projects. It prevents the complete mess that is the result of team members using their own indentation settings on source code. It analyses the text file, and provides reliable hints what indentation settings (tabs vs spaces, number of spaces etc) were used by the author. There is another plugin for the same purpose, which is basically the vimscript port of IndentFinder, but this latter is extremely slow, especially on large files (it seems vimscript was not really meant for parsing thousands of lines every time you open a file).

What I did not really like about IndentFinder was the way it invokes the python interpreter (thus increasing startup time and leaving room for errors), even if python bindings are supported in vim. One solution could have been simply to create a vim plugin from the python source, but then this would have meant increased maintenance efforts (copying changes from the python source to the vim plugin). Another possibility was to generate the vim plugin via setup.py from the python source.

First I added a new method to the main class in indent_finder.py:

--- a/indent_finder.py
+++ b/indent_finder.py
@@ -125,16 +125,21 @@ class IndentFinder:
         self.clear()^M
         f = open( fname )^M
         l = f.readline()^M
         while( l ):^M
             self.analyse_line( l )^M
             l = f.readline()^M
         f.close()^M
 ^M
+    def parse_buffer( self, buf ):^M
+        self.clear()^M
+        for l in buf:^M
+            self.analyse_line(l)^M
+^M
     def clear( self ):^M
         self.lines = {}^M
         for i in range(2,9): self.lines['space%d' % i] = 0^M
         for i in range(2,9): self.lines['mixed%d' % i] = 0^M
         self.lines['tab'] = 0^M
 ^M
         self.nb_processed_lines = 0^M
         self.nb_indent_hint = 0^M

It just makes it possible to call the parser with a buffer instead of a file name: vim opens the file and reads its contents into memory, thus it’s easier to pass the buffer itself to IndentFinder.

The second part is modifying setup.py to build the plugin from a fixed header, the python source and a fixed trailer. It turned out distutils is flexible enough to do this easily: you can subset the build class and do your special stuff in it. Easy:

#!/usr/bin/env python

from distutils.core import setup
from distutils.command.build import build
import os
import indent_finder

HEADER = """if exists('indent_finder') || !has('python')
	finish
endif
let indent_finder=1

python << EOS
"""

TRAILER = """EOS

augroup indent
	au!
	au BufRead *.* :py import vim
	au BufRead *.* :py fi = IndentFinder()
	au BufRead *.* :py fi.parse_buffer(vim.current.buffer)
	au BufRead *.* :py vim.command(fi.vim_output())
augroup END
"""

These are our fixed header and trailer parts. We just make sure python is supported in vim, and make IndentFinder parse the opened buffer. The result from IndentFinder is an actual vim command that adjusts indentation settings.

class build_vimscript(build):
	vimscript = "indent_finder.vim"
	pyscript = "indent_finder.py"

	def run(self):
		build.run(self) # call superclass first

		if self.is_vimscript_outdated():
			self.update_vimscript()

	def is_vimscript_outdated(self):
		pyst = os.stat(self.pyscript)
		try:
			vimst = os.stat(self.vimscript)
		except OSError:
			return True
		else:
			if pyst.st_mtime > vimst.st_mtime:
				return True
		return False

	def update_vimscript(self):
		pyf = open(self.pyscript, "r")
		vimf = open(self.vimscript, "w")

		vimf.write(HEADER)
		for line in pyf:
			vimf.write(line)
		vimf.write(TRAILER)

		vimf.close()
		pyf.close()

Here we check that the vim plugin is older than the python source of IndentFinder. If it is, then the plugin is rebuilt. Calling the superclass in our run() method makes sure the other part of the build process is handled the usual way.

setup( 	name = "Indentation Finder",
		version = indent_finder.VERSION,
		description = "Finds of check the indentation used in programming source files",
		author = "Philippe Fremy",
		author_email = "phil at freehackers dot org",
		maintainer = "Philippe Fremy",
		maintainer_email = "phil at freehackers dot org",
		license = "BSD license",
		url="http://www.freehackers.org/Indent_Finder",
		py_modules = [ "indent_finder" ],
		cmdclass={"build": build_vimscript},
		)

Here we’ve just added a new parameter to setup cmdclass tells distutils to use our class for building the package.

And that’s it. The .vim plugin is built and can be copied to
~/.vim/plugin/.