Aki's Blog

Boilerplate-free Functional Lenses for Scala

2012-12-07T00:00:00+09:00

This post introduces boilerplate-free functional lens creation to Scala using a library called Rillit. Lenses are composable getters and setters for immutable data structures, which can significantly ease your pain when writing functional code. Rillit makes lens creation easier than any other lens library I know of by utilizing new features from Scala 2.10: macros and dynamic invocations.

Why lenses?

Say you have the following case class structure for describing a person. The nesting seems unncessary in this small example, but that’s what you need to do with larger data structures, so bear with me:

case class Email(user: String, domain: String)
case class Contact(email: Email, web: String)
case class Person(name: String, contact: Contact)

val person = Person(
  name = "Aki Saarinen",
  contact = Contact(
    email = Email("aki", "akisaarinen.fi"),
    web   = "http://akisaarinen.fi"
  )
)

Now, say you want to modify the user of the email address from ‘aki’ to ‘john’. And because we’re working with immutable data structures, we can’t just assign a new value, but we want to create a new instance of Person with the user field updated. This is a common pattern when writing functional code.

Here’s the idiomatic solution using the copy method provided by case classes:

scala> person.copy(contact = person.contact.copy(email = person.contact.email.copy(user = "john")))
res0: Person = Person(Aki Saarinen,Contact(Email(john,akisaarinen.fi),http://akisaarinen.fi))

The field gets updated, but the syntax is verbose and ugly. Lenses can ease the pain. They provide a lens to a certain part of a larger immutable data structure, allowing you to get and set that particular value.

The beauty of Rillit

This is what you could do in Rillit:

scala> Lenser[Person].contact.email.user.set(person, "john")
res1: Person = Person(Aki Saarinen,Contact(Email(john,akisaarinen.fi),http://akisaarinen.fi))

This performs the exact same thing as our nested copy soup above, but looks a lot more civilized.

Now, what exactly just happened there? Lenser creates a new lens for your user field using a combination of macros, dynamic invocations and implicit conversions.

Lenses in Rillit are defined as follows (with some irrelevant details omitted):

trait Lens[A, B] {
  def get(x: A): B
  def set(x: A, v: B): A
}

A Lens provides a view to a particular part of a larger immutable data structure. In our example the larger part, A, is a Person and the lensed part, B, is the user of the email (whose type is a String).

A lens can be used to get or set the part of Person that it points to. Exactly what happened in our example above. And the cool thing is that Rillit’s Lenser gives you an easy way to build lenses to arbitrary nested case classes.

Let’s split the earlier expression so it’s easier to understand what happens:

val lenser = Lenser[Person]
val userLens = lenser.contact.email.user
val updatedPerson = userLens.set(person, "john")

So we first instantiate a Lenser for Person, then describe the fields for which this lens should be built (this is all type-safe, done in compile-time using macros), and finally use that lenser to create a new Person with username updated. If you want to know the details how the Lenser works, look at the source Luke.

What’s next?

There is a lot more benefits to using lenses than just getting rid of ugly copys. You can for example compose your lenses together, forming new lenses.

There are also many other implementations of lenses for Scala, including ones in Scalaz, Shapeless and Macrocosm. At the moment they have far more advanced lens implementations than Rillit, except for one part: their creation of lenses requires more boilerplate. So I focused on implementing a boilerplate-free Lenser.

As lenses seem an incredibly useful feature in functional programming, I would hope to see a boilerplate-free, production-ready, stand-alone lens library for Scala soon after 2.10 is released. I’m planning on using Rillit for myself, we’ll see if someone else is interested in it as well. And maybe the same Lenser approach could be adopted to fit Scalaz and Shapeless as well.

Oh, and the name, Rillit? It’s Finnish for glasses, and happens to sound funny to me.

The mismatch between OO design and function composition

2012-11-24T00:00:00+09:00

Scala is a multi-paradigm language, giving developers the ability to mix both functional and object-oriented designs.

Combining these two can be painful. I have a hunch that part of this friction comes from the difficulty of using function composition together with object-oriented designs.

It’s not that you can’t combine these two, it’s just that OO designs nudge you away from function composition. This post is trying to exemplify that mismatch.

Implicit receivers lead to method chaining

Methods of a class (in object-oriented sense) are functions, which have one implicit parameter, this. They are always tied to a specific instance of the class.

Let’s look at a simple example. Say we want to implement a numerical algorithm, which first adds 8 to a number, then takes the square root and sine of the result.

Also, for the sake of making of a point, let’s assume we are implementing this API ourselves and won’t rely on any standard library interfaces. I’m only using standard library in the implementations.

I am assuming we are civilized people here and don’t implement this by mutating the state of Number, but return a new immutable instance. This is what our OO solution could look like:

object OO {
  class Number(val value: Double) {
    def add(x: Double) = new Number(value + x)
    def sqrt           = new Number(math.sqrt(value))
    def sin            = new Number(math.sin(value))
  }

  def algorithm(x: Number) = x.add(8).sqrt.sin

  val result = algorithm(new Number(17)).value
}

Functions lead to new functions

If we would write this in a purely functional manner, i.e. not binding the implicit receiver, a comparable solution in Scala would look something like this:

object FP {
  type Number = Double

  val add: (Number => Number) = _ + 8
  val sqrt                    = math.sqrt _
  val sin                     = math.sin _

  def algorithm = add andThen sqrt andThen sin

  val result = algorithm(17)
}

Note: I updated the FP version from the original a bit, as suggested by Antoine Gourlay in the comments, to make the function composition less clumsy.

Where’s the difference?

Now, these solutions both give you the same result, -0.9589242746631385. They also look kind of similar, both having a chain of add, sqrt and sin in the algorithm part.

There are conceptual differences, though. In the OO version, the algorithm is constructed by chaining a sequence of method calls for the given instance of Number. To do this, we naturally need to have a handle to the instance of Number. That’s where the chaining starts.

In the FP version, we are taking functions, and creating a new function without ever needing to know the instance of Number while doing the composition. The parts we are composing don’t have to have anything in common, but we can still compose them to create a new function. In this sense function composition is more flexible.

Which one is better?

The disappointing answer is that it depends. I think the functional approach is more flexible, and when I’m writing Scala I try to start with that, but there is no single answer as what to do.

The thing to keep in mind is that when you write something in Scala as a method (or a function), you are making a choice. You define what will be the most natural way of using your piece of code.

The natural way of using methods will be to chain calls on object instances. Functions, on the other hand, will naturally lead you into thinking about new functions.

Now, what do you want to see done in your codebase?

Thinking in terms of data transformations

2012-09-22T00:00:00+09:00

This is a successor to my previous post, Haskellify your Scala, giving another example of how to design Scala in a Haskellish fashion. I am describing a way of thinking design in terms of data transformations, instead of a sequence of instructions to mutate state.

Imperative programming focuses on executing a series of statements to mutate the program state into something desirable. It is fundamentally based on the idea of executing instructions on the CPU. Modern languages provide high level abstractions for specifying the instructions, of course. Java programmers rarely need to care about the Java bytecode instructions ¹, and even less so about the actual CPU instructions.

However, none of these abstractions change the underlying mindset. The program is still a sequence of statements, albeit well structured sequence of statements, that are executed in the defined order to achieve the goal.

Functional programming offers a different way of reasoning about programs. Instead of thinking in terms of sequential statements, programs can be viewed as ways of transforming input data into something more useful than the input itself.

It’s about first modelling the problem by defining appropriate data structures, such as lists, queues and trees, and then defining functions, which transform this data.

A tale of two Pong bots

To illustrate my point, I’ll show you two imaginary designs that could take part in the Hello World Open competition. In Hello World Open, programmers compete against each other by writing bots for the game of Pong. Bots receive events over the network about the current state of the game, while controlling their own paddle by sending commands to the server.

The point of this example is not to describe a great Pong bot, but the difference between designing imperatively and functionally. I am presenting two ways of structuring a Pong bot in Scala.

An imperative approach to implementing the core bot logic and the main loop could look something like this (Full source as Gist).

class Bot(connection: PongConnection) {
  var myDirection      = 0
  var missileInventory = 0
  var enemyPosition    = 0

  def update(event: Event) {
    if (conditionA(event)) {
      goUp()
    } else if (conditionB(event)) {
      shootMissile()
    }
  } 
    
  private def goUp() {
    myDirection = 1
    connection.moveUp()
  }

  private def shootMissile() {
    missileInventory -= 1
    connection.shootMissile()
  }
  
  private def conditionA(event: Event): Boolean = sys.error("not implemented")
  private def conditionB(event: Event): Boolean = sys.error("not implemented")
}

object Pong extends App {
  val connection = new PongConnection
  val bot        = new Bot(connection)
  while (connection.isConnected()) {
    val event = connection.readEvent()
    bot.update(event)
  }
}

A functional version, structured around the idea of transforming data, could look more like this (Full source as Gist).

object Bot {
  def actions(state: State, event: Event): (State, Seq[Action]) = (state, event) match {
    case (s,e) if conditionA(s,e) => State.up(state)    -> Actions.up
    case (s,e) if conditionB(s,e) => State.shoot(state) -> Actions.shoot
    case _                        => state              -> Actions.none
  }

  def conditionA(state: State, event: Event): Boolean = sys.error("not implemented")
  def conditionB(state: State, event: Event): Boolean = sys.error("not implemented")
}

object Pong extends App {
  val events       = createSource()
  val initialState = State()
  
  events.foldLeft(initialState) { case (state, msg) =>
    val (newState, actions) = Bot.actions(state, msg)
    actions.foreach(execute)
    newState
  }

  private def createSource(): Iterator[Event] = sys.error("not implemented")
  private def execute(action: Action) = sys.error("not implemented")
}

Key differences in the two designs

There are countless ways of making both of these implementations better, but I want to concentrate on the key differences between the thought process behind them. This is not a competition between the imperative and the functional version, but an illustration of the differences.

The functional version is built around the concept of transforming the current state and incoming event into a sequence of formalized actions. The imperative version, on the other hand, is executing a series of instructions by calling other objects, in hopes of leading the bot to a victory.

Structuring program as data transformations leads to some nice things. In the functional version it’s easy to isolate all side-effects out of the core bot logic. The decision function is only transforming (State, Event) tuples into new a new state and a sequence of Actions. The imperative version, in contrast, is using Connection to move our world into the desired new state. Nothing prevents you from implementing similar separations for the imperative version, but it usually doesn’t feel as natural.

Functional data transformations are also conveniently testable using techniques like ScalaCheck. It generates a set of random inputs and asserts that given properties are true for all outputs. Similar testing is harder to achieve for imperative code, even if you would utilize e.g. mock objects.

Summary

Thinking in terms of data transformations can lead to better designs. It is one of the ways you can Haskellify Scala code. Transformations, and the data-centric approach in general, forces you to clarify and formalize concepts. It also makes isolating side-effects easy.

Unless you’re doing performance optimizations, at which point javap becomes an invaluable tool, especially with Scala. I could rant a lot about this, but this is not the post for that :)
↩

Haskellify your Scala

2012-09-10T00:00:00+09:00

After working with Scala daily for almost 3 years, I am still struggling to find the best practices for using it. Scala is a language of many choices and compromises: it does not do much to enforce a certain style.

One of the better guidelines was suggested by my colleagues, Pekka Enberg and Jussi Virtanen

Always consider what you would do in Haskell.

Let’s take an example.

Separate data and logic

In Haskell, the first step to solving any problem is to design the data types. That is followed by writing an appropriate set of referentially transparent functions, which transform that data model as necessary. A very clear separation of data and logic.

In imperative object-oriented programming, data and logic are more mixed. Classes contain data, but also provide the operations for manipulating that data.

How to apply the Haskell principle to Scala? One concrete example is that case classes should only contain data, while data manipulation functions live their own life in the companion object.

Here’s an example:

    case class Portfolio(positions: Map[Instrument, Quantity]) {
      def get(instrument: Instrument): Quantity = positions.getOrElse(instrument, 0L)
    }

    object Portfolio {
      def add(a: Portfolio, b: Portfolio): Portfolio = {
        // whatever (sometimes complex) logic to add two portfolios
      }
      def subtract(a: Portfolio, b: Portfolio): Portfolio = {
        // whatever (sometimes complex) logic to subtract two portfolios
      }
    }

Complex portfolio manipulation functions are not part of Portfolio case class, but part of the companion object instead. This provides a natural separation for data and logic. The example also contains one instance method, get, for the case class. There are two reasons for that. First, it is effectively part of the data model by defining the values for non-defined instruments. Second, it keeps the code using portfolios concise because get is called repeatedly for Portfolio objects all over the place.

To put this simply:

Do not write data manipulation logic to case classes

Following this convention establishes two things:

When writing code, it encourages to focus more on the data model
When reading and reviewing the code, the data model is more visible

While Haskell does not always have the right answers, keeping it as a role model certainly guides you towards more functional and data-oriented habits. Hence when writing scala, always imagine what you would do in Haskell. After that, use your judgement.

Functional Finite-state Machines using Akka

2012-05-13T00:00:00+09:00

This post is about combining functional programming, finite-state machines and the actor model of computation using Akka FSM.

Finite-state machine, or FSM for short, is a mathematical model for structuring a computer program. Many programmers have come across them, either in a CS course in university or in practical applications. FSM is a useful tool, because it forces the programmer to lay out the states and flow between them explicitly. I often use FSMs as a mental model for reasoning about programs, even if the program would not be explicitly structures as one.

Actor model of computation, extensively used in e.g. Erlang, is another interesting concept: application is modelled as a group of independent actors, who only communicate with each other using asynchronous message passing. Actors are an old concept, first presented in 1973, but are becoming increasingly popular as a way of handling concurrency and parallelism as the number of cores increase. There is a great talk about actors on Channel 9, involving Carl Hewitt, the inventor of actors, and Eric Meijer from Microsoft Research. I recommend watching it, regardless of your level of experience with the actor model.

Regarding the third component in this post, functional programming, Michael Feathers recently wrote about hybridizing object-oriented and functional design:

Object-orientation is better for the higher levels of a system, and functional programming is better for the lower levels.

Michael throws in the usual disclaimers about this not applying to every case, to which I agree, but I still think there is something to it. I love functional programming: filtering, mapping and folding lists allow me to write very concise code, fast. However, pure code with no side-effects is ultimately of no use on itself: someone somewhere must see the results. How to orchestrate the side-effects and build the larger application around the functional pieces is a very interesting question, one which does not have a clear-cut answer.

In the spirit of Michael’s post, but instead of using traditional objects on the higher level, could we structure the application as a set of functional finite-state machine actors? Independent entities, whose behaviour is well specified in terms of states and state transitions. Entities which communicate asynchronously using actor message passing. Entities whose state transitions are written as purely functional pieces of code. This seems like an intriguing concept.

The idea itself is nothing new. The Erlang design principles, for example, describe how to implement finite-state machines using Erlang actors. I have, however, rarely seen this as the base for application design.

A few weeks ago I was excited to find Akka FSM, a finite-state machine implementation library for Akka, an actor framework for Scala.

Scala is well suited for functional programming, and Akka FSM does hence provide the possibility to combine the aforementioned three pieces into a single clean package: implementing the application as a set of actors which are internally functional finite-state machines. I have been writing code using Akka FSM for a few weeks now, and it seems like a natural way of structuring programs. So far I am very pleased.

The official documentation contains plenty of examples, so I won’t replicate them here. Check the docs and give it a go.

Running this site with Jekyll, Compass and Twitter Bootstrap

2012-04-09T00:00:00+09:00

During the Easter holidays I updated the technology for this site.

My goal was to find a simple and elegant solution for hosting two things: my static home page and this blog. Until yesterday, the blog was hosted at Posterous. Posterous was recently acquired by Twitter for their talent, which means the service is likely no longer actively developed. On a related note, there was good discussion in Hacker News about talent acquisitions.

In addition to the fact that Posterous platform is dying, I was never really happy with writing the blog posts in a WYSIWYGish web interface. Writing in Vim with basic Markdown syntax is an attractive alternative.

Here’s what I came up with. A pretty standard stack these days, I guess. I have been reading about almost all of these tools for a while and it was nice to see how well everything worked. Not always the case with hyped tools.

Everything is based on the static site generator, Jekyll. Installation is straightforward using RubyGems, just install RubyGems in your preferred way and run gem install jekyll. Then all you need to do is run jekyll and the whole site is compiled to static HTML, ready to be hosted where ever you want. I am using Kapsi for hosting, but basically any provider, like Github pages or Heroku would do. Nice thing about this setup is that it requires nothing except a basic HTTP server from the hosting provider.

Jekyll contains support for Markdown based page rendering and Liquid templates out-of-the-box. There are at least two major projects that try to bring one-click blog hosting around Jekyll: Octopress and Jekyll-Bootstrap. I decided against using these, for two reasons. First, I wanted to keep things as simple as possible and second, I wanted to learn in the process.

To make the site look like it does, I used two tools: Compass and Twitter Bootstrap.

Compass is a CSS Authoring Framework, as they call it. In practice, it lets me write style sheets in Sass. Sass an excension of CSS3 which compass then compiles to standard CSS3. End result is cleaner stylesheet syntax and less duplication than with traditional CSS files. Being also a rubygem, installation is again simple: gem install compass.

Basic layout is achieved using Twitter Bootstrap styles, with some custom CSS thrown in. In case you are not already familiar with Bootstrap, I suggest you take a look.

Sources are hosted in a git repository and deployed to production using a simple shell script. Previewing posts and changes locally is easy:

compass watch
jekyll --server --auto

This results into compass watching for changes in Sass files and compiling them to CSS as soon as the file is modified. Jekyll starts a standalone HTTP server in port 4000 and lets me preview the latest changes by accessing http://localhost:4000/. The --auto flag automatically re-compiles the site as soon as there are any changes.

The feedback loop for testing is extremely fast, something I value a lot. Writing and publishing this blog post is now all sunshine and happiness.

Smooth application development with Qt

2011-12-02T00:00:00+09:00

I wrote an article about C++ development with Qt, together with my colleagues Sami Rosendahl and Tuomas Järvensivu at Reaktor. Check it out at Reaktor’s site.

It’s interesting to see what happens to Qt in the future. It’s clearly one of the best C++ libraries I’ve used and I’m sincerely hoping they’ll stay on the right path.

A cleaner Google Reader

2011-11-07T00:00:00+09:00

The new design of Google Reader has been debated a lot. I was also frustrated by unnecessary large margins and useless bars, especially when reading from my 13” laptop screen.

I made a simple Chrome extension to fix things up a little. You can install CleanReader from the Chrome Web Store for free. Sources are available at GitHub.

Little things that make the customer feel good

2011-10-30T00:00:00+09:00

Yesterday I wanted to add a new post to the social bookmarking service, trunk.ly, by emailing a link to my username+magicnumber@post.trunk.ly address. I was typing the address out of my memory and unfortunately sent my link to @trunk.ly instead of the correct address at @post.trunk.ly.

In many services a mistake like this could result in a silent failure and a lost email. Or some technical “Deliveration failed: error 0xF3F3”. Afterall, I did email to a stupid address, right?

Trunk.ly did something different. I received a personalized reply, with trunk.ly co-founder Tim Bull as the sender:

From: Tim Bull <tim@trunk.ly>
To: Aki Saarinen <..>
Subject: Re: LiveReload

Hi Aki,

Just FYI - the links need to mailed to post.trunk.ly NOT trunk.ly :-)

Cheers,
Tim Bull

How amazing is that! It made me feel that they really want to take good care of me and my links. In a technical sense things like this are not hard to implement, it’s just that we often don’t even think about it.

DNS for internal service resolution and configuration

2011-09-03T00:00:00+09:00

Ricardo Vice Santos gave an interesting presentation about the Spotify architecture yesterday at Reaktor Dev Day.

What struck me as a clever yet rarely used idea was his description of their usage of DNS for internal service resolution and configuration in the Spotify backend.

Connections from clients come through an access point, a relatively thin proxy to various backend services. Services provide authentication, decryption keys, track information and of course the actual music data, among other things. How did they configure the whole network of services? They used DNS. It’s pretty easy to store configuration parameters as TXT records in DNS alongside other records. Access point queries the internal DNS server and then decides where to connect the incoming request.

Setting up a DNS server is really easy and DNS servers are one of the most battle tested pieces of code. When was the last time you had to worry about DNS or heard about DNS server failure? You almost never do, because it just works.

Project Ghettoblaster: How we built a kick-ass portable boombox

2011-08-31T00:00:00+09:00

Just finished an article about building a portable boombox with no previous experience with DIY audio systems. Check it out!

Running oprofiler in Nexus One with Android 2.2

2011-08-09T00:00:00+09:00

Oprofiler is a system-wide profiling tool for Linux that utilizes hardware performance counters in the CPU to provide a wide range of interesting statistics about program execution. Oprofiler is not enabled in the standard Android distribution, but with some modifications, it can be used under Android also.

I was able to run oprofiler under Nexus One and Android 2.2, with one limitation. I could only profile once after a restart, further attempts to start the profiler would not succeed in collecting any data. This limits the usability of oprofiler, but it was enough for my immediate profiling needs. If you manage to fix this issue, let me know.

Compile kernel with support for profiling

Oprofiler needs support in the Linux kernel, and the prebuilt kernel for Nexus One doesn’t include this support. You need to compile your own version of the Android kernel for your device, with profiling support enabled.

You can check what kernel version your phone is currently running from Settings->About Phone. I suggest you try to compile the same version of the kernel that the phone is running as default. A good article about building the kernel and platform for Nexus One is available here. I used kernel version 2.6.32 for my phone with Android Froyo 2.2 (FRG83).

You should use standard kernel configuration tools (“make menuconfig”) to enable the support, but you can compare results with this patch to see you’re enabling the right things (CONFIG_OPROFILE, etc).

In order to keep the kernel small enough (there’s a limitation what can be fit into the phone), I additionally needed to disable ext3 and ext2 support.

Interrupt bug in 2.6.32 kernel for Nexus One

Oprofiler module in 2.6.32 kernel is missing an interrupt for Nexus One CPU. The bug is discussed in the android-platform mailing list and original patch are available here. I also have a mirror of the patch, just in case. Apply the patch, include the kernel with your custom Android 2.2 distribution and start profiling.

Android radio firmware checksums for Google Nexus One

2011-08-08T00:00:00+09:00

I recently needed to downgrade my radio firmware of an european Google Nexus One for research purposes. To my surprise, HTC or Google didn’t provide official packages of the older radio firmware. If you want to get an older radio firmware, you’re on your own. Only way of getting them is to dig deep into Android developer forums and download a radio image from unofficial sources.

Because flashing the radio firmware is one of those things that has a serious risk of bricking the phone, it’s not too fun to try with shady images and see what happens. If the image is incompatible or damaged, you’ll end up with a brick.

I can’t distribute any radio images, because their license doesn’t allow redistribution. But I can publish the checksums of the images I successfully used, so you can verify the files you’ve got to see if it’s the same that I’ve been using. I hope this can be another data point you can use to decide whether a radio image is reliable or not.

Note that I don’t have any visibility what’s inside these, they’re just a result of a few days worth of forum searches and experiments. They could even contain malware. What I can tell is that they did not brick my phone and the radio was still working after these were in. The checksums are of the radio.img file only.

I have a newer Nexus One model (the one with SLCD), microp firmware version 0c15, and hboot version 0.35.0017.

Radio 5.12.00.08

Tested to work with Android 2.2.1 Froyo (FRG83).

MD5: 263fb298d747f9e5632b373c69ce7893
SHA1: 2ad521b954178f0962d25c13ba45014df7d2c455

Radio 5.08.00.04

Tested to work with Android 2.2.1 Froyo (FRG83).

MD5: dee19eddd42cd0166398bcab37663f62
SHA1: 802656e261433400d3a56a978b0350b180bc8884

Radio 4.04.00.03_2

Tested to work with Android 2.1 Eclair (ERE36B).

MD5: 310d85c4998163818f7dcdef730c2a12
SHA1: 1bc692631d33f8b885a5152d602cb3f2e812250d

Booting Android 2.1 in Nexus One with SLCD

2011-08-05T00:00:00+09:00

I wanted to downgrade my Google Nexus One momentarily back to stock Android 2.1 (called Eclair), in order to build my own version of the TaintDroid system. I first needed the stock Eclair to fetch the proprietary 2.1 compatible libraries from the phone. These libraries are not allowed to be distributed by anyone but the vendors, which means they are only embedded into the stock images. I had Android 2.2 installed to the phone, but the libraries there were incompatible with 2.1.

I wrote this post because there’s quite a lot confusing and contradictory information about downgrading, scattered around various forums. Hopefully someone can learn from my experiences. I’m using a phone with unlocked bootloader and most of the tricks only work after unlocking, so if you want to keep your phone locked, this is not the way for you.

Using these steps I managed to boot up stock 2.1 Eclair in Nexus One, even though admittedly I not in a 100% working shape. It kept throwing some exceptions into my face, but it was in a good enough condition to use adb to download the libraries. After that I successfully compiled and installed the modified 2.1 in TaintDroid, which worked just fine. Probably the stock installation could’ve fixed too, but I didn’t invest my time in finding what was wrong with it.

SLCD or AMOLED?

First piece of the puzzle when downgrading is that there are two kinds of Nexus Ones. An older generation with an AMOLED display and a newer one with an SLCD display. You can boot up your phone to the bootloader and check microp version to see which one you have: 0b15 is AMOLED, 0c15 is SLCD.

I have the newer one, which becomes a problem: the kernel in stock Android 2.1 does not support SLCD, so you need to compile a kernel by yourself, as described later on in the post. Android 2.1 can be booted up with SLCD, you just have to have support for it in the kernel. I have no experience with the AMOLED Nexus One, so I’m not sure about that, but I believe it will work out-of-the-box.

Hboot, microp and radio firmware

I had a recent version of the bootloader firmware, also called hboot. My bootloader is in version 0.35.0017. Some sources claim the newer bootloader wouldn’t work with Eclair. I had no problems with the newer bootloader, so I have proof that it does work just fine. Nexus one with SLCD also has newer version of the microp firmware (0c15) than originally used for by Eclair (0b15), and it also works just fine. No need to worry about those.

A possible troublemaker could be the radio driver. I had some problems along the way, which led me to downgrade my radio firmware back to 4.04.00.03_2. When I was all finished, though, I also successfully booted up Eclair with radios 5.08.00.04 and 5.12.00.08, so I believe they should all work and my problems were cause by other factors at the time.

If you run into trouble you can’t figure out, you can try downgrading the radio, some checksums of working radio images available in my other post.

Getting a kernel with SLCD support

In order to boot Android 2.1 Eclair with the SLCD, you need to have a new enough kernel. I used 2.6.32, newer ones might do as well.

Simplest way to get a working kernel is to use the boot image from Android 2.2 (Froyo) distribution. You can get one for Nexus one from developer.htc.com. I used Nexus One FRG83 system image. Unzip the zip, and you get a boot.img loaded with 2.6.32 kernel, which you can eventually flash into the device with ‘fastboot flash boot boot.img’.

If you want, you can also compile your own 2.6.32 kernel. This will become handy (and even a necessity) later on, if you want to customize your platform at all, but it’s not strictly necessary at this point. If you don’t want to compile your own kernel, you can skip the next section.

Compiling your own 2.6.32 kernel

A good writeup about how to compile 2.6.32 kernel for Android is available at here. First part of the post is about compiling Android 2.2 distribution, but take a look at the kernel part. After compiling the kernel, you can use ‘fastboot boot /path/to/my/kernel/zImage’ to boot up Android with your custom-built kernel instead of the one flashed to the device. This is a handy way of testing a new kernel, without having to flash it.

I compiled my kernel while still having Android 2.2.1 installed on the phone. This was convenient, as the phone already had a 2.6.32 kernel, so I could use that as a baseline for my kernel configuration. That way I could also modify one thing at once, so if the system didn’t boot, I knew the reason was my kernel, and the kernel only. Only after having a fully functioning 2.6.32 kernel image did I proceed with installing Eclair.

Downloading and flashing stock Eclair image

You can download the stock image for Android 2.1 from developer.htc.com, I used Nexus One (ERE36B) Official OS Image.

In order to flash the zip with ‘fastboot update NexusOne_ERE36B_TMOUS.zip’, you need to unzip the package, modify android-info.txt so that you allow the newer bootloader and microp versions (and radio if you didn’t downgrade):

require version-bootloader=0.33.2012|0.33.0012|0.35.0017
require version-microp=0b15|0c15
require version-baseband=4.04.00.03_2|5.12.00.08

…and then zip it back in.

This will just tell fastboot that you want to flash the images even though you have a newer bootloader, microp and radio versions. I’m not that experienced on this subject, so I’m not 100% sure if there’s a way this can screw things (flashing always has a hint of danger to it), but I had no problems myself. I believe this to be safe, if the flashing is done properly, but be warned though, I’m not going to take the blame if you brick your phone.

Booting Eclair with custom kernel

After flashing the stock Eclair, you can try booting up the phone, but with an SLCD you’ll end up with a black screen. My phone also seemed to jam up so I had to take out the battery before being able to reset it.

What you need now, is the 2.6.32 kernel image. If you want to use the one from stock Froyo, just issue ‘fastboot flash boot boot.img’ for the boot-image coming with froyo and you should be good to go.

If you compiled it yourself, simplest way is to use ‘fastboot boot myKernelImage’, so you don’t have to craft a boot.img to be flashed. If you want to be able to boot the 2.1 without some assistance from a computer and the fastboot utility, you’ll need to do that eventually, too. I didn’t because I only used the stock 2.1 for a while in order to fetch the proprietary drivers.

Fixing network drivers

Since the bundled bcm4329 network device driver in the system image is compiled to work with the exact kernel version thats bundled with the distribution, it won’t work when booting up with a different kernel using ‘fastboot boot’. To make networking work, you need to patch the bcm4329.ko from your custom kernel to the system partition of the phone, where it’s loaded from in the boot procedure. This might apply to other drivers too, however I was only experiencing problems with networking. If you only want to fetch the proprietary drivers, you won’t even need the networking, anyway.

A working Android 2.1 in Nexus One

If everything went well, you should now have booted up stock Android 2.1 in your Nexus One. Happy hacking, whatever it is you wish to do with it!