Branch and Bound

Automatic-Module-Name: Calling all Java Library Maintainers

2017-12-07T00:00:00+01:00

With the Java 9 release, developers can use the new module system to create modular applications. However, in order to modularize applications, libraries should be usable as modules as well.

Creating modular applications using the Java module system is an enticing prospect. Modules have module descriptors in the form of module-info.java, declaring which packages are exported and what dependencies it has on other modules. This means we finally have explicit dependencies between modules at the language level, and can strongly encapsulate code within these modules. The book Java 9 Modularity (O'Reilly) written by Paul Bakker and me explains these mechanisms and their benefits in detail.

That, however, is not what this post is about. Today, we'll talk about what needs to be done to move the Java library ecosystem toward modules. In the ideal world where all libraries have module descriptors, all is well. That's not yet where we are in the Java community.

What You Need To Do Now

We need Java library maintainers to step up! Ultimately, it would be best for your library to have a module descriptor so that modular applications can depend on it. Getting there isn't trivial in all cases. Fortunately, support for the Java module system can be incrementally added to libraries. This post explains the first step you can take as library maintainer on your way to becoming a Java module. This first step boils down to picking a module name, and adding it as Automatic-Module-Name: <module name> entry to the library's MANIFEST.MF. That's it.

With this first step you make your library usable as Java module without moving the library itself to Java 9 or creating a module descriptor for the library, yet. It doesn't even require re-compilation. So, do yourself a favor and do it now. If you're not a library maintainer, encourage libraries you use by opening an issue and pointing to this post. Then, if you'd like to know why this is a good idea and how it actually works, keep reading.

Automatic Modules

Traditional applications are packaged into JARs and run from the classpath. Java 9 still supports this, but also opens the door to more reliable and efficient deployment. Modular applications on Java 9 and later are packaged into JARs which contain module descriptors (modular JARs) and run from the module path. Code in modular JARs on the module path can't access code in traditional JARs on the classpath. So what happens when a modular application wants to use a library living on the classpath, which doesn't have a module descriptor yet? The modular application can't express such a dependency in its module descriptor.

It would be unworkable if the only resort for application developers were to wait for the library maintainer to write a module descriptor, or worse, attempt to patch the JAR themselves with a module descriptor. To prevent this possibly indefinite waiting game, a feature called automatic modules was introduced. Moving a traditional JAR (without module descriptor) from the classpath to the module path turns it into an automatic module. Such an automatic module exports all of its packages and depends on all other resolved modules. Additionally, automatic modules themselves can still access code on the classpath.

So, instead of waiting for libraries to be modularized, application developers can take matters into their own hands. Traditional JARs can be used as if they were modules. For an example of automatic modules in action, look at Paul's post where he uses Vert.x JARs as automatic modules in an application.

Automatic Module Name Derivation

Still, the question remains how you can express a dependency on an automatic module from application modules. Where does its name come from?

There are two possible ways for an automatic module to get its name:

When an Automatic-Module-Name entry is available in the manifest, its value is the name of the automatic module
Otherwise, a name is derived from the JAR filename (see the ModuleFinder JavaDoc for the derivation algorithm)

That second option probably had you shaking your head. Filenames are not exactly the hallmark of stability, and your library may be distributed in many ways that could lead to different filenames on the user's end. (Maven's standardized approach to artifact naming alleviates this a bit, but is still far from ideal.) Moreover, the module name derived from a filename might not be your ideal pick as module name. That's exactly why you're reading this call to action for adding Automatic-Module-Name to libraries. Pick an explicit module name, put it in the MANIFEST.MF and ensure a smooth ride into the modular age for users of your library. This way, you're not forcing users of your library to depend on an 'accidental' module name.

Naming Library Modules

Naming is hard. Picking the right module name for your library is important; module descriptors will refer to your library module by this name. It's effectively part of your API—once you pick a name, changing it constitutes a breaking change.

For libraries it's essential to pick a globally unique module name. A long-standing practice in Java is to use reverse DNS notation for packages (e.g. com.acme.mylibrary.core). We can apply the same to module names. Name your module after the root package of your module. This is the longest shared prefix of all packages in the module (for the previous example it might be com.acme.mylibrary). Read Stephen Colebourne's excellent advice on why this is a good idea. Ensure your module name is valid, meaning it consists of one or more java identifiers separated by a dot.

If you want to see examples of libraries who've already gone through this process, look at the module name discussion for Google Guava and the Spring Framework.

Practical Tips

Most likely your library is built using Maven or Gradle. Adding a manifest entry to the resulting JAR is a breeze in both build tools. For Maven, make sure the jar plugin has the following configuration:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>
  <configuration>
    <archive>
      <manifestEntries>
        <Automatic-Module-Name>com.acme.mylibrary</Automatic-Module-Name>
      </manifestEntries>
    </archive>
  </configuration>
</plugin>

With Gradle, you can configure the jar plugin as follows:

ext.moduleName = "com.acme.mylibrary"

jar {
    inputs.property("moduleName", moduleName)

    manifest {
       attributes  'Automatic-Module-Name': moduleName
   }
}

Sanity-Check Your Library

Is this really all there is to do as a first step toward modularization? Ideally, yes. But if you want your library to be used as automatic module on Java 9 and later, there's a few other potential issues you need to verify:

Make sure your library doesn't use internal types from the JDK (run jdeps --jdk-internals mylibrary.jar to find offending code). JDeps (as bundled with Java 9 and later) will offer publicly supported alternatives for any use of encapsulated JDK APIs. When your library runs from the classpath on Java 9 and later, you can still get away with this. Not so if your library lives on the module path as automatic module.
Your library can't have classes in the default (unnamed) package. This is a bad idea regardless, but when your library is used as automatic module, this rule is enforced by the module system.
Your library can't split packages (two or more JARs defining the types in the same package), nor can it redefine JDK packages (javax.annotation is a notorious example, being defined in the JDK's java.xml.ws.annotation module but also in external libraries).
When your library's JAR has a META-INF/services directory to specify service providers, then the specified providers must exist in this JAR (as described in the ModuleFinder JavaDoc)

Addressing these concerns is a matter of good hygiene. If you encounter one of these issues and you can't address those, don't add the Automatic-Module-Name entry yet. For now, your library is better off on the classpath. It will only raise false expectations if you do add the entry.

What You Need To Do Next

While your library can now be used as automatic module, it isn't really a great module. Every package is exported and it doesn't express its dependencies yet. That's why your next step is to add a module-info.java file describing which packages must be exported and which dependencies on other modules the library has. This might even entail some refactoring, by dividing up your code into API (exported) packages and internal packages.

If your library has no external dependencies, creating and compiling this module descriptor is relatively straightforward (see this real-world example). However, if you have external dependencies, you'll have to wait until those libraries have added module descriptors (or at least have an Automatic-Module-Name themselves). Writing your module descriptor to depend on filename-derived automatic module names is a sure way to break things for your users. When these transitive dependencies do start modularizing with a different module name, users of your library will experience breakage.

In Chapter 10 of our book, we give in-depth advice on how to migrate a library to a proper module step-by-step. For example, we explain how you add a module descriptor to your library without having to target Java 9+ when compiling your code. Features like the new --release flag and Multi-Release JARs are very helpful. That all goes far beyond the scope of this post. So, please be a good citizen of the Java community. Decide upon a module name and add it to your library's manifest. Then, keep the ball rolling by reading up on the module system and adding a real module descriptor.

Special thanks to Alex Buckley for commenting on a draft version of this post.

Java 9 Modularity (O'Reilly) now available

2017-09-11T00:00:00+02:00

With the imminent release of Java 9, we're happy to announce the O'Reilly book Java 9 Modularity is available as ebook, and soon in print as well. More info about how to get the book can be found at javamodularity.com.

About a year ago, the Early Release version of Java 9 Modularity was announced by Paul Bakker and me. It contained a rough version of the first three chapters. A lot has happened since. The book has grown to encompass 14 chapters, highlighting all aspects of the upcoming Java module system. At javamodularity.com you'll find an overview of the chapters, as well as information on how to get the book.

Since the Early Release, the module system implementation itself has seen quite some changes, especially in the area of migration support. The book has been updated with all recent developments and shows you not only how to create modular applications with Java 9, but also how to approach a migration to Java 9 from earlier versions. Java 9 Modularity is the definitive guide to the Java module system.

Why Modules?

The module system has been a boon to the JDK itself. Once a gigantic monolithic codebase, it is now neatly modularized into recognizable components:

Modularizing the JDK makes it future proof. Through strong encapsulation in modules, internal packages can truly stay private, allowing evolution of internal code as was envisioned from the start. Before strong module boundaries, it was all too easy to 'accidentally' rely on non-public APIs. Explicit dependencies ensure there's no backsliding into a big ball of mud.

The nice thing is, with Java 9 you can use the very same module system used to modularize the JDK to modularize your own applications. Translate those boxes and arrows on your whiteboard into actual Java modules, with module boundaries and dependencies enforced by the Java compiler and runtime. In the book we approach both new development with Java 9, as well as migration scenarios.

But I Won't Be Using Modules (Anytime Soon)

Modules are optional in Java 9. You can keep running applications on the classpath, or you can choose to create modular applications. However, applications on the classpath still have to run on top of the modularized JDK. This brings some new challenges when migrating to Java 9.

Especially when the application uses libraries that poke into JDK internals (and there are many of these), you need to know how to address these situations. The book covers all scenarios you need to handle when migrating existing codebases to Java 9. For library authors, it is even more important to support Java 9 as soon as possible. A special chapter on migration of libraries offers help.

Book Signing Event at JavaOne

If you're at JavaOne this year, be sure to stop by at one of our talks as well:

Designing for Modularity With Java 9 (CON2606)
Modules or Microservices (CON1450)
Migrating to Java 9 Modules (CON1455)

Besides these technical sessions, we will also be doing a book signing session at the O'Reilly booth on Wednesday afternoon, October 5th. It will be the first time we hold the print version of the book in our hands as well, so it's going to be special. Hope to see many people there!

Java 9 Modularity: O'Reilly Early Access Release

2016-08-30T00:00:00+02:00

Early this year my colleague Paul Bakker and I started working on the first drafts of Java 9 Modularity. We're proud to announce that the firsts bits of what will become the final book are now publicly available!

You can now order the Early Access release of our upcoming book through O'Reilly's webshop: http://shop.oreilly.com/product/0636920049494.do. It is also available on Safari Books online, if you have a subscription there. If prefer reading the final edition on real paper, you can even pre-order at Amazon already.

When you order the Early Access release, obviously you will get updates on the content as they come. No worries, there's more in the pipeline already. Be sure to follow @javamodularity if you're interested in updates, or to give us - much appreciated! - feedback.

As of now, the first three chapters are available. If you happen to be at JavaOne this year, mark your calendar for Tuesday September 22nd, 1:30-2PM (PDT). Paul and I will be at the O'Reilly booth to sign printed copies of the early access release. Come by and say hi!

What's in there?

So, what's all the fuss about? Java's upcoming module system has been many years in the making. With Java 9 it's finally coming to fruition, and that's a big deal.

On the one hand, it allows new ways of creating strictly more modular applications. The module system introduces a stronger notion encapsulation into the platform and allows for explicit dependencies between modules. On the other hand, it's a logical continuation of Java's existing features and philosophy on large-scale software development. Access control (private/protected/public), programming to interfaces and so on were, and still are, important enablers of modular application development. The module system takes these features to next level.

The first chapter will take you from how Java works today, to how modules address pain points in the current situation (classpath hell, anyone?!). Then, the second chapter continues with an introduction of how the JDK itself was modularized using the very same module system application developers can use. Meanwhile, you'll gain an understanding of the most important concepts of the Java module system. After reading those two chapters you know why and what, so in the third chapters it's time for the how. Here, you'll learn how to create your own modules and work with them in practice.

What can you expect?

Of course, it doesn't end there (although it does end there, for now, in the early access release). The outline shown at the O'Reilly site is somewhat outdated already, but gives a good indication nevertheless.

In subsequent chapters we discuss how modules and interfaces together do not fully solve the problem of decoupling consumers and providers of services. One of the solutions lies in the Services and ServiceLoader model that has been updated in Java 9 to work with modules.

The next chapters look at so-called modularity patterns. Existing wisdom, viewed in the light of the new Java module system. In these chapters, some of the more advanced APIs of the module system will be discussed as well.

Currently, a big focus area for us is migration of existing codebases to Java 9. Fortunately, the Java module system is designed with migration in mind. That doesn't mean it will be a walk in the park in all situations. The book gives practical advice on how to migrate in several steps, both for application developers and library authors.

Last, you can expect an overview of how the Java tooling eco-system interacts with Java 9 modules. Since this is very much in flux at the moment, those chapters will probably be among the last to be pushed out in the early access release process.

Moving on

The process of writing Java 9 Modularity is quite challenging, since the final specifications of the module system have not been nailed down yet. Even before the Early Access release we already had some re-writing to do based on developments in the JDK 9 Jigsaw prototype. We are very much indebted to Alan Bateman and Alex Buckley from the Oracle JDK team, who have been graciously reviewing our work and provided valuable feedback. Obviously Paul and I will stay on top of any developments in the period until the specification is stable, which is hopefully not too far off.

We are writing this book because we truly believe modular development is an enabler of both agile and reliable application development. Through this book, we want to show how you can take advantage of age-old principles of modularity in the shiny new version of Java. Please let us know what you think of the Java 9 Modularity Early Access release. It's through feedback of early adopters like you we can shape the book into a useful resource for the whole Java community.

The final release of the book is still set for February 2017, right ahead of the Java 9 GA release. Meanwhile, follow @javamodularity for interesting news about the Java 9 module system and updates on the book!

Exploring Angular2

2015-11-10T00:00:00+01:00

AngularJS 1.x gained an unprecedented following in the past few years. We use it heavily in our applications, too. All the more reason to dive into the next iteration of this widely popular framework: Angular2.

The Angular team is currently chipping away at their backlog, promising a beta version 'real soon now'™. Meanwhile, I've been playing around with their alpha releases. Yes, that is as painful as it sounds. Still, it gave me a solid understanding of where the framework is conceptually heading. Check out the code, then come back for some much-needed context!

The sample app provides an alternative front-end to YouTube. You can search for live music by providing an artist name. Search results can either be stored, or played directly. The playlist is backed by localstorage. It's not a large application by any means, but it's not a toy example either. This is 'YouTube live!' in action:

TypeScript

First thing you'll notice when looking at the code is that YouTube live! is written in TypeScript. Why? Well, first of all I think TypeScript is a huge improvement over plain JavaScript. Watch my talk on TypeScript to see why. This post assumes some familiarity with ES6 and TypeScript. Again, watch the talk if you want to brush up on that. It has lots of live coding, so I promise you won't be bored.

The biggest reason for writing Angular2 apps in TypeScript is that Angular2 itself is written in TypeScript. You can still write Angular2 apps in plain JavaScript (ES5 or ES6), but you'll miss out on some syntactic niceties. And miss out on full compile-time type-checking of your clientside app. Trust me, it's a big deal.

After cloning the repo, you can install and run the app with npm install && npm start. The layout of the application then looks like this:

src/
├── index.html
├── tsconfig.json
├── node_modules/
├── lib/
└── ytlive/
    ├── playlist/
    │   ├── PlaylistBackend.ts
    │   ├── PlaylistComponents.ts
    │   ├── playlist.html
    │   └── playlistentry.html
    ├── search/
    │   ├── SearchComponents.ts
    │   ├── YTLiveBackend.ts
    │   ├── search.html
    │   └── searchresult.html
    ├── ytlive.html
    └── ytlive.ts

The code is nicely modularised using ES6 modules, supported by TypeScript. Angular2 does away with its own module system. This solves the awkward problems of duplicate module definition when combining AngularJS 1.x with module loaders like require.js. Notice that Angular2 is distributed as npm package so installing (and later upgrading) is a breeze.

Components

When opening up the source files, it is immediately clear that Angular2 is a completely different framework from AngularJS 1.x. Conceptually you'll recognize some things, but at a technical level it's a clean slate. If that scares you a bit: I sympathize wholeheartedly. However, the component-based approach of Angular2 definitely is a step up. Starting over was a bold move by the Angular team, and it pays off as evidenced by some preliminary performance figures.

Reminiscent of React, your whole application is constructed as a tree of components:

Angular2 components replace a whole host of abstractions we know from AngularJS 1.x. It essentially unifies services, controllers and directives. A component is an annotated class, that can refer to an associated HTML template:

PlaylistComponents.ts:

import { Component, View, NgFor } from 'angular2/angular2';
import { LocalStoragePlayList } from './PlaylistBackend';

// PlaylistEntryComponent class definition omitted for brevity.

@Component({
  selector: 'playlist',
  providers: [LocalStoragePlayList]
})
@View({
  templateUrl: "ytlive/playlist/playlist.html",
  directives: [NgFor, PlaylistEntryComponent]
})
export class PlaylistComponent {

  constructor(private playlistService: LocalStoragePlayList) { }

  get entries(): ConcertSummary[] {
    return this.playlistService.getPlaylist();
  }

}

In particular, the @View annotation contains a reference to this template:

playlist.html:

<div class="playlist row">
  <div *ng-for="#entry of entries">
    <playlist-entry [entry]="entry"></playlist-entry>
  </div>
</div>

Together, the template and component class form a reusable whole. Through the selector property on the @Component annotation, we control how this component can be instantiated in templates. In the template above, we similarly use the playlist-entry element to instantiate nested components for each entry we have in the PlaylistComponent. These entries come from the getter method entries() on that component. Using the [entry]="entry" syntax we pass the current entry in the iteration to the nested component instance's entry property (which is just a plain class member on the PlaylistEntryComponent class).

Note that we use two custom elements in the template: ng-for and playlist-entry. Looking at the PlaylistComponent class, you see these are explicitly listed under directives in the @View annotation. No more guessing where the 'magic' elements are coming from! It's right there. And not just as strings, but properly imported and referenced from the file they are defined. In this case, ng-for hails from Angular2 itself, and PlaylistEntryComponent is defined earlier in the same file (omitted above). You might be wondering about the slightly funky syntax with asterisks and brackets. There's a method to the madness, fortunately. Read this post for a more in-depth treatment of Angular2 template syntax. And yes, it is 100% valid HTML attribute syntax, in case you were wondering.

One fair warning when working with components: component declaration order within a single source file does matter. I started out defining PlaylistComponent first, and the PlaylistEntryComponent later in the file. It seemed so logical, but it broke at runtime. There's a forward reference to a class that doesn't exist yet in the directives property of PlaylistComponent. That makes for some nice error messages and stacktraces in the console, I can tell you. (for the unlucky googler who is suffering from this problem: 'EXCEPTION: Unexpected directive value 'undefined' on the View of component 'PlaylistComponent' was the error with Angular2.alpha45 and earlier)

Moral of the story: define (or import) your components before referencing them in other components. Or resort to ugly workarounds.

Component interaction

So we have a component tree, components encapsulate data and can render themselves initially. Next question: how does anything get done? How do components interact with the user and each other?

With AngularJS 1.x, you're used to 2-way databinding by default. In Angular2, by default data flows uni-directionally, from the root component to the children. We already saw an example of passing down data to child components through their attributes, which end up on component class members. This is a one-way street. You have two main ways of communicating between arbitrary, non-hierarchical components: events, and shared components.

This example uses shared components. It is also possible to define custom events and trigger behavior throughout the component tree. However, not all custom events are propagated correctly yet in the alpha-versions I worked with. You will not find an example of using custom events in YouTube live!, but you can find more information in this post.

An example of shared components in action is playing a video in YouTube live. It's possible to start a video both from the playlist entries and the search results. This shared functionality can be achieved by simply creating a VideoPlayer class with the appropriate methods and state:

export class VideoPlayer {
  public isPlaying = false;
  public currentVideoUrl: string

  public playConcert(id: string) {
    this.isPlaying = true;
    this.currentVideoUrl = this.concertIdToEmbedUrl(id);
  }

  public stop() {
    this.isPlaying = false;
    this.currentVideoUrl = undefined;
  }

  private concertIdToEmbedUrl(id: string): string {
    return yt_embed + id + '?showinfo=0&autoplay=1';
  }
}

It's just a plain class, no special Angular annotations necessary. There is no view attached. One caveat: if we wanted to inject other components into this class, an @Injectable annotation would have been necessary. We can inject this VideoPlayer class into existing components through their constructors. It's a bit like services in AngularJS 1.x.

Take for example the SearchResult component, showing the constructor injection:

@Component({
  selector: 'search-result',
  properties: ["concert"],
  providers: [LocalStoragePlayList]
})
@View({
  templateUrl: "ytlive/search/searchresult.html",
  directives: []
})
class SearchResultComponent {
  concert: ytbackend.ConcertSummary

  constructor(private playlistService: LocalStoragePlayList,
     private videoPlayer: ytbackend.VideoPlayer) {}

  addToPlaylist(concert: ytbackend.ConcertSummary) {
    this.playlistService.addConcert(concert);
  }

  playConcert(id: string) {
    this.videoPlayer.playConcert(id);
  }
}

Two things are injected into the constructor: LocalStoragePlayList (so we can save search results) and VideoPlayer (so we can play search results). In the @Component annotation, you can see that the injection of LocalStoragePlaylist is setup in the providers property. But VideoPlayer is not mentioned there. How come? When you define a provider, that is also the level where the to-be-injected component is instantiated. This instance is then available to the component and all its child components for injection. Therefore, the VideoPlayer provider is setup in the root YTLiveComponent. This way, the same instance of the VideoPlayer is injected into all components that request it in their constructors. That's good, because there is only one viewport for the videos. One video can be played at the time, which makes the VideoPlayer is a shared resource that's used by multiple other components.

Playing a concert is as simple as calling the playConcert method on the SearchResultComponent from the searchresult template:

searchresult.html:

<!-- lots of stuff omitted -->
<button title="Play now" class="play btn btn-success">
    <span (click)="playConcert(concert.id)" class="glyphicon glyphicon-play-circle"></span>
</button>

It binds the click-event on this span to the playConcert method on the SearchResultComponent, which in turn calls the shared VideoPlayer component. Binding to DOM-events like this is the primary means of user-interaction. Obviously, higher-level components are available for easily integration input components et cetera.

The state of the VideoPlayer instance is watched in another component/template:

search.html:

<!-- lots of stuff omitted -->
<div *ng-if="playing" id="concerts" class="row">
  <iframe width="100%" height="100%" [src]="embedUrl" frameborder="0" allowfullscreen></iframe>
</div>

The [src] syntax binds the src property of the iframe to the embedUrl property of the component for this template. If the embedUrl changes, the src of the iframe is automatically updated (but not the other way around).

Http service

Angular is more than just a front-end component framework. In AngularJS 1.x there was an $http service to do backend calls. The same applies to Angular2. Instead of returning (their own flavor) of Promises like in 1.x, the new Http component returns RX Observables. Angular2 adopts RxJs as core dependency, you see it popping up in several APIs. It takes some getting used to, but RxJs is a proven library offering a great way to compose asynchronous data flows.

In YouTube live!, we use an injected Http component to do the YouTube API calls:

@Injectable()
export class ConcertService {

  private concerts: ConcertSummary[];

  constructor(private http: Http) { }

  public findConcerts(artist: string, duration = Duration.FULLCONCERT): any {
   var ytDuration: string
   var searchString = yt_search + ytDuration + '&q=' + encodeURIComponent('live ' + artist);

   return this.http.get(searchString).map((res: any) => {
      var ytResults: {items: YTSearchResult[] } = res.json();
      var transformedResults = ytResults.items.map(this.toConcertSummary)
      this.concerts = transformedResults;
      return transformedResults;
    });
  }
}

Again, we see a viewless component, but this time with the @Injectable annotation since we need Angular to inject the Http component in the constructor. After performing a get call, the result is transformed using map on the observable. This returns another observable, now containing data in a format we can use. One slight annoyance is that the Http.get returns any in the current typing definition of Angular2. It would be nice to use the RxJS type definitions for Observables, so we can get some compile-time sanity back here as well.

The resulting Observable is used in the searchConcerts method on SearchComponent:

export class SearchComponent {

  private concerts: ytbackend.ConcertSummary[] = [];

  constructor(private concertService: ytbackend.ConcertService,
      private videoPlayer: ytbackend.VideoPlayer) { }

  searchConcerts(): void {
    this.videoPlayer.stop();
    this.concertService
      .findConcerts(this.searchTerm)
      .subscribe((results: ytbackend.ConcertSummary[]) => this.concerts = results);
  }
}

Since the ConcertService returns an observable, we cannot assign it directly to a class member of type ConcertSummary[]. Instead, we subscribe to the observable and assign the result once our subscriber is called when results are available. The template automatically detects changes to concerts and shows the new results from the API call. It would be nice if this manual 'unwrapping' of Observables would not be necessary.

Wrapping up

This post barely scratches the surface of what features are in Angular2. There's a whole new approach to Forms, a new Router and much more. You will find the documentation to be inadequate though. There's also lots of outdated information on the web, especially given the pace of the alpha releases and the amount of breakage between releases. This article itself will be no exception, probably.

Still, a more stable period is forthcoming with the Angular2 beta nearing. Now is a good time to start learning the concepts of Angular2, but don't expect it to be a beginner-friendly experience. There's definitely some rough edges to Angular2, but all in all it looks very promising to me.

Play around with the code for YouTube live and let me know what you think!

From OSGi to Jigsaw

2015-10-27T00:00:00+01:00

In my last post I introduced the new Java Platform Module system with a small example. As of September, there is an implementation to play with, codename Jigsaw. After some introductory toy examples, I did what any curious coder does: port a familiar (and not entirely trivial) codebase to the new shiny technology. In this case, I took a small dynamic dashboard based on OSGi and implemented it using the proposed Java Platform Module System.

If you want an introduction on what the new Java Platform Module System (referred to as JPMS hereafter) entails and how to get started, read 'The Java Module system: a first look'. This post assumes you're familiar with the basics of the proposed module system. And if you're the kind of person who just wants to see the code: here you go.

The original OSGi application

Before diving into the Jigsaw port, let's have a look what the original application is all about:

As you can see, it mimics a car entertainment system dashboard. However, the requirement is that this car entertainment system can be dynamically extended and updated. Each of the 'apps' come from a separate OSGi bundle. The dashboard only collects applications that are currently provisioned to the OSGi runtime and shows them. You can click on the icons to access the underlying 'app' (music player, navigaton, phone), which come from the same bundle that contributes the dashboard icon. This example has served as demo for a conference talk by myself and my colleague Paul Bakker called 'Provisioning the IoT'. In this talk, we use Apache ACE to dynamically update and provision OSGi bundles to running instances of the car entertainment system on multiple devices. It's actually really cool to see your system update in real-time without restarting. If you want to see it in action I recommend watching the talk. The demo starts around the 11 minute mark.

Technically, the dynamic dashboard looks up all instances of the App interface in the OSGi service registry. This interface is almost the only piece of code that is publicly shared between bundles. In turn, bundles containing an App implementation register themselves upon bundle start, and unregister when the bundle is stopped. This makes full use of the dynamic life-cycle afforded by OSGi. The dashboard gets App instances from the service registry without having to know about the implementation classes. Inversion of control in action! Each app implementation bundle also provides its own resources such as images. You can check out the original application on GitHub.

Finding the right modules

The question is, how hard is it to port this modularized OSGi application to the JPMS using the Jigsaw prototype? Being as dynamic as OSGi isn't a goal of the JPMS. So to keep expectations in check, I'm already happy if we can port the module and service structure at startup. Adding and removing new modules dynamically will have to wait for now.

Our challenge is to translate the OSGi bundles into equivalent Jigsaw modules. The first step for re-creating this example in the JPMS is to find out what should go into the module-info.java descriptors. These module descriptors contain the dependency information for Java modules. It is similar to the OSGi meta-data in the manifest file of OSGi jars.

The most straightforward module definition is the one for the API bundle:

module carprov.dashboard.api {
   exports carprov.dashboard.api;
   requires public javafx.graphics;
}

You can find the full code for the Jigsaw version of the dashboard on GitHub if you want to follow along. It compiles and runs on build b86 of the Jigsaw-enabled JDK.

We declare a module with the name carprov.dashboard.api, exporting a package of the same name. Meaning the interface and helper class inside this package are visible to all modules that import this module. Next, we need to declare what this module needs in terms of dependencies. Since the App interface uses JavaFX types, these need to be required somehow. An important goal of the JPMS is to modularize the itself JDK as well. Therefore we cannot just import types from the JDK without specifying which module they come from. Note that unlike the exports-clause, the requires-clause takes a module name rather than a package name.

So how do you find the right module to require amongst the ~80 modules that currently comprise the JDK in the Jigsaw prototype? Fortunately, we can do better than trial and error. The JDK provides a tool called jdeps, which analyzes the dependencies of an existing Java class. You provide the class name and an appropriate classpath that contains the class:

$ jdeps -module -cp carprov.dashboard.api.jar carprov.dashboard.api.App
carprov.dashboard.api -> java.base
carprov.dashboard.api -> javafx.graphics
   carprov.dashboard.api (carprov.dashboard.api)
      -> java.lang                                          java.base
      -> javafx.scene                                       javafx.graphics

The last two lines indicate that the App interface imports from the java.lang and javafx.scene packages. By providing the -module option, jdeps also outputs the source modules (on the far right). This way, you can identify the modules providing the packages that the analyzed class depends on. In this case, the dashboard module should require the java.base module and the javafx.graphics module from the JDK. That's exactly what we did in the module-info.java descriptor earlier. Except, the java.base module is always implicitly required for all modules. You can't live without it.

Another option for finding the right modules is to peruse the module overview page of the early access Jigsaw build. It gives a comprehensive overview of all JDK modules and their dependencies. To get a feeling for the new modularized Java platform, it's indispensable.

There's on last twist: what does the public in requires public mean in the module descriptor? Let's have a look at the App interface:

import javafx.scene.Node;

public interface App {
   String getAppName();
   int getPreferredPosition(); 

   Node getDashboardIcon();
   Node getMainApp();
}

If only the carprov.dashboard.api package would be exported by the Dashboard API module, what happens if another module imports it and tries to use it? That consuming module is then forced to also require the module containing javafx.scene.Node (in this case javafx.graphics). Since Node is used as return type in App, the interface cannot be used without access to this class as well. You could document this as part of the Dashboard API module, but that's error-prone and generally unsatisfactory. The public keyword in the requires-clauses solves this. Effectively, it re-exports the public packages from the required module as part of the Dashboard API module. Now, the app implementation modules can require the Dashboard API module without having to worry about requiring javafx.graphics. Without the public keyword, compilation fails unless the consuming module itself imports the javafx.graphics module.

This re-exporting mechanism solves the same problem that OSGi 'uses-constraints' solve. It goes a bit further though. With the re-exporting mechanism in the JPMS, you can create an 'empty' module that acts as a façade. The public exports in the module descriptor of this empty module can aggregate several other modules. As an example, you can use this mechanism to split a module into multiple modules without breaking consumers. They still require the same module, only now it 'delegates' to other modules.

However, we're getting off track. Back to porting the dashboard example. How do the apps actually end up on the dashboard using the JPMS?

Services with ServiceLoader

So far, we've talked about a single module and its dependencies: the dashboard API. However, the diagram above shows 5 modules in the sample application. What about the dashboard implementation module, and the App implementation modules? We explicitly do not want the dashboard to know about the concrete App implementation classes. It just needs to gather instances of those implementation classes, without doing the instantiation itself. Loose coupling, remember?

This means we don't require any App implementation modules in the dashboard's module-info:

module carprov.dashboard.jfx {
   requires carprov.dashboard.api;
   requires javafx.base;
   requires javafx.controls;
   requires javafx.swing;

   uses carprov.dashboard.api.App;
}

The interesting part is the last line of the module descriptor: uses carprov.dashboard.api.App;. With this uses-clause, we tell the JPMS that we are interested in instances of App interface. Subsequently, the dashboard can use the ServiceLoader API to retrieve these instances:

Iterable<App> apps = ServiceLoader.load(App.class);

for(App app: apps) {
  renderDashboardIcon(app);
}

Instances are created by the module system. Of course, the big question is: how does the module system locate service providers?

Let's look at an example of a module providing an App service. The Phone module exposes its App implementation as follows:

module carprov.phone {
   requires carprov.dashboard.api;
   requires javafx.controls;

   provides carprov.dashboard.api.App with carprov.phone.PhoneApp;
}

The magic happens in the last line. It indicates that we want to expose an App instance, using the concrete PhoneApp implementation class. Note that PhoneApp's package is not exported. Nobody can instantiate it but the JPMS, or another class inside the same module. There is one requirement for a service class: it must have a default no-arg constructor. You can even provide services and consume them in the same module. See the actual source of the Dashboard implementation for an example of both a uses and provides-clause in the same module descriptor.

Now the JPMS knows that the dashboard implementation module wants to see App instances, and the Phone module (and others) provide these instances. If at any time an additional service implementing the App interface is put on the modulepath, the dashboard will pick it up without modifications to the module descriptor. It's not as dynamic as the original OSGi application, though. Only after a restart of the whole application (JVM) are these new modules loaded.

For those who know the OSGi service model, statically describing service dependencies is quite a difference. OSGi services come and go at run-time. On the one hand, this is more powerful and dynamic. On the other hand, the JPMS approach could provide errors in case wiring is not possible at startup time. By the way, the current prototype does not appear to do so. Declaring a uses-clause on an interface without any implementations on the modulepath at runtime does not trigger any warnings or errors. It's still on my list to experiment with the Layer construct of the JPMS. Let's see how close it can bring us to loading additional modules on-the-fly.

In short, the ServiceLoader mechanism allows us to hide implementations in a modular world. It's not quite dependency injection but it is a form of inversion of control. I'm sure dependency injection models will be built upon this foundation.

Resources

Modules can encapsulate more than just code. In this application, we need images as well. Loading resources using Class.getResourceAsStream still works, with some caveats. The class calling this method must be in the same module that contains the resource. Otherwise, null is returned.

The original OSGi implementation delegated loading resources to a helper class in the Dashboard API bundle. It did this by passing the BundleContext of the requesting bundle to this helper class. The BundleContext provides access to the bundle and its meta-data.

public static ImageView getImageByFullname(BundleContext bundleContext, String name) {
  URL entry = bundleContext.getBundle().getEntry(name);
  try {
    Image image = new Image(entry.openStream());
    ImageView view = new ImageView(image);
    view.setPreserveRatio(true);
    return view;
  } catch (IOException e) {
    throw new RuntimeException(e);
  }
}

I tried to emulate this by passing a Class object from the requesting module to a similar helper class in the JPMS version:

public static ImageView getImageT(Class<?> loadingContext, String name) {
  Image image = new Image(loadingContext.getResourceAsStream(name));
  ImageView view = new ImageView(image);
  view.setPreserveRatio(true);
  return view;
}

However, the access checks do not seem to care about the Class object which getResourceAsStream is invoked on, but rather on which class is on top of the call-stack. That's of course the module that contains the helper class, which cannot read resources from the module that called the helper method. In that case, getResourceAsStream just returns null. That lead to some interesting NullPointerExceptions and confused looks on my face. In the end, I just had my requesting modules call getResourceAsStream and pass the resulting InputStream to the helper instead:

public static ImageView getImage(InputStream stream) {
     Image image = new Image(stream);
     ImageView view = new ImageView(image);
     view.setPreserveRatio(true);
     return view;
}

After talking to Mark Reinhold at JavaOne, I learned this behavior is by design. There is an alternative that looks more like the BundleContext solution: you can also pass a java.lang.reflect.Module to a helper method like the one above. This reified module instance effectively allows the recipient to do anything they would like with the calling module. Including getResourceAsStream on that module.

List loaded modules

The original dashboard had an app that lists the loaded OSGi bundles comprising the whole application. Naturally, that needs to be ported as well. There is a new API for introspecting modules of the JPMS. Using it is fairly straightforward:

Layer layer = Layer.boot();
for (Module m: layer.modules()) {
  if(m.getName().startsWith("carprov")) {
     String name = m.getName();
     Optional<Version> version = m.getDescriptor().getVersion();
     // Show it in the ui
  }
}

Modules are organized into Layers. Since we do not specifically create a module Layer ourselves, the loaded modules are part of the boot-layer. We retrieve this layer, and ask it for all the loaded modules. Then, we only process modules that start with "carprov", in order to not show JDK modules in the overview.

Conclusion

It's going to be interesting to see how the current JPMS prototype will morph into a production-ready module system for Java 9. One thing is sure: it's a big step forward for the Java platform.

All in all I was pleasantly surprised how far I could come with the Jigsaw prototype. Yes, it is less dynamic than the OSGi original. On the other hand, it is also vastly less complex. OSGi service dynamics are cool, but it makes you handle lots of (concurrency) edge-cases. Do you really need these dynamics all the time? Nevertheless, my next challenge will be to bring some of the original dynamics back using the JPMS. Stay tuned!

The Java Module system: a first look

2015-09-10T00:00:00+02:00

A module system for Java has been a long time coming. Late 2014, a new JSR requirements document (JSR-376) was created to this end. The changes are slated for Java 9. However, no working prototype was available. Until yesterday, that is. There now is an OpenJDK early access build that includes Project Jigsaw.

Together with my co-worker Paul Bakker I gave a talk on the proposed Java Module system at JavaZone yesterday. We had to base this entirely on the JSR-376 requirements document and other tidbits of information floating around. While proposing this talk at the beginning of the year, we definitely thought a prototype would be available to showcase. However, that didn't quite pan out the way we thought. Instead, the prototype was released just hours after our talk ended (phew). Which means some things we say in the talk are already outdated, but the main ideas still stand. If you're completely new to the Java Module system proposal, I recommend you watch our talk before reading on. It explains the current proposal and puts it in a broader context by comparing it to OSGi.

Why modules?

So what are modules, and why do we want them? If you want an in-depth discussion, read the 'State of the module system' or watch our talk. For the uninitiated, here's the Cliff's notes version.

Java has jar files. But really, these are just glorified zip-files containing classes which in turn are inside packages. When you assemble and run an application consisting of different jar files (read: every non-trivial application), you put them on the classpath. And then hope for the best. Because there's no way to tell if you put everything on the classpath that your application needs. Or whether you inadvertently put the same classes (in different jars) on the classpath. Classpath-hell (analogous to DLL-hell) is a real thing. This leads to bad situations rearing their ugly head at runtime. Also, the knowledge that a class was ever in a jar file is lost at runtime. The JRE just sees one big collection of classes. But jars need other jars. It's just not encoded explicitly in any form of meta-data at the moment. Ideally, you would also be able to hide implementation classes inside your jar and only expose your public API. The proposed module system for Java aims to solve these issues:

modules become first-class citizens that can encapsulate implementation details and expose only what is needed
modules explicitly describe what they offer, and what they need (dependencies), hence dependencies can be verified and resolved automatically during all phases of development

Having such a module system greatly improves maintainability, reliability and security of large systems. Not in the least of the JDK itself. Given such a system, a module graph can be automatically constructed. This graph contains only the necessary modules to run your application.

Installing JDK9 early access

If you want to follow along with the example code yourself, you need to install the JDK9 early access build that includes the Jigsaw prototype. On OSX, this means extracting the archive, and moving the extracted directory to /Library/Java/JavaVirtualMachines/. Then, you need to adjust your path and JAVA_HOME environment variable to point to the JDK9 directory. I'm using the excellent setjdk bash script to switch between Java installations on the command line. You most certainly don't want to use this early access build as your daily Java installation. You can verify that the installation works by executing java -version. The output should read something like:

java version "1.9.0-ea"
Java(TM) SE Runtime Environment (build 1.9.0-ea-jigsaw-nightly-h3337-20150908-b80)
Java HotSpot(TM) 64-Bit Server VM (build 1.9.0-ea-jigsaw-nightly-h3337-20150908-b80, mixed mode)

As long as it includes the phrase Jigsaw, you're good to go. The resulting code for the example coming up can found at https://github.com/sandermak/jigsaw-firstlook.

A small example

You can still use JDK9 in 'legacy-mode' with just classes, jars and the classpath. But obviously we want to work with modules. So we'll create a project that produces two modules, where module1 uses code from module2.

The first thing to do, is to structure your project so that modules are clearly separated. Then, meta-data needs to be added to modules in the form of a module-info.java file. Our example is structured as follows:

src\
  module1\
     module-info.java
     com\test\TestClassModule1.java
  module2\
     module-info.java
     com\moretest\TestClassModule2.java

Effectively, this introduces another layer (module1, module2) on top of the package layering that you already do in Java. In these 'module directories', we find the module-info.java descriptor at the root. Furthermore, note that the two classes are in distinctly named packages.

Let's look at the code for TestClassModule1:

package com.test;

import com.moretest.TestClassModule2;

public class TestClassModule1 {
   public static void main(String[] args) {
     System.out.println("Hi from " + TestClassModule2.msg());
   }
}

Looks pretty vanilla, right? Nothing related to modules going on here. There is an import for the TestClassModule2, on which the main method later calls the msg() method:

package com.moretest;

public class TestClassModule2 {
   public static String msg() {
     return "from module 2!";
   }
}

For now, we'll leave the module-info.java files empty.

Compiling Java modules

Now for the next step: actually compiling our modules and associated source-files. To make this work, a new javac compiler flag is introduced:

javac -modulesourcepath src -d mods $(find src -name '*.java')

This assumes you run the command from the parent directory of the src dir. The -modulesourcepath flag switches javac into module-mode, rather than 'legacy' mode. The -d flag indicates the output directory for the compiled modules. These are output by javac in an exploded directory format. If we later want to deliver modules as jars, that's a separate step.

So what happens if we execute the above javac invocation? We get errors!

src/module1/module-info.java:1: error: expected 'module'
src/module2/module-info.java:1: error: expected 'module'

The empty module-info.java files are wreaking havoc here. Some new keywords are introduced for these files, the most important being module. These new keywords are scoped to the module-info.java definition. You can still use variables called module in other Java source files.

We update the module descriptors to contain the minimal amount of information necessary:

module module1 { }

and for module2:

module module2 { }

Now the modules are explicitly named in their definitions, but do not contain any other meta-data yet. Compiling again leads to new errors:

src/module1/com/test/TestClassModule1.java:3: error: TestClassModule2 is not visible because package com.moretest is not visible

Encapsulation in action! By default, all classes/types inside a module are hidden to the outside world. That's why javac disallows the usage of TestClassModule2, even though it is a public class. If we were still in a flat classpath world, everything would be fine and dandy. Of course we can fix this, by explicitly exposing TestClassModule2 to the outside world. The following changes are necessary in module2's module-info.java:

module module2 {
  exports com.moretest;
}

That's not enough. If you compile with this change, you still get the same error. That's because module2 now exposes the right package (and thereby all it's containing public types), but module1 does not yet express its dependency on module2. We can do that by changing module1's module-info.java, too:

module module1 {
   requires module2;
}

Requirements are expressed on other modules by name, whereas exports are defined in terms of packages. Much can be said about this choice, but I won't go into this for a first look. After making this change, we have our first successful compilation of a multi-module build using the Jigsaw prototype. If you look inside the /mods directory, you see the compiled artifacts neatly separated into two directories. Congratulations!

Running modular code

Just compiling is not much fun of course. We also want to see the app running. Fortunately, the JRE and JDK have also been made module-aware in this prototype. The application can be started by defining a modulepath rather than classpath:

java -mp mods -m module1/com.test.TestClassModule1

We point the modulepath to the mods dir that javac wrote to. Then, -m is used to indicate the initial module that kickstarts the resolving of the module graph. We also tack on the name of the main class that should be invoked, and there we have it:

Hi from from module 2!

Future

This first look gives a taste of what you can do with modules in Java 9. There's lots more to explore here. Like packaging: besides jars, there is a new format coming called jmod. The module system also includes a services layer that can bind service providers and consumers through interfaces. Think of it as inversion of control where the module system fulfills the role of service registry. It's also very interesting to see how the module system was used to modularize the JDK itself. This in turn enables nice things like creating a run-time image that contains just the JDK and application modules that your app needs, nothing more. Lower footprint, more options for whole-program optimization, etc. It's all very promising.

The next step for me is to try and port a sample OSGi application that uses several modules and services to the Java 9 module system. Stay tuned!

On monoliths, microservices and critical thinking

2015-06-10T00:00:00+02:00

Something very interesting happened last week. At first I wasn't sure whether it was a misguided attempt at humor. Sure enough, Twitter timelines switched en masse from advocating microservices to glorifying monoliths. What a strange world we live in.

Microservices

Microservices burst onto the conference scene, blogosphere and other echo-chambers with blunt force in the past years. Like all movements, the microservices movement offers some good points amongst the propaganda. We are dealing with increasingly complex problems at an increasingly large scale. It's good to pause and reflect on these challenges.

Except, that's not what happens. Instead, microservices frameworks, microservices books and other microservices paraphernalia convert large swathes of developers into 'early adopters' of the golden age. Vendors create microservices platforms, including some sweet lock-in of course. Suddenly we're counting lines of code like a weightwatcher counts calories. Must. Not. Exceed. <insert arbitrary figure here>. It's unhealthy.

Monoliths

Then, pendulum swings back. Martin Fowler publishes his MonolithFirst post. And the people rejoice. It's ok to officially push back against the cool kids again. Now you're scolded for ever starting from scratch with a service oriented architecture. YAGNI, silly! And if you do need it, just chop up your monolith and season to taste. Easy, right?

As you might have guessed by now, this post is not about technical arguments either for or against monoliths and microservices. If you're looking for that, read Fowler's post. Then, read Stefan Tilkov's counterpoint (graciously hosted on Fowler's site as well). I'm much more interested in what this all means for our profession. What does it mean if public software engineering opinion flips 180 degrees in a matter of weeks? It's too easy to chalk it all up to people needing authority figures. Yes, I know: not everybody was all over microservices. But you have to admit there's something fundamentally unsound going on here.

The art of problem solving

Software development (and by extension, software architecture) is often characterised as a form of problem solving. There's a problem (requirements, input), and we need to create a solution (software, output). It's not wrong per se, but framing it like this primes your thinking in a dangerous way. If there's a problem, there must be an exact solution, right?

However, we all know that there's no single perfect solution. There's a vast solution space and it is our task to navigate it and arrive at a reasonable location in this space. All hypes like microservices are doing, is prematurely pruning our solution space. Without knowing anything about the problem. That's wrong. We all know it. Still, the solution space is often so enormous, it is tempting to think "this time it's different" when a new voice of (apparent) reason appears. Hopefully, the monolith vs. microservices farce brings us back to reality. Critical thinking cannot be done for you.

Risk management

Instead of problem solving, a much better characterisation of software architecture is risk management. There are trade-offs in every dimension of the solution space. A software architect must quantify, disqualify and hedge the risks. This is a tough job. There's no substitute for experience here. Books like 'The architecture of open-source applications', containing case-studies, are much more valuable than books on architecture patterns in that sense. Managing risks means building up your architecture from first principles. Not arriving at some architecture through elimination of fashionable trends.

Architectural patterns are tools for navigating the solution space we must explore. Ultimately, the problem we're solving dictates which direction to go. Not our philosophical preference for certain solutions, unless it is based on similar problems we have solved before. If technical debt is like that irresistible car loan - costly but typically harmless -, then choosing the wrong architecture is more like a sub-prime mortgage. Hey, everybody does it. And when you are left homeless, there's nothing else to blame than your own critical thinking skills.

The Road to Data Science

2014-10-27T00:00:00+01:00

Data is invariably a large part of the applications we deliver. Thinking about data, we often are concerned with ACID, scalability and other operational aspects. Building an (enterprise) app means we want our data to be safe, accessible and acted upon by our business rules. Sure, there is the occasional reporting need, but usually nothing a few well-placed queries won't be able to solve.

(this article appeared in an edited form in DZone's Big Data guide)

Shifting sands

But the world has changed. Gmail sorts our priority inbox for us. Facebook decides what's important in our newsfeed on our behalf. E-commerce sites are full of recommendations, sometimes eerily accurate. We see automatic tagging and classification of natural language resources. Ad-targeting systems predict how likely you are to click on a given ad. The list goes on and on. Applications are getting smarter with their data. It's not just about managing and consuming data anymore. It's all about deriving knowledge and insights from existing data, predicting future data and creating a customised user experience. And not just in fancy reports for management (think 'Business Intelligence'), but as an integral part of the application. Data drives the user experience directly, not after the fact. This is what data science is all about. Granted, the term is incredibly hyped, but there's a lot of substance behind the hype. So we might as well give it a name and try to figure out what it means for us as developers.

All the applications discussed above arose in the context of the large web-giants (Google, Yahoo, Facebook) and lots of startups. Yes, these places are filled to the brim with very smart people, working on the bleeding edge. But make no mistake. This trend will trickle down into 'regular' application development just as well. Users' expectations for business applications increase, if only because they interact with slick and intelligent apps every day at home. For enterprise applications it's not a matter of if, but when.

From developer to data scientist

How do we cope with these increased expectations? It's not just a software engineering problem. You can't just throw libraries at it and hope for the best. Yes, there are great machine learning libraries, like Apache Mahout (Java) and scikit-learn (Python). There are even programming languages squarely aimed at doing data science, such as the R language. But it's not just about that. There is a more fundamental level of understanding you need to attain to wield these tools.

No, this article will not be enough to gain this level of understanding. It can, however, show you the landmarks along the road to data science. This diagram (adapted from Drew Conway's original) shows the lay of the land:

As software engineers we can relate to hacking skills. It's our bread and butter. And that's good, because from this solid basis you can branch out into the other fields and become more well-rounded. Let's tackle domain expertise first. It may sound obvious, but if you want to create good models for your data, you need to know what you're talking about. This is not strictly true for all approaches. For example deep learning and other machine learning techniques might be viewed as an exception. In general though, having more domain-specific knowledge is better. So start looking beyond the user-stories in your backlog and talk to your domain experts about what really makes the clock tick. Beware though. If you 'only' know your domain and can churn out decent code, you're in the danger zone. Meaning you're at risk of re-inventing the wheel, misapplying techniques and many other ways of shooting yourself in the foot.

The elephant in the room here is of course math & statistics. The link between math and the implementation of features such as recommendation or classification is very strong. Even if you're not building a recommender algorithm from scratch (which hopefully you wouldn't), you need to know what goes on under the hood in order to select the right one and to tune it correctly. As the diagram points out, the combination between domain expertise and math & statistics knowledge is traditionally the area of researchers or analysts within companies. But when you combine these skills with software engineering prowess, many new doors will open.

What can you do as developer if you don't want to miss the bus? Before diving head-first into libraries and tools, there are several areas where you can focus your energy:

Data management
Statistics
Math

We'll look at each of them in the remainder of this article. Thinks of these items as the major stops on the road to data science.

Data management

Features like recommendation, classification and prediction cannot be coded in a vacuum. You need data to drive the process of creating/tuning a good recommender engine for your application, in your specific context. So it all starts with gathering relevant data. It might already be in your databases, or you might have to setup new ways of capturing relevant data. Then comes the act of combining and cleaning data. This is also known as data wrangling or munging. Different algorithms have different pre-conditions on input data. You'll have to develop a good intuition for good data versus messy data.

Typically, this phase of a data science project is very experimental. You'll need tools that help you quickly process lots of heterogeneous data and iterate on different strategies. Real world data is ugly and lacks structure. Dynamic scripting languages are often used for these tasks because they fit this challenge perfectly. A popular choice is Python with Pandas or the R language.

It's important to keep a close eye on everything related to data munging. Just because it's not production code, doesn't mean it's not important. There won't be any compiler errors or test failures when you silently omit or distort data, but it will influence the validity of all subsequent steps. Make sure you keep all your data management scripts, and both mangled and unmangled data. You can then always trace your steps. Garbage in, garbage out applies as always.

Statistics

Once you have data in the appropriate format, the time has come to do something useful with it. But your data is typically just a sample. You want to create models that handle yet unseen data. How can you infer valid information from this sample? How do you even know your data is representative? Enter the domain of statistics, a vitally important part of data science. I've heard it put like this: 'a Data Scientist is a person who is better at statistics than any software engineer and better at software engineering than any statistician'.

What should you know? Start by mastering the basics. Understand probabilities and probability distributions. When is a sample large enough to be representative? Know about common assumptions such as independence of probabilities, or when values are expected to follow a normal distribution. Many statistical procedures only make sense in the context of these assumptions. How do you test the significance of your findings? How do you select promising features from your data as input for algorithms? Any introductory material on statistics can teach you this. Then, move on the Bayesian statistics. It will pop up more and more in the context of machine learning.

It's not just theory. Did you notice how we conveniently glossed over the 'science' part of data science up till now? Doing data science is essentially setting up experiments with data. Fortunately, the world of statistics knows a thing or two about experimental setup. You'll learn that you always should divide your data in a training set (to build your model) and test set (to validate your model). Otherwise, your model is no good for unseen data: the problem of overfitting. Even then, you're still susceptible to pitfalls like multiple testing. There's a lot to take into account.

Math

Statistics tells you about the when and why, but for the how, math is unavoidable. Many popular algorithms such as linear regression, neural networks and various recommendation algorithms all boil down to math. Linear algebra, to be more precise. So brushing up on vector and matrix manipulations is a must. Again, many libraries abstract over the details for you. But it is essential to know what is going on behind the scenes, in order to know which knobs to tune. When results are different than expected, you need to know how to debug the algorithm.

It's also very instructive to try and code at least one algorithm from scratch. Take linear regression for example, implemented with gradient descent. You will experience the intimate connection between optimization, derivatives and linear algebra when researching and implementing it. Andrew Ng's Machine Learning class on Coursera takes you through this journey in a surprisingly accessible way.

But wait, there's more...

Besides the fundamentals discussed so far, getting good at data science includes many other skills as well. Such as clearly communicating results of data-driven experiments. Or scaling whatever algorithm or data munging method you selected across a large cluster for large datasets. Also, many algorithms in data science are 'batch-oriented', requiring expensive recalculations. Translation into online versions of these algorithms is often necessary. Fortunately, many (open-source) products and libraries can help with the last two challenges.

Data science is a fascinating combination between real-world software engineering and math & statistics. This explains why the field is currently dominated by PhDs. On the flipside, we live in an age where education has never been more accessible. Be it through MOOCs, websites or books. If you want read a hands-on book to get started, read Machine Learning for Hackers. Then, move on to a more rigorous book like Elements of Statistical Learning. There are no shortcuts on the road to data science. Broadening your view from software engineering to data science will be hard, but certainly rewarding.

Facebook announces Apollo at QCon NY 2014

2014-06-11T00:00:00+02:00

Today I attended a talk 'How Facebook Scales Big Data Systems' at QCon NY 2014. It turns out that Jeff Johnson took this opportunity to announce a new project by Facebook: Apollo. The following is a rough transcript of the talk.

Apollo is a system Facebook has been working on for about a year in their New York office. The tagline of Apollo is 'consistency at scale'. Its design is heavily influenced by their experience with HBase.

Why?

Facebook generally favors CP systems. They have four datacenters using master/slave style replication. Missing the 'A'vailability usually isn't a big deal within a datacenter, but across DCs it becomes troublesome.

The wishlist that lead to Apollo is roughly as follows:

we need some kind of transactions
acked writes should be eventually visible and not lost
availability

So the question really boils down to: can we layer AP on top of CP?

Apollo

Apollo itself is written in 100% C++11, also using thrift2. The design is based on having a hierarchy of shards. These form the basic building block of Apollo. Essentially like HDFS (regions) are the building block for HBase. It supports thousands of shards, scaling from 3 servers to ~10k servers.

The shards use Paxos style quorum protocols (CP). Raft is used for consensus. Fun sidenote: this turned out to be not much simpler than multi-paxos, even though that was the expectation.

RocksDB (a key-val store, log-structured storage) or MySQL can be used as underlying storage. The storage primitives offered by Apollo are:

binary value
map
pqueue
CRDTs

Use cases

The apparent sweetspot for Apollo is online, low-latency storage of aforementioned data structures. Especially where atomic transformations of these data structures are required. The size of 'records' should range from a byte to about a megabyte.

Every operation on the Apollo api is atomic. For exampe, you can pass in conditions into a read/write call. As an example, a condition can assert that a map contains a certain key. The operation will only go through if the condition (atomically) holds.

First write call mimics a traditional last-write wins KV store. Second one does optimistic concurrency, last one allows for complex transactional behavior. Note that a write can also return (consistent) reads!

Atomicity works across shards, at varying levels:

The idea is that Apollo provides CP locally and AP between remote sites.

Fault tolerant state machines

In addition to just storing and querying data, Apollo has another unique feature: it allows execution of user submitted code in the form of state machines. These fault tolerant state machines are primarily used for Apollo's internal implementation. E.g. to do shard creation/destruction, load balancing, data migration, coordinating cross-shard transactions.

But users can submit FTSMs too. These are persistently stored, so it tolerates node failures. A shard owns a FTSM. A state machine may have side effects (e.g. call an external API), but all state changes are submitted through the shard replication mechanism. The exact way to create these FTSMs were not discussed.

Applications of Apollo at Facebook

One of the current applications of Apollo at Facebook is as a reliable in-memory database. For this use case it is setup with a RAFT write-ahead-log on a Linux tmpfs. They replace memcache(d) with this setup, gaining transactional support in the caching layer.

Furthermore it powers TACO, an in-memory version of their TAO graph. It supports billions of reads per second and millions of writes this way.

A more persistent application of Apollo is to provide reliable queues, for example to manage the push notifications.

Open source?

Currently, Apollo is developed internally at Facebook. No firm claims were made during the talk that it will be opensourced. It was mentioned as a possibility after internal development settles down.

This was a quick write-up, twenty minutes after the talk ended. Any factual errors are solely caused by me misinterpreting or mishearing stuff. I expect more information to trickle out of Facebook engineering soon. That said, I hope this post paints an adequate initial picture of Facebook's Apollo!

Generating JUnit test reports with Jasmine 2.0

2014-03-06T00:00:00+01:00

While setting up a new project at work, a seemingly small upgrade from Jasmine 1.3 to Jasmine 2.0 broke a vital feature: the ability to generate JUnit XML reports for JavaScript unit tests. Long story short: I created jasmine2-junit to bring this feature to Jasmine 2.0 again.

Jasmine is famous for its soothing HTML based reports:

Running your specs in a browser is great during development. You can easily get an overview of your tests and debug failing tests. However, we also want to run the tests in our CI build and have it report success and failure alongside other automated tests. In that case, a pretty HTML page isn't terribly useful. Fortunately, there's a de facto standard for reporting test results: JUnit XML files. Despite its Java heritage, the JUnit XML format can be (ab)used for all kinds of test results. Most CI servers support it out of-the-box. And to top it off, there's a Jasmine reporters project on GitHub that provides a JUnit XML reporter. So all is well, right?

Until you start using Jasmine 2.0, that is.

Jasmine internals

Jasmine 2.0 was released December 2013 and was pretty much a total overhaul of the project. This blogpost from a Jasmine team member grudgingly admits that the impact on 3rd party add-ons is significant. Still, the concept of additional reporters stands in Jasmine 2.0. So it's just a matter of porting the 1.3 version to the 2.0 reporter interface, right?

Well, no. That's only part of the story. What happened is that Jasmine 2.0 moved away from having a monolithical implementation with a global jasmine object by default. That's good. However, Jasmine is now modularized using a roll-your-own JavaScript module pattern implementation. That's not so good. Originally, you could grab the jasmine global object and register your custom reporter. All you had to do was add the script tag with your own reporter and registration code after including Jasmine's script (see this example). In the new situation, using Jasmine is already a two-step process. First include the core Jasmine scripts. Then, you have to include a boot.js script. This script initializes all the Jasmine modules and exposes the Jasmine API on the appropriate scope. But it also runs this gem:

var currentWindowOnload = window.onload;

window.onload = function() {
  if (currentWindowOnload) {
    currentWindowOnload();
  }
  htmlReporter.initialize();
  env.execute();
};

That's right, it directly starts executing the JavaScript tests. Besides the fact that this horribly breaks with AMD module loading (which we use), there's no window to add additional third-party reporters when using the stock boot.js script. It seems that the idea is to take boot.js and customize it to your needs. This type of copy/paste/modify development mystifies me. Why not split up assembling and initializing the Jasmine internals and the actual configuration and running of tests in two parts? That would make Jasmine much easier to use in the face of third-party extensions.

jasmine2-junit

Sorry if that came about a bit ranty, but these things cost too much time to figure out. I just want to write my tests and be happy... But there's good news: the work I did to port the JUnit XML reporter to Jasmine 2.0 and to create a custom boot.js is now yours for the taking: jasmine2-junit. Setting up the HTML for running your specs is easy:

<!-- Include Jasmine 2.0's assets -->
<link href="lib/jasmine.css" rel="stylesheet" type="text/css">
<script src="lib/jasmine.js"></script>
<script src="lib/jasmine-html.js"></script>

<!-- The JUnit reporter should go before the boot script -->
<script src="../src/jasmine2-junit.js"></script>
<!-- This boot.js is a modified version of Jasmine's default boot.js! -->
<script src="../src/boot.js"></script>

<!-- Include your spec files here -->>
<script src="spec.js"></script>

You can find a working example in the repo. Obviously, running this in a browser will not generate any files because the JavaScript sandbox doesn't offer a file API (you'll see a bunch of errors). Though you can still see the visual report of course. To get the JUnit XML output, run a spec file with the provided runner script for PhantomJS:

phantomjs jasmine2-runner.js spec.html

Running this from a CI build is trivial. I also created a gulp plugin to do so. If there's any interest I might push that to NPM as well. (update: I just did, see gulp-jasmine2-phantomjs)

The Jasmine 2.0 JUnit XML Reporter is a bit less configurable than the original it was based on. Let me know if you need anything, and we'll make it work.

Happy JavaScript testing!

ng-conf 2014: the world's first AngularJS conference

2014-01-20T00:00:00+01:00

Last week I attended the world’s first ever dedicated AngularJS conference in Salt Lake City. More than 700 JavaScript developers gathered in the Little America hotel for two days to learn from Angular core-devs and community members.

Angular is going mainstream, that’s for sure. It was 'Google IO hard’ to get tickets for ng-conf: they sold out in a few minutes tops. The organizers were overwhelmed by this as well and it wouldn’t surprise me if next year’s ng-conf is significantly bigger. Fortunately, a few days before the conference it was announced that the talks were to be live-streamed. Now they are all available on YouTube for your viewing pleasure.

Conference format

I really liked the conference format. It was single-track, so no switching between (and looking for) rooms in between talks. Really amazing what a difference in atmosphere this gives, as everybody has the same experience throughout the day. Also, the talks were short: 20 minutes with 5 minute breaks. You would think that it’s hard to go deep in 20 minutes. And it definitely required a lot of skill from the presenters, but most pulled it off. Another advantage of 20 minute talks is that non-interesting topics or speakers are over before you know it.

(Large ballroom filling up for ng-conf)

AngularJS seems to attract a very bright and friendly community. Everything went super-smooth even though it was the first time around this conference was organized. Another thing that really worked well was a private Google+ community that the organizers used for communication. It was used heavily throughout the event and proved to be a good way to discuss talks in more depth afterwards.

Future of Angular

Of course the keynote was delivered by two Angular core-team members. Brad Green and Miško Hevery started by giving some context and history around Angular. For example, I never knew that Angular originally included a whole backend and data-syncing solution as well. Or that the name Angular was chosen because HTML has lots of angle brackets. But however interesting the past is, I am more interested in the future of Angular. Miško gave a very clear overview of what’s in store for Angular 1.3 and beyond. In short, Angular 1.3 will be the Angular for the current browsers and AngularJS 2.0 will be for the future browser:

embracing Web Components
using ES6 to enhance Angular modules and dependency injection
using object.observe instead of dirty-checking for two-way databinding

All three points are exciting improvements, the first two for increased modularity and the last for increased performance. Slides for the keynote can be found here or you can just watch it.

(Brad and Miško keynoting on the first day)

As you can guess from the picture above, improved support for mobile is also forthcoming.

AngularDart

It was interesting to see that none of the attendees reacted favorably to Dart as as JavaScript replacement. Much less the fact that AngularDart was created. The Angular team recognized this sentiment and explained the reasoning behind all this. Internal demand at Google spurred the development of AngularDart. Still, most of the 14 core developers currently working on Angular focus on AngularJS rather than AngularDart. Also, the team sees AngularDart as a fertile breeding ground for new concepts and innovations. In fact, many of the points I described on the future of Angular are prototyped in the AngularDart implementation. All of the successful AngularDart experiments will flow back into AngularJS (2.0). The bottomline is that AngularDart is an important part of Angular's future, even if the current focus is mostly on AngularJS.

Community

An open source project is only as good as its community is. Angular definitely had a bumpy journey in that regard. For example, the 1.2 release was severely delayed without any communication around it. This turned out to be due to an internal experiment with server-side pre-rendering for Angular, which was ultimately abandoned. The sudden introduction of AngularDart is another example of poor communication of longer term plans.

A lot of these issues stemmed from the lack of resources within the core-dev team. Pull-requests withered and GitHub issues were sometimes overlooked. In the past few months, a lot has changed. Besides extending the core team to 14 developers, an automated build/verification bot checks all pull-requests. Also, the Angular team has weekly issue triage meetings so nothing gets overlooked. Finally, an Angular working group will be assembled consisting of non-core developers, giving the community a voice to steer the direction of Angular. If you'd like to be an active participant in the Angular community, do check out angularjs.org/i-want-to-help.

Recommended watching

I’ll round off this post by listing some of my favorite talks:

Writing a Massive Angular App at Google: Google’s DoubleClick team shows their code-organization patterns
End to End Angular Testing with Protractor: the upcoming integration-testing framework for Angular
Angular performance: some great insights to improve performance of your app.
Dependency Injection: Vojta Ina explains the current prototype of Angular injection based on ES6 classes and annotations.

It's exciting to see that AngularJS is gaining so much momentum. Going to ng-conf was a great way to get inspired to write solid Angular code and to be prepared for what's coming.

A failed experiment: improving the Builder pattern

2013-10-21T00:00:00+02:00

Sometimes, you want to try something new. Like last week, when I wanted to implement a builder for a domain object. In Java, lest you wonder why the code samples in this post are so long.

The rationale for wanting builders for domain objects goes roughly like this. You want:

domain objects that are never in an inconsistent state
immutable domain objects, preferably
to avoid 'telescoping' constructors' taking all combinations of optional fields on your object upfront
a nice (fluent) API for building your domain objects

Granted, you could just use Scala's case classes with named parameters and call it a day. Alas, this was no such day.

On the shoulders of giants

Obviously the builder pattern has been belaboured by many who are greater than I am. In fact, the original Gang-of-Four description dates back to 1995.

But here we are in 2013. Let's say we want a domain object modeling pizza orders. The usual solution looks somewhat like this, using a static inner class to implement the builder:

public class PizzaOrderOldStyle {

<span class="kd">private</span> <span class="kt">int</span> <span class="n">size</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">pepperoni</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">chicken</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">mushroom</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">peppers</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">cheese</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">;</span>

<span class="kd">public</span> <span class="kd">static</span> <span class="n">Builder</span> <span class="nf">pizzaOrder</span><span class="o">(</span><span class="kt">int</span> <span class="n">size</span><span class="o">,</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">,</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">)</span> 
<span class="o">{</span>
    <span class="k">return</span> <span class="k">new</span> <span class="n">PizzaOrderOldStyle</span><span class="o">.</span><span class="na">Builder</span><span class="o">(</span><span class="n">size</span><span class="o">,</span> <span class="n">sauce</span><span class="o">,</span> <span class="n">orderFor</span><span class="o">);</span>
<span class="o">}</span> 

<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Builder</span> <span class="o">{</span>

    <span class="kd">private</span> <span class="kt">int</span> <span class="n">size</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kt">boolean</span> <span class="n">pepperoni</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kt">boolean</span> <span class="n">chicken</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kt">boolean</span> <span class="n">mushroom</span><span class="o">;</span>
    <span class="kd">private</span> <span class="kt">boolean</span> <span class="n">peppers</span><span class="o">;</span>
    <span class="n">String</span> <span class="n">cheese</span><span class="o">;</span>
    <span class="kd">private</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">;</span>
    <span class="kd">private</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">;</span>

    <span class="kd">public</span> <span class="nf">Builder</span><span class="o">(</span><span class="kt">int</span> <span class="n">size</span><span class="o">,</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">,</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">size</span> <span class="o">=</span> <span class="n">size</span><span class="o">;</span>
        <span class="k">this</span><span class="o">.</span><span class="na">sauce</span> <span class="o">=</span> <span class="n">sauce</span><span class="o">;</span>
        <span class="k">this</span><span class="o">.</span><span class="na">orderFor</span> <span class="o">=</span> <span class="n">orderFor</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withPepperoni</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">pepperoni</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withChicken</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">chicken</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withMushroom</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">mushroom</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withPeppers</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">peppers</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withCheese</span><span class="o">(</span><span class="n">String</span> <span class="n">cheese</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">this</span><span class="o">.</span><span class="na">cheese</span> <span class="o">=</span> <span class="n">cheese</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">PizzaOrderOldStyle</span> <span class="nf">build</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">return</span> <span class="k">new</span> <span class="n">PizzaOrderOldStyle</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
    <span class="o">}</span>
<span class="o">}</span>

<span class="kd">private</span> <span class="nf">PizzaOrderOldStyle</span><span class="o">(</span><span class="n">Builder</span> <span class="n">builder</span><span class="o">)</span> <span class="o">{</span>
    <span class="n">size</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">size</span><span class="o">;</span>
    <span class="n">pepperoni</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">pepperoni</span><span class="o">;</span>
    <span class="n">chicken</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">chicken</span><span class="o">;</span>
    <span class="n">mushroom</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">mushroom</span><span class="o">;</span>
    <span class="n">peppers</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">peppers</span><span class="o">;</span>
    <span class="n">cheese</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">cheese</span><span class="o">;</span>
    <span class="n">sauce</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">sauce</span><span class="o">;</span>
    <span class="n">orderFor</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">orderFor</span><span class="o">;</span>
<span class="o">}</span>


// Omitted getters
// Omitted equals/hashCode
}

It's fairly straightforward. The constructor for PizzaOrderOldStyle is private and takes an instance of the builder. In this constructor, the fields on the domain object are initialized to the values from the builder. Only Builder can be instantiated by the user of the API and it takes the non-optional values directly. The with*() methods on the builder expose a Fluent API by returning this. Since the resulting domain object has no setters, it is effectively immutable after it is returned from build().

Building the domain object is now as simple as:

pizzaOrder(10, "tomato", "Sander")
    .withPepperoni()
    .withMushroom()
    .withCheese("parmesan").build();

(I added a static convenience method pizzaOrder to instantiate the builder)

Problems

While the above solution seems reasonable, it contains some flaws. The most obvious one is that this pattern just shifts mutability to the Builder class. If you don't share builders across threads (which seems reasonable) we can live with this. What bothers me most is that we have three (!) locations where all the properties of the domain model are enumerated. First, the field definitions are repeated inside the Builder class. Second, the private constructor copies each of domain model properties. Way too many places to screw up. In fact, I only found out through a unit test that I forgot to copy the cheese property from the builder into the domain object.

Granted, there are several IDE plugins or other forms of code generators that can automate builder generation. Generating code, however, opens up a whole different can of worms.

Can we do better?

A feeble attempt

Seeing this problem got me thinking. What if the builder is just a façade on top of the domain object? It could use the fields of the domain object as 'intermediate' storage, without exposing the domain object as a whole before it is ready. Java allows non-static inner classes to mess with private state of the outer object. So that was a good starting point:

public class PizzaOrder {

<span class="kd">private</span> <span class="kt">int</span> <span class="n">size</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">pepperoni</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">chicken</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">mushroom</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">boolean</span> <span class="n">peppers</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">cheese</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">;</span>

<span class="kd">private</span> <span class="nf">PizzaOrder</span><span class="o">()</span> <span class="o">{</span>
    <span class="c1">// Prevent direct instantiation</span>
<span class="o">}</span>

<span class="kd">public</span> <span class="kd">static</span> <span class="n">Builder</span> <span class="nf">pizzaOrder</span><span class="o">(</span><span class="kt">int</span> <span class="n">size</span><span class="o">,</span> <span class="n">String</span> <span class="n">sauce</span><span class="o">,</span> <span class="n">String</span> <span class="n">orderFor</span><span class="o">)</span> 
<span class="o">{</span>
    <span class="k">return</span> <span class="k">new</span> <span class="n">PizzaOrder</span><span class="o">().</span><span class="na">new</span> <span class="n">Builder</span><span class="o">(</span><span class="n">size</span><span class="o">,</span> <span class="n">sauce</span><span class="o">,</span> <span class="n">orderFor</span><span class="o">);</span>
<span class="o">}</span>

<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Builder</span> <span class="o">{</span>
    <span class="kd">private</span> <span class="n">AtomicBoolean</span> <span class="n">build</span> <span class="o">=</span> <span class="k">new</span> <span class="n">AtomicBoolean</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span>

    <span class="kd">public</span> <span class="nf">Builder</span><span class="o">(</span><span class="kt">int</span> <span class="n">_size</span><span class="o">,</span> <span class="n">String</span> <span class="n">_sauce</span><span class="o">,</span> <span class="n">String</span> <span class="n">_orderFor</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">size</span> <span class="o">=</span> <span class="n">_size</span><span class="o">;</span>
        <span class="n">sauce</span> <span class="o">=</span> <span class="n">_sauce</span><span class="o">;</span>
        <span class="n">orderFor</span> <span class="o">=</span> <span class="n">_orderFor</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withPepperoni</span><span class="o">()</span> <span class="o">{</span>
        <span class="n">throwIfBuild</span><span class="o">();</span>
        <span class="n">pepperoni</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withChicken</span><span class="o">()</span> <span class="o">{</span>
        <span class="n">throwIfBuild</span><span class="o">();</span>
        <span class="n">chicken</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withMushroom</span><span class="o">()</span> <span class="o">{</span>
        <span class="n">throwIfBuild</span><span class="o">();</span>
        <span class="n">mushroom</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withPeppers</span><span class="o">()</span> <span class="o">{</span>
        <span class="n">throwIfBuild</span><span class="o">();</span>
        <span class="n">peppers</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">Builder</span> <span class="nf">withCheese</span><span class="o">(</span><span class="n">String</span> <span class="n">_cheese</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">throwIfBuild</span><span class="o">();</span>
        <span class="n">cheese</span> <span class="o">=</span> <span class="n">_cheese</span><span class="o">;</span>
        <span class="k">return</span> <span class="k">this</span><span class="o">;</span>
    <span class="o">}</span>

    <span class="kd">public</span> <span class="n">PizzaOrder</span> <span class="nf">build</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">build</span><span class="o">.</span><span class="na">compareAndSet</span><span class="o">(</span><span class="kc">false</span><span class="o">,</span> <span class="kc">true</span><span class="o">))</span> <span class="o">{</span>
            <span class="c1">// check consistency here...                </span>
            <span class="k">return</span> <span class="n">PizzaOrder</span><span class="o">.</span><span class="na">this</span><span class="o">;</span>
        <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
            <span class="k">throw</span> <span class="k">new</span> <span class="n">IllegalStateException</span><span class="o">(</span><span class="s">&quot;Build may only be called once!&quot;</span><span class="o">);</span>
        <span class="o">}</span>
    <span class="o">}</span>
    <span class="kd">private</span> <span class="kt">void</span> <span class="nf">throwIfBuild</span><span class="o">()</span> <span class="o">{</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">build</span><span class="o">.</span><span class="na">get</span><span class="o">())</span> <span class="o">{</span>
            <span class="k">throw</span> <span class="k">new</span> <span class="n">IllegalStateException</span><span class="o">(</span><span class="s">&quot;Cannot modify builder after calling build()&quot;</span><span class="o">);</span>
        <span class="o">}</span>
    <span class="o">}</span>
<span class="o">}</span>


// Omitted getters
// Omitted equals/hashCode    
}

Interestingly, the exposed API is identical since we again offer a static convenience function pizzaOrder hiding the slightly funky new PizzaOrder().new Builder(..) that is necessary to instantiate the builder:

pizzaOrder(10, "basil", "Fred")
    .withPepperoni()
    .withMushroom()
    .withCheese("mozzarella").build();

Even though from a syntactic standpoint the API hasn't changed, from a usage perspective there are differences. After a call to build() any other call on the builder results in an exception. This must be the case, since the underlying fields are the actual fields of the domain object. We don't want those to change after the builder is done. I used an AtomicBoolean to 'seal' the builder and the underlying domain object. Compare this with the original approach, where you can build as many times as you want with the same Builder instance. Whether this is a good or bad thing is debatable. In practice it doesn't make much difference since you tend to use a builder only once anyway.

Builder failure?

So why do I call this a failed experiment? First of all, I expected this solution to be more concise than the original. It isn't. Check the linecounts in this gist. Indeed, the fields are not enumerated three times. On the other hand, we have to add bookkeeping to manage the state of the builder façade to each with*() method. It's easy to forget the call to throwIfBuild and if you do this threatens the immutability of the domain object.

Second, non-nested inner classes keep an implicit reference to their outer containing objects. This means that the builder object itself may prevent the domain object from being garbage-collected. Retaining a reference to the builder in the original pattern doesn't have this problem, since inner static classes are instantiated without an outer instance and hence don't point to an outer instance.

Pattern failure?

So the result is not groundbreaking. Still, it's a nice variation on the builder pattern. Here's the gist containing this post's code if you want to play around with it. More than anything, it reminds us that design patterns only serve to point out weaknesses in languages. Creating and maintaining builders for every domain object is just too much hassle. Languages like Scala and C# are much better equipped in this regard.

One improvement I've been thinking of is to use a nested empty instance of the domain class in the builder to store the data. We can gradually build up the object by modifying its private fields until we return it from build(). In this variation the builder can be made static again. In fact, while writing this I decided to implement this variation and it looks like the cleanest approach so far.

Leave a comment if you see other improvements that I missed!

JavaOne 2013 trip report

2013-09-30T00:00:00+02:00

Last week, San Francisco was sprawling with developers. It was that time of the year again: JavaOne 2013 took place. Thousands of Java developers gathered September 22-26 to learn new things and meet new people.

I started visiting JavaOne in 2010, just after Oracle took over the reigns from Sun. In that sense, my past four JavaOnes gave a consistent experience. Not incredibly exciting, but lots of good content when you look hard enough at the enormous content catalog. Whether this is a good or bad thing is a matter of taste. Sun over-promised and under-delivered, Oracle has a consistent and oftentimes conservative message. At least they deliver, with schedule slippages of months rather than years.

Announcements

This can be pretty short: there were no big announcements at the JavaOne keynotes this year. It was all about consolidating the progress made in the past few years. Delivering Java EE 7, JavaFX and in the near future Java 8 is certainly something to be proud of. The demo's in the technical keynote were fun and engaging. However, the punch-line was missing, so to say.

(Just before the keynote in Moscone)

There was actually one new piece of information: an update on project Avatar. Project Avatar was originally announced without any details in 2011. Last year, it became clear that Avatar is a JavaScript framework to tie the browser and Java EE backends together more easily. This year, it finally became tangible. It is now open-sourced at java.net. No further story, that was it. Honestly, it feels a bit like this baby was left at the doorstep of the hospital. Who's going to take care of it, and will it ever grow up to be important?

Finally, while not really an announcement, it was interesting to note that Oracle is shifting the attention for Java to the 'Internet of things.' This term is used often to explain their heavy focus on embedded Java development. The cynic might say it's because this is the only part of Java that is traditionally well-monetized (with Java ME), so it makes sense to secure this cashflow for the future. Regardless of the motivation, I'm happy to see that a unified VM for Java 8 for all platforms is in the cards.

Java 8

Java 8, and specifically its Lambda's took the center stage. With the GA date set for March 2014, now is the right time to familiarize yourself with Lambda's. Especially nice is the fact that the collections library has been revamped to support a more functional style. Even cooler, Streams are added on top of the improved collections library, providing stream fusion:

int sum = widgets.stream()
                      .filter(w -> w.getColor() == RED)
                      .mapToInt(w -> w.getWeight())
                      .sum();

In this example, there is only one traversal of the widgets collection because the operations are performed on a stream. Stream fusion means that the implementation collects all the transformations up until a terminal operation (like sum). The terminal operation then forces the evaluation of the preceding chain of expressions with a single pass over the actual underlying collection.

Conference sessions

As to be expected with a huge amount of parallel tracks, the quality of the sessions varied. I did attend some excellent sessions. Particularly interesting was the talk from one of the Jetty founders. Jetty 9.1 pushes the performance boundaries for Java-based webservers, with SPDY and SPDY-Push support. All this is supported on top of the Servlet spec. You only need to configure the right endpoints.

One of the other great talks I saw was by Ben Christensen of Netflix. He showed how RxJava enables functional reactive programming on the JVM. RxJava ports the Reactive Extensions from the .Net world to the JVM world. Slides are here.

Another cool talk was by the Typesafe guys on SLICK (Scala Language-Integrated Connection Kit.) In short, SLICK maps standard Scala collection methods and for-comprehensions to database access code in a type-safe manner. Kind of what Linq-to-SQL does for .Net, but slightly more ambitious. Definitely something to consider when accessing relational databases from Scala code without relying on a more traditional Java-inspired ORM approach. Slides can be found here.

Talking at JavaOne

Last year I had the opportunity to speak at JavaOne for the first time. Since it was an enjoyable experience, fate decided that I should double the amount of talks this year (hopefully this trend will not continue for too many years.)

My first talk was 'Data Science with R for Java developers' on Tuesday. It was fun to introduce a completely new language to a group of Java developers. Understanding the data that lives in your application deeply is increasingly important. The main takeaway of the talk is that learning R gives you a powerful tool for doing exploratory data analysis. Here are the slides:

embed

On Wednesday, I presented 'Modular JavaScript' with my co-worker Paul Bakker. Modular development is something we value. For large Java codebases, we use modular designs on top of OSGi to enforce runtime modularity as well. In JavaScript land, codebases are getting bigger and bigger too. Hence, you need a principled approach to modular development there as well. In this talk we introduce the primitives for modularity in JavaScript and compare several module systems for JavaScript.

embed

During both talks there was a lot of audience interaction, which is always an encouraging sign. All in all, JavaOne 2013 was a great experience. Hope to see you in 2014!

Some AngularJS pitfalls

2013-08-05T00:00:00+02:00

In the past few months I've been heavily using AngularJS. It's a marvelous framework for modern web applications. But as with every new toy, there are some rough edges that you have to be aware of. In this post I'll round up some gotcha's I encountered. Assuming you already know your way around Angular, these may be helpful.

The flickering UI

Angular's automatic data-binding feature is awesome. However, it does have a flipside: before Angular is initialized, your page may show unparsed expressions to the user. When the DOM is ready and Angular calculates and replaces the values, this results in an ugly flickering effect:

The above situation is a direct result of rendering this example from the Angular tutorial:

<body ng-controller="PhoneListCtrl">
  <ul>
    <li ng-repeat="phone in phones">
      {{ phone.name }}
      <p>{{ phone.snippet }}</p>
    </li>
  </ul>
</body>

If you have a single page application, this problem only occurs on the first page-load. Fortunately, it's easily prevented: don't use inline {{ }} expressions. Instead, use the ng-bind directive:

<body ng-controller="PhoneListCtrl">
  <ul>
    <li ng-repeat="phone in phones">
      <span ng-bind="phone.name"></span>
      <p ng-bind="phone.snippet">Optional: visually pleasing placeholder</p>
    </li>
  </ul>
</body>

You need a tag to apply this directive, so I added a <span> around the phone name. What happens is that initially, the value inside the tag is shown (but you can choose to leave it empty.) Then, when Angular initializes and replaces the innards of the bound tag with the result of the expression. Note that you don't need to add the curly braces inside the ng-bind property. Much cleaner! If you need a compound expression, use ng-bind-template instead. With this variant you do need the curly braces though, in order to differentiate between string literals and expressions.

Another option is to completely hide elements, or even your whole application, until Angular is ready. Angular provides ng-cloak to this end. It works by injecting a CSS rule very early in the initialization phase into the page. Alternatively you can include this CSS hiding rule in your own stylesheet so it loads early. Angular will the remove the cloak style when it is ready to roll, unveiling your application (or designated elements) at once and fully rendered.

jQuery and Angular

Angular doesn't depend on jQuery. In fact, the Angular source contains an embedded lightweight alternative: jqLite. Still, when Angular detects the presence of a jQuery version in your page, it uses that full jQuery implementation in lieu of jqLite. One direct way in which this manifests itself is with Angular's element abstraction. For example, in a directive you get access to the element that the directive applies to:

angular.module('jqdependency', [])
  .directive('failswithoutjquery', function() {
    return {
      restrict : 'A',
      link : function(scope, element, attrs) {
               element.hide(4000)
             }
    }
});

(see this example in action in this plunkr)

but is this element a jqLite or jQuery element? It depends. The manual states:

All element references in Angular are always wrapped with jQuery or jqLite; they are never raw DOM references.

So it will be a jqLite element only if jQuery was not detected by Angular. The hide() method only exists for jQuery elements, meaning this example only works when Angular uses jQuery. If you (accidentally) switch the order of AngularJS and jQuery script tags in your page, this code breaks! Shuffling script tags around may not happen very often, but it bit me when I started modularizing our codebase. Especially when you start using a module loader (like RequireJS), this can rear it's ugly head. For RequireJS, I solved it by explicitly declaring that Angular does in fact depend on JQuery in the shim config.

Another takeaway might be that you shouldn't call jQuery specific methods through Angular's element wrapper. Instead you can use $(element).hide(4000) to make it abundantly clear that you do, in fact, depend on jQuery. In that case, changing the script tag order won't break the directive.

Minification

Special care should be taken for Angular applications that need to be minified. Otherwise error messages like 'Unknown provider: aProvider <- a' will hound you. Like with so many things, this information is hidden somewhere in the official documentation. In short, Angular depends on function parameter names to perform dependency injection. The minifier, blissfully unaware of the special semantics of formal parameter names in Angular, mangles them to the shortest possible identifier. Solution? Use the minify-friendly method of method injection. From this:

module.service('myservice', function($http, $q) {
   // This breaks when minified
});

to this:

module.service('myservice', [ '$http', '$q', function($http, $q) {
   // Using the array syntax to declare dependencies works with minification!
}]);

The array syntax is accepted wherever a to-be-injected function is accepted. My advise is to use this syntax from the get-go. It will prevent a lot of pain later on if you decice to minify your JavaScript. There also seems to be a automatic rewriter, although I don't know how well it works.

One last bit of advice: when you need to rewrite your functions using the array syntax, apply it at every location where Angular injects dependencies. This includes directives, but also controllers on directives. A spot that is easily missed (speaking from experience):

// the directive itself needs array injection syntax:
module.directive('directive-with-controller', ['myservice', function(myservice) {
    return {
      controller: ['$timeout', function($timeout) {
        //  but this controller needs array injection syntax, too!  
      }],
      link : function(scope, element, attrs, ctrl) {

  <span class="p">}</span>
<span class="p">}</span>


}]);

Note that the link function doesn't need the array syntax, since it is not actually an injection target. It is a fixed-arity function called directly by Angular. Injections from the directive level can be used inside the link function though.

Directives are never 'done'

One irritating issue with directives is that you don't get a hook for when the directive is 'done'. When integrating a jQuery plugin inside a directive, such a hook would be very helpful. Let's say you want to display a table with dynamic (ng-repeat) data as a jQuery datatable. You need to call $('.mytable').dataTable() at some point when all data is actually materialized in an HTML table (and preferably before it is inserted into the DOM), but you can't.

Why is that? Angular's data-binding is powered by a continuous digest-cycle which can lead to updates and re-rendering of directives. As such, there is never really a moment that the framework is 'at-rest' when applying directives. One (admittedly ugly) workaround is to schedule the jQuery dataTable call outside the current digest-cycle by using timeout:

angular.module('table',[]).directive('mytable', ['$timeout', function($timeout) {
    return {
      restrict : 'E',
      template: '<table class="mytable">' +
                   '<thead><tr><th>counting</th></tr></thead>' +
                   '<tr ng-repeat="data in datas"><td></td></tr>' +
                '</table>',
      link : function(scope, element, attrs, ctrl) {
         scope.datas = ["one", "two", "three"]
         // Doesn't work, shows an empty table:
         // $('.mytable', element).dataTable()   
         // But this does:
         $timeout(function() {
           $('.mytable', element).dataTable();
         }, 0)
      }
    }
}]);

(see this example in action in this plunkr)

In our codebase we've even encountered a situation where a doubly-nested $timeout was necessary. Another crazy work around is to add a <script> tag in the template which does a callback to an Angular function using scope.$apply(). Suffice to say, it's less than ideal. Due to Angular's nature this is unlikely to change.

All this being said, Angular is still my favorite client-side JS framework. Did you encounter other pitfalls while using Angular? Or did you solve similar problems in a different way? Leave a comment!

The Java Modularity Story

2013-07-18T00:00:00+02:00

So you've got a growing Java application with a nice feature set. Unfortunately adding new features gets harder over time and things start breaking in unexpected places. Chances are that your applications is not as modular as it could be. Relax, it's not (just) your fault. Plain Java is notoriously lacking in the modularity department. But it doesn't have be this way.

Modularity leads to more maintainable, composable and extensible systems. When you have clearly demarcated module boundaries and explicit contracts between modules, life is good. Functionality can be tested in isolation, and divide-and-conquer can be applied at the code and team-level. This speeds up development, not just in the first year of the system, but throughout its whole lifecycle.

From architecture to software

So how do you get to that point? We'll get to a specific solution later, but first I want to take the time to define the problem clearly. Modularity plays a big role at many levels of abstractions. At the systems architecture level, we have Service Oriented Architecture. When done right, SOA means explicit and versioned public interfaces (mostly webservices) between loosely coupled subsystems that hide their internal details. These subsystems possibly even run on completely disparate technology stacks and are easily replaceable on an individual basis.

However, when building the individual services or subsystems in Java, a monolithic approach is often irresistible. Java's own runtime, rt.jar is unfortunately a prime example. Sure, you may partition your monolithic application into the three obligatory layers, but that's a far cry from true modularity. Just ask yourself what it would take to swap out the lowest layer of your application for a completely different implementation. Oftentimes, this would ripple through the whole application. Now, try thinking of how you would do this without disrupting the other parts of your application, when hot-swapping at run-time. Because why should this be possible in a SOA context, but not inside our applications?

True modularity

So what is this true modularity I alluded to in the preceding paragraph? Let me state some defining characteristics of a true module:

a module is an independent unit of deployment (reusable in any context),
it has a stable, reified identity (for example name and version),
it states requirements (dependencies),
it advertises capabilities for other modules to use while hiding implementation details.

Ideally, these modules live in a lightweight runtime environment that matches up the requirements and capabilities of modules for us according to our desired composition. In short, we want our application to use modules to get the good parts of a Service Oriented Architecture on a smaller scale. And not just on the whiteboard, but also while coding and running the application. We want our physical implementation to follow our logical design all the way through production. What's more, the composition of modules shouldn't be static: applications need to be resilient and extensible without downtime or full redeployment.

Objects: true modules?

What about the lowest layer of structural abstraction in Java: classes and objects. Doesn't OO provide identity, information hiding and loose coupling through interfaces in Java? Yes it does, to a degree. However, object identity is ephemeral and interfaces are unversioned. Classes are most certainly not an independent unit of deployment in Java. In practice, classes tend to get way too familiar with each other. Public means visible to literally every other class on the JVM's classpath. Which is probably not what you want for anything but truly public interfaces. To make matters worse, Java's visibility modifiers are weakly enforced (think reflection) at runtime.

Reuse of classes outside their original context is hard when nobody forces the cost of their implicit external dependencies on you. I can practically hear the words 'Dependency Injection' and 'Inversion of Control' racing through your mind now. Yes, these principles help to make dependencies of a class explicit. Unfortunately their archetypical implementations in Java still leave your application as a big ball of objects wired together statically at runtime by a big ball of configuration. I highly recommend reading 'Java Application Architecture: Modularity Patterns' if you wan to learn more about modular design patterns. But you'll also find that applying these patterns in Java without additional runtime enforcement of modularity is fighting an uphill battle.

Packages: true modules?

But then what is the unit of modularity in Java inbetween applications and objects? You could argue that packages must be it. The combination of package names, import statements and visibility modifiers (e.g. public/protected/private) gives the illusion that at least some of the characteristics of true modules are present. Unfortunately packages are purely a cosmetic construct, providing namespacing for classes. Even their apparent hierarchy is actually an illusion.

So yes, by all means use packages to structure your code base in logical chunks. Just don't count on packages to improve modularity beyond pretty names. To be fair, there are tools that can help with enforcing package dependencies through static verification at development time. But that's hardly a satisfying solution.

JAR files: true modules?

Surely the true unit of modularity for Java applications then must be JAR files (Jars). Well, yes and no. Yes, because Jars are the independent units of deployment for Java applications. No, because they fail on the three other characteristics. Jars have a filename, and sometimes a version in the MANIFEST.MF. Neither are part of the runtime model and hence do not form an explicit reified identity. Dependencies on other Jars can't be declared. You have to make sure any dependencies are on the classpath. Which, by the way, is just a flat collection of classes: gone is the link to the originating Jars. This also explains another big problem: any public class in a Jar is visible to the whole classpath. There is no 'jar-scope' modifier to hide implementation details inside a Jar.

All of the above means that Jars are a necessary, but not sufficient mechanism for modular applications. Many people are successfully building systems out of lots of Jars (by applying modular architecture patterns), managing the identities and dependencies with their favorite compile-time dependency management tool. Take for example the Netflix API, which is composed of 500 JARs. Unfortunately, your compile-time and run-time classpath will diverge in unforeseen ways, giving rise to the JAR-hell. Alas, we can do better than this.

OSGi bundles

Clearly, plain Java doesn't offer enough in terms of modularity. It's an acknowledged problem and with project Jigsaw there might be a native solution on the way. However, it has missed the Java 8 train (and Java 7 before that), so it will be quite some time before we can use it. If it ever arrives. Enter OSGi: a modular Java platform that's mature and battle-hardened. It is used by the likes of application servers and IDEs as the basis for their extensible architectures.

OSGi adds modularity as a first-class citizen to the JVM by amending Jars and packages with the necessary semantics to achieve all of our stated goals for true modularity. An OSGi bundle (module) is a Jar++. It defines additional fields inside a Jar's manifest for a (preferably semantic) version, bundle-name and which packages of the bundle should be exported. Exporting a package means you give the package a version, and all public classes of the package are visible to other bundles. All classes in non-exported packages are only visible inside the bundle. OSGi enforces this at runtime as well by having a separate classloader per bundle. A bundle can choose to import packages exported by another bundle, again by defining its imported dependencies in the Jar's manifest. Of course such an import must define a version (range) to get meaningful dependencies and guide the OSGi bundle resolving process. This way, you can even have multiple versions of package and its classes running simultaneously. A small example of a manifest with some OSGi parameters:

And the accompanying manifest for the service bundle:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MyService bundle
Bundle-SymbolicName: com.foo.service
Bundle-Version: 1.0.0
Import-Package:  org.apache.commons.logging;version="[1.0.4, 2.0.0)"
Export-Package:  com.foo.service.api;version="1.0.0"

And there you have it: OSGi provides an independently deployable Jar with a stable identity, and the possibility to require or advertise dependencies (ie. versioned packages). Everything else is strictly contained inside bundles. The OSGi runtime takes care of all the gritty details to enforce this strict separation at runtime. It even allows bundles to be added, removed and hot-swapped at run-time!

OSGi services

So, OSGi bundles take care of dependencies defined on the package level, and defines a dynamic lifecycle for bundles containing these packages. Is that all we need to create a SOA-like solution in the small? Almost. There is one more crucial concept before we can really have modular micro-services with OSGi bundles.

With OSGi bundles, you can program to an interface that is exported by a bundle. But, how do you obtain an implementation of this interface? It would be bad to export the implementing class, just so you can instantiate it in consuming bundles. You could use the factory pattern and export the factory as part of the API. But having to write a factory for every interface sounds… boiler-platey. Not good. Fortunately, there's a solution: OSGi services. OSGi provides a service-registry mechanism, where you can register your implementation under its interface in the service registry. Typically, you register your service when the bundle containing the implementation is started. Other bundles can request an implementation for a given public interface from the service-registry. They will get an implementation from the registry without ever needing to know the underlying implementation class in their code. Dependencies between service consumers and providers are automatically managed by OSGi in much the same way as the package-level dependencies are.

Sounds good, right? There's one slight bump in the road: using the low-level OSGi service API correctly is hard and verbose, since services can come and go at runtime. This is because bundles that expose services can be started and stopped at-will and even running bundles can decide to start and their services at any time. That's extremely powerful when you want to build resilient and long-lived applications, but as a developer you have to stand your ground. Lucky for us, many higher-level abstractions have been created to take care of this problem. So, when using OSGi services, use something like Declarative Services or Felix Dependency Manager (which is what we use in our projects) and create and consume micro-services with ease. You can thank me later.

Is it worth it?

I hope you agree with me that having a modular codebase is a worthy goal. And no, you don't need OSGi to modularize a codebase. Conversely, modular runtimes like OSGi can't rescue a non-modular codebase. In the end, modularity is an architectural principle that can be applied in almost any environment, given enough willpower. It takes additional effort to create a modular design. It's only natural to use runtime in which modules and their dependencies are first-class citizens from design-time through run-time to ease the burden.

Is it hard in practice? There definitely is a learning curve, but it's not as bad as some people make it out to be. Tooling for OSGi-based development has improved tremendously the past few years. Especially bnd and bndtools deserve a mention for this. If you'd like to get a feel for what it means to develop a modular Java application, watch this screencast from my co-worker Paul Bakker. He is also the co-author of an upcoming O'reilly book on this topic that you can preorder here.

Modular architectures and designs are increasingly getting attention. If you want to apply this now in your Java environment I encourage you to follow some of the links in this article and give OSGi a spin. Again, it won't come from for free. But remember: OSGi isn't hard, true modularity is.

Annoyances in the daily life of a developer

2013-06-24T00:00:00+02:00

I'm not complaining and I love my job, but sometimes...

You know it's a bad day when you experience all of these.

Exceptional errors

So you run into a really obscure error or exception. Google it. Result: many promising threads. Except you'll only find: 'I ran into this problem, help plz!' Bonus points for the sadistic 'never mind, solved it!' follow-up without an actual solution. Consider it a challenge.

Scratch this

Work on a feature all day long, only to discover that with a few lines of well-placed code somewhere else all your work is made redundant. Actually, this is enlightenment. But man, does it feel annoying.

Breaking bad

That awkward moment when you realize you pushed something completely unacceptable that will break the build.

Git woes

That awkward moment when you realize you forgot to push something completely acceptable that your co-worker needs.

Heisentools

Strange and hard to reproduce bugs happen. But often it's not you, it's them. Flippant browser caches. Unpredictable IDEs (I'm looking at you, Eclipse). Fumbling operating systems. Did you try refreshing/restarting/rebooting?

Coding, interrupted

Email. Twitter. Co-workers. Bosses. Recruiters. The first two are easily taken care of... yeah, right.

You start talking, I stop listening

"Can't we just ..." or, "how hard can it be?", or "it's just a small additional feature". Nope. Nip it in the bud, it only goes downhill from here. Skip the weasel words and have a realistic discussion, please.

Documentation fail

As in: no library documentation available. Or, if there is, it's wildly out-of-sync. Even worse: up-to-date auto-generated documentation. Come on library authors, documentation is the most important part of your API. We'll conveniently forget about the state of the documentation of our own code for now. No hard feelings.

Finally: I'm still not complaining, and I still love my job!

JEEConf 2013 trip report

2013-05-25T00:00:00+02:00

Like last year, I had the opportunity to speak at JEEConf Kiev, the leading Java conference in Eastern Europe. More than 700 people from countries like Ukraine, Russia and Belarus attended JEEConf. Most talks are therefore in Russian. However, there was just enough English content to keep me entertained as well.

This year, the conference lasted two days rather than one. Somehow, this provided a bit more relaxed atmosphere. Unfortunately I had to miss out on some good content the second day, since I had to leave early to catch my flight.

(the conference venue)

The content

JEEConf started with a keynote by Reza Rahman on Java EE 7 and 8. To be honest, I think this talk is getting a bit stale. Not that the content is bad, but I've heard (slight variations) of this talk by different Oracle people for more than two years now. It's good to tell a consistent story, but a bit excitement might be good for a change.

I enjoyed the session on Arquillian and some of its extensions. Arquillian isn't really new to me, but I hadn't been actively following its development. This was a nice status update, showing many improvements that are very useful in integration test scenario's. For example, being able to have data-driven integration tests using the Persistence extension (based on DBUnit). Another cool extension is the Selenium integration of Arquillian, called Arquillian Drone. You can deploy your app with Arquillian, get the resulting URL injected into your test class and run Selenium tests for different browsers easily.

Another highlight were the talks bij Venkat Subramaniam, also known as venkat_s. His talks are packed with practical knowledge, an enormous amount of energy and humor as well. His performance was even more impressive given the fact that his laptop broke down minutes before the session. So Venkat did all his live coding on a borrowed machine from one of the participants. The first talk was a practical guide to picking a JVM language like Scala, Groovy or Clojure. This lead to a good discussion on whether such a switch is even feasible for 'typical' Java developers. Yakov Fain articulates his point in a blog post. Even though I don't fully agree with his conclusions, it is a good discussion.

(Venkat_s in action)

The second talk by Venkat was on concurrency with Akka actors. While it's easy to label something as the Next Big Thing™, I believe Akka really is a big deal. It has solid engineering behind it and opens up new doors for highly concurrent applications. With their cluster-implementation maturing as well, we're going to hear a lot more about Akka in the near future.

Modularity in the Cloud

My talk 'Modularity in the Cloud: a case study' was on the fourth timeslot in the second biggest room. This was my first conference session as a representative of Luminis Technologies which I joined March this year. Around 150 people attended the session and judging by the insightful questions and positive feedback, I think the message came across very well. Here are the slides:

embed

The case study focussed on PulseOn, an educational system that Luminis Technologies is building. If you want to find out more about this system, check out the PulseOn website. On the technical side, the open-source Amdatu project is where you should look. I'll write a dedicated post on the actual content of the talk later. Applying modular architecture is a very relevant message for contemporary Java developers, even though there are no easy answers. So watch this space.

The people

Besides technical content, the other important part of a conference is meeting people and exchanging ideas. Unlike last year there was a speaker's dinner which provided ample opportunities for exchanging ideas and having a good time.

It was nice to see Yakov Fain again, whom I met previous year at JEEConf as well. I should also mention Mohamed Taman, an active Java community member who is doing some great work for the JCP. I also talked to Vladimir Miguro (aka vova), who is working on the new MongoDB 3 Java driver. It's interesting to hear how it will to totally revamped to provide a more fluent and idiomatic Java interface. Fear not, there will be a v2 compatibility layer for those who have non-modular applications that cannot easily cope with changes in underlying API's :-)

Last but not least, props to Mikalai and Aleksey, who once again organized a great Java conference!

Launching Scalabitz

2013-05-12T00:00:00+02:00

Today marks the launch of my lingering weekend-project: scalabitz.com. It tries to surface interesting Scala content through the API of link-shortener Bit.ly. The site was built using Scala and Play (and a bit of Akka) with a MongoDB storage layer and can be found on GitHub.

Before you read on you may want to take a look at scalabitz.com to get an idea what the rest of this post is all about. You can also follow Scalabitz on Twitter.

Finding content

One of the harder problems on the web is how to find relevant content. Link aggregators like Reddit and Hacker News solve part of this problem by crowd-sourcing links. Twitter and Facebook can be more tailored, but only work as good as the people you follow. Startups like Prismatic create a more personalized solution using machine learning and there are many solutions combining different approaches. In short, this is not a solved problem. A while ago I stumbled across the Bit.ly API and wondered whether it could provide an alternative. Can we find interesting content from the data behind a link-shortening service? Hence the start of my experiment. So far I believe the answer is a cautious yes, since people tend to shorten links they care about.

But is it relevant?

The Bit.ly API offers a search endpoint. If you start feeding it 'Scala' as keyword, surprisingly many interesting Scala-related links surface. But this also gives a lot of false positives. For example, it turns out there are many clubs and restaurants called 'Scala' announcing events through Bit.ly links. You could use 'Scala programming' as search phrase, but then you miss a lot of content.

For scalabitz.com I went with just 'Scala' as search phrase. This means I have to do something about the false positives. Currently, the results on the frontpage are manually curated. There is a small keyword-matching heuristic on content that helps me to quickly see whether content is relevant:

(a partial screenshot of the admin page, green is good, red is bad...)

Fortunately the volume of new links discovered through the Bit.ly API is manageable for now. To avoid flooding the frontpage after a moderating session by yours truly, approved links are released one-by-one in a timed fashion. Hence the 'prepublish' action on the screenshot above, which puts the link into the publishing queue. Obviously, this is all far for from perfect. My goal is to make the curation process fully automatic. As a next step I will try to implement a supervised machine learning algorithm when I have a larger corpus of training data with correct publish/reject decisions available.

Additionally, a different Bit.ly endpoint provides data on how many click-throughs the links received. This count is updated periodically by the Scalabitz backend and shown on the frontpage.

Why Play?

One of the reasons I chose the Play framework was that I wanted to explore building a webapp that is completely asynchronous and non-blocking. Play helps you to write reactive applications on all levels: from handling HTTP requests, to performing the backend REST calls and querying the database. The first two are native Play features and for the latter you can use the ReactiveMongo driver. Even though the driver is not at 1.0 yet, it is a pleasure to use.

The main building block for this application is scala.concurrent.Future. Introduced in Scala 2.10, the Future API provides a very nice way to do async programming. It has been adopted by both Play and Akka, so all the pieces fit together perfectly. To give a taste of how easily you can do non-blocking and asynchronous programming with Scala and Play, consider the following snippet:

ScalabitzService.getPublishedArticles(0, Some(100)).flatMap { articles =>
   val clickFutures = for(article <-articles) yield clickWSCall(article)
   Future.sequence(clickFutures)
}

(see the code in context on GitHub)

The getPublishedArticles method returns a Future[List[ScalabitzArticle]], essentially promising that this list of articles will be available from Mongo at some point in the future. Next we must specify what should happen to the result inside the Future when it is succesfully completed. We can do this by defining a transformation, in this case using the flatMap method. In fact, flatMap is a monadic bind operation, making it a general mechanism for composing actions with other Futures like we are going to do here.

On to the actual transformation. For each of the articles we want to retrieve the click count. Since there is no single Bit.ly endpoint accepting multiple identifiers at once, we need to do a single call to Bit.ly for each article. Obviously we don't want to do this blocking and sequentially. Therefore, the for-comprehension builds up a List[Future[(String, Int)]] based on the articles we got from the outer database call. The method clickWSCall uses Play's non-blocking WS API to perform these calls, ultimately returning a Future. Now the variable clickFutures represents the result of all these calls at some point in the future. However, there's no blocking or busy waiting for any WS call. Then, we feed the resulting list of type List[Future[(String, Int)]] to the Future.sequence method. Et voila, this creates a new Future[List[(String,Int)]] (notice the inversion in the type?), eventually containing all click counts associated with their id in a tuple. It is asynchronously completed when all individual futures are completed automatically.

The topic of reactive applications deserves a blogpost of its own, which will surely follow... After some adjustments the paradigm feels almost as natural as doing it the 'old way'. Almost, with the exception of error handling, which I haven't really tackled in an elegant manner yet. Another area of improvement is to find a nice model for dependency injection of services (may be the Cake pattern?), currently there's just singleton objects wired directly to each other.

Deployment

I've had many toy-projects over the past years, but none of them made it out of the comfort of my own machine. Therefore I'm still looking for the perfect production environment. Heroku is the current deployment platform for Scalabitz because it supports Play apps natively, is low-friction and it's free. Deploying the app is as simple as doing a Git push to Heroku, which is nice. Since the app is rebuilt on Heroku before it is deployed and started, it is a bit slower than ideal. Also, I'm not sure how long the free tier will work for this application. I guess I will find out soon enough.

Future developments

Next up is automatically feeding the @scalabitz twitter account with published links. So Follow @@scalabitz to get a nice feed of Scala articles! On the backend side of things the next priority is to get a more accurate classification of incoming links. Fortunately I've been notching up my machine learning skills in the past year, so that should be a nice challenge. One other interesting idea is to generalize this 'platform', since the same could be done not just for Scala but for a range of topics.

In short, I have tons of ideas for developing Scalabitz further. But the most important question is: what do you think? Promising? Pointless? Let me know in the comments or on Twitter!

A year of blogging analyzed with R

2013-03-23T00:00:00+01:00

Today it's exactly one year ago that I published my first blogpost on branchandbound.net. During this year I've written 12 posts, including this one. Since we live in the age of analytics, I thought it would be nice to show how this blog has fared during its first year.

Like most people I use Google Analytics to track what's happening on this site. However, I'm not going to show you screenshots of the GA Dashboard. Not that it's bad (it's actually pretty good), but where's the fun in that? Since this is a programming blog we're going to hook into the Google Analytics API. And this time, without using my standard tools of trade. Instead of using a JVM based language, I'm turning to R to do some number crunching.

Learning R

The R language is geared towards statistical programming and data analysis. Someone described it to me as a headless version of Excel on steroids. I think this gives the right feeling for what R is. If you're coming from a statically-typed background or expect a perfectly crafted language, you're in for a culture shock. R lives firmly in the camp of 'shut up, it works' and its virtues lie in the vast amount of statistical libraries that are available. Let's just say that the intersection on the Venn diagram covering statisticians and kick-ass language designers/software engineers is very small:

(the intersection is inhabited though: check the Julia language)

I picked up R while following two Coursera courses that both use R as main language: Computing for Data Analysis and Data Analysis. The former is more an introduction to programming with R, whereas the latter course focuses on applied statistics and machine learning with R. I don't have a formal background in statistics, hence Data Analysis course was much more of a challenge for me.

Setup Google Analytics with R

Enough about learning R, let's try to load some analytics data. If you want to jump right into the code and results, you can view the RMarkdown output of my exploratory analysis. RMarkdown enables literate programming, by combining Markdown syntax with R code. You can see that it shows my narrative interspersed with actual code and the output that it produces. For the remainder of this post I'll discuss some of the results from this exploratory analysis without diving too deep into the mechanics behind it.

How do you get the Google Analytics data into R? We could work with the Analytics REST API directly in R, but there is also a nice wrapper library call rga. Did I mention there's a package for everything in R? Here's how you call the Google Analytics API:

library(rga)
rga.open(instance = "ga", where = "~/temp/ga-api")
# the call above triggers the oAuth flow, and the resulting  
# key is cached in the specified file.

ga.df <- ga$getData("XXXXXXXX", # replace with your GA profile id
                    start.date = "2012-03-23",
                    end.date = "2013-03-23",
                    metrics = "ga:visits",
                    dimensions = "ga:date",
                    max = 1500,
                    sort = "ga:date")

Yes, that instance="ga" on the second line specifies the name of a variable that will pop into existence, later to be used in ga$getData(). Life's full of dynamic surprises with R.

We're assigning the result of the ga$getData call to a variable called ga.df. Here, df is short for dataframe, a handy datatype which resembles the familiar relational table. Because R has named parameters, the actual call is more or less self-explanatory. It returns the visits by date over the specified period. All parameters with "ga:" prefixes follow the dimension and metrics name from the Google Analytics API directly.

Actually, the hardest part was finding out the right profile id to use for calling the API. Just so you know, this id is not the same as the tracking id you use on your page. Anyway, a quick way to find all your profile id's is to head to Google's OAuth2 playground, authorize the Analytics API and call:

https://www.googleapis.com/analytics/v3/management/accounts/~all/webproperties/~all/profiles

Visitors

Now on to some actual numbers. In the past year, this blog received 23,903 visits (from 18724 unique visitors), generating 27,860 pageviews. As you can see, there are some huge spikes in the number of visitors:

(again, check the RMarkdown to see how the graphs are generated)

The highest datapoint was measured in December 2012 when Martin Odersky tweeted my Scala is like Git post, driving a record 2077 visits to my blog in one day. The other peaks can all be explained by referral traffic from social link sharing sites such as Reddit and DZone. Alas, 15, the median number of visits per day paints a better picture of the actual events for most days. Fortunately, the trendline (a linear regression line) is sloped upwards.

It's also interesting to see how the visits are distributed over the days of the week, using a boxplot on the same data:

The horizontal lines inside the boxes represent the median number of visitors for that particular weekday. The boxplot is deliberately zoomed in on the y-axis to show the interesting parts. You can imagine the outlier of 2077 visits on a single day (it was on a Thursday) otherwise makes the plot unreadable. It's pretty clear that my blog is not as popular on the weekends as it is during weekdays. Apparently programming blogs are read mostly at work, which makes sense. Mondays and Thursdays are the best performing weekdays. Are people reading away their Monday blues on a programming blog? I don't have an explanation why Tuesday is relatively weak in comparison to the other weekdays.

Traffic patterns

Most of the traffic originates from social sharing sites. This becomes clear by looking at the percentage of visits by medium, over time:

The blue area represents referral traffic. On the Google Analytics dashboard I can see that Reddit is the top referrer (29%), followed by Twitter (11%) and DZone (9%). Some posts were submitted to Hacker News as well, but unfortunately never really gained traction.

What's more interesting to me is that the percentage of "organic" traffic (ie. search traffic) is steadily increasing. That makes sense, since I started out with virtually no content and zero PageRank, but that has changed in the course of the year. In the complete analysis you can also find a graph that corroborates this finding using absolute numbers of search traffic.

Keep on blogging

It certainly was a good learning experience to free myself from the Google Analytics dashboard and try to work with the analytics data directly in R. Obviously, there's plenty more to explore. I'll get to that when my statistics and R knowledge has made sufficient progress.

While it's nice to put some numbers on what's happening with this blog, in the end it doesn't really matter. It's not a business and I'll keep writing anyway. It's fun, engaging and helps me to verbalize my thoughts whether someone reads it or not. Meanwhile, I hope this post has provided some inspiration on how to use R for analyzing and visualizing Google Analytics data.

Book review: Infinity and the Mind

2013-01-31T00:00:00+01:00

Infinity and the Mind provides a rare mixture of mathematical, scientific and philosophical explorations. It gives both an account of the theory of transfinite numbers and discusses its implications from a philosophical point of view. With 'The science and philosophy of the infinite' as its subtitle, it is one of the most challenging books I've read in a while. Granted, this may be because I'm neither a mathematician nor a philosopher, just a software developer who has some interest in both fields. And yes, the book is relatively old, but it has stood the test of time.

What is infinity? Does it even exist? How can there be more than one infinity? Using vivid imagery Rucker shows that throughout the ages man has pondered these questions. Ranging from Plato to Cantor, infinity has inspired awe, fear and feelings of futility. Through a fun exercise of 'name the largest natural number' the book starts exploring the concept of different levels of infinity. From the 'smallest' countable infinity ω (omega) to the whole succession of Cantor's א (aleph) numbers. Discussions are accompanied by drawings that are sometimes enlightening and other times mind-boggling. Every chapter concludes with puzzles and paradoxes for the reader. Fortunately the answers are included in the back, since many questions left me scratching my head.

Absolute Infinity and the Mindscape

It doesn't take long for Rucker to mix philosophy into his - so far mostly mathematical - discussion of infinity and transfinite numbers. Starting with the concept of Absolute Infinity, or Ω (Omega), the inconceivable infinity that contains all ordinals. Now, if you're asking yourself how can you sensibly talk about something inconceivable, read Rucker's response:

A skeptical reader could, rightly, demand to know how it is possible to discourse rationally about an inconceivable object like Ω. I would respond that Ω is a given, an object of our immediate pre-rational experience. And to use the tools of symbolic logic to investigate an empirically existing phenomenon is not to commit a category mistake, anymore than it is to look at living cells through the inanimate lenses of a microscope. We have a primitive concept of infinity. This concept is inspired, I suspect, by the same deep substrate of mind that conditions religious thought. Set theory could even be viewed as a form of exact theology.

One other philosophical notion I especially enjoyed is the Mindscape. Here Rucker conjectures that, like our bodies all occupy part of the same physical universe, our thoughts occupy part of a shared Mindscape. Your own thoughts are like a private room, and you can expand your view of the Mindscape by exploring it. Like in the physical universe, a single 'point' in the Mindscape can only be occupied by a single consciousness at the same time. But at the same time thoughts can be observed by many. Every possible thought is in the Mindscape, whether someone has thought of it or not. Of course the Mindscape is discussed in tandem with the Absolute Infinity, using set theory to formalize the notion.

Paradoxes

After discussing the theory and philosophy of infinities, the book continues exploring some famous paradoxes. The Berry paradox, Richard's paradox and the Liar paradox (think: 'this sentence is false') are treated in great detail. However, the goal is not to bask in the glory of these paradoxes. No, Rucker wants to persuade the reader that the infinite regresses that underly these paradoxes point to mental concepts defying any exact formalization. Yet we as humans can understand these paradoxes and reason about them anyway.

Thinking machines and Gödel

If the last sentence left you wondering what this means for artificial intelligence and whether machines can ever think like humans, you're thinking in the right direction. The book introduces Gödel's incompleteness theorems in a chapter titled 'Robots & souls'. Ultimately, we are lead to the conclusion that we cannot possibly formalize all of our mathematical intuitions in a finite way. Because to formulate our intuitions is to create a formal system. This formal system must be incomplete and its consistency (even if we know it is consistent) cannot be proven by the system itself according to Gödel's theorems. However, this does not mean that such a 'thinking' machine for all our mathematical intuitions cannot exist. It just means we won't be able construct or verify it. Rucker goes as far as stating that:

… rationality can never penetrate to the final, ultimate truth.

What I loved about this book is that Rucker shares his first-hand conversations with Kurt Gödel. It turns out Gödel had a mystic turn of thought, which Rucker shares throughout this book. It's less fuzzy than it sounds at first:

… "mysticism" does have a precise meaning [..] not to be confused with occultism, strange rites and so on. Mysticism is just the simple awareness of the direct identity of the individual soul and the Absolute. Too much rationality quickly becomes inane and boring. What is needed is some kind of bridge between the two.

And that is exactly what the book sets out to achieve. First using the pure mathematics of infinity, set in its historic background. But also by acknowledging that mathematics and science cannot answer all interesting questions posed by the existence of Absolute Infinity. As a Christian I recognize many of the conclusions Rucker draws while exploring the boundaries of our knowledge.

Math overflow?

Now, I'm a programmer. In my case that means I have some affinity for math, but clearly I'm out of my depth in several of the more rigorous mathematical interludes. But that's ok. The surrounding narrative is easy to follow, even if some of the finer points escape me. Especially if the alternative is for Rucker to hand-wave his way through important details, I prefer rigor. I might even have learned a thing or two along the way. What made this book most interesting is the fact that it goes beyond just the math. It tries to derive meaning from notions like (absolute) infinity, relating it concepts like consciousness and the One/Many problem.

This is not a book for casual reading. You really have to make an effort, but when you do it is a very mind-expanding book. Which reminds me, I really should finish Gödel, Escher and Bach as well some day... Since I only have finite space for this review, let me conclude by saying that I thoroughly enjoyed reading this book. If you have any interest in mathematics, philosophy and the intersection of these two, I highly recommend reading 'Infinity and the Mind'.

Scala is like Git

2012-12-20T00:00:00+01:00

Scala is often portrayed as being too complex. Many times these discussions about complexity remind me of similar arguments around the complexity of Git. I know, the comparison between a source control system and a programming language falls flat quickly when pushed too hard. But bear with me, because I do believe it can put the Scala complexity discussion in a new perspective.

Both Git and Scala are power tools, meant to disrupt the status quo rather than incrementally improving it. Git isn't just a better CVS/Subversion: its distributed nature makes it fundamentally different. The same goes for Scala: it's not just a better Java, but a daring entrant into the hybrid FP/OO arena. Interestingly, the complexity arguments are often framed by comparing Git or Scala to fundamentally more limited technologies. Yes, obviously whatever you were doing in Subversion or Java may be much simpler. What should be at the forefront of these discussions though, is what more you can achieve using Git or Scala. It's not just about the (complexity) costs, but also about the benefits. Do you buy into the vision behind these power tools?

Power tools require investment

It has taken me a solid year of actually working with Git in a team-setting to appreciate all the advantages it has to offer. Before this period, I played around with it, mainly using GitHub, but many questions remained (why do I need a staging area? Why are there three ways to do the same thing?). Trusting people who went before me and embracing Git brought me a whole new level of productivity in the end. Scala's learning curve is similar. Even though I have a background in functional programming from my university days, there still was a lot to take in. But like with Git, for me the advantages far outweigh this awkward period of embracing a tool you don't fully 'get' yet.

Such an investment definitely poses a barrier to entry. However, Git has overcome this barrier by offering something very desirable: effortless branching and merging (or rebasing, love that). Likewise, Scala's gateway drug is pervasive parallelization and an integrated stack of Scala, Akka and Play.

Clean core

What I like most about Git and Scala is that both embrace the approach of having a clean core of basic concepts. Git has its Directed Acyclic Graph of commits, trees and blobs. Every operation is essentially a transformation of the graph (and/or refs into the graph, to be precise) . Complex? Maybe, but once you grasp it, you have so much power. A similar case can be made for Scala. Even though the amount of concepts is larger - it is a programming language after all, not a VCS - to me all language features show regularity both in the small and in the large. Everything can be nested and composed just like you would expect.

That Scala has a relatively small core is illustrated by the fact that the Scala Language Specification is quarter the size of the Java Language Specification. Scala's grammar is smaller as well. Let me be quick to add that specification size is by no means a token of superiority in and of itself (cf. Brainfuck). But it is congruent with this great quote by Martin Odersky:

I have always tried to make Scala a very powerful but at the same beautifully simple language, by trying to find unifications of formerly disparate concepts.

This philosophy really resonates with me. Admittedly Scala doesn't fully unify everything that it could. It's also a very practical language with practical tradeoffs, for example concerning Java interop. Such is life.

Complexity

Even though Git and Scala have clean and composable primitives, the surfaced complexity is relatively high. Git's CLI has an abundance of commands and is notorious for its inconsistent flag usage. Scala's syntax has some surprises for newcomers as well. And yes, Scala may have a feature or two too many. If you know the underlying core model that ties everything together, such issues are annoying but you'll get over them. It's a form of accidental complexity. I do applaud the efforts of the Scala team to modularize language features, since it allows for gradual rectification of these issues!

In the end, being able to work with the elegant core models of these tools is worth the price of admission to me and many others. Yes, you will have some rope to hang yourself with both Git and Scala. Deal with it.

GUIs

But it doesn't stop there. One of the most important things for wide adoption, especially when taking into account the 'enterprise crowd', is having good GUI support. Git GUIs and IDE support for Scala have long been an afterthought. And I'm not even sure whether GUIs for Git actually reduce complexity. I tried a few, but didn't like it. For Scala IDE plugins though, it's a no-brainer. Being able to navigate types and implicits in large codebases is invaluable. Fortunately, the Scala Eclipse plugin is now backed by Typesafe and is moving forward quickly. IntelliJ's Scala plugin is still going strong as well. I'd say Scala is ahead of Git in this regard, though not ahead of other programming languages to be fair.

Real-world usage

One thing that becomes apparent when adopting Git is that it doesn't prescribe a particular workflow. Hence, many different workflows have been championed. Some even including tool support, like git-flow and Legit. Does this indicate a shortcoming with Git? Is it too complex on its own? I don't believe so. It's just that people are looking for common ground and don't want to reinvent the wheel. But it is telling that none of these workflows and supporting tools have really won out. Diversity is key it seems.

For Scala, things are a bit different. Adoption is increasing, but the community is still looking for a common 'tone-of-voice'. For a programming language this is much more crucial, especially an expressive one like Scala. Java, by contrast, has had almost two decades to mature and develop widely accepted idioms and accompanying styleguides. I believe Scala needs to go through a similar process. The signs that this process has started are there. What we need is a definitive Effective Scala book that can lead the way, like Effective Java did for Java.

The enemy of good is good enough

Why should people switch from SVN to Git, or from Java to Scala? After all, Subversion got merge-tracking and Java 8 will get lambdas. The point is that these additions do not change the fundamental core model of the 'older generation' tools. If anything, it adds unanticipated complexity without truly reaching the next level. It will be good enough for a lot of people though, and I recognize that changing VCS or programming language is not something to be taken lightly. I do urge you to explore new possibilities and see if good enough still cuts it for you.

Then there are the 'current generation' competing technologies. Git had to battle Mercurial and Bazaar. Scala faces potential competition from Kotlin and Ceylon. All these alternatives advertise simplicity or 'intuitiveness' as big selling point. And they may very well be more intuitive initially. I'm very curious whether they have to compromise on their core model and internal design philosophy to surface this intuitiveness. Fusing object-orientation and functional programming is a hard job. I'll definitely revisit this once these new languages are properly released.

In the end, Git has succeeded because it offers tangible benefits, despite its surfaced complexity. So can Scala.

Unify client-side and server-side rendering by embedding JSON

2012-11-21T00:00:00+01:00

Not many applications I work on these days are solely using the traditional server-side rendering model. Neither do they employ 100% client-side rendering and templating. Usually it's a mix in which the 'old' world meets the new world, giving rise to some interesting design decisions. In this article I want to explore a solution that combines both worlds while minimizing the duplication of rendering logic by embedding JSON in the view.

Let's introduce a simple example to illustrate the JSON embedding trick. It's on GitHub, with several branches showing different approaches. Say we have a screen containing a product list with paging:

After the initial request to the server, the first page of products is shown. Once the page is shown, we want the paging links to be AJAX-enabled: no full page reloads. The server just returns JSON representing the products and the client renders them in a table.

What duplication?

We have two main goals:

Code for rendering the table should be written once
User-experience must be as smooth as possible (i.e. no visible flickering for AJAX updates of the initial page)

Think about how you would implement this example. Strictly speaking, you can't really prevent duplicated logic. On the one hand, you need to render the table with the products from page one server-side on the initial request. Like in this JSP/JQuery based example:

<table>
    <thead>
        <tr>
            <th>Product</th>
            <th>Description</th>
            <th>Price</th>
        </tr>
    </thead>
    <c:forEach var="product" items="${products}">
        <tr>
            <td><c:out value="${product.name}" /></td>
            <td><em><c:out value="${product.description}" /></em></td>
            <td><c:out value="${product.price}" /> Euro</td>
        </tr>
    </c:forEach>
</table>
<!-- paging links omitted -->

Then you also need to construct the table again after a successful paging AJAX call:


var products = // JSON result of AJAX call
$.each(products, function(index, product) {
    productTable.append($('<tr><td>'+product.name+'</td><td><em>'
          +product.description+'</em></td><td>'+product.price+'</td></tr>'));
});

(only do this when you can guarantee the concatenated strings are properly escaped!)

It's not just the grunt work of building the table rows again. Notice how we need to add the em-tag around the description in both templates. We can already see how specific duplication is creeping in. And real-world examples have an order of magnitude more complexity! By the way, the point I want to make in this post is agnostic of the Javascript framework used. You can use your favorite Javascript templating/MVC library (too many to choose from!), I just picked plain JQuery since everybody is familiar with it. Of course the same goes for the server-side library: this issue crops with Rails, Django, and any 'traditional' webframework.

The obvious solution

How to prevent this mess? One option is to emit an empty placeholder table on the server-side, and to trigger an AJAX call from the client-side once the page has loaded to fill it with page one products. Twitter popularized this method for a while with its web-based timeline. Even though this moves all rendering of the paginated calls to the client, the solution unfortunately violates our second goal: a smooth user-experience. The user first sees an empty table which is later filled:

Depending on the network latency the effects range from nearly unnoticeable to very annoying. In my example I added a 400ms delay to illustrate the effect of the additional roundtrip. Deferring the rendering of the complete page until the Ajax call completes is even worse. And think of how this would scale with many components on the page using this approach (hint: there is a limit to the amount of concurrent connections browsers make).

Another option is to not pass JSON to the client, but return server-side rendered (partial) HTML for just this component. The trick is then to modularize your server-side code so that you can have a single template for both the initial request and subsequent AJAX calls. Component-based frameworks like JSF favor this approach. However, the consensus seems to moving towards consuming JSON on the client, giving the added benefit of a real API rather than a specific rendering band-aid.

The lazy option is to call it a day and revert to the old-skool server-side rendering model, where everything triggers a full page refresh. But that would be giving up, whereas there still is another option...

Embedded JSON

While the general idea of client-side only rendering is good, the execution can be improved. What if the server makes the product data available in a format digestible by the client in the first request, but without needing to render the actual HTML? It turns out we can do that by embedding JSON in the page rendered on the server:


<html>
<head>
    <!-- other header tags omitted -->
    <script>
        // Where the magic happens:
        var initialProducts = ${productJSON};
        // Fetch products using Ajax call and update html.
        function getProducts(page, pageSize) {
               // snipped
        }
        // Render products from JSON
        function updateProducts(products) {
              // similar to previous JS snippet, render table rows
        }
        // On document.ready, render first products.
        $(function () {
            updateProducts(initialProducts);
        });
    </script>
</head>
<body>
   <!-- body containing an empty table and paging links etc. -->
</body>
</html>

(View the full source of this page here.)

The initialProducts JS variable is initialized with a JSON literal. This JSON literal is composed on the server in the controller, and exposed with the variable productJSON to the server-side template. Once the page is on the client, it can use the client-side rendering logic without performing an AJAX request first. Other product pages are retrieved using AJAX calls, then using the same client-side rendering logic encapsulated in the updateProducts function. Observe the sequence diagram, with one less roundtrip to the server:

With this trick it's possible to blend server and client-side rendering where appropriate, without any duplication. In the complete example you will see that the pagination links are actually still rendered with a server-side forEach loop.

As an added bonus, it is trivial to revert to the initial state of the page without a roundtrip to the server (the embedded JSON never changes). Since that's not always appropriate I didn't implement it in this sample app.

Drawbacks

Of course after I 'discovered' this trick it turned out I wasn't the first. There are some drawbacks you should keep in mind. The biggest disadvantage is that whenever your JSON contains the string '', this trick breaks. The browser closes the script block the moment it encounters this tag, whether it's a part of a JS string or object literal, regardless of context. Escaping on the server is in order, since cross-site scripting is otherwise possible.

Because most webframeworks either render JSON or HTML as response, not a combination, you may have to look around for a solution. In my example I use the Jackson ObjectMapper directly from the controller, with some small trickery to force the result into a string. Other frameworks and languages provide similar capabilities, for example simplejson.dumps in Python.

One of the attractive properties of the client-side only approach is that you can do away with server-side rendered templates completely. Instead, HTML can be served from an edge node in a CDN for example. Obviously, when you want to embed dynamic JSON this won't fly. May be Edge Side Includes can help, but it was never accepted by the W3C.

Lastly, when you're trying to unify client and server-side rendering for accessibility issues (i.e. the site must work without JS enabled), this is not the trick for you. Even though all information is provided in a single request, you still need JS to do the rendering. It's not progressive enhancement by any definition. If that's you want, you might enjoy LinkedIn engineering's blog post on how they went the other way around: falling back to 'client-side' templates on the server in these cases.

Like with any technique you'll have to weigh the pros and cons for each application. In my opinion, avoiding the ubiquitous first AJAX callback while still maintaining only a single template is worth a lot.

JFall 2012: Dutch Java awesomeness

2012-11-01T00:00:00+01:00

The J-Fall conference has been a stable factor in the Dutch Java community for many years now. The 2012 edition was again jam-packed with excellent content. But above all, it's a place where more than thousand like-minded developers can meet (again) and exchange fresh new ideas!

Right from the beginning it was clear that this year's keynote would be much more dynamic than last year. Stephen Chin drove his motorcycle (yes, you read that right) into the keynote hall. What followed was a whirlwind of short presentations, outlining the major themes for Java in the coming years: Java 8/9, JavaFX 2.2, Java Embedded for ARM and Project Easel for deep HTML5 integration in NetBeans. Nothing that wasn't announced at JavaOne, but still a good overview.

Morning sessions

The first break-out session I attended was on Unfiltered. Wilfred Springer aptly called it an 'unframework' because it is so small, and focused solely on HTTP request handling. After some live coding (with audience participation where necessary) it became clear that this Scala framework is quite coherent and capable.

Next up was a 'Calling time on performance measurement' by William Louth. I was half expecting a product pitch on JXInsight, but fortunately he focused not just on the tool but also on an interesting philosophy about adaptive systems in general. Most applications are fairly 'dumb' in a sense that they keep repeating sub-optimal behavior. Things like hammering a database that is down, or even fairly advanced cases like sub-optimal use of a HashMap by program code. Louth proposed Signals as a way to build a runtime profile of such behaviors. In turn, the program can use these profiles to guide subsequent executions on a better path. Interesting, but I'm not sure how usable the whole Signals approach currently is. Even though the demo's were kind of jittery and (too) fast-paced, this session gave me some food for thought.

NightHacking

I 'missed' the second keynote by Microsoft because I was being interviewed by Stephen Chin about my upcoming cross-build injection attacks session. Stephen is traveling Europe with a motorbike on his way to Devoxx 2012, called the NightHacking tour. Meanwhile he live-streams interviews with various Java experts. It was cool to be part of the NightHacking tour, and at least I have the sticker to prove it :)

You can watch the recorded interview here:

embed

CQRS and event-sourcing

One last talk I want to highlight is 'High performance and scalable architecture' by Allard Buijze. He showed us a real-life example of a CQRS and event-sourcing based architecture for BridgeBig.com, an online bridge gaming site. The talk was very down-to-earth with some good examples on how to split up your system in functional areas that exchange (stateless) Commands and Events. Since Allard is the author of the Axon CQRS framework, it was used heavily in this system. Even in combination with the Disruptor pattern for high throughput within a single sub-system. I will have to investigate Axon, but think I think the same can be achieved with for example Akka actors and an accompanying event-sourcing library. That would be my personal preferred stack, on top of Scala.

Cross-build injection revisited

For the fourth year in a row I was able to present at J-Fall. This year I did a (Dutch) re-run of my JavaOne talk 'Cross-build injection attacks: how safe is your Java build?'. The session was scheduled on the last slot of the conference. Some people already left the building, but still there was a nice crowd. I always ask how many people have heard about cross-build injection and build security. Typically about 2-3% of the audience raises their hand, which is heartening to me because I can teach something new. Good questions where asked on actual steps people should take to prevent cross-build injection. I believe developers are starting to see that what's most convenient for them in the build process is not necessarily what's in the best interest of the safety of the end-product.

(me at the start of the session, photo courtesy of @byteheads)

It was especially nice that I could point out that Maven Central is now https accessible as well. This is a big step forward in comparison to the previous incarnation of this talk at JavaOne. I'll blog some more details about this later.

See you next year

Overall, I was impressed with the quality of content delivered by the Dutch Java community and the various international speakers at J-Fall. But it was also a great meeting point for old friends, (former) colleagues and many other interesting people. Until next year!

Cross-Build Injection in action

2012-10-08T00:00:00+02:00

A previous post introduced Cross-Build Injection (XBI) attacks. Recently, I've had the opportunity to present this work at JavaOne. In that session, I used a live example to show what could go wrong. By popular request, this post outlines the example XBI attack that I used as demonstration.

The setup consists of a trivial Maven project called 'hacked' (download here) with a single class that prints the exciting message "Hi there"... or does it? Along the way, we'll introduce a nefarious version of the maven-compiler-plugin which mixes things up a bit!

Hello world under attack

So how do we get this innocent snippet of Java:

public class Test {

  public static void main(String... args) {
    System.out.println("Hi there");
  }

}

to print "Hi, you've been p0wned" rather than the bland "Hi there" message using a Cross-Build Injection attack? The central premise of my Cross-Build Injection Attacks talk is that this depends on how secure your build setup is. Let's take a look at a typical Maven-based setup. When we do a "mvn clean install" on the project containing the Test class, the following happens:

Maven downloads both the dependencies of your application (none in this case) and the plugins (including any dependencies) it needs for its own functioning. We defined version 2.5.1 of the maven-compiler-plugin in the pom.xml, so it is retrieved from Maven Central over a regular http connection. Of course this is prone to all kinds foul play, as discussed in the original article.

Use the source

Rather than creating a network setup that intercepts and replaces Maven request/responses, let's just place a compromised artifact in our local repository manually, to simulate the effects of an XBI attack. Since the maven-compiler-plugin orchestrates the actual invocation of the Java compiler, it seems like a good place to subvert the build process. Fortunately the source is just an svn checkout away. A little digging finds us the AbstractCompilerMojo class, containing an execute() method which ultimately invokes the compiler. So let's get to it and change it to our liking:

public abstract class AbstractCompilerMojo extends AbstractMojo {

  // ... snipped lots of code ...

  public void execute()
    throws MojoExecutionException, CompilationFailureException {

<span class="n">File</span> <span class="n">src</span> <span class="o">=</span> <span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="n">basedir</span> <span class="o">+</span> <span class="s">&quot;/src/main/java/Test.java&quot;</span><span class="o">);</span>
<span class="n">String</span> <span class="n">original</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
<span class="k">try</span> <span class="o">{</span>
    <span class="n">original</span> <span class="o">=</span> <span class="n">FileUtils</span><span class="o">.</span><span class="na">fileRead</span><span class="o">(</span><span class="n">src</span><span class="o">);</span>
    <span class="n">String</span> <span class="n">replaced</span> <span class="o">=</span> <span class="n">original</span><span class="o">.</span><span class="na">replace</span><span class="o">(</span><span class="s">&quot; there&quot;</span><span class="o">,</span> <span class="s">&quot;, you been p0wned!&quot;</span><span class="o">);</span>
    <span class="n">FileUtils</span><span class="o">.</span><span class="na">fileWrite</span><span class="o">(</span><span class="n">src</span><span class="o">,</span> <span class="n">replaced</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">throw</span> <span class="k">new</span> <span class="n">RuntimeException</span><span class="o">(</span><span class="n">e</span><span class="o">);</span>
<span class="o">}</span>

<span class="k">try</span> <span class="o">{</span>
    <span class="c1">// original body of execute() here, calling the </span>
    <span class="c1">// compiler with the replaced source...</span>
<span class="o">}</span> <span class="k">finally</span> <span class="o">{</span>
    <span class="k">try</span> <span class="o">{</span>
        <span class="n">FileUtils</span><span class="o">.</span><span class="na">fileWrite</span><span class="o">(</span><span class="n">src</span><span class="o">,</span> <span class="n">original</span><span class="o">);</span>
    <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">IOException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="n">RuntimeException</span><span class="o">(</span><span class="n">e</span><span class="o">);</span>
    <span class="o">}</span>
<span class="o">}</span>


  }

}

(For the lazy bunch, here's the code (including the modified maven-compiler-plugin) in a convenient download.)

The first try-block reads the original source, stores it for later use and overwrites the original source with our replaced code. When the actual Java compiler later on reads the source, it sees our modified version. Of course, this is horribly hardcoded towards our example, but you can make this as smart as you want (discover and parse all sources, transform them conditionally et cetera). Then, the last try-block calls the original code of execute(). Since the original code has many return points, we use the finally-block to ensure we always write back the original source to Test.java. Nobody will see what just happened to the source being compiled...

To bring our version of the maven-compiler-plugin in place, execute

:maven-compiler-plugin$ mvn clean install -DskipTests

in the maven-compiler-plugin project that we just 'improved'. The skipTests flag is necessary since our 'improvement' obviously fails any sensible unit test in the compiler plugin. Now this nefarious version of the compiler plugin lives in your local repository.

Compile and behold

Just two more steps to see our 'improved' compiler plugin in action. Perform a mvn clean install on the 'hacked' project, and run it:

:hacked$ mvn clean install
.. snipped ..
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hacked ---
[INFO] Compiling 1 source file to .../hacked/target/classes
.. snipped ..
[INFO] -------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -------------------------------------------------------------------
.. snipped ..

:hacked$ java -cp ./target/classes Test
Hi, you've been p0wned!

There you have it: when compiled using this maven-compiler-plugin implementation, our original class seems to be untouched but the resulting .class file will print the unexpected message. And remember, we use a plain and uncompromised install of both Maven and the Java compiler. However, because Maven downloads a plugin before compiling, arbitrary code may be executed if you are not careful!

Exploiting XBI

Now of course this example would be easily caught by your friendly QA department. But think about it. What could really happen if arbitrary code were injected into your build process, or into your application? How about a backdoor? Or, if a hacker is really smart... Replace an up-to-date library with a previous version containing a known vulnerability, while pretending it is the current version. As you can see, there's plenty of options. Here's some more inspiration if you're still not convinced. So make sure you verify what you put into your build system and application. Trust is in short supply. Use it sparingly.

For some more context, here are the slides of the talk I gave at JavaOne on Cross-Build Injection attacks:

embed

Have fun, and remember: build safely!

Verify dependencies using PGP

2012-08-22T00:00:00+02:00

Many builds use automatic dependency retrieval from repositories like Maven Central. In an earlier post, I've outlined the dangers of so-called cross-build injection attacks when using automatic dependency management. In this post, I'll show how you can counter such attacks by verifying PGP signatures of artifacts downloaded from Maven Central.

There are various articles that show how to sign your artifact before uploading it to Maven Central. However, we're more likely to be library users than library authors. Unfortunately, information on how to verify PGP signatures of artifacts is surprisingly scarce. Even StackOverflow seems to dodge the bullet. So let's fix that.

Pretty Good Privacy

Pretty Good Privacy (PGP) can be used to encrypt or sign data. It uses asymmetric public/private keypairs. Since libraries on Maven Central are open source, encryption is pointless. Signing, on the other hand, can be used to prove the authenticity of a library. We can use the library authors' public key to verify that the artifact is signed with the corresponding private key. This may sound easy, but in practice this is a multi-step process. If you want to follow along with the signature verification steps in this post, you'll have to install a PGP implementation. I'm using GnuPG ('brew install gpg' on OSX).

Getting the signature

Before we can verify an artifact, we must first obtain its signature. So let's say we want to use the plexus-cipher library. First, add the dependency to the build (groupId: org.sonatype.plexus, artifactId: plexus-cipher and version: 1.7). Any buildtool that uses Maven Central to retrieve dependencies will do. Of course this can be Maven itself, but also SBT, Gradle or even Ant+Ivy (I'll assume Maven for the remainder of this post). Adding this dependency will prompt Maven to download:

http://repo1.maven.org/maven2/org/sonatype/plexus/plexus-cipher/1.7/plexus-cipher-1.7.jar
http://repo1.maven.org/maven2/org/sonatype/plexus/plexus-cipher/1.7/plexus-cipher-1.7.pom

In order to get the accompanying signatures, append .asc to these urls and download them:

$ wget http://repo1.maven.org/maven2/org/sonatype/plexus/plexus-cipher/1.7/plexus-cipher-1.7.jar.asc
$ wget http://repo1.maven.org/maven2/org/sonatype/plexus/plexus-cipher/1.7/plexus-cipher-1.7.pom.asc

Even though these signatures live alongside the artifacts themselves, Maven does not download them automatically. Also, beware that not all artifacts in Maven Central have .asc signatures. Up to three years ago, Maven Central did not require library authors to provide PGP signatures.

Verifying the signature

We have the artifact and the signature, but we still need to obtain the public key of the library author before we can continue. The gpg verify command dutifully tells us just that:

$ gpg --verify plexus-cipher-1.7.jar.asc ~/.m2/.../plexus-chipher-1.7.jar

gpg: Signature made Tue Jul 26 20:06:33 2011 CEST using DSA key ID 8DD1BDFD
gpg: Can't check signature: public key not found

Now we could of course ask the author for his public key by email, or try to download it from the project's website. That's not really a scalable solution though... Fortunately, Maven Central also requires its uploaders to publish their public key to a PGP keyserver. Specifically, MIT's keyserver. The following diagram shows how it works:

Armed with this knowledge, we can import the key with ID 8DD1BDFD (as we learned during our failed verification attempt) into our keyring:

$ gpg --keyserver pgp.mit.edu --recv-key 8DD1BDFD

gpg: requesting key 8DD1BDFD from hkp server pgp.mit.edu
gpg: key 8DD1BDFD: public key "Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1

We can make our lives even easier (depending on your tolerance for long commandlines...) by telling gpg to find and import keys automatically as they are referenced in the signature. Note that the following command combines the previous two steps into one:

$ gpg --auto-key-locate keyserver --keyserver pgp.mit.edu --keyserver-options auto-key-retrieve --verify plexus-cipher-1.7.jar.asc ~/.m2/.../plexus-chipher-1.7.jar

gpg: Signature made Tue Jul 26 20:06:33 2011 CEST using DSA key ID 8DD1BDFD
gpg: requesting key 8DD1BDFD from hkp server pgp.mit.edu
gpg: key 8DD1BDFD: public key "Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>" imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Good signature from "Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2BCB DD0F 23EA 1CAF CC11  D486 0374 CF2E 8DD1 BDFD

Our long-winded command rewards us with an even longer output, telling that this is a 'good signature'. Victory! Congratulations, you're now part of a select elite who actually verifies what they download from the internet before putting it into their mission-critical application:

Web of trust

But all is not well. In case you missed it, there's a huge warning in the verification output, ending with 'There is no indication that the signature belongs to the owner'. This means that even though everything checks out, we can not be certain that the key is actually from 'Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>'. Anyone could have uploaded such a key to the MIT keyserver. This is a hard problem™ to solve. With SSL certificates, the other well-known public/private key infrastructure, this conondrum is solved by having ultimately trusted root Certificate Authorities. This creates a centralized system which only works because browsers distribute the trusted certificates of these selected root CAs to end-users.

PGP takes a different, decentralized approach known as the web of trust. In essence, PGP allows public keys to be signed by other people. By doing so, they certify that that key is valid. This web of trust can work transitively. For example, if I sign person A's key, and person A signs person B's key, I transitively trust person B as well (there are different levels of trust, but I'll leave that aside for now). The idea is that we only sign another person's key if we met them in person and verified their identity. Obviously, it is benefial to have a large web of trust, to increase the chances that it contains a trusted someone close to yourself.

Let's try to visualize the situation in the case of Sonatype's release key: Th red keys are private, the blue keys are public and arrows indicate signing. Three different individuals have signed the public Sonatype release with their private key. You can lookup this information on the MIT keyserver. And, in turn, other people may have signed the public keys of these three individuals and so on. Now in order for gpg to fully trust the Sonatype release key, either we must sign it with our own key, or we must have signed any key that has a path in the web of trust to this release key. Signing the Sonatype release key directly would be the easiest, but not the most correct approach. How would we meet 'Sonatype, Inc.' and verify their identity? (hint: they might post their key fingerprint on a secure website). Let's assume instead that we know Brian F. Therefore, we can import and trust his key to put the web of trust in motion:

gpg --keyserver pgp.mit.edu --recv-key 3C062231

gpg: requesting key 3C062231 from hkp server pgp.mit.edu
gpg: key 3C062231: public key "Brian E Fox <brianf@apache.org>" imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Total number processed: 1
gpg:               imported: 1 

gpg --edit-key 3C06223

<snipped some certificate details>
gpg> trust
<snipped some certificate details>
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu

Your decision? 4

Instead of assigning trust to this key, we could have also signed it. Note that you need your own keypair to do this. Now lo and behold, no more warnings when verifying the signature:

$ gpg --verify plexus-cipher-1.7.pom.asc  ~/.m2/.../plexus-cipher-1.7.pom

gpg: Signature made Tue Jul 26 20:06:33 2011 CEST using DSA key ID 8DD1BDFD
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   1  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   1  signed:   1  trust: 0-, 0q, 0n, 0m, 1f, 0u
gpg: depth: 2  valid:   1  signed:   0  trust: 1-, 0q, 0n, 0m, 0f, 0u
gpg: Good signature from "Sonatype, Inc. (Sonatype release key) <dev@sonatype.com>"

Automatic verification?

By now, you must be thinking 'that is an awful lot of work just to verify a single dependency'. And you're absolutely right. Applications typically use many dependencies, and checking them all by hand quickly becomes tedious. In my opinion, there is a huge opportunity for Maven-based build tools to support automatic PGP signature verification. Until that is the case though, you can also use Sonatype's Nexus repository manager as a proxy to Maven Central. It can automatically check the PGP signatures for proxied artifacts and refuse to serve them when the signature check fails. Unfortunately, this is only possible using the commercial version Nexus Pro, not with the open source version.

Wrapping up

Before you dismiss everything above as wildly impractical, I urge you to read up on cross-build injection attacks. Decide for yourself whether your applications constitute a target for such attacks, and what the impact will be. If you'd like to hear more about this, I'm also giving a talk called 'Cross-build injections attacks: how safe is your Java build' (CON3892) at JavaOne 2012. May your build be secure!

Book review: DSLs in Action

2012-08-04T00:00:00+02:00

Domain-Specific Languages are not the most used tools in the toolbox of developers. According to the author of 'DSLs in Action' this is a missed opportunity: these little languages can deliver lots of benefits. In this book he argues this position with fervor. But be warned: this is not a book for beginners.

DSLs in Action consists of three main parts and a well-filled appendix. The first part introduces the term DSL and shows the benefits of DSL-driven development. It's also made clear that the purpose of DSLs is not to allow the business to program, but to help developers write code that can be a means of communication with the business. Ultimately, a DSL is a way to achieve 'beautiful abstractions'. The introduction is followed by concrete examples of DSL implementations in Java and Groovy. But obviously, without a domain there is no DSL. Therefore, throughout the whole book the stock trading domain is used in all examples. The author takes the time to thoroughly describe this domain, so the examples are non-trivial and really fit the narrative.

Internal DSLs

In the second part the so-called 'internal DSLs' are described. These are languages embedded in an existing programming language. Of course the expressiveness of the existing programming language determines how successful such an embedding can be. Examples are provided using both dynamic languages and statically typed languages. The dynamic examples use concepts such as meta-programming, macros, and monkey-patching using Groovy, Clojure and Ruby. The statically typed examples are all written in Scala, and clearly demonstrate the value of the type-system for enforcing domain-specific rules. Fortunately you don't have to be an expert in those languages, because the book contains appendices with an introduction and a cheat sheet for each of the mentioned languages.

External DSLs

The second part then continues with 'external DSLs', which are separate languages with their own language definition and associated parser. This is often what people think of first when talking about DSLs. But it is not without reason that this alternative is discussed later. External DSLs require much more effort because you are not reusing the infrastructure of an existing programming language. Also, the added flexibility offered by external DSLs is not always necessary.

The first example of an external DSL is developed with ANTLR, a parser generator framework. Meanwhile, much is explained about parsing and grammars. The author doesn't shy away from the underlying theoretical foundations where necessary. Next, an example is given for the Eclipse XText framework. XText language definitions deliver both a language parser and an Eclipse editor plugin for your DSL, with syntax highlighting, code completion and everything you're used to with Eclipse for Java.

Finally parser combinator libraries are discussed. Parser combinators allow you to express a grammar as internal DSL in a general purpose language. Scala's parser combinator library is used to show how you can create a parser without code generation, unlike the previous two external DSL examples.

Closing remarks

The last part looks at expected developments in the field of domain-specific languages. It feels a bit strained and doesn't offer many new insights. But that's okay, since the previous parts are more than excellent. Seldom have I read a technical book which is both wide, deep and practical. Even if you do not believe in DSLs as a concept, then this book is packed with useful information. For example, it is also an excellent introduction to 'polyglot programming' (combining different programming languages).

A similar book has been published by Martin Fowler: Domain-Specific Languages. However, Fowler explicitly focuses on an Object-Oriented approach to DSLs, whereas DSLs in Action takes both the OO and functional approach. In my opinion you can't ignore functional languages when talking about (internal) DSLs.

All in all, DSLs in Action is a top-notch book that I heartily recommend.

(a Dutch version of this article was print in the Dutch Java Magazine)

Modern concurrency and Java EE

2012-07-23T00:00:00+02:00

Managing your own threads within a Java EE container is not recommended and even illegal in some circumstances. But where does that leave us when we want to use the shiny new Java concurrency frameworks like Fork/Join and Akka in Java EE applications? The addition of new concurrency facilities to the Java EE 7 spec may open some doors.

The past year I've been diving into new concurrency options on the JVM. Java SE 7 brought us Fork/Join concurrency, and other other approaches like actor-based concurrency with for example Akka gain popularity as well. I did some talks about Fork/Join (talk on Parleys, slides) and Akka (talk on Parleys, slides) and one question I always get is: 'can this be used in a Java EE applications?'

Since both Fork/Join and Akka start and stop their own thread(pool)s, the answer is: 'it usually works, but it is not recommended within Java EE containers'. Unsatisfactory answer? You bet. But fortunately things may change with the next release of Java EE.

JSR-236: Concurrency Utilities for the Java EE platform

Most application servers already offer the possibility to get a managed (sort-of, kind-of) threadpool through the WorkManager API. It was never standardized as part of Java EE though. And even if it was, it would not be enough for the Fork/Join framework. The only pluggability you get when instantiating a ForkJoinPool is a custom thread factory. For the record, it seems that Akka can be configured with WorkManagers.

However, standardization of application-facing, container-managed threading may be upon us. Work on JSR-236 commenced in 2003. Yes, that's almost a decade ago. The first draft of the Concurrency EE spec appeared in 2006, but unfortunately it languished ever since. In April of this year, however, the JSR was resuscitated and an updated version was published last month. Even though the delta with the 2006 draft is minimal (e.g. 'update package name from javax.util.concurrent to javax.enterprise.concurrent'), it is good to see things are back on the agenda.

javax.enterprise.concurrent

So what's in this elusive new javax.enterprise.concurrent package? I salvaged the following diagram from a 2006 JavaOne presentation, with the new concurrency facilities marked yellow:

The two most important interfaces for enabling alternative concurrency frameworks are ManagedExecutorService and ManagedThreadFactory, both extending their non-managed java.util.concurrent counterparts from Java SE. What does this Managed prefix buy us? It means that tasks submitted and threads created through these interfaces will be fully aware of the Java EE contextual services. Things like transaction management, security context and so on are all available when the task is executed or the thread is used. The container is fully aware of the lifecycle of these managed threads, and can offer generic monitoring and tuning options.

Another interesting aspect of the propesed ManagedExecutorService is that it must support distribution. One of the following properties can be configured:

Local: the task is run in the same process/server that submitted the task.
Distributable: the task may be run in any process/server including the one that submitted the task.
Distributable with Affinity: the task may be run in any process/server including the one that submitted the task. All tasks will then run on the selected process/server.

Suddenly we have the building blocks for an asynchronous, distributed computation engine at our fingertips! Obviously, the tasks need to be Serializable for this to work.

Connecting the dots

You might be wondering why it is important to have these concurrency utilities, since Java EE 6 already provides the @Asynchronous annotation on EJB's and async Servlet support is also available. However useful these are for application development, the constructs are not meant to bootstrap other concurrency libraries and frameworks. With the proposed utilities, however, you could construct your own java.util.concurrent.ThreadPoolExecutor backed with managed threads:

 // Can also use JNDI lookup in method instead of injection
 @Resource
 ManagedThreadFactory mtf;

 public ExecutorService getManagedThreadPool() {
   // All threads created will be managed
   return new ThreadPoolExecutor(5, 10, 5, TimeUnit.SECONDS,
       new ArrayBlockingQueue<Runnable>(10), tf);
 }

Similarly, ForkJoinPool can be parameterized with a custom ForkJoinPool.ForkJoinWorkerThreadFactory implementation that delegates thread creation to the ManagedThreadFactory. While trying to implement such an adapter I unfortunately ran into the issue that Fork/Join uses a custom Thread subclass (ForkJoinWorkerThread). This makes the task a bit harder, it's constructor being package protected. And obviously I can't test the code, since I'm not aware of any (public) JSR-236 implementation. Furthermore, I believe such adapters for java.util.concurrent should be part of the spec. No need to force application developers to reinvent the wheel again and again!

Now what?

When reading the JSR-236 spec it is clear that it was crafted in the ancient J2EE times. Just look at the code examples in the draft spec (EJBHome interfaces, really?). CDI is never mentioned, and neither is Fork/Join or other relevant newer technologies. Still, it's clear that this spec will lay the groundwork for integrating modern Java concurrency frameworks with Java EE. But exactly how this will work out is too early to tell.

Another obvious candidate for using the javax.enterprise.concurrent functionality is the Batch Processing JSR. There's lot's of work to do it seems. Let's hope that expert groups find enough time to join forces and move the Java EE platform forward.

Preparing a technical talk

2012-06-06T00:00:00+02:00

We software engineers love giving talks. Fortunately, the abundance of technical conferences shows that we like listening too. A lot of talks on these conferences are given by software engineers who love sharing knowledge, rather than professional speakers. This especially holds true for conferences organized by user groups. Since I'm one of those software engineers who likes to give talks, I'd like to share my thoughts on what you can do to prepare a technical talk.

In the past few years I've gained some experience with presenting at conferences. In fact, I started writing this post en-route to JEEConf where I'll give two talks (trip report here). While I enjoy doing this, I'm by no means a professional speaker. It's really a matter of learning-by-doing. To me, giving a talk in front of a like-minded audience is exhilarating, and striking up interesting conversations afterwards can be really rewarding. But don't be fooled: even though your performance may last for only an hour, you'll be investing a lot of time to make it go just right.

How to start

Let's say you have a wonderful topic in mind for a talk: 'Peer-to-peer IDEs using Websockets and Git' (fictional of course, but it sounds cool. Someone should do this). You've worked on the topic for some time, but it's time to share it with the world. Now, what will be your first step? Making slides? Coming up with a witty title? Start hacking on your demos? No. You start by asking yourself the two very important questions:

Who is my audience?
What is my message?

Is your audience into specific programming languages and frameworks? Does it consist of experienced coders, or a mix of technical and 'business' people? What are their preconceptions? All of this information helps you determine the central message of your talk before you've made a single slide. For example, our topic is a bit out there and wouldn't really fit a conference with your typical 'enterprise developers' audience. You might tone down the topic a bit in this case, focusing on a single aspect, or simply choose a different audience...

After you've profiled the audience, it is time to formulate the message of your talk. No, your title is not your message: titles can be provocative teasers, whereas your message succinctly describes what you want to convey with your talk. Think of it as the core sentence of your abstract. Of course this core message must be tailored to your audience. The message defines the focus of all remaining efforts. Our fictional topic, for example, can be brought with several different messages:

'Explain how Websockets enable cool applications like a collaborative P2P IDE that leverages Git'
'P2P collaborative IDEs are the way of the future, and here's how you could implement them using Websockets and Git'
'How Git can be used as building block for new applications'

Each of these messages leads to a vastly different narrative. Reach back to the core message every time you need to decide whether to include or omit a sub-topic. Does it strengthen your message? Above all, remember: the message should answer a question that is relevant to your audience. If not, both you and the audience will be wasting time.

Focus

A big pitfall for me is that I tend to focus too hard on the subject. This may sound strange, so let me try to explain. Doing research is necessary when preparing a technical talk. However, it is very easy to overdo this. It's not necessary to read every single blogpost ever written on the topic at hand, to delve into the source of a library, or follow a framework's mailinglist religiously. You'll tell yourself that you are getting a deeper understanding by researching so much, but remember you're not doing this (just) for you!

In the end what counts is giving a clear and concise talk to your audience. Having more information in your head does not automatically lead to a better presentation. Let the message guide the direction and amount of research that you perform on each sub-topic. There will always be someone in the audience who knows more about the topic than you. It'll be fine, trust me.

Slides

There's lots of excellent advice already on using and designing slides which I won't repeat here. A lot of it boils down to 'do less'. I have a lot of respect for speakers who deliver talks without a slide deck. But it's not for me, at least not now. Having slides to guide me through a talk is essential for me. Additionally, having visuals like diagrams and graphs always helps in technical talks. Avoid the stereotypical slides with endless bulleted lists and you're ahead of the pack already.

Code examples

Remember the P2P IDE using Websockets and Git? You can talk about it all you want, but showing code is infinitely more effective. That is, if done right. One of the worst presentations I ever saw was simply playing a screencast for the demo parts. Aside from the unreadable pixelated text (if you ever do this, use a proper codec) it was just plain boring to watch a pre-cooked movie. Keeping the narration in sync with the screencast was awkward. Just don't do this, however safe it may feel.

The other extreme is live coding. Doing live coding in front of an audience is risky, but the rewards can be commensurate. What's better than watching a skilled craftsman doing some awesome coding? Not every topic lends itself for this approach, though. You need to be able to build something from scratch without too much distracting boilerplate. Preferably building out a single example throughout the talk.

I tend to choose the happy medium: walking through some actual code in the IDE which is prepared beforehand, occasionally adding a bit. You run the risk of overwhelming the audience with this approach, so there's a few tricks I picked up. First, whenever possible introduce the demo with a diagram showing the big picture before switching to the IDE. Second, minimize the amount of scrolling back and forth and switching between files. This takes some careful planning, and IDE features such as code folding can really help in this regard. And, on a more practical note, make sure the font is readable throughout the whole room!

When code is not really essential to the talk, or you just need to show some standalone snippets, put it into a slide. Yes, it's a bit more static, but not having to switch between slides and IDE all the time is worth something as well. Just promise me you'll leave the laserpointer at home. Number the lines and refer to these linenumbers if you need to. Anything is better than a jittery red dot on the screen.

Showtime

So you've got your slides and demos ready. Now what? Some people like to do a 'dry-run' of the presentation just by themselves. You'll familiarize yourself with the slides and the flow of the presentation, and find out whether you finish early or late. While this can be an excellent preparation, for me this doesn't really work. There's something slightly odd about talking to a void. If I decide to do a dry-run, I try to organize a few colleagues. As added bonus you'll get early feedback to polish your talk. When this is not feasible, I try to verbalize my talk for at least the first three slides. This makes me think less when getting up in front of the audience. Having a flying start is a real confidence booster for the rest of the talk.

In the end giving good technical talks is a matter of doing it often. Combining deep technical knowledge with the performance art of public speaking is a valuable skill, and not just in the context of giving talks at conferences. Whenever you communicate with your customers about software requirements and technical issues, you'll find yourself using the same skillset. So I urge you to just try and prepare a talk on a framework or programming language you love. Present it to your co-workers, or at a user group meeting and keep improving!

JEEConf 2012 trip report

2012-05-20T00:00:00+02:00

What do Eastern European developers do on their free Saturday? Well, at least 650 of them go and attend a technical conference: JEEConf in Kiev, Ukraine. This is the second year that JEEConf is organized, and they've already grown from 400 attendants last year to 650 this year, coming from 9 countries surrounding Ukraine. I was invited to give two talks, one on Hibernate Performance Tuning and another on Scala & Lift.

How on earth does a Dutch guy end up on a Eastern European Java conference? Yeah, I had the same question. Turns out that the Hibernate Performance Tuning presentation I put on Slideshare a while ago is quite popular: more than 12,000 views to date. Among those viewers was one of the organizers of JEEConf, who happened to stumble upon a Hibernate performance problem. And then he thought it might be a good idea to let me explain it in person. Thanks Mikalai!

Conference day

My first talk was at 11am, right after the opening by Oracle's Arun Gupta on the future of Java EE. The nice thing about my Hibernate Performance Tuning talk is that almost everyone can relate to it. Since Hibernate practically is the de facto standard implementation for Object Relational Mapping, almost the whole audience had experience with it. Whether those are good or bad experiences I'll leave up for discussion... This led to some nice discussions and questions afterwards. Below you can view the slides, it's a slightly updated version of the original presentation:

embed

The Scala & Lift talk was scheduled later in the afternoon. Fortunately there were a few other talks in English, because any Russian beyond 'da', 'njet' and 'spasiba' is lost on me. I got lots of question during my session on Lift, people were really interested. It will be interesting to see how Lift usage develops now that Typesafe has 'officially blessed' Play 2 as official webframework for Scala. Anyway, here are the slides:

embed

A nice touch at JEEConf is the fact that every attendee gets some 'thank you' cards to write a comment on for the speakers. The idea is to hand them over directly after the talk. Therefore the act of giving feedback gets a personal touch!

Meeting new people

To me, the most interesting part of going to conferences is meeting new people and exchanging experiences. For example, I discovered that Marktplaats.nl (Dutch Craigslist equivalent) outsources a lot of its development to Ukraine. Interestingly, they're migrating from Java/Hibernate to Scala/Squeryl with great success.

Since we shared the same hotel, I also got to hang out with Dejan Bosanac. He works for FuseSource and is a committer on several Apache projects. We had great fun discussing topics ranging from the next-generation Scala-based implementation of ActiveMQ (ActiveMQ Apollo) to Serbian and Dutch politics.

Kiev

Since I arrived a day early and left the day after the conference, I had some spare time to explore the city. By the way, arriving in Ukraine is a sobering experience: I couldn't read a single sign (Cyrillic) or understand a single word. Fortunately the organisation arranged a taxi to take me to the hotel. After settling in I explored some of the sights in Kiev's city center. All in all it was a great experience, a great conference and met some cool new people. What more could you ask for? Well, may be for some pictures to finish this post:

The JEEConf venue

Arun Gupta presenting Java EE 7 to a packed room

Audience assembling before my Hibernate Performance Tuning talk

Colorful churches enough in Kiev!

Closing the main street for the weekend? Good idea!

Beautiful Kiev University building

Cross-build injection attacks: how safe is your build?

2012-03-23T00:00:00+01:00

Imagine a world in which people blindly trust binaries uploaded to the internet by random strangers. Certainly not a world in which we as software engineers want to live. Except that many of us, including myself, do exactly that. On a regular basis. In our most precious environment: the software build process.

Update 2012-08-22: I've posted a follow-up on how to verify dependencies using PGP.

What follows is a tale of trust and naiveté, leading to vulnerabilities that we rarely talk about: the injection of malicious code into our own applications. Enter the wonderful world of 'Cross-build injection attacks' (XBI). Aptly named by Fortify (pdf) after website attacks such Cross-site Scripting and Cross-site Request Forgery, but without even nearly as much mindshare among developers.

Typical builds

Most software uses other software: libraries and frameworks enable us to quickly write the good stuff rather than the tedious and boring stuff. This implicitly means we trust the code inside these libraries, based on for example the credentials of the library authors or because we've used it before (or dare I say, because we've verified the sourcecode of said library...) But how do these libraries physically end up in our own software? Let's look at the typical setup of a Java project (the remainder of this post discusses XBI on the Java platform, but towards the end we'll see that other platforms have similar issues). Most projects use Maven as their build system, or one of the fancier new build tools like SBT or Gradle which rely on the same Maven repository infrastructure. This results in the following situation:

Maven has, for all its warts, provided us with a centralized canonical repository called Maven Central that contains many popular Java libraries. That's a good thing. But how can we be sure that what Maven downloads into our local repository is actually what was put into Maven Central by the library authors? Or, put differently, how would an attacker exploit this system in order to inject malicious code into our builds?

Attacking a build

The first obvious XBI attack is to man-in-the-middle requests to Maven Central. Since Maven Central only allows http access (I guess https is cost-prohibitive CPU-wise when you are streaming binaries to the world for free), this is relatively easy. Maven does have a checksum facility, but an attacker can just man-in-the-middle the file containing the checksum as well, which is stored alongside the binary in Maven Central. In this sense, the checksums only help to detect transport problems and cannot be relied on to verify the authenticity of the binary.

Another, more involved XBI attack would be to 'poison' Maven Central by replacing binaries that are served. This only works when the security of the Maven Central servers is somehow compromised, which would be a big deal for everyone.

Cutting all ties

One obvious reaction to these threats is to not depend on Maven Central. Many companies I've worked with setup their own internal Maven repository, so the build server needn't be connected to the internet. Of course, the next question is: how do dependencies end up in this internal repository?

Indeed, by downloading them from Maven Central. The good thing is that you've decreased the attack surface since you only need to download once. But it's still not really satisfactory, since the attacks mentioned earlier still apply, albeit less so. Additional checks and measures before placing dependencies into the internal repository can help, but these are rarely in place. Looking at this survey only 50% of the companies have an open-source policy in place, and if there is one it is mainly concerned with legal aspects, not technical.

No, in practice, someone is tasked to mindlessly perform the download and shove it into the internal repository. At least now you have someone to scapegoat should it be malicious code, but really, this solves nothing. Also, this 'offline' workflow is sort of annoying during development, especially when you have manually chase transitive dependencies and do other things Maven would normally do automatically. But then again, security always has a cost.

Cryptography

Can we do better? Fortunately, yes. Starting three years ago, Maven Central requires PGP signing of uploaded artifacts. An .asc file containing the cryptographic signature is added alongside the binary. This means we can use the public key of the uploader to verify the signature derived from the binary with the author's private key. Of course, the public key should be obtained from a trusted source (such as MIT's public key server). Now we can be sure that we download the exact file that was uploaded by the library author. We're safe, even if the repository we download from has been compromised! Unless the artifact is older than three years, that is...

There's one downside: Maven does not check these PGP signatures automatically. A fully automated solution is offered by Sonatype with Nexus Professional, a commercial repository manager that can be installed inside a company. Didn't I just mention that security always has a cost?

Other platforms

I took the example of Maven and Maven Central because it's the eco-system I'm most familiar with. However, all platforms with automatic dependency management have to deal with this issue. Look, for example at this quote from the Ruby Gems manual:

1.3 Security Issues
Doesn’t this open a huge security hole? How can I trust the Gems
which are automatically downloaded from the net?

The same way you can trust all other code you install.
(I.e. ultimately, you can’t.)

Or take Perl's CPAN, which hosts Perl modules from just about anyone. Checksums are as good as it gets on CPAN. Many people even execute CPAN scripts as root for ease of use. Discussing this dire situation leads to nice quotes such as:

'Over the last 7 years, we haven't had any problems of this nature and hopefully it will remain that way ...'

or:

'How much can you trust code from CPAN? The bad news is that it is true that
there is little quality control on CPAN .. Technically, someone could place
malicious code on CPAN.'

In other words: don't sweat it, nobody would do such a thing, right?

Really?

I know, this whole post may sound far-fetched. But that doesn't make the threat less real. There are many high-profile software builds (think banks, government, law enforcement etc.) depending increasingly on open-source libraries distributed through the internet. In the end the only way to be really safe is to download the library sources, verify them by hand, build them from source and place them into your build. But then again, who has the resources to do that? I'm very curious if there have been actual XBI victims, either with Maven or similar build systems.