Draconian Overlord

Services Should Come with Stubs

2013-04-13T00:00:00-07:00

Services Should Come with Stubs

At Bizo, we do our fair share of service-oriented development, where instead of one big monolithic application, we have lots of small applications that talk to each other.

The services might integrate with each other via JSON or Thrift or what not, almost always via HTTP, but whatever the underlying wire format/protocol is, it is always hidden from the client code behind a service interface.

For example, let’s imagine a very trivial data service that provides an interface:

trait DataService {
  def saveData(id: String, data: Array[Byte])
  def getData(id: String): Array[Byte])
}

Real services would of course have more interesting contracts, but this is good enough for illustration purposes.

So, the DataService codebase is going to ship a jar, say data-service.jar, with both it’s DataService interface in it, as well as the implementation, say, DataServiceJsonImpl:

class DataServiceJsonImpl(server: String) extends DataService {
  override def saveData(id: String, data: Array[Byte]) = {
    // do JSON serialization
  }
  override def getData(id: String): Array[Byte]) = {
    // do JSON serialization
  }
}

So, this is all well and good; the downstream client codebase can program against the DataService contract, and while testing use a fake, and in production use the real DataServiceJsonImpl.

Okay, so let’s look at the fake implementation…what should it look like? Per some other posts, I generally prefer stubs, so we might write:

class DataServiceStub extends DataService {
  private val data = Map[String, Array[Byte]]()
  override def saveData(id: String, data: Array[Byte]) = {
    data.put(id, data)
  }
  override def getData(id: String): Array[Byte]) = {
    data.get(id).getOrElse { sys.error("Not found") }
  }
}

Great! We’re done.

So, the interesting thing is that the stub we wrote, DataServiceStub, is actually fairly generic; that is nothing specific about our client code in it (as written anyway). And the “fake” semantics actually very accurately mimic the real semantics (of course, as otherwise the implementation of the DataService contract would be nonsensical and our tests would be much less coherent).

What we’ve been doing recently is realizing that, instead of each client rewriting its own stub implementations of the same contract, the upstream service should just ship it’s own stub DataService implementation.

There are a few interesting aspects of this:

Obviously code reuse, as each downstream project can reuse the DataServiceStub.
Since the stub implementation is shared, the pooled effort of maintaining just one fake implementation leads to a higher quality stub that covers more of the API (instead of only the parts that each particular project needed).
The upstream developers, who are implementing DataServiceJsonImpl can also implement DataServiceStub, which makes sense as they will be most familiar with the semantics of the DataService contract.
Per a comment by Mark Dietz, we could envision an upstream service shipping the stub before the actual implementation, allowing downstream projects to start integration sooner, and using the stub’s fake-but-accurate semantics to flush out assumptions in both the upstream and downstream projects.

So, that’s the idea; I’d be delighted if I could pull a service-oriented project’s jar from Maven central and have stubs included in the jar and ready to go.

As it is today, we end up building our own stubs for a number of services–basically any internal or external service/API we touch. Some are simple, others are more work. Either way our implementations (which we open source like fakesdb when possible) are rarely application-specific, so it a shame we can’t reuse existing ones.

Somewhat tangentially, I also think services coming with/sharing stubs would help tilt the scales towards stubs over mocks–stubs obviously require some up-front investment, where as mocks are cheap right out of the box. But I think once you have more than a few tests, stubs start paying off. And so if sharing stubs could spread out this up-front cost, perhaps more projects would choose to test with stubs.

So, that’s what we’ve been trying, and I think we’ve liked it so far. I don’t know that I’ve really seen another projects do this before, which makes me think it’s somewhat novel.

Thinking about it more; I suppose most services that are exposed via HTTP (or what not) are meant to be used in a language agnostic manner; which usually means there are either multiple language-specific client binding implementations (e.g. a Ruby client implementation, a Java client implementation, etc.), all of varying quality, or no client bindings at all, and writing them is left as an exercise to the user.

Given this, where even language-specific client bindings are hit and miss, perhaps it is not surprising that few/if any projects also take the time to write language-specific stub implementations (as, the nature of stubs is that they are in-memory and so use each language’s given idioms and data structures).

Bizo is a JVM shop, so it makes sense that, for us, writing a language-specific DataServiceStub implementation in the upstream project is worth it because we know it will be consumed downstream by a Java/Scala project.

So perhaps our situation is unique. But I would think that surely highly-used client bindings (like the Amazon aws-java-sdk) should have enough users to support a shared stub implementation.

East-Oriented Programming

2013-04-12T00:00:00-07:00

East-Oriented Programming

In chatting with a coworker today about mocks vs. stubs (prompted by Fowler’s Mocks Aren’t Stubs), I was reminded about the concept of “east”-oriented programming.

To understand “east”, think of a map, like north, south, east, west; the idea, as far as I understand, is that west-oriented is stateful (you call methods and work on the return values), while east-oriented is stateless (you pass lambdas or interface implementations).

As an example, this is normal/stateful/return-oriented programming:

class Customer {
  val name: String
  val description: String 
}

// print a customer
println(customer.name)
println(customer.description))

Where as this is stateless/east-oriented programming:

class Customer {
  private val name: String
  private val description: String 
  def printOn(writer: CustomerWriter) {
    writer.printName(name)
    writer.printDescription(description)
  }
}

trait CustomerWriter {
  def printName(name: String)
  def printDescription(name: Description)
}

// print a customer
customer.printOn(new CustomerWriter() {
  override def printName(name: String) {
    println(name)
  }

  override def printDescription(desc: String) {
    println(desc)
  }
});
// could have a suite of CustomerWriter implementations,
// e.g. System.out/JSON/etc., decorators, etc.

The topic of east-oriented programming came up because east-oriented programming is the one style where I think you could use a mocking library extensively and not have crappy tests.

To explain, and as the More East post notes, as soon as you start wanting your program to use return values, mocking starts to suck. Each individual test needs to start anticipating ahead of time (when setting up the mock expects/etc. clauses) the needed return values.

Taken to an extreme, each test starts re-implementing the stateful behavior of the collaborators, but via the mocking library’s DSL, and things just get verbose and ugly. In my opinion, this is when stubs start looking awfully nice (you can implement the fake stateful behavior simply/directly in the language itself, and reuse it across all of your tests).

But, if like east-orientation, your style avoids/limits return values, and uses more event-/interaction-style contracts between collaborators, then mocks aren’t so bad.

I believe east-oriented programming is also the style preached (perhaps not with that name) by the Growing Object Oriented Software with Tests book, whose mock-heavy style I’m not a huge fan of, but whose authors I really respect because they write good code and are the only other people in Java land these days that eschew DI containers.

As far as my personal opinion of east-oriented programming, to me it looks like you’re just moving the coupling around–instead of the coupling being in return values, the coupling is in your interface contracts.

Granted, east-orientation means you get to say nifty things like “the internal implementation details of the object are protected” (e.g. there are no getters), but is calling a getter really any different than being passed the same value as a method parameter? Especially on some object that suddenly starts looking like a visitor (translation, that is not a good thing)?

Furthermore, in the above example, the CustomerWriter API and is now dictating how your client code must look (it must be spread through the printXxx methods) instead of being arranged however naturally fits your problem (where you could call getters on a whim).

Thinking more about it, I think this “replace getters with an XxxWriter” example probably is not doing east-orientation justice. If I end up re-reading GooS again, I will keep an eye out for examples I can past in here.

Nonetheless, I’m left thinking that east-orientation is great if you really want to use a mocking library. Then it will make your tests suck less. But, in my opinion, stubs will also make your tests suck less, and not require absolving your code of return values.

(One flaw of stubs if that you have to be careful to not start having unit tests rely on having “the whole system but just with stubs” to test their individual component, and instead keep things isolated. Mocks force you to do this because mocking out this much behavior would be impossible. With stubs, it is possible, but just takes discretion to not abuse it.)

Anyway, I really enjoy reading topics like east-orientation, that try to find “a better way” of structuring systems. I’m glad it’s working out for some, and since it came up again, I’m going to try and keep it in mind while I’m writing code to see if I see any patterns emerge. But, for now, I don’t think it’s quite the game-changing idiom that I was thinking it might be when I originally read about it.

Google's Build System is a Giant Maven Repo

2013-03-23T00:00:00-07:00

Google’s Build System is a Giant Maven Repo

At Bizo, we occasionally mull over our build system and how it could be improved.

Tangentially, right now our build is based on Ant and Ivy. Yes, yes, I was admittedly skeptical at first too, but our per-project build.xml files are less code than most Maven pom.xml files I’ve seen. We use an internal fork of Spring’s common-build project, and it is a surprisingly nice setup.

Anyway, I occasionally think about Google’s internal build system, and what ideas/approaches we could potentially reuse if we were to revamp our own build system. Their build system isn’t open source, but every once in awhile a blog post goes by, e.g. this Build in the Cloud post.

These Google blog posts describe an interesting setup where:

Conceptually, there is one large trunk/ directory that everyone checks out and builds all of their project’s upstream dependencies locally.
In reality, they cache builds (based on hashes of the input files/compile settings) across machines so you only build locally what has changed.

At first I thought this was a huge breakthrough (always build from trunk!), and while I still think it is, what’s interesting is that you can conceptually model it like a huge Maven repository.

Look what happens in Maven:

You check out trunk/ and download dependencies where org=foo.com, module=blah, and version=X.Y

This is basically the same thing as in Google, except:

You check out trunk/ and download dependencies where org=google, module=path/to/BUILD, and version=(hash of file inputs).

Google has basically automated the process of keeping version up to date, where each time you svn up, you’re updating the version/input hash of your upstream dependencies, and so will now download (or build) the new version.

(I’m sure there are a lot of intricacies here that I’m missing, e.g. how their FUSE file system only downloads the files you actually need, etc., but I believe this is conceptually correct.)

What’s interesting is that Maven’s SNAPSHOT and Ivy’s latest.integration (which we use) basically try to do the same thing (“always give me the latest upstream dependency”), but have to rely on expensive network pings to check “what’s the latest version?” each time you build (or they only check every N minutes, which means you risk stale results).

(Of course, the Maven/Ivy approach is very understandable, given they don’t have the benefit of a canonical, company-wise source repo.)

And, granted, as a whole, the Google build system is very different from Maven proper (cross-technology, can locally build anything missing, other things I don’t have a clue about), but I was just surprised that the “repository in the cloud” aspect is not as different as I had originally thought.

The main innovation of their repository approach is leveraging their unique position of having all source code locally to use hashes for identifying artifacts.

Stay in the Language

2013-03-08T00:00:00-08:00

Stay in the Language

I made a “gumpy old man” comment, on a G+ post from Mike Brock this morning.

The topic was some new IntelliJ support for Errai, JBoss’s GWT framework. The new IDE support allows cool stuff like auto-completion in templates based on the user’s annotated view classes.

It does this because these annotations/glue code are filled in later at GWT compile time, so it’s very useful to provide the user with instant/up-front feedback that something will break later on. Which is a great goal. Faster feedback cycles.

Here’s my comment:

“This reminds me of Groovy, where the language was dynamic, but, contrary to the delusions of the Groovy committers, in reality the majority of Groovy users really did want (in 90% of their code) static type checking/intellisense/etc.

So, since it couldn’t be in the language, they built static checking into the IDEs.

IntelliJ is really awesome at this game, of finding languages and frameworks that are dynamic enough that users love the static tooling they can bring to the table.

…but at the end of the day, it seems like all of this inference/etc. should be in the language proper, instead of glued on the side and rewritten by each IDE (IntelliJ/Eclipse/Netbeans).

Granted, templates are a different beast, and I’m sure this is a boon to Errai users, so that’s awesome. The IntelliJ guys are amazing at what they do.

But, personally, I just like approaches to templates, and frameworks in general, that, as soon as possible, handle control back over to language proper, so that the default tooling/language semantics can take over.”

This ties in a lot with my feelings of code generation–to me, code generation “done well”, scans external artifacts (templates, databases, schemas, etc.) and turns them into representations that the user-code can program directly against.

This is a great approach, because as soon as the user-code has an in-language representation (some generated interface/class), it’s “business as normal”, and all of the existing tools (auto-complete, type checking, etc.), “just work”.

In contrast, code generation approaches that are “post-compile” (e.g. GWT’s deferred binding, which runs after the user’s code is compiled, or most annotation/CDI/runtime bytecode-generated based approaches) do not create in-language representations, or at least ones that users can program directly against.

The result is that built-in tooling doesn’t work, so you need to rely on IDEs plugins that re-implement/re-use 90% of the code generation logic, but this time only to provide extra hints that tell the user “oh right, when code generation eventually runs, this is going to fail”.

In my opinion, if we run the code generation up front, a framework should get all of that error checking for free.

This opinion is the basis for a lot of my open source projects:

Joist (an ORM that does all code generation up-front off the schema),
Tessell.org (GWT framework that, among other things, parses templates up-front), and
dtonator (DTO generator that parses a YAML file up-front, plus some reflection, to pre-generate DTOs + mapping code).

I won’t say these any of these projects are perfect, but in my experience the approach has worked well.

I am biased, and of course intimately familiar with these projects, but the approach seems both simpler to implement and simpler to debug. Because if/when things go wrong, you have visible output on your hard drive to “control-click” through, instead of “crazy stuff happened at runtime”.

Of course, YMMV, there are multiple ways to skin a cat, etc.

What Makes Spark Exciting

2013-01-22T00:00:00-08:00

What Makes Spark Exciting

(Cross-posted on the Bizo dev blog.)

At Bizo, we’re currently evaluating/prototyping Spark as a replacement for Hive for our batch reports.

As a brief intro, Spark is an alternative to Hadoop. It provides a cluster computing framework for running distributed jobs. Similar to Hadoop, you provide Spark with jobs to run, and it handles splitting up the job into small tasks, assigning those tasks to machines (optionally with Hadoop-style data locality), issuing retries if tasks fail transiently, etc.

In our case, these jobs are processing a non-trivial amount of data (log files) on a regular basis, for which we currently use Hive.

Why Replace Hive?

Admittedly, Hive has served us well for quite awhile now. (One of our engineers even built a custom “Hadoop on demand” framework for running periodic on-demand Hadoop/Hive jobs in EC2 several months before Amazon Elastic Map Reduce came out.)

Without Hive, it would have been hard for us to provide the same functionality, probably at all, let alone in the same time frame.

That said, it has gotten to the point where Hive is more frequently invoked in negative contexts (“damn it, Hive”) than positive.

Personally, I admittedly even try to avoid tasks that involve working with Hive. I find it to be frustrating and, well, just not a lot of fun. Why? Two primary reasons:

1. Hive jobs are hard to test

Bizo has a culture of excellence, and for engineering one of the things this means is testing. We really like tests. Especially unit tests, which are quick to run and enable a fast TDD cycle.

Unfortunately, Hive makes unit testing basically impossible. For several reasons:

Hive scripts must be run in a local Hadoop/Hive installation.

Ironically, very few developers at Bizo have local Hadoop installations. We are admittedly spoiled by Elastic Map Reduce, such that most of us (myself anyway) wouldn’t even know how to setup Hadoop off the top of our heads. We just fire up an EMR cluster.
Hive scripts have production locations embedded in them.

Both our log files and report output are stored in S3, so our Hive scripts end up with lots of “s3://” paths scattered throughout in them.

While we do run dev versions of reports with “-dev” S3 buckets, still relying on S3 and raw log files (that are usually in a compressed/binary-ish format) is not conducive to setting up lots of really small, simplified scenarios to unit test each boundary case.
Hive scripts do not provide any abstraction–they are just one big HiveQL file. This means its hard to break up a large report into small, individually testable steps.

Despite these limitations, about a year ago we had a developer dedicate some effort to prototyping an approach that would run Hive scripts within our CI workflow. In the end, while his prototype worked, the workflow was wonky enough that we never adopted it for production projects.

The result? Our Hive reports are basically untested. This sucks.

2. Hive is hard to extend

Extending Hive via custom functions (UDFs and UDAFs) is possible, and we do it all the time–but it’s a pain in the ass.

Perhaps this is not Hive’s fault, and it’s some Hadoop internals leaking into Hive, but the various ObjectInspector hoops, to me, always seemed annoying to deal with.

Given these shortcomings, Bizo has been looking for a Hive-successor for awhile, even going so far as to prototype revolute, a Scala DSL on top of Cascading, but had not yet found something we were really excited about.

Enter Spark!

We had heard about Spark, but did not start trying it until being so impressed by the Spark presentation at AWS re:Invent (the talk received the highest rating of all non-keynote sessions) that we wanted to learn more.

One of Spark’s touted strengths is being able to load and keep data in memory, so your queries aren’t always I/O bound.

That is great, but the exciting aspect for us at Bizo is how Spark, either intentionally or serendipitously, addresses both of Hive’s primary shortcomings, and turns them into huge strengths. Specifically:

1. Spark jobs are amazingly easy to test

Writing a test in Spark is as easy as:

class SparkTest {
  @Test
  def test() {
    // this is real code...
    val sc = new SparkContext("local", "MyUnitTest')
    // and now some psuedo code...
    val output = runYourCodeThatUsesSpark(sc)
    assertAgainst(output)
  }
}

(I will go into more detail about runYourCodeThatUsesSpark in a future post.)

This one liner starts up a new SparkContext, which is all your program needs to execute Spark jobs. There is no local installation required (just have the Spark jar on your classpath, e.g. via Maven or Ivy), no local server to start/stop. It just works.

As a technical aside, this “local” mode starts up an in-process Spark instance, backed by a thread-pool, and actually opens up a few ports and temp directories, because it’s a real, live Spark instance.

Granted, this is usually more work than you want to be done in an unit test (which ideally would not hit any file or network I/O), but the redeeming quality is that it’s fast. Tests run in ~2 seconds.

Okay, yes, this is slow compared to pure, traditional unit tests, but is such a huge revolution compared to Hive that we’ll gladly take it.

2. Spark is easy to extend

Spark’s primary API is a Scala DSL, oriented around what they call an RDD, or Resilient Distributed Dataset, which is basically a collection that only supports bulk/aggregate transforms (so methods like map, filter, and groupBy, which can be seen as transforming the entire collection, but no methods like get or take which assume in-memory/random access).

Some really short, made up example code is:

// RDD[String] is like a collection of lines
val in: RDD[String] = sc.textFile("s3://bucket/path/")
// perform some operation on each line
val suffixed = in.map { line => line + "some suffix" }
// now save the new lines back out
suffixed.saveAsTextFile("s3://bucket/path2")

Spark’s job is to package up your map closure, and run it against that extra large text file across your cluster. And it does so by, after shuffling the code and data around, actually calling your closure (i.e. there is no LINQ-like introspection of the closure’s AST).

This may seem minor, but it’s huge, because it means there is no framework code or APIs standing between your running closure and any custom functions you’d want to run. Let’s say you want to use SomeUtilityClass (or the venerable StringUtils), just do:

import com.company.SomeUtilityClass
val in: RDD[String] = sc.textFile("s3://bucket/path/")
val processed = in.map { line =>
  // just call it, it's a normal method call
  SomeUtilityClass.process(line) 
}
processed.saveAsTextFile("s3://bucket/path2")

Notice how SomeUtilityClass doesn’t have to know it’s running within a Spark RDD in the cluster. It just takes a String. Done.

Similarly, Spark doesn’t need to know anything about the code you use witin the closure, it just needs to be available on the classpath of each machine in the cluster (which is easy to do as part of your cluster/job setup, you just copy some jars around).

This seamless hop between the RDD and custom Java/Scala code is very nice, and means your Spark jobs end up reading just like regular, normal Scala code (which to us is a good thing!).

Is Spark Perfect?

As full disclosure, we’re still in the early stages of testing Spark, so we can’t yet say whether Spark will be a wholesale replacement for Hive within Bizo. We haven’t gotten to any serious performance comparisons or written large, complex reports to see if Spark can take whatever we throw at it.

Personally, I am also admittedly somewhat infutuated with Spark at this point, so that could be clouding my judgement about the pros/cons and the tradeoffs with Hive.

One Spark con so far is that Spark is pre-1.0, and it can show. I’ve seen some stack traces that shouldn’t happen, and some usability warts, that hopefully will be cleared up by 1.0. (That said, even as a newbie I find the codebase small and very easy to read, such that I’ve had several small pull requests accepted already–which is a nice consolation compared to the daunting codebases of Hadoop and Hive.)

We have also seen that, for our first Spark job, moving from “Spark job written” to “Spark job running in production” is taking longer than expected. But given that Spark is a new tool to us, we expect this to be a one-time cost.

More to Come

I have a few more posts coming up which explain our approach to Spark in more detail, for example:

Testing best practices
Running Spark in EMR
Accessing partitioned S3 logs

To see those when they come out, make sure to subscribe to the blog, or, better yet, come work at Bizo and help us out!

DHH vs. Fowler

2013-01-07T00:00:00-08:00

DHH vs. Fowler

Just a quick note on a topic going around recently; David Heinemeier Hansson first asserted Dependency injection is not a virtue, calling into question a practice that is in some circles sacrosanct.

Marcel Weiher responded that, no, Dependency Injection is a Virtue. And HN commenter smoyer commented ”I’ll choose MF (Martin Fowler) over DHH anytime both have an opinion”.

So, the discussion has devolved to “DI sucks! Nu-huh! Uh-huh!”

What I believe is missing in the conversation is context. From my perspective, DHH/Rails and Fowler address two very separate types of systems.

DHH and Rails are all about pragmatic choices for fairly simple, maybe a little complex webapps built by 1-5-ish programmers (I’m sure there are large Rails projects, but I think they’re a minority).

The whole point of Rails is that it makes choices for you, guided by the assumption you’re making “just yet another webapp”. One of these choices, IMO, is playing a little fast and loose with sacred practices like decoupling (which is fine, all decoupling has a cost, which most people don’t realize).

DHH understands Rail’s (admittedly very large) niche:

“There’s nothing wrong or shameful with nailing a single use case, like VB did for Windows desktop or PHP for web scripts. It’s beautiful!” (on twitter)
“Rails is omakase” (blog post)
“This nonsense (using transaction scripts) is what happens when you actually start believing that “Rails should be a detail” in your Rails app” (on twitter)

DHH’s opinions, and Rails’s approach here, are perfectly fine–they’re directly responsible for the huge productivity wins that comes from using Rails for your “just yet another webapp”. If your app fits that sweet spot, it’s great.

Fowler, on the other hand, is more old-school/enterprise, where applications are not “just a webapp” (but probably have a webapp in them), and are built by teams an order of magnitude larger (10-20 programmers) than most Rails projects.

Fowler’s realm is where decoupling, dependency injection, etc., are held sacred, and more rightly so (even if I do think DI frameworks are too complex, I appreciate the problem of global state). This is where having global Time variables starts to suck, because there are so many a person can’t keep them in their head.

(And I do not think it’s a language issue; Java can and is used wrong, to build systems with overly complex abstractions that cost more than they are worth, but it can have a singleton Time.now that defers to a stub-able implementation without DI frameworks or reams of XML.)

So, I don’t find it very surprising that both camps have different tools and tastes.

It seems like DHH gets pissed off when people try to apply Fowler-/enterprise-patterns to Rails, which is fine, and I understand why–but by the same token, I think he should understand why people get pissed off at him for trashing (without context, i.e. he portrays them as fundamentally flawed) the patterns they’ve found to work well in their own contexts.

Spark Test

2012-12-29T00:00:00-08:00

Spark Test

At Bizo we’re currently evaluating Spark as a replacement for Hive for reporting.

So far we’re still in the prototype stage, but it’s looking promising.

As an example of some Spark code, I thought I’d copy/paste one of my “let’s try Spark” unit tests:

object SparkTest {
  val sc = new SparkContext("local", "unit test")
}

class SparkTest extends ShouldMatchers {
  import SparkTest._
  import SparkContext._

  @Test
  def test() {
    // make collection (table) a, with 3 rows, each
    // row is a tuple of (key, value), or (Int, String)
    val a = sc.parallelize(List((1, "a"), (1, "b"), (2, "c")))

    // make collection (table) b, also with 3 rows
    val b = sc.parallelize(List((1, 5.00), (1, 6.00), (3, 7.00)))

    // typical join, all that match in both a and b
    a.join(b).collect() should be === Array(
      (1, ("a", 5.00)),
      (1, ("a", 6.00)),
      (1, ("b", 5.00)),
      (1, ("b", 6.00)))

    // typical left join, include all from a
    a.leftOuterJoin(b).collect() should be === Array(
      (1, ("a", Some(5.00))),
      (1, ("a", Some(6.00))),
      (1, ("b", Some(5.00))),
      (1, ("b", Some(6.00))),
      (2, ("c", None)))

    // typical right join, include all from b
    a.rightOuterJoin(b).collect() should be === Array(
      (3, (None, 7.00)),
      (1, (Some("a"), 5.00)),
      (1, (Some("a"), 6.00)),
      (1, (Some("b"), 5.00)),
      (1, (Some("b"), 6.00)))

    // cogroup, which is the primitive, and returns each
    // key with the key's elements from both a and b
    a.cogroup(b).collect() should be === Array(
      (3, (Seq(), Seq(7.00))),
      (1, (Seq("a", "b"), Seq(5.00, 6.00))),
      (2, (Seq("c"), Seq())))
  }
}

A few things to note:

Spark’s primary API is a Scala DSL, with a collection-like RDD type, so our reports are mostly map, filter, and join calls.
Holy shit this is a unit test–completely unlike Hive, Spark can run locally (see the "local" parameter to SparkContext) very trivially, by being embedded in a unit test.

The above test takes ~2 seconds to spin up Spark in local mode (which is not actually in-process) and execute the tests. Amazing!

This is a huge reason as to why I am liking Spark much more than Hive, where our reports were basically not under test.
The above test is just kicking the tires of cogroup, join, leftOuterJoin, and rightOuterJoin.

While the join flavors are recognizable from SQL, it is interesting that cogroup is actually the primitive operation, and the join operations are implemented as secondary reorganizations of the data (via map/flatMap calls) returned from cogroup.

Refreshingly, you can read the Spark implementation of the various join operations, and they really are just 4-5 lines of very readable Scala code.

This is obviously a small example, where I was just loading some very small, hard-coded data into Spark (via the parallelize calls). Eventually I hope to post more Spark-related techniques/results as the report we’re working on gets more flushed out.

High Level Assertions

2012-12-26T00:00:00-08:00

High Level Assertions

It is often said that we should treat test code like production code, and have it be DRY, readable, well refactored, etc.

But I think, compared to production code, there is generally less discussion about how to actually do this within test code. (Well, maybe not, if you count all of the TDD/BDD posts, but those tend to be more process oriented that code oriented.)

Anyway, the pattern I strive for with most tests is (ha) BDD inspired:

public void someTest() {
  // given <some business condition>
  setupTheBusinessCondition();

  // when <some business event happens>
  invokeTheEvent();

  // then <some business artifact is observable>
  assertSomething();
}

Where ideally the comments of “given, when, then” document the why, the meaning of what’s going on, so that 6 months from now, any programmer, yourself included, could glance at the comments and follow what is going on.

(Unfortunately I don’t know any programming language that is so fluent/high-level that comments are unnecessary; given that mainstream programming languages always instruct “how”, I think comments of “why” will always have their place.)

Okay, to the point, within this idiom, it’s import for each step (given, when, then) to be as small as possible, so that the reader only has to deal with 7+/-2 lines of code to comprehend the test.

Of course, this 7+/-2 limitation is hard to accomplish.

Approach for Assertions

One trick I’ve occasionally used, specifically within the “then” assertion section, is to make custom, higher-/business-level assertions.

E.g. often you’ll see code that wants to assert “there are two $50 credit transactions”, but it takes 5 lines of code to accomplish:

assertThat(txns.getSize(), is(5));
assertThat(txns.get(0).getAmount(), is(Money.dollars(50.00));
assertThat(txns.get(0).getType(), is(CREDIT));
assertThat(txns.get(1).getAmount(), is(Money.dollars(50.00));
assertThat(txns.get(1).getType(), is(CREDIT));

We just ate more than half of our 7 LOC budget.

So, in thinking how we can make this assertion simpler, and more direct, I’ve wound up occasionally using strings (gasp) to encode a high-/business-level meaning, e.g.:

assertTxns(txns, "1/1 $50 CREDIT", "1/1 $50 CREDIT");

Or, if you have attributes about an account (like overpaid or overdue), you might do:

// then the 1st account is bad
assertAccount(account1, "#1234 CHECKING (overpaid, overdue)");
// and the 2nd account is okay
assertAccount(account2, "#5678 CHECKING");

Note how ideally information you normally wouldn’t care about, like an account being overpaid (which is hopefully unusual), is not included in the default description, as it would be noise that most test cases don’t care about.

Pros

I think the benefit of this approach is that you’re able to pack a lot of information into a single LOC. It’s like making a mini-DSL for your assertions.

It is similar to FIT table-based assertions, where the assertions are declarations of desired output that a business person would understand, and not manual, imperative checks.

I think these high-level assertions can be very nice to read, which is useful especially if you’re quickly scanning tests, trying to understand each boundary condition.

Cons

Granted, there are some things to watch out for:

You have to carefully pick and choose what the relevant information you include in the description.

If you choose too little information, the assertAccount will not be useful, and tests will have to fall back to low-level assertions.

If you choose too much information, then each assertAccount call has a lot of extra noise, which is distracting to read (“Do we really care about attribute X for this test case? Or is it just in the description by default?”).

You also risk having a whole lot of assertions fail if they include information they don’t technically care about, but somehow changes due to an otherwise unrelated change.
These are strings, so are hidden from refactoring and “find caller” references, and can generally be a pita to update if they have to change.

(As a minor justification for this, I can at least know that all of these strings are evaluated at test time.)
Making a DSL has a cost that will only pay off if you have a lot of similar assertions.

This likely means it’s only worthwhile for larger, more complex projects that have many use cases for the same entities. E.g. a banking system would surely have an awful lot of assertions against account balances/attributes.

Potential Alternative

I haven’t tried it yet, but I could see addressing the cons with an assertion method that used optional parameters, e.g. in Scala:

assertAccount(account1, id = 1234, type = Checking);

// here's the custom assertion method:
def assertAccount(
  account: Account,
  id: Int = 0,
  type: AccountType = null,
  overpaid: Boolean = null) {
  // only assert against parameters that were provided
  if (overpaid != null) {
    assertThat(account.getOverpaid, is (overpaid))
  }
}

This way each test method could opt-in to only asserting the attributes it cares about. …although this sounds very similar to the imperative approach, and is just kind of hacking it to be one 1 line.

I’m also not sure how this would work with multiple entities (like asserting against a list of accounts), although maybe the compromise is that you just have to have 1 assertion per entity.

Conclusion

Anyway, my basic thought is that “then” assertion sections of tests can easily become too many lines of code, with too many repetitive, imperative assertions, and so it’s worth keeping in mind how/when you could switch over to a more declarative DSL-/FIT-style approach, even within the confines of regular xUnit tests.

Good Test, Bad Test

2012-12-15T00:00:00-08:00

Good Test, Bad Test

I caught myself writing a bad test yesterday, but while setting up the code review thought, er, if I was reviewing this code, I would be less than thrilled.

So, I refactored the test and updated the code review to include it.

The difference in test quality was so much better, IMO, that I thought I’d post it here as an example.

The Code

So, what I am testing is a substringBetween utility method, with just finds two tokens and replaces what is between them with a new string.

Note that the places we use this are very limited/special, so we’re assuming the tokens will always exist, and so ignoring some boundary conditions that would otherwise need handled.

Anyway, here’s the code:

public static String replaceBetween(
    final String s,
    final String startToken,
    final String endToken,
    final String replacement) {
  final int i = s.indexOf(startToken);
  final int j = s.indexOf(endToken, i + startToken.length());
  return s.substring(0, i + startToken.length())
    + replacement
    + s.substring(j);
}

Bad Test

So, this is the test I started out with:

@Test
public void testReplaceBetween() {
  assertThat(
    replaceBetween("one two three four", "two", "three", "!"),
    is("one two!three four"));
  assertThat(
    replaceBetween("aaaabccaa", "b", "a", "CC"),
    is("aaaabCCaa"));
  assertThat(
    replaceBetween("starttoken ... tend", "starttoken", "t", " !!! "),
    is("starttoken !!! tend"));
}

This test covers 3 boundary cases that I cared about, but while setting up the review, I found several things confusing about it:

The strings/tokens are non-sensical to each case
The strings/tokens change arbitrarily between each case
The strings/tokens are arbitrarily longer/shorter between each case

The result is that it’s very hard to quickly glance at these tests and tell what is actually being tested.

Good Test

This is what I ended up refactoring the test to:

@Test
public void testReplaceBetween() {
  assertThat(
    replaceBetween("start middle end", "start", "end", "!"),
    is("start!end"));
}

@Test
public void testWhenEndTokenIsAlsoBeforeTheStartToken() {
  assertThat(
    replaceBetween("end start middle end", "start", "end", "!"),
    is("end start!end"));
}

@Test
public void testWhenEndTokenIsWithinTheStartToken() {
  assertThat(
    replaceBetween("startendstart middle end", "startendstart", "end", "!"),
    is("startendstart!end"));
}

I think this version is demonstrably better:

The boundary cases are highlighted in the method names
The strings under test and tokens are very simple and align with the concepts they are testing
The strings under test contain minimal noise that is not directly related to the boundary cases being tested

A big improvement.

I think this is important because the function of unit tests is not just to put boundary cases under test, it’s also to explain these boundary cases to future maintainers so they can understand why the code (and tests) work the way they do.

Perhaps the bad test, for this small/simple code snippet, is not going to ruin our codebase. But as the code being tested starts getting larger/more complex, and the tests start getting larger/more complex, I think the time taken to make the tests just as clean and readable as the main codebase is time well spent.

(Obviously this is not a new concept; hopefully I am preaching to the choir.)

Scala Implicit Conversion with Tuples

2012-12-14T00:00:00-08:00

Scala Implicit Conversion with Tuples

I had not seen this before, but while reading a blog post about this Magnet Pattern, they showed how Scala’s implicit conversions will convert a parameter list into a tuple while searching for implicit conversions. E.g.:

object ImplicitTest {
  // we have some type Foo
  case class Foo(o: AnyRef)

  // and a method that only takes Foos
  def needsAFoo(foo: Foo) = foo.toString

  // and we can convert tuples to Foo
  implicit def tupleToFoo(t: (Int, Int)) = Foo(t)

  def main(args: Array[String]) {
    // obviously, calling foo with a tuple works, as the
    // Scala compiler looks for implicit defs that would
    // take this arg that doesn't compile (Tuple2[Int, Int])
    // and returns it as a type that would compile, i.e. Foo
    println(needsAFoo((1, 2)))
    // becomes:
    // needsAFoo(tupleToFoo(new Tuple2(1, 2))))

    // but we can also pretend we're passing the arguments
    // directly to needsAFoo, and the Scala compiler will
    // group "1, 2" together into a Tuple2[Int, Int], and
    // find the same implicit def as before
    println(needsAFoo(1, 2))
  }
}

This is pretty neat, and something I’ll try and remember.

The Magnet pattern looks pretty cool, too, but I haven’t had a lot of time to study it yet.

One Click VNC

2012-12-01T00:00:00-08:00

One Click VNC

I did some of the usual family tech support over the Thanksgiving holiday, this time for a family member who lives in another state.

While the “issue” (too many icons in the IE favorites bar) was taken care of in short order, it seemed inevitable that something else would come up, and I was not looking forward to remote phone support.

Initially, I was quite willing to sign up for Fog Creek’s Copilot program, but ran into two issues:

It seems geared towards on-the-fly support, with the “here, we’ll email them a code and they can download it” approach.

For my situation, that was too many steps–instead, I wanted to set it up on the remote computer ahead of time, for free, and then be able to use it for a nominal daily fee if ever needed.

(Perhaps Copilot actually allows this, and I didn’t RTFM, but…)
They don’t support Linux.

Um…wtf? All Fog Creek did was pay some intern to bundle a VNC client/server into Window EXEs, hook up some admittedly vary useful firewall poking scheme via their remote servers, and hey, they have a product.

Which, fair enough, smart business execution…but Copilot is VNC and VNC is from the Linux ecosystem–so, seriously, just let me fire up TightVNC, even if on my end I have to, oh no, type in a crazy host name and port number. Whatever.

So, that led to an hour or so of poking around, before coming across exactly what I was looking for: One-Click VNC.

With One-Click VNC, you can setup the relative’s computer a head of time, with a simple “here, when I tell you to, just click on this desktop icon”.

When they click on it, One-Click starts a VNC server on their local computer, and then, via a config file, calls out to your computer’s IP address, so your waiting VNC client can accept the incoming connection.

And, tada!, you see their screen. Awesome.

Note that the server establishing the connection is not how VNC normally works, but it’s great as it means One-Click will poke a hole through the remote computer’s firewall, NAT, ISP, etc.

Which leads to the only wrinkle–your VNC client won’t have poked a hole in your firewall, it’s just waiting for an incoming connection, so you pretty much have to have a static IP (since it is hard-coded in the One-Click config file) which can forward the One-Click 5500 port back to your machine.

Which, hey, I have a static IP, so good enough.

Getting this to work if you have a dynamic IP is left as an exercise to the reader, but leave a comment if you figure it out (other than “just use Copilot on a Mac”).

Reinvent 2012 Bezos Notes

2012-11-30T00:00:00-08:00

Reinvent 2012 Bezos Notes

As a continuation of my AWS re:Invent keynote notes, here are some from Werner Vogel chatting with Jeff Bezos.

Bezos: we only win when our customers win, for example:

Kindle hardware is break even, so we only make money when our customers actually use it, not just have it sit around in a drawer.
This aligns our goals with the customer’s–enjoy it as much as possible, by encouraging them with useful content, even if it’s an ancient 1st gen device.
(My comment: this is a less evil view than “oh, they just want to make more money on the consumption of content”, but if the customer is choosing to consume the content anyway, and this alignment means amazon.com is incentived to line up more content for the consumer to consume, vs. letting their device rot away un-updated on Android 3.x cough carriers cough, then perhaps the customer is actually getting a better deal, and more realized value, out of it than an up-front cost of the device anyway, even if the content cost ends up being higher over time.)

AWS utility pricing is the same way, aligned with customers. Helps customers drive their own costs down–that will lead to more usage, more adoption, more revenue in the long run.

Flywheels:

Don’t ask “what’s going to change in 10 years”…ask “what’s not going to change in the next 10 years”?
That’s what you can build a business around: stability.
For retail this is stuff like low prices, a huge catalog.
Customers will always want low cost goods. Impossible to imagine customers wishing for slower delivery.
So spinning up, investing in, these sorts of fly wheels is worth the long-term investment.
No one will say AWS should be less reliable, or more expensive, or have less than APIs.

Good ideas are not obvious, and its hard to see the obvious all of the time.

Principles of innovation:

It’s a point of view.
Select people who want to explore, want to invent (vs. people whose view point is, say, destroying the competition–that can be successful too, but is different).
Work backwards from customers–which customers can we please today.
What is not as fun:
- Willingness to fail, to be misunderstood.
- Both well-meaning and self-interested critics will question you–that’s okay and expected.
Ramp up rate of experimentation–try until you find things that customers care about.
How do you organize your life and time to increase your rate of experiments?

Biggest surprise about AWS: early adoption within government, enterprise, and education. Knew startups would adopt it (although much quicker than expected), but thought government would be a hard sell.

On business models:

Assume customers have access to perfect information.
Because they do: the Internet empowers users with better information each year (e.g. price comparison–used to require driving store to store, then phone calls, now just Google)
Balance of power is shifting from producers of goods to consumers.
If your business model depends on customers being misinformed, you need a new model.
AWS is a huge success, with no salesforce like HP/IBM/Oracle–consumers find new options that work by themselves.

Lean:

Defect reduction, don’t let defects move downstream, muda
But requires information to make decisions.
Data centers historically have been information free, no visibility–AWS changes that.
About line workers “stopping the line”:
- Stopping the line was just not done in Detroit, scene as insane, stopping the output
- But defects were still being made, so instead of being fixed, they just got skipped over and passed along.
- amazon.com customer service rep knew a customer was going to want to return a table (“everyone returns this table”)–but no communication was happening with distrubtion. This sort of communication doesn’t happen automatically. Now customer service reps can pull products from website until customer complaints are addressed.
Kaizen training: fiery “insultant”. Don’t sweep, eliminate the source of the dirt.
High margins cover a lot of sins, so you don’t need to be efficient. And it’s more fun to align with customers anyway.

How has entrepreneurship changed since founding Amazon:

Rate of change is even faster.
The heart of entrepreneurship is still risk taking:
- Entrepreneurs are all about taking and then reducing risk
- Take a business concept and drive the risk about of it until it’s sustaintable.
- Systematically identifying risk.
Empowering people with new tools can lead to new interesting results (AWS utility model enabled many, many unforeseen uses and adoption.)

Can AWS serve startups and enterprise at the same time? Of course, it already does.

The 10,000 year clock:

In a mountain in Texas, it should run for 10k years.
The symbol is about long term thinking.
Allows solving things that you wouldn’t even consider otherwise. Time horizons matter.
If asked to solve world hunger in 5 years, it’s impossible. But in 100 years? You can start envisioning gradual, long-term change.
Ask amazon.com team: could you deliver a book to Mars in 8-10 years? Super-excited to answer. What about Mongolia? “Oh right.”

Blue Origin: you can’t practice with non-reusable vehicles, they will never be low cost.

Advice:

Never chase the hot thing, you need to be there already to catch the wave when it starts.
So make sure you do what you are passionate about, so you can do it anyway, before it’s hot, and even if it never does get hot.
Prefer missionaries (doing something they love anyway) over mercenaries (chasing the cash). Ironically, missionaries usually end up with more money anyway.
Start with the customer and work backwards.

Reinvent 2012 Keynote Notes

2012-11-29T00:00:00-08:00

Reinvent 2012 Keynote Notes

The Bizo engineering team is attending AWS re:Invent this year, which has generally been awesome.

I thought I’d post the meandering notes I took during the keynotes. My disclaimer is that I’m erring on the side of posting incomplete/off-the-cuff notes vs. a more curated, edited version, as the latter just wouldn’t happen.

Andy Jassy, Senior VP, Keynote

Every day AWS is adding enough capacity to run all of amazon.com circa 2003, which was then a $5 billion business.

Segment showing how the NASA’s Mars Curiosity lander used AWS to stream video from the lander to JPL into AWS through Simple Workflow, S3, EC2, to their website. Self-hosted NASA website went down, so they redirected all traffic directly to the AWS-based Curiosity video feed (which is where all the users wanted to go anyway), which scaled and stayed up.

Pitching AWS/the cloud:

Allows trading CapEx (large, up-front server costs) for OpEx (on-going costs), with a smaller OpEx than up-front servers anyway.
Economies of scale (within cloud provider) as adoption increases.
AWS has lowered prices 23x, 25% off S3. Utility pricing. No contracts.
AWS is extremely disruptive to the economics of traditional hardware companies (Oracle, HP, IBM) which are used to 60-80% margins. AWS is low margin, high volume.
Easy to experiment (new apps, new services) with low risk (not stuck with later-unneeded, wasted infrastructure).
Don’t waste time on non-differentiating infrastructure.

Conversation with Reed Hastings:

Aside about Nicolas Carr, there used to be VPs of Electricity, as each company generated its own electricity. VPs of Data Centers/etc. will go the same way.
Netflix went from 1 million to 1 billion hours streamed/month (1000x increase) in 4 years.
Reed’s future trends:
- Picking instance types should go away, just like picking register allocation went away with compilers
- Moving running instances at scale is hard, but will happen
- Trend of cloud-assisted personal computers, e.g. Siri, taking input and using the cloud to make it useful.
- Netflix’s UI has a 10” window through which customers must choose from 50k videos–ranking and suggestions (first two home pages) are extremely important to their business

New service: RedShift, data warehouse.

Reduces costs from ~$19k/TB/year for traditional setup to $1k/TB/year.

Werner Vogel Keynote

Amazon.com being a retail, low margin business meant having 75% unused capacity was too expensive. (E.g. handling Black Friday spike of 4x traffic with traditional hardware means other 364 days of the year, only 1/4th of the traditional server capacity would be used.)

Led to avoiding business decisions, like time-based promotions (millions of users hitting F5 at 11:59pm), because hardware capacity wasn’t unavailable.

Research for fault tolerance was done, complete, 15 years ago, but only people at scale could implement it. So no one did.

Werner asserts that dynamic resources will reduce risk, complexity, etc., and increase the historically bad (30% etc.) software project completion rates. (I am skeptical.)

Commandants for 21st century architectures:

Controllable: small, loosely coupled, stateless building blocks.
- These blocks are the basis of your scale and recovery.
- Example of architecture inversion:
  - amazon.com had been calling IMDB for DVD stats with a sync call
  - Meant amazon.com load affects IMDB load
  - Change to IMDB pushing HTML into S3 buckets, amazon.com pulls from S3, now decoupled
- Architect with cost in mind–not big O, but real dollars. Utility pricing. Cost grows inline with business.
- Demo graphing cost back to a business goal like “cost per image uploaded”
Resilient
- Protecting customers is the first priority (privacy, encryption, HTTPS all the time).
- Use availability zones.
- Integrate security from the ground up (firewalls are not enough anymore, we don’t use moats), ports are closed by default.
- Build, test, deploy continuously
  - amazon.com deploys every 10 seconds, 30k instances receiving updates
  - amazon.com used to do phased deployments, roll a new version gradually through each availability zone
  - Now flips back/forth. Rollback is possible as the old machines are still around.
- Don’t think about single failures. Don’t treat failure as an exception.
Adaptive * “Use occam’s razor” and “Assume nothing”–what?
Data driven: instrument everything, all the time.
- Track business metrics, not just JVM heap size.
- Average sucks…it means half of your customers are getting something worse. * Look at the whole distribution, look at the 99th percentile. It will raise all boats.
- Put everything in logs, including business information.

Some other pithy quotes:

Thou shalt use new concepts to build new applications.
Thou shalt automate your applications and processes. * If you have to log in to an instance, you’re doing it wrong. Business metrics (latency) should drive scaling, not ops.
Thou shalt turn off the lights. * SimpleGeo graphed total cost of running clusters (prod, test, qa) on a dashboard–meant devs actually shut down test cluters when they went home.

Segment on Amazon S3, the “10th world wonder”:

At large scale even low probability events happen a lot.
1 in a billion means a few 1000 times per day.
Runs per AZ, and PUTs handle syncing across regions?
Rolled out new storage back end incrementally, everyone got it for free. No “S3 v2”.

New instance types:

240gb of RAM, 240gb of SSDs
48TB local storage.

New service: AWS data pipeline, automates moving data around. Scheduling, retry.

Joist Execute Class

2012-11-02T00:00:00-07:00

Joist Execute Class

I was porting a Ruby script to Scala today, and was reintroduced to how shockingly low-level the Java process APIs are.

For example, things you probably take for granted, like:

Resolving a simple executable name to an absolute path via PATH variable
Passing the current process’s environment variables on to the child process
Managing the stdin, stdout, stderr buffers to ensure they don’t get full (which blocks the child process)

Are not done by Java’s java.lang.Runtime class out-of-the-box. (Which I suppose is understandable, given the APIs must be cross-platform, so cannot make per-platform assumptions.)

This was nonetheless surprising for me, as some of these things I assumed were so basic to process management that it was the OS performing the logic, so they would just magically happen. But no.

So, to the point, Joist has a utility class that is perfect for this: Execute.

I originally wrote it to invoke PgAdmin/MySQL commands to backup/restore the developer’s local database, but since it has come in handy for a number of other things.

The usage is very straight forward:

val result = new Execute("ls")
  .arg("-l")
  .arg("/some/path")
  .toSystemOut()
println(result.exitCode)

So, if you’re doing similar things, you might find it useful. You can check out:

Execute.java
ExecuteTest.java
joist-util in the Joist maven repo

In other news, the ~70 lines of Ruby code I was ported ended up being ~50 lines of Scala code. Nice!

Admittedly, I cheated and put some abstractions that were used by another script into a trait, but that’s even better anyway.

Strangeloop 2012--VoltDB

2012-10-14T00:00:00-07:00

Strangeloop 2012–VoltDB

I really enjoyed going to Strangeloop this year. The signal-to-noise ratio of the presentations and conversations with other attendees was great.

Embarrassingly, I’m finally getting around to writing some blog posts about it. I had wanted to do a single review post, but that turned out to be unrealistic, so I’ll do a series of smaller ones about the specific topics/presentations I enjoyed.

The opening keynote was VoltDB, a next-generation “NewSQL” relational database built by database industry guru Michael Stonebraker.

A Look at “OldSQL” Architecture

I was initially skeptical of this keynote, as keynotes are not usually product-based, but most of Stonebraker’s talk ended up as a convincing argument that the fundamental architecture of current-generation databases has hit its scalability limits.

While this is not really news in terms of NoSQL/BigData products these days, it was nonetheless interesting to hear, from a relational database expert, a logical explanation of exactly what parts of current relational database architecture are slow and why.

For example, based on profiling (covered in OLTP Through the Looking Glass), Stonebraker’s assertion is that only 10% of time is spent doing real work, like updating the core data structures and indexes, and the other 90% is secondary concerns like waiting for disk IO, locking, etc.

Which means that traditional ways to speed up databases (novel B-tree algorithms/etc.) really can’t do much if the rest of the incidental cruft remains.

Deadpans NoSQL

Given Stonebraker is an old-school RDBMS guy, he also had a unique perspective on NoSQL.

As he tells it, he lived through a phase in the industry (late 60s/70s?), before the relational model “won”, and before ACID was taken for granted, where there were competing technologies and data models, each vying for prominence.

His point was that, looking back, relational- and ACID-based models won for a reason–they greatly simplify life for the application programmer.

So, he asserts that NoSQL-style eventual consistency isn’t the answer, it’s actually a step backwards, with a cheeky quote that “eventual consistency means ‘creates garbage’” (ha!).

I am inclined to agree, and have hoped for awhile that eventual consistency will be a passing fade until database technology catches up today’s operational constraints. Products like VoltDB and Google’s Spanner make me optimistic that this will be the case.

VoltDB as “NewSQL”

So, finally, Stonebraker talked about VoltDB as a “NewSQL” approach, which forgoes the traditional architecture by being single-threaded within a partition (no locking), in-memory (no disk, network-based replicas), and moving the computation to the data (stored procedures).

By doing this, VoltDB still provides ACID, but with impressive vertical scalability improvements, and, via partitioning, horizontal scalability as well.

I have to admit that the architecture sounds pretty sexy–it has all of the things you look for in a system that can scale–in-memory, little contention, horizontal scaling.

Where’s the Middleware?

My biggest concern about VoltDB is that it requires a huge change to how applications are currently built–you can only invoke stored procedures.

This is because you no longer get cross-wire call transactions, like in SQL/JDBC where you can do “begin transaction, select …, select …, update …, commit”, interspersing business logic with your SQL calls, and still have it all complete transactionally and with some amount of read isolation.

Instead, with VoltDB, every wire-call to the database is its own transaction. That’s it. This is where VoltDB gets some of its big wins, because it doesn’t have to do any locking/versioning to keep a transaction “in flight” while waiting for your application’s next wire call (which may take awhile or never come back).

Which makes sense–but it’s a huge change to today’s N-tier/middleware-based architectures, as you have to move any logic that must be transactional into VoltDB’s stored procedures.

This is a tough pill to swallow; any sort of domain model/ORM-based architecture goes away, as they are usually predicated on a chatty SQL connection that still provides cross-wire call transactions.

Moving Computation to Data?

Stonebraker asserts this requirement for stored procedures is “moving the computation to the data”, which is a popular approach to BigData; instead of moving TBs of data to your client/middleware machine, you ship your code directly to the database machine.

But VoltDB seems different–the types of computation stored procedures allow you do are not general purpose computations, e.g. making calls to other systems (you can’t do anything that will block), nor do any real heavy calculations (again would block), it’s just a way to batch a few SQL operations together.

I suppose this is similar to Hive, but the limitation seems more natural for Hive because your almost always doing just variation SQL-ish transformations on your data, and not real business logic.

Anyway, perhaps it is just my bias, but I’d prefer to keep computation at the middleware layer.

Potential Compromise

If I were to pick up VoltDB, I think I would try to keep a traditional domain model/ORM-ish architecture, and just use optimistic locking to enforce transaction isolation.

So, if my middleware did something like:

orm.beginTxn();

// one call to VoltDB
val b1 = BankAccount.load(1);
b1.balance += 10;

// another call to VoltDB
val b2 = BankAccount.load(2);
b2.balance -= 10;

// sends update b1 and b2 as 1 call/transaction
orm.commitTxn();

The UPDATEs for b1 and b2 would happen atomically.

But what about read isolation? I think optimistic locking would work for this, e.g. the SQL on the wire would really be:

-- b1 = BankAccount.load(1)
SELECT id, balance, version FROM bank_account WHERE id = 1;
-- b1.balance += 10

-- b2 = BankAccount.load(2)
SELECT id, balance, version FROM bank_account WHERE id = 2;
-- b2.balance += 10

-- orm.commitTxn
UPDATE bank_account SET balance = 20, version = 2
  WHERE id = 1 AND version = 1;
UPDATE bank_account SET balance = 0, version = 3
  WHERE id = 2 AND version = 2;

So, now if anyone else has touched either bank_account in between my read and my write, the version = 2 clause will fail, and I’d know the data is stale.

The trick would be that I’d need VoltDB to fail the whole transaction if the UPDATE modified count for any of the statements was zero.

This is basically moving isolation enforcement to the client, meaning it would have to fail or retry if the optimistic lock failed. I would be fine with that though, as optimistic locking is easy to build into an ORM.

With a bit of work, I could see an ORM like Joist supporting VoltDB as a backend just like the traditional MySQL/Postgres backends.

Unfortunately, I don’t think VoltDB can do this today–the client API can only invoke stored procedures, so it would require a sort of meta-stored procedure that took a list of tables/values to update and iteratively eval‘d them.

Operational Concerns

My only other concern with VoltDB is that it’s a new piece of infrastructure software that requires learning the ins/outs of. And since it owns your data, you want to make especially sure you don’t mess something up.

This is not VoltDB’s fault, it’s just the reality of relying on a new software package. I’ve seen a few bad things happen before when deploying new software that were not the software’s fault, but a configuration misunderstanding or error. You just hope that you catch these sort of things before your data is gone.

In that regard, I’d enjoy seeing a RDS-style offering for VoltDB–I don’t want to log into servers, configure clusters or logging, or whatever, I just want a GUI that says “give me X many servers, go!”.

Tangentially, it would be awesome if Amazon RDS let vendors build their own integrations, so you really could have a “VoltDB Engine” drop-down option in RDS, but supported by VoltDB instead of Amazon staff.

It is not realistic for Amazon RDS themselves to support/integration all of the myriad of new databases available these days, so it would be great if RDS was more of an open marketplace through which vendors themselves could make their databases available as a PaaS.

Conclusion

I’ve really enjoyed looking into VoltDB.

I haven’t had time to play with the community edition locally yet, but I’m going to try that soon and see how it goes. I’m not really sure what the application development/test/deploy cycle will look like, which I’m sure one of their tutorials will cover.

So, in retrospect I was surprised, but I thought the Stonebraker’s VoltDB keynote was very good, and it has me checking out their product. I definitely recommend watching the video, which I’ll link to here in a few weeks when Strangeloop makes it available online.

Brief Dependency Injection Skeptic

2012-10-07T00:00:00-07:00

Brief Dependency Injection Skeptic

I saw this go by on Twitter the other day:

“Someone needs to know how my collaborators get wired up, but it doesn’t need to be me. Now you’re ready for Dependency Injection.” (link)

This leads to a very succinct description of why I’m a DI-skeptic: I actually do care about how my collaborators get wired up.

Perhaps I’m naive, or haven’t worked in a system large enough, or complicated enough, or whatever, but so far my opinion and experience has been any that overhead or tedium that comes from manual wiring (which is little if done well, IMO) is a wash compared to fighting a framework to get it to do the same thing for you.

This is especially true of legacy DI containers, e.g. Spring.

Perhaps Guice is better, but I am still skeptical.

Dagger has potential though. The biggest reason I think so is that code generation makes the magic visible. And statically verified at build-time. Then both myself and the compiler can see what is happening. And trust it.

Aggregate Roots in SQL

2012-09-16T00:00:00-07:00

Aggregate Roots in SQL

I’m reading through a paper on Google’s Spanner, a distributed database that maintains transactions.

One interesting snippet is how they’ve added aggregate roots as a first-class concern in the schema, which is still SQL-eque, e.g.:

CREATE TABLE Users {
  uid INT64 NOT NULL,
  email STRING
} PRIMARY KEY (uid), DIRECTORY;

CREATE TABLE Albums {
  uid INT64 NOT NULL,
  aid INT64 NOT NULL,
  name STRING
} PRIMARY KEY (uid, aid),
INTERLEAVE IN PARENT Users ON DELETE CASCADE;

Where DIRECTORY declares Users as a top-level aggregate root, and INTERLEAVE IN declares Albums as a member of the Users aggregate root.

(I’m using “aggregate root”, as terminology from Evan’s Domain Driven Design.)

I think this is a pretty exciting, albeit somewhat obvious, evolution of relational schemas to handle NoSQL-scale datasets.

Obviously people have been doing this by hand for a long time, but moving it into the database just makes sense. (It’s likely other systems I’m not aware of had already done this; this is just the first I’ve seen, and especially one that builds it into the SQL schema).

I haven’t finished the paper yet, so I don’t know if they support cross-aggregate root joins, but they seem to definitely support cross-root transactions.

So, this has me wondering: if giving the database locality hints works so well, why wasn’t this done before? E.g. for just regular/pre-NoSQL/on-spinning-disks databases?

Even without being distributed, it seems like it’d make sense for a database to be able to layout related data (all of the records within an aggregate root) close together on disk, to minimize IO.

E.g. if all of an employee’s transactions where in basically-sequential order on disk, retrieving them would be cache-/IO-friendly and so very fast.

A few thoughts:

With SSDs removing the spinning disk, I suppose physical layout on disk isn’t that important anymore, at least for single-machine databases. Although that doesn’t answer why it wasn’t done before…
It would require the database to constantly reorder data to fit the aggregate root grouping; this would likely be expensive, especially to do real-time.

However, this reminds me of Google’s BigTable, which, after committing the data to a redo log to ensure consistency, delays the larger on-disk data reordering/compaction so that it’s only done infrequently and not as part of a client request.
Perhaps this is actually doable and pretty routine/old-hat with various databases out there?

I haven’t noticed it before, but that does not mean much.
Perhaps it actually would not help that much.

I’m not an expert on RDMBS literature, or a wide variety of systems, so perhaps this was tried already and deemed not worth it. Seems unlikely to my current intuition, but I suppose I should trust 30 years of RDMBS research over my 5 minutes of musing.

Seems like a good question for an RDMBS implementation expert, not that I know any to go bug them about it..

Building Sane Rich User Interfaces

2012-09-15T00:00:00-07:00

Building Sane Rich User Interfaces

I’ve been meaning to write a “lessons learned” or “best practice” sort of post on how I think rich UI applications can be made simpler and less buggy by changing your mindset from an imperative one to a declarative one.

But, until I do that, I thought it might be better to start with a concrete example. Sometimes its easier to think about something small and concrete before abstracting into the theoretical.

So, do that, I thought I’d show an example refactoring of changing some UI code from an imperative approach to a declarative approach.

The Example Problem

We’ll use a relatively small problem that I think is still a good illustration: managing a list of tabs.

E.g. the tabs might be “Tab A”, “Tab B”, and “Tab C”, and clicking on each tab hides the previous tab’s content and shows the new tab’s content.

Instead of just building this functionality into a larger page, we’ll break it out into a separate component, Tabs.

The Imperative Approach

Jumping straight to the code, this is a slightly simplified version of an imperative approach to building the tabs:

public class Tabs extends CompositeIsWidget {

  private IsTabsView view = newTabsView();
  private ArrayList<IsWidget> panels = new ArrayList<IsWidget>();

  public Tabs() {
    setWidget(view);
  }

  public void addTab(String tabName, IsWidget panel) {
    IsTabView itemView = newTabView();
    panels.add(panel);
    if (panels.size() == 1) {
      itemView.listItem().addStyleName("active");
      show(panel);
    } else {
      hide(panel);
    }
    itemView.anchor().setText(tabName);
    itemView.anchor().addClickHandler(new ClickHandler() {
      public void onClick(ClickEvent event) {
        for (IsWidget p : panels) {
          hide(p);
        }
        for (int i = 0; i < view.list().getWidgetCount(); i++) {
          view.list().getIsWidget(i).removeStyleName("active");
        }
        show(panel);
        itemView.listItem().addStyleName("active");
      }
    });
    view.list().add(itemView);
  }
}

This code might be a little foreign if you’re not used to GWT/Tessell development, but I think in general it’s pretty easy to follow.

The newXxxView calls are instantiating templates from GWT’s ui.xml UiBinder files, so newTabsView() returns a styled ul tag, and newTabView() returns a styled li tag for each tab.

In general, I don’t think there is anything terribly wrong with this code; the variable names are good, the methods aren’t egregiously long, and it’s bug free.

However, it is very imperative, in that it’s reactive to user behavior instead of being proactive. For example, what makes it seem imperative is code like:

Whether a panel is hidden or shown is done reactively in 4 places (show+hide on initial add, show+hide on each click).
The active CSS class is added in two places, and removed in one place.
We have an anonymous inner class that does multiple things in response to a click

The result is actually not that bad in this case, but I think the approach, when used in larger components and applications, eventually leads to spaghetti code.

You end up with lots of “oh, remember to do this here and there and there” lines in each place that responds to user input or logic changes. It becomes easy to miss places where updates need to happen, or not make the updates consistent, and results in bugs.

The Declarative Approach

So, now let’s try and refactor this code to being more declarative.

In taking on a declarative mindset, we want to think, without reacting to user input, how can we declare that some behavior of our UI should just happen?

For example, let’s think of the behaviors of our tabs:

When I am the current tab, show my panel
When I am the current tab, set active on my li tag
When I am clicked, make myself the current tab

Looking at these behaviors, the notion of a “current tab” is pretty apparent. Really all of our behaviors are based around which tab is the current tab.

So, let’s pull out that notion into an abstraction; let’s make a Tab and a currentTab:

public class Tabs extends CompositeIsWidget {

  private IsTabsView view = newTabsView();
  private Tab currentTab;

  public void addTab(String tabName, IsWidget panel) {
    Tab tab = new Tab(tabName, panel);
    if (currentTab == null) {
      setCurrentTab(tab);
    }
    view.list().add(tab.view);
  }

  private void setCurrentTab(Tab tab) {
    if (currentTab != null) {
      // unstyle old active tab;
    }
    // style new tab
    currentTab = tab;
  }

  private class Tab {
    private IsTabView view = newTabView();
    private Tab(String tabName, IsWidget panel) {
      view.addClickHandler(new ClickHandler() {
        public void onClick(ClickEvent e) {
          setCurrentTab(Tab.this);
        }
      });
    }
  }

This is better. We’ve moved the styling updates into one place, setCurrentTab, so things are not as spread out.

However, we’re still stuck in the reactive model; we’re having to remember to call setCurrentTab in all the right places. (Which, granted, in this small example is only two places).

The stumbling block is that currentTab is just a field–we can’t react to its changes unless we manually add code before/after our own setting of the field.

Rich UI frameworks, Tessell included, solve this by promoting simple fields into stateful properties, which can watch their current value, and call observers when it is changed. The property is the foundation for models in traditional MVC.

So, let’s change currentTab to a property:

private BasicProperty<Tab> currentTab =
  basicProperty("currentTab");

Since we have an abstraction around the value instead of just the value, we can now setup declarations around the abstraction, the property, and not just the value itself.

To see how well this works out, we can re-examine our 3 behaviors, and see how they can be translated into declarations. (We’ll use Tessell’s DSL, but the same abstractions could be done in any MVC framework.)

“When I am the current tab, show my panel” can look like:
```
binder.when(currentTab).is(this).show(panel);
```
“When I am the current tab, set active on my li tag” can look like:
```
binder.when(currentTab).is(this).set(active).on(view.li());
```
“When I am clicked, make myself the current tab” can look like:
```
binder.onClick(view.anchor()).set(currentTab).to(this);
```

And that’s it.

With a property-based DSL, like in Tessell, we have a 1-to-1 mapping between a behavior and 1 line of code.

This behavior should then “just work” as the program runs. Whether in response to user input, or other business logic code changing the model, we don’t care; our behavior should implicitly stay correctly applied.

The Final Code

So, here’s the full refactored code example:

public class Tabs extends CompositeIsWidget {

  private IsTabsView view = newTabsView();
  private Binder binder = new Binder();
  private BasicProperty<Tab> currentTab = basicProperty("currentTab");

  public Tabs() {
    setWidget(view);
  }

  public void addTab(String tabName, IsWidget panel) {
    Tab tab = new Tab(tabName, panel);
    currentTab.setIfNull(tab);
    view.list().add(tab.view);
  }

  private class Tab {
    private IsTabView view = newTabView();

    private Tab(String tabName, IsWidget panel) {
      view.anchor().setText(tabName);
      binder.when(currentTab).is(this).show(panel);
      binder.when(currentTab).is(this).set("active").on(view.listItem());
      binder.onClick(view.anchor()).set(currentTab).to(this);
    }
  }
}

When This Works

Obviously the success of this approach depends on the robustness of the binding DSL, as the DSL has to support the various behaviors you want to perform in a generic way.

Specifically for Tessell, I won’t assert that Tessell’s DSL is as refined as something like Hamcrest, which I think is the prototypical DSL in the Java world. But it supports a pretty wide array of behaviors, and is getting more when they are added as needed.

Besides the DSL, you also have to represent your problem domain as these stateful properties, basically models, to allow them to be passed declaratively into the DSL and for the DSL to be able to register observers and react to them.

I don’t think either of these are large stumbling blocks, but they do require massaging your codebase to fit the MVC/property/DSL approach, instead of just using raw DTOs.

Rules of Thumb

I have a few rules of thumb for thinking declaratively:

If you find yourself both doing a behavior and later undoing that behavior (like applying then unapplying a style), this is a smell there there’s some higher level abstraction that you could setup declaratively instead of performing it imperatively in multiple places.
Manual event listeners (e.g. anonymous inner classes for addClickListener in GWT, or anonymous functions for element.onclick in JavaScript) should be used sparingly, and, if really needed, be as tiny as possible.

Anonymous inner classes (or callback functions in JavaScript) are a smell that you’re doing something reactively, e.g. responding to specific UI input and events, that you’ll probably have to react to elsewhere in a different way, and so there might be an abstraction you could pull out.

(Tangentially, I think this minimizes the downside of using the Java language, which lacks closures, for rich UI development. Even if you had closures for reacting to user input, it’s still a reactive model, and so will lead to the same spaghetti code, albeit with less anonymous inner class boilerplate.)
Try and think “1 behavior == 1 line of code”, and then extend your DSL as necessary to fit that.

Even if your behavior is not terribly generic, I think having it in your DSL and out of your controller/presenter code is a better separation of concerns anyway.

Conclusion

So, those are my thoughts on building a rich UI in a sane way.

I don’t think this is terribly novel, as MVC is old-hat 1980s/90s stuff, albeit reinvented in Web 2.0 apps, and I think everyone generally acknowledges that declarative programming can be, for the right problems, much more succinct than imperative programming.

But I think, even if its not novel, its easy for developers to stay in an imperative mindset if you’re not consciously thinking of looking for abstractions and pulling out declarations as they become apparent.

Screencasts in Linux

2012-09-14T00:00:00-07:00

Screencasts in Linux

A few times a year, I try and record a screencast for one of my open source projects, and I always waste the first 15-30 minutes re-learning how to do screencasts in Linux.

Using ffmpeg

After trying a variety of GUI programs (like gtk-recordMyDesktop) with various degrees of success, I’ve had the best results just running ffmpeg from the command line. Which, given this is Linux, I guess shouldn’t be too surprising.

For my own future reference, the magic incantation I’ve been using is:

ffmpeg
  -f alsa -ac 2 -ar 44100 -i pulse
  -f x11grab -r 15 -s 1920x1080 -i :0.0
  -qscale 4 output.flv

This records directly to Flash, which, yeah, yeah, isn’t HTML5, but means, for now, it just works in all browsers.

Two tips for getting high quality screencasts with ffmpeg are:

Ensure you have a recent version of ffmpeg.

I have had good luck with 0.10.4-6:0.10.4-0ubuntu0jon2~oneiric1, which, IIRC, is newer than the one in the default Ubuntu repos, which I was originally having quality issues with.
Ensure your mic isn’t over-amplified.

I was getting terrible sounding audio for a long time before I realized my mic was over-amplified. I’m not sure how it got that way; it seems like some applications take it upon themselves to adjust it.

You can check your mic settings in Sound Settings > Input, where there is a slider. I find “Unamplified” is too low, but “100%” is way too high–you get a constant buzz throughout the screencast.

Putting the slider at ~20% works best for me, but it should be easy to find the ideal setting for you by doing a few test screencasts.

Using FlowPlayer

FlowPlayer is a free Flash movie player that I admittedly struggled to setup. Especially in pages that are generated by jekyll, as FlowPlayer is very finicky about the HTML being exactly what it expected.

So, along with the regular setup instructions of including their JavaScript file/etc., my current incantation that works in a jekyll/Markdown page is:

<p>
  <a href="http://.../screencast.flv" style="display:block;width:520px;height:330px;margin-left:1em;" id="player"> </a>
  <script type="text/javascript"><!-- 
    flowplayer("player", "casts/flowplayer-3.2.7.swf", { clip: { autoPlay: false } });
  --></script>
</p>

The subtle but important thing to note here is the space before the closing </a> tag; this keeps jekyll from munging the HTML into something that the FlowPlayer JavaScript doesn’t like.

Good Luck

So, that is what works for me. YMMV. If you end up recording any screenshots of your programming projects with this, post a link in the comments, I’d enjoy checking it out.

Tessell Gets Member Changed Events

2012-08-30T00:00:00-07:00

Tessell Gets Member Changed Events

Tessell got a feature I’d been meaning to add for awhile: member changed events.

Tessell has always had property changed events, e.g. StringProperty gets a new value, so it fires a change event, which, via data-binding, auto-updates the view:

binder.bind(someStringProperty).to(view.someNameField);

Which has handled a surprising number of use cases over Tessell’s lifetime.

But the other hip JavaScript MVC frameworks usually have this notion of “member changed” (my made-up term), where if you have a model:

class EmployeeModel {
  StringProperty name;
  StringProperty city;
}

You can, besides listening to name/city change events, listen to change events on the model, and get notified when any of the properties of the model changes.

This also typically works recursively up a model tree, e.g. if you add a parent/child model:

class EmployeeModel {
  StringProperty name;
  List<AddressModel> addresses;
}

class AddressModel {
  StringProperty city;
}

Then any change of an address’s city property will not only fire the event on AddressModel, but also on the parent EmployeeModel.

Well, after thinking Tessell should have that for awhile, I finally got around to it:

Add MemberChangedEvent to percolate changes up the model tree.

Coincidentally, I had a chance to use this today, with some data-binding:

binder.onMemberChange(employeeModel).execute(saveCommand);

So that any time any property on the EmployeeModel changes, we send it to the server (yes, doing this manually is terribly unhip, compared to transparent persistence frameworks like Meteor).

…in retrospect, ModelChangedEvent probably would have been a better name. Although ListProperty also fires these “something I own changed” events, so that is probably why I shied away from “model”.

I had also thought of reusing PropertyChangedEvent, instead of creating a new event type. This seems to be what some of the other frameworks do, e.g. Backbone, which just have change, which is used to denote either a property itself changing or the model implicitly changing. Perhaps this would have been better, but for now I thought the semantic distinction seemed worth making it a separate event type.