I went into describing how to implementing Command Query Responsibility Segregation in Play, but there was a fairly long question and answer section about Play as well. I couldn’t go into detail on some of the answers and missed some others, so I’ll fill in the details here.

Video

Slides

Core API

The core API is Action, which take in a Request and return a Result. The Request is immutable, but you can wrap it with extra information, which you’ll typically do with action composition. 2.1.1 introduced EssentialAction, which uses (RequestHeader => Iteratee[Array[Byte], Result]) instead of Action’s (Request => Result) and makes building Filters easier.

Again, Play’s core is simple. About as simple as you can get.

Streaming

Streaming is handled by Iteratees, which can be a confusing topic for many people. There are good writeups here and here. lila is the best application to look at for streaming, especially for sockets and hubs.

Also worth nothing: the Streaming API has changed repeatedly, and is probably still under active development. Somehow lila is typically ahead of any mailing list or documentation, so if something breaks in a new version of Play, then check if lila changed the API interaction recently.

Having good streaming primitives is something that I didn’t get into that much in the talk, but is still vitally important to ”real time web” stuff.

Templating

Play comes packaged with its own template language, Twirl, but you’re not required to use it. There is an integration into Scalate that gives you Mustache, Jade, Scaml and SSP. There’s also an example project that shows how to integrate Play with Freemarker.

One thing that Play doesn’t address directly is how to set up a structure for page layouts. Play provides you with index.scala.html and main.scala.html, but doesn’t provide you with any more structure than that. If you set up a header and footer and allow for subdirectories to use their own templates, you can minimize the amount of confusion in the views.

There’s an example in RememberMe, and this is the approach that lila takes as well.

Another thing is that Play’s default project template is intentionally minimal. If you use Backbone and HTML5 templates, then a custom giter8 template like mprihoda/play-scala may suit you better.

JSON

Play’s JSON API is very well done, and is a great way to pass data around without getting into the weeds or having to resort to XML. It goes very well with case classes.

The documentation isn’t bad, but Pascal Voitot (the author of play-json) has a series of blog posts that go the extra mile: reading JSON with JsPath, writing JSON formats, transforming JSON, and even defining JSON macros.

Forms

Form handling is always painful. The documentation helps, but really if you want to know how to do validation, using the sample forms application is the best way to pick things up. There are many useful nuggets that aren’t explicitly discussed in the documentation. In particular, the ability to make custom constraints is extremely useful.

Routing

There’s only one routing API replacement that I know of, Play Navigator, a routing DSL for REST services. However, you can use custom data types in the routing table using QueryStringBindable and PathBindable, and save yourself some “string2foo” conversion.

Optimization

Optimizing Play can be tricky, for a couple of reasons.

The first reason is that it’s easy to think you’re optimizing something that you’re not. Sadek Drobi gives a nice overview, and there’s an exhaustive mailing list discussion about asynchronous code in Play.

The second bit of trickiness is that Play 2.0 and Play 2.1 do not use Akka in the same way.

Play 2.0 uses Akka for almost everything internally.

Play 2.1 does not use Akka to handle incoming requests, or iteratees, or internal code. It uses scala.concurrent.Future instead with its own thread pools.

Play 2.1 also uses a default thread pool, which is Akka backed – ActorSystem("play") – and is used for the application code, i.e. the stuff inside Action.

This is important, because blog posts like James Ward’s Optimizing Play 2 for Database Driven Apps are only applicable to Play 2.0, not 2.1. For 2.1, use the thread pools documentation.

In addition to the “play” actor system, there’s a Play Akka plugin. The Akka plugin is actually packaged with Play itself, and you can find it under play.api.libs.concurrent.Akka.

So, if Play already uses Akka, then why define an Akka plugin?

I believe it’s because the Akka plugin defines a distinct ActorSystem("application") that can be used for backend tasks like sending email, and can be configured without impacting the “play” ActorSystem. The Akka plugin provides a useful default and enforces seperation between Play’s actors and the application’s actors.

CQRS

Given that most of the CQRS talks I’ve read have been from the enterprise perspective, it was nice to talk about CQRS in the context of functional programming and statelessness.

Message passing is something that is typically mentioned in inter process communication, or in message oriented middleware. Akka – a message passing architecture on the thread level – allows us to build “zero coupling” systems . As message passing patterns, CQRS and DDD are a good set of idioms to think about domain logic together, especially since they already assume eventual consistency and indeterminate time.

Authentication

If you’re using Scala, there are two good authentication options, RememberMe (ahem) and SecureSocial. SecureSocial has better documentation and has been around longer, but RememberMe has better security resistance to some attacks. I’m working to integrate RememberMe’s functionality into SecureSocial, but you’ll want to check out both of them.

There’s also a pure Java authentication option: Play Authenticate. I haven’t used this, but the code looks reasonable.

If you’d rather go it alone or need a basic starter application, you may find Play20StartApp useful (password reset, account confirmation, etc.)

Authorization

Deadbolt 2 is the best known authorization framework. You can use things like Shiro, but you’re better off with something specifically designed for Play.

Security

Play does fairly well on security compared to other frameworks. For example, it will set a CORS header to protect against clickjacking, will sign the session cookie with an HMAC to protect against broken authentication, supports SSL, etc.

However, there are some things that Play doesn’t do.

Play doesn’t encrypt the session cookie, so you shouldn’t store any sensitive information in there.

Play won’t protect you from replay attacks, as Play is stateless by default. You can specify a nonce or request counter to counteract this, and RememberMe uses a token based approach for persistent login cookies.

Play won’t protect you against injection attacks. You can specify value classes to validate your input against raw strings.

Play won’t protect you against security misconfiguration. You should have a release checklist.

Play won’t protect you from insecure cryptography practices. Education helps, but there’s a lot of misinformation out there as well; watch this video (and slides) and be wary of things you read on Stack Overflow and Hacker News.

Play won’t protect you from failure to restrict URL access; that’s up to the authorization framework.

Play does have cross site request forgery protection, but it will only be effective if you enable the filter and explicitly pass the CSRF helper function in through every single form. There is an authenticity token approach as well, though I haven’t used it.

Most importantly, Play won’t tell you about how web application security fails. I recommend The Tangled Web as an excellent overview on how web applications are stitched together out of different technologies, and how to secure them.

Logging

The underlying logger for Play is Logback. Logback is one of the few hardcoded dependencies in Play, which has caused some issues. Fortunately, Play uses Logback through the SLF4J logging API, but there’s no option built into Play to allow Logback to be swapped out easily. There are reports of people swapping out Logback for other logging frameworks, but I haven’t tried them.

There have also been issues with the logging configuration conflicting in places or being unclear. One thing that has tripped people up repeatedly is that all the logging configuration must be done in one place. You can’t have some logging configuration in application.conf and some configuration in logger.xml.

While Play uses SLF4J under the hood, it doesn’t expose SLF4J functionality in play.api.Logger. In fact, there are only two method signatures for logging:

1
2

  def error(message: => String) : Unit
  def error(message: => String, error: => Throwable) : Unit

This doesn’t really cover the way I like to log, and it doesn’t provide even the features that are available in SLF4J, such as parameterized logging. My own answer was to ignore the Play logging API entirely and write a Logging wrapper directly against SLF4J (with kestrel combinators, natch), but you may want to use something out of the box.

For example, Typesafe Logging, uses SLF4J and provides you with this:

1
2
3
4
5
6

  def error(message: String): Unit
  def error(message: String, params: AnyRef*): Unit
  def error(message: String, t: Throwable): Unit
  def error(marker: Marker, message: String): Unit
  def error(marker: Marker, message: String, params: AnyRef*): Unit
  def error(marker: Marker, message: String, t: Throwable): Unit

Or you can use loglady, which uses the Python API style with printf syntax:

1
2

  def error(message: String,  params: Any*) : Unit
  def error(thrown: Throwable, message: String,  params: Any*) : Unit

WAR packaging

I said in the Q&A that I didn’t think you could package Play 2 applications as WAR files. Well, it turns out that there is a plugin available, and it works with Servlet 3.0 and 2.5 containers (Tomcat 6/7, Jetty 7/8/9, JBoss 5/6/7, etc). You may need to tweak the logger to work in the container correctly.

I don’t know how Play’s performance is affected by running inside a servlet container; let me know if it works for you.

Email

Email is one of those things that I think should be divorced as much as possible from Play. It’s backend and async by nature, and this makes it something that is best handled through Akka.

akka-email is available on Github and gives you a starting place to build up a message passing infrastructure for email.

Metrics

Instrumenting applications is important. The Typesafe Console is the best thing to use, but that depends on having a Typesafe subscription if you want to use it in production.

However, there are other options:

• Ostrich, the Twitter metrics library.
• Metrics, with the metrics-scala from Erik Van Oosten, cross-compiled for multiple versions. This is what I use.
• Pillage, which has a Scala option (I have not tried this).
• statsd module for Play 2.

Profiling

When you’re load testing, it helps to be able to profile the application as well. I usually use Apache JMeter to load the system, and then use jconsole to connect to Play through JMX. YourKit is apparently a good option for profiling as well.

Deployment

There are a number of different ways to deploy Play projects. Using play dist gets you most of the way, but you may want to deploy with Ansible or Chef or Fabric. Or you can use upstart or even git hooks.

If you just want to push changes to a staging server as they happen, you can do this with rsync -avz --delete -e ssh $deployed_code staging:/opt/play-app, although this isn’t so great for production.

More?

If you have suggestions or want to point something out, please email me at will.sargent@gmail.com, and I’ll fill out this post with more details.

First, let’s define what we mean by failure.

• Unexpected internal failure: the operation fails as the result of an unfulfilled expectation, such as a null pointer reference, violated assertions, or simply bad state.
• Expected internal failure: the operation fails deliberately as a result of internal state, i.e. a blacklist or circuit breaker.
• Expected external failure: the operation fails because it is told to process some raw input, and will fail if the raw input cannot be processed.
• Unexpected external failure: the operation fails because a resource that the system depends on is not there: there’s a loose file handle, the database connection fails, or the network is down.

Java has one explicit construct for handling failure: Exception. There’s some difference of usage in Java throughout the years – IO and JDBC use checked exceptions throughout, while other API like org.w3c.dom rely on unchecked exceptions. According to Clean Code, the best practice is to use unchecked exceptions in preference to checked exceptions, but there’s still debate over whether unchecked exceptions are always appropriate.

Exceptions

Scala makes “checked vs unchecked” very simple: it doesn’t have checked exceptions. All exceptions are unchecked in Scala, even SQLException and IOException.

The way you catch an exception in Scala is by defining a PartialFunction on it:

1
2
3
4
5
6
7
8
9
10
11
12

val input = new BufferedReader(new FileReader(file))
try {
  try {
    for (line <- Iterator.continually(input.readLine()).takeWhile(_ != null)) {
      Console.println(line)
    }
  } finally {
    input.close()
  }
} catch {
  case e:IOException => errorHandler(e)
}

Or you can use control.Exception, which provides some interesting building blocks. The docs say “focuses on composing exception handlers”, which means that this set of classes supplies most of the logic you would put into a catch or finally block.

1
2
3

Exception.handling(classOf[RuntimeException], classOf[IOException]) by println apply {
  throw new IOException("foo")
}

Using the control.Exception methods is fun and you can string together exception handling logic to create automatic resource management, or an automated exception logger. On the other hand, it’s full of sharp things like allCatch. Leave it alone unless you really need it.

Exceptions don’t get mentioned very much in Scala, but they’re still the bedrock for dealing with unexpected failure. For unexpected internal failure, there’s a set of assertion methods called require, assert, and assume, which all use throwables under the hood.

Option

Option represents optional values, returning an instance of Some(A) if A exists, or None if it does not. It’s ubiquitous in Scala code, to the point where it fades into invisibility. The cheat sheet is the best way to get a handle on it.

It’s almost impossible to use Option incorrectly, but there is one caveat: Some(null) is valid. If you have code that returns null, wrap it in Option() to convert it:

1

val optionResult = Option(null) // optionResult is None.

Either

Either is a disjoint union construct. It returns either an instance of Left[L] or an instance of Right[R]. It’s commonly used for error handling, where by convention Left is used to represent failure and Right is used to represent success. It’s perfect for dealing with expected external failures such as parsing or validation.

1
2
3
4
5
6
7
8
9
10

case class FailResult(reason:String)

def parse(input:String) : Either[FailResult, String] = {
  val r = new StringTokenizer(input)
  if (r.countTokens() == 1) {
    Right(r.nextToken())
  } else {
    Left(FailResult("Could not parse string: " + input))
  }
}

Either is like Option in that it makes an abstract idea explicit by introducing an intermediate object. Unlike Option, it does not have a flatMap method, so you can’t use it in for comprehensions – not safely at any rate. You can use a left or right projection if you’re not interested in handling failure:

1

val rightFoo = for (outputFoo <- parse(input).right) yield outputFoo

More typically, you’ll use fold:

1
2
3
4

parse(input).fold(
  error => errorHandler(error),
  success => { ... }
)

You’re not limited to using Either for parsing or validation, of course. You can use it for CQRS.

1
2
3
4

case class UserFault
case class UserCreatedEvent

def createUser(user:User) : Either[UserFault, UserCreatedEvent]

or arbitary binary choices:

1

def whatShape(shape:Shape) : Either[Square, Circle]

Either is powerful, but it’s trickier than Option. In particular, it can lead to deeply nested code. It can also be misunderstood. Take the following Java lookup method:

1

public Foo lookup(String id) throws FooException // throw if not found or db exception

Scala has Option, so we can use that. But what if the database goes down? Using the error reporting convention of Either might suggest the following:

1

def lookup() : Either[FooException,Option[Foo]]

But this is awkward. If you return Either because something might fail unexpectedly, then immediately half your API becomes littered with Either[Throwable, T].

Ah, but what if you’re modifying a new object?

1

def modify(inputFoo:Foo) : Either[FooException,Foo]

If you’re dealing with expected failure and there’s good odds that the operation will fail, then returning Either is fine: create a case class representing failure FailResult and use Either[FailResult,Foo].

Don’t return exceptions through Either. If you want a construct to return exceptions, use Try.

Try

Try is similar to Either, but instead of returning any class in a Left or Right wrapper, it returns Failure[Throwable] or Success[T]. It’s an analogue for the try-catch block: it replaces try-catch’s stack based error handling with heap based error handling. Instead of having an exception thrown and having to deal with it immediately in the same thread, it disconnects the error handling and recovery.

Try can be used in for comprehensions: unlike Either, it implements flatMap. This means you can do the following:

1
2
3
4
5
6

val sumTry = for {
  int1 <- Try(Integer.parseInt("1"))
  int2 <- Try(Integer.parseInt("2"))
} yield {
  int1 + int2
}

and if there’s an exception returned from the first Try, then the for comprehension will terminate early and return the Failure.

You can get access to the exception through pattern matching:

1
2
3
4
5
6
7
8

sumTry match {
  case Failure(thrown) => {
    Console.println("Failure: " + thrown)
  }
  case Success(s) => {
    Console.println(s)
  }
}

Or through failed:

1
2
3

if (sumTry.isFailure) {
  val thrown = sumTry.failed.get
}

Try will let you recover from exceptions at any point in the chain, so you can defer recovery to the end:

1
2
3
4
5
6
7
8

val sum = for {
  int1 <- Try(Integer.parseInt("one"))
  int2 <- Try(Integer.parseInt("two"))
} yield {
  int1 + int2
} recover {
  case e => 0
}

Or recover in the middle:

1
2
3
4
5
6

val sum = for {
  int1 <- Try(Integer.parseInt("one")).recover { case e => 0 }
  int2 <- Try(Integer.parseInt("two"))
} yield {
  int1 + int2
}

There’s also a recoverWith method that will let you swap out a Failure:

1
2
3
4
5
6
7
8

val sum = for {
  int1 <- Try(Integer.parseInt("one")).recoverWith {
    case e: NumberFormatException => Failure(new IllegalArgumentException("Try 1 next time"))
  }
  int2 <- Try(Integer.parseInt("2"))
} yield {
  int1 + int2
}

You can mix Either and Try together to coerce methods that throw exceptions internally:

1
2

val either : Either[String, Int] = Try(Integer.parseInt("1")).transform({ i => Success(Right(i)) }, { e => Success(Left("FAIL")) }).get
Console.println("either is " + either.fold(l => l, r => r))

Try isn’t always appropriate. If we go back to the first exception example, this is the Try analogue:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

val input = new BufferedReader(new FileReader(file))
val results = Seq(
  Try {
    for (line <- Iterator.continually(input.readLine()).takeWhile(_ != null)) {
      Console.println(line)
    }
  },
  Try(input.close())
)

results.foreach { result =>
  result.recover {
    case e:IOException => errorHandler(e)
  }
}

Note the kludge to get around the lack of a finally block to close the stream. Victor Klang and Som Snytt suggested using a value class and transform to pimp Try:

1
2
3
4
5
6
7
8

implicit class TryOps[T](val t: Try[T]) extends AnyVal {
  def eventually[Ignore](effect: => Ignore): Try[T] = {
    val ignoring = (_: Any) => { effect; t }
    t transform (ignoring, ignoring)
  }
}

Try(1 / 0).map(_ + 1) eventually { println("Oppa Gangnam Style") }

Which is cleaner, at the cost of some magic.

Try was originally invented at Twitter to solve a specific problem: when using Future, the exception may be thrown on a different thread than the caller, and so can’t be returned through the stack. By returning an exception instead of throwing it, the system is able to reify the bottom type and let it cross thread boundaries to the calling context.

Try is new enough that people are still getting comfortable with it. I think that it’s a useful addition when try-catch blocks aren’t flexible enough, but it does have a snag: returning Try in a public API means exceptions must be dealt with by the caller. Using Try also implies to the caller that the method has captured all non fatal exceptions itself. If you’re doing this in your trait:

1

def modify(foo:Foo) : Try[Foo]

Then Try should be at the top to ensure exception capture:

1
2
3

def modify(foo:Foo) : Try[Foo] = Try {
  Foo()
}

Because exceptions must be dealt with the caller, you are placing more trust in the caller to handle or delegate a failure appropriately. With try-catch blocks, doing nothing means that the exception can pass up the stack to a top level exception handler. With Try, exceptions must be either returned or handled by each method in the chain, just like checked exceptions.

To pass the exception along, use map:

1
2
3
4
5

def fooToString(foo:Foo) : Try[String] = {
  modify(foo).map { outFoo =>
   outFoo.toString()
  }
}

Or to rethrow the exception up the stack if the return type is Unit:

1
2
3

def doStuff : Unit = {
  val modifiedFoo = modify(foo).get // throws the exception if failure
}

And you want to avoid this:

1
2
3
4
5
6
7
8

modify(foo) match {
  case Failure(f) => {
    // database failure?  don't care, swallow exception.
  }
  case Success(s) => {
    ...
  }
}

If you have a system that needs specific error logging or error recovery, it’s probably safer to stick to unchecked exceptions.

TL;DR

• Use Exception to signal unexpected failure.
• Use Option to return optional values.
• Use Option(possiblyNull) to avoid instances of Some(null).
• Use Either to report expected failure.
• Use Try over Either to return exceptions.
• Use Try for handling unexpected failure.
• Use Try when working with Future.
• In a public API, throwing an exception is probably safer than returning Try.

It’s a tough question to answer off the bat. If I point to the language features, it doesn’t get the experience across. It’s like explaining why I like English by reading from a grammar book. I don’t like Scala because of its functional aspects or its higher kinded type system. I like Scala because it solves practical, real world problems for me.

You can think of Scala as Java with all the rough edges filed off, with new features that make it easier to write correct code and harder to create bugs. Scala is not a purist’s language – it goes out of its way to make it easy for Java programmers to dip their toes in the pool. You can literally take your Java code and hit a key to create working Scala code.

So what problems does Scala solve?

Let’s start with the single biggest problem in programming, the design flaw that’s caused more errors than anything else combined. Null references.

Solving for Null

Scala avoids null pointer references by providing a special type called Option. Methods that return Option[A] (where A is the type that you want, i.e. Option[String]) will give you an object that is either a wrapper object called ‘Some’ around your type, or None. There are a number of different ways you can use Option, but I’ll just mention the ones I use most. You can chain Options together in Scala using for comprehensions:

1
2
3
4

  for {
     foo <- request.params('foo')
     bar <- request.params('bar')
  } yield myService.process(foo, bar)

or through a map:

1

  request.params('foo').map { foo => logger.debug(foo) }

or through pattern matching.

1
2
3
4

  request.params('foo') match {
    case Some(foo) => { logger.debug(foo) }
    case None => { logger.debug('no foo :-(') }
  }

Not only is this easy, but it’s also safer. You can flirt with NPE saying myOption.get, but if you do that, you deserve what you get. Not having to deal with NPE is a pleasure.

Right Type in the Right Place

What’s the second biggest problem in programming? It’s a huge issue in security and in proving program correctness: invalid, unchecked input.

Take the humble String. The work of manipulating strings is one of the biggest hairballs in programming – they’re pulled in from the environment or embedded in the code itself, and then programs try to figure out how best to deal with them. In one case, a string is displayed to the user and it’s done. In another case, an SQL query is embedded as a query parameter on a web page and passed straight through to the database. To the compiler, they’re just strings and there is no difference between them. But there are some types of strings that are suitable to pass to databases, and some which are not. Ideally, we’d like to tell the compiler that SQL and query parameters have different types. Scala makes this easy.

With the Shapeless library, you can add distinguishing type information to objects and ensure that you can’t pass random input in:

1
2
3

import shapeless.TypeOperators._
type SqlString = Newtype[String, Any]
val x: SqlString = newtype("SELECT * FROM USER")

I’ve called out strings because it’s a good example, but you can also do this for repository IDs. No more this:

1
2

  case class User(id: Int, firstName:String)
  def lookup(id:Int) : User

When you can have this:

1
2

  case class User(id: Id[User], firstName:String)
  def lookup(id:Id[User]) : User

You can also use this to validate input on the front end. One of the big problems with regular expressions is that when you parse a random string for certain kinds of input, you get back… more strings. You may be validating a string as a username (no spaces, no odd characters), but what you’ve got at the end is a string that says it’s a username.

1
2
3
4

val rawInput = request.params('foo')
if (isUsername(rawInput)) {
  val username = rawInput
}

You can replace that with something nicer.

1

   val email : Option[Username] = parseUsername(rawInput)

This embeds the constraint in the type itself. You can design your API to accept Username instead of String, and so enforce a kind of whitelisting.

Can you do this in Java? Yes, but it’s inconvenient. Scala’s type system makes it easy for you, and in 2.10 there will be Value Classes, which will provide this functionality in the core language itself.

Doing the gruntwork for you

The previous example can be improved though. Really, we just want a Username at the end – we don’t want to have to call parseUsername on it. Fortunately, Scala rewards the lazy with implicit conversions. If you define a method like this and use the implicit keyword:

1

  implicit def string2username(input : String) : Option[Username] = parseUsername(input)

And do this:

1

  val email : Option[Username] = rawInput;

Then the compiler is smart enough to see that a String isn’t an Option[Username], and looks through any implicit methods available to do the conversion.

There is an element of ‘magic’ to implicit conversions, especially when you’re reading someone else’s code and trying to figure out where the conversion is happening. You can find the appropriate implicit through the REPL, or through IDEA.

Providing Context

There are many cases in programming where everything depends on a Context object in some way: either you’re using a database connection, or you rely on a security principal, or you’re resolving objects from a request or JAAS / LDAP / Spring context… the list goes on. Whatever it is, it’s passed in by the system, it’s absolutely essential, and you can count on most of your API to depend on it in some way. A typical Java way to deal with this is to make it part of the parameter list, or try to ignore it and make it a ThreadLocal object.

1

   public void doStuff(Context context);

Scala has a better way to deal with this: you can specify implicit parameters on a method.

1

   def doStuff(implicit context:Context)

which means that anything marked as implicit that is in scope will be applied:

1
2

   implicit val context = new Context()
   doStuff  // uses val context automatically.

This is all handled by the compiler: just set up the implicits and Scala will do the rest.

A place for everything

So now you have a number of implicit methods, value classes and type definitions and wotnot. In Scala, there’s a place to keep all this stuff that is so intuitive, you may not think of it as a place at all. It’s the package object.

Package objects are supremely useful. You define a file called package.scala, then in the file you put

1
2
3

package object mypackagename {
  implicit def string2username(input : String) : Option[Username] = parseUsername(input)
}

and after that point, anything with ‘import mypackagename._’ will import the package object as well. One less thing to think about.

Free Data Transfer Objects

Case classes. So called because they’re used in case statements (see below).

1

case class Data(propertyOne:String, propertyTwo:Int)

Immutable, convenient, and packed with functionality. They make creating data types or DTOs trivial. They’re cool.

Free Range (Organic) Checking

Scala contains a powerful pattern matching feature. You can think of it as a switch statement on steroids.

1
2
3
4

a match {
   case Something => doThis
   case SomethingElse => doThat
}

There are so many things that feed into pattern matching – extractor objects, aliases, matching on types, regular expressions and wildcards – it’s the ‘regexp’ of Scala. It takes in an object as input, filters it, and manipulates it in exactly the way you want.

But the thing I really like about pattern matching is what it doesn’t let you do. It doesn’t let you miss something.

There’s a feature called sealed classes which lets you define all the valid types in a file. If you define a trait with the sealed keyword inside a file, then any classes you define inside that file that extend that trait are the ONLY classes that will extend that trait.

1
2
3

sealed trait Message { def msg: String }
case class Success(msg:String) extends Message
case class Failure(msg:String) extends Message

The compiler knows this, and so when you write use pattern matching against that trait, it knows that it must be one of the case classes defined. If not all of the case classes are defined in the match, it will print out a warning method saying that you don’t have an exhaustive match.

1
2
3
4

def log(msg: Message) = msg match {
  case Success(str) => println("Success: " + str)
  case Failure(str) => println("Failure: " + str)
}

And More

But that’s enough for now. I hope this gives you an idea of why I like Scala. If you have any features dear to your heart, add them to the comments and let me know what makes you happy.

That being said, it’s a new web framework, and the biggest issue right now is all the boring infrastructure that goes on top of it to make a framework deal with authentication, authorization, and even boring things like resetting a password.

On the Java side, Yvonnick Esnault has a good starter application (disclaimer; I contributed some code), or you can use Play Authenticate.

On the Scala side, play20-auth is a good starting point for an authentication system. However, it didn’t do token based authentication, aka “Remember Me” cookies. Adding this feature turns out to be tricky if you’re new to Scala, because extending the request pipeline in Play 2.0 Scala requires that you know a functional style of programming called ”action composition”.

So here’s a boilerplate project play20-rememberme that does authentication with remember me functionality (although it doesn’t have the password reset or confirm features added to Play20StartApp).

UPDATE: Now works with Play 2.1.

Issue #1: Public wireless networks are not secure.

The biggest risk is that someone steals your login credentials while you’re surfing the web. There are a number of ways I deal with this:

• I have a VPN service, which will encrypt all my traffic. Here’s a list.
• I use HTTPS, backed up with HTTPS Everywhere and Convergence.
• I use Privoxy, an HTTP proxy that blocks tracking websites… or all websites.

This last one is worth noting if you have a tendency to procrastinate: using a whitelist is be far more effective for me than using Freedom or SelfControl. When I go to a coffeeshop, I’m going to work; there’s no point in randomly browsing websites, so I block everything with Privoxy, and then set up a whitelist.action file.

1
2
3
4

{ +block{WORK MOAR} }

{ -block }
  .turntable.fm # Add whitelisted domain names here

You can install Privoxy using Homebrew and it’s good to go. Set up a custom location in Network Preferences, and set up the HTTPS and HTTP proxy to localhost:8118 to pass it through.

Issue #2: Public Wireless networks are flakey.

I still can’t quite believe how terrible open networks can get. So:

• I have a Verizon Mifi mobile hotspot. I don’t typically need it, but it’s worthwhile when I do.
• I use another HTTP proxy, Polipo, that caches content for me.
• I use a DNS proxy, pdnsd, that caches IP addresses from DNS, and Google DNS.

I used the instructions from Yesudeep Mangalapilly to set these up, with the modification to chain Privoxy to Polipo. See “Is it possible to run Polipo together with Privoxy?” in the FAQ.

Polipo is extremely useful, but the version on Homebrew is out of date. The trunk version is recommended if you run into any bugs. The pdnsd install on Homebrew is almost totally seamless, and Polipo recommends it.

Using Polipo and pdnsd means my exposure to the network is somewhat limited, and browsing is somewhat faster. Minimizing network traffic is important when I’m using the mobile hotspot – it’s limited to 5 GB a month, which is very easy to blow through if you’re not careful. There is some duplication of effort as browsers will typically cache content internally… but then again, if I’m checking the same page in Chrome & Firefox, they’re not sharing their internal caches.

That, and I’d really been hankering for a good science talk. There’s nothing quite like science; even when it comes to something as wishy-washy as consciousness, it can still give you surprising answers.

And video!

The short version of the slides:

When you see, you see what’s already been processed and filtered. Illusions are when the system doesn’t work; you don’t see when it does work. In other words, we see “car accident” as presented to our consciousness – we don’t consciously put it together from our visual input. I’ll spare you the customary link to the “You wouldn’t know if a Gorilla showed up” study, but it’s fairly clear the brain only passes on the Cliff Notes version to the executive layer.

Consciousness lags well behind. When scientists measure the movement of a finger, the electrical potential rises a full second before the finger moves. But we report making the decision to move half a second before the finger moves (Libet). We become aware of making the decision after it’s already happened.

Some scientists conjecture that consciousness may simply be unnecessary (Rosenthal). Others think that consciousness may be a result of conflicting subconscious systems (Morsella, Hofstader, Metzinger), and the Watts commentary points out that consciousness seems to be strongly associated with inner conflict and/or pain, although I’m not spoiling his punchline.

Despite what Heph says, I don’t think the talk is depressing. When you think about consciousness, you assume that it’s a good thing, but realistically we’re far happier and productive in flow, without that nagging voice inside our heads. Rather than life being suffering, suffering itself is the act of consciousness.

The talk itself went down well, with the coveted seal of approval. The 5MoF itself was surprisingly wide ranging – Eclair Bandersnatch showed up in a barbie mask and wig to talk about art, Danny O’Brien gave a talk on The Cosmopolitan Anarchist and recapped the news on Byron Sonne, Liz Henry read poetry from her new book, and Josh Juran presented FORGE, a GUI based on manipulating what appeared to be symbolic files on the filesystem – a programming paradigm that apparently came from Plan 9 and hurts my brain every time I think about it.

We’re doing the same thing next month, and I’ll probably be talking about Transcranial Direct Current Stimulation (if I’m not, y’know, drooling in a corner). So! If you have a thought that’s been burning a hole in some mental sidepocket, you should sign up.

UPDATE: Class went well. I’ve put a website up at http://webapp-hardening.heroku.com/ and will be updating it as I go.

This is a hard book to get hold of, but a worthwhile one. Systemantics doesn’t make a lot of sense without the context of Systems Theory, which is responsible for the word “cybernetics” and a whole bunch else, mostly talking about systems in the context of the complex feedback loop of a nuclear power plant.

Systemantics is a little bit different: it talks about the feedback loop involved in organizations, and how the system has an independent life (and will to live) outside of any of its participants. It’s a book about how systems actually behave, and how what an observer may consider to be a bug looks like appropriate behavior to the system. It’s about the system as you know it at 2 am, the system complete unto itself in all its ineffable complexity.

That being said, much of it is applicable to complex computer systems as well – in fact I’d say that Systems Theory is far more applicable to my day job than most CS Theory is, and if there’s ever going to be a Software Engineering curriculum then I’d want it to include this book.

There are a few famous interview puzzles out there. Microsoft has “Why are Manhole Covers Round?” Google has “You are shrunk to the height of a nickel and your mass is proportionally reduced so as to maintain your original density. You are then thrown into an empty glass blender. The blades will start moving in 60 seconds. What do you do?”

The standard answer to the first puzzle is “So they don’t fall in.” The answer to the second is that assuming you have the same proportionate strength, you can jump out. (There’s an inverse square root effect between muscle and body length, so density is not a factor.)

But here’s the thing. The first answer is incorrect. Manhole covers are round because they can be round. They could be square or triangular just as easily.

So as an interviewer, if you pick random interview puzzles out of a book and you think you know the “right answer” to the puzzle, you run the chance of not hiring someone because the answer he gave was actually the correct one.

A more common problem is that a clever interview puzzle is usually well-known. As soon as you figure it out, you’ll tell all your friends, and they tell their friends. Eventually, you can google for the answer.

Back in the day, Zen Schools had the same problem. They were looking for insight and flashes of realization, initially, and wanted a way to test for this. Some people thought up excellent questions that could test the subtle understanding of self, reality and perception required of Zen students. More and more, the koans were used as the standard by which students’ understanding could be measured. Eventually, someone had the bright idea to put together a book of koans – complete with “acceptable responses” – and through years of formalization, the age old question “What is the sound of one hand clapping” turned into a meaningless ritual.

Even if you don’t use well known puzzles, interviewing with logic puzzles in the long run optimizes for them. Through discussion, shared experiences and research, people will generally know that they should study for a general class of logic puzzle. And the company will start getting more people that will do well at those puzzles… but that doesn’t mean they know programming any better.

Fermi questions, those ”How many elevators are in New York City?” questions that are popular in interviews, have a clearer structural weakness. They have no right answer at all, and can be completely circumvented with the right training. Once you know the rules, you can come up with completely the wrong answer and still be “correct” according to the law of the game.

I also have a philosophical problem with Fermi questions. Yes, they’re pointless, but it’s not just that they’re pointless. Asking a Fermi question says that you don’t really care what the answer is. Asking a Fermi question tells your candidate that you want them to guess.

Engineers are trained out of guessing. Engineers are trained to nail down as much as they can before solving any problem, because invalid assumptions and requirements are dangerous and expensive. But that’s not why engineers don’t like to guess.

Why is because most engineers remember vividly what happened the last time someone came up to them and said “When do you think we can go live? Just make a guess.” When it comes to Fermi questions, a cagey and hesitant engineer isn’t a bad candidate, but an experienced one.

So what do I recommend? This book is good to get a broader sense of what interviews are supposed to do, and you can ask spot questions about language and system knowledge, then rate them on the Programmer Competency Matrix. But the best way to figure out how someone attacks code is to bring out some buggy code and ask for help debugging it, or bring out a design and talk about how you’d implement it, talk about the engineer’s background, print out the engineer’s github project and ask about it. They’ll appreciate it, and so will you.

So, I cheat. Here’s how I set it up with Vagrant and VirtualBox so I can carry several different development environments around. Warning: severe nerdery and technical detail below.

ATG has some very specific requirements. If you’re developing on a different platform, you can be bitten by unexpected bugs when you test in a different environment. However, if you’re running an upgrade, you need to a) have fresh environments configured exactly to spec. You have a bit of wiggle room on the OS. You have very little wiggle room when it comes to the JDK and JBoss versions: the EA version of JBoss will NOT serve as an acceptable substitute, and ATG will stress out the JDK in some very specific ways that are painful to debug given the wrong patch version.

Here’s what you can do with VMs:

• Package up your project-specific VMs as boxes (basically templates), then distribute them to your team for an immediately runnable system.
• Create automated boxes from your VMs as “snapshots” of the system at various points in time.
• Create disposible, run-once instances of your project to try out destructive changes and upgrades to the system.
• Create a snapshot of the system in a broken or buggy state and it make available for debugging.
• Move a box over to a big iron server with lots of memory / CPUs when you want to run batch processes.

I use VirtualBox as my VM platform, but you can get the same effect by using VMWare or Parallels. I prefer VirtualBox as it’s free and (with the addition of Vagrant) easy to configure in scripts and on the command line, and works the same in Windows, MacOS and Linux.

Here’s how you do it:

• Install VirtualBox.
• Install Vagrant.
• Review your target ATG platform.
• Download (preferably) or build (not so preferably) your vagrant box.
• Create an instance from your box that you’ll add more stuff onto.
• Package up your instance as another ATG box.

The first step is to make sure you have VirtualBox installed and you feel comfy with it. Read through this.

So notice how you had to install the OS by hand, starting with the ISO and walk through everything right from the very beginning, ending with a system that has a bunch of stuff that you don’t care about. Vagrant short-circuits that by providing you with “boxes” that have already been through the OS install and let you hit the ground running. This will teach you about Vagrant. You will also want the vagrant docs.

So the next step is to install Vagrant. Once you’ve installed Ruby (I use RVM, you may want something else), this is simple:

1
2
3
4

$ gem install vagrant
$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init
$ vagrant up

The important bit to note here is the second command: we told Vagrant to add a box called “base” to its library, from the URL http://files.vagrantup.com/lucid32.box. When we typed “vagrant init” then that created a Vagrantfile with some default options, and then “vagrant up” started up the VM. Vagrant has far more features than that, of course, but it’s a good place to start.

The next step is to make sure you have the files you need for your environment, as close as possible.

The dependency matrix for ATG (not counting the various patches, hotfixes, etc.):

ATG 2007

• JBoss EAP 4.0.5,
• RHEL 4.0update2 (32 bit),
• Oracle 10g,
• Sun JDK 5.0u11

ATG 9.3

• JBoss EAP 4.3.0 CP09,
• RHEL 5.1 (32 or 64 bit),
• Oracle 10g/11g,
• Sun JDK 1.6_22,
• Oracle Thin 11.2.0.x XA (for all versions).

ATG 10.2

• JBoss 5.1.0 EAP,
• RHEL 5.4 (64 bit),
• Oracle 10.2.0.2 / 11.2.0.1.0,
• Oracle Thin 11.2.0.1 XA (for all versions),
• Sun JDK 1.6.0_22

If you are doing this for development or want to try it out, you may find it easier to download a CentOS box from http://www.vagrantbox.es/search/?q=centos

If you’re hardcore, then you’ll be downloading RHEL 4 or RHEL 5, which aren’t publically available. If so, you’re going to want to know the details of building a box with Vagrant.

Building a base box

Your first Vagrant box is always going to be a bit wonky. Get comfortable and knock out a few before you think about putting together a base box you’re going to settle down with.

Once you have your box, then you’ll want to do the following:

1
2
3

mkdir my_instance; cd my_instance
vagrant init
vi Vagrantbox

• Set up a vagrant instance with memory / CPU (see the vagrant docs for this.
• Provision your vagrant box with basic packages and port forwarding.

Because you’re using CentOS, you’ll need some extra settings.

Note that you may run into a problem with CentOS being slow but this will fix it:

1
2
3
4

config.ssh.max_tries = 50
 config.ssh.timeout   = 300
vagrant up
vagrant ssh

Now you have a vagrant instance up and you can connect to it. Start installing packages. NOTE: If you’re doing this all from scratch, you may want to look at this.

Installing ATG 10:

• Create ATG user.
• Install Sun JDK.
• Install the version of JBoss.
• Copy all your ATG files into the VM, and then install them.
• Install the licenses into $DYNAMO_HOME/localconfig
• Install any patches and hotfixes for JBoss and ATG.

Packaging your base ATG Commerce box

Once you’ve got a base ATG install, you can package your instance up as another box:

1
2
3
4

mkdir atg10_box; cd atg10_box;
vagrant package --base atg10
# vagrant box remove atg10
vagrant box add atg10 package.box

You now have a correct, well known ATG template that you can take with you and install on any project.

Installing Oracle

In addition, you’ll also need Oracle installed. Installing Oracle is enough of a pain that I’ve seen bumper stickers about it. You will want to do this on a different VM than your ATG install, just for ease of use, not least because Oracle has its own specific memory and disk space requirements and it’s easier not to have to manage them. oraToolkit is a godsend here, and I can only imagine the work it took to make this all pan out.

Do not use Express Edition. It only takes 2GB databases. If someone hands you a 20GB dump of their database, you’re out of luck.

You may want to increase maximum number of sessions, or add more memory to the VM to the Oracle instance, but for right now we won’t worry about it.

Setting up and running new project

• Create a new vagrant instance using your ATG base box
• Create a new vagrant instance using your Oracle instance
• Set up all the ATG tables in Oracle
• Set up your project to have deployment sftp the EAR file to the ATG box
• Set up your project to have an rsync daemon send JSP files to the ATG box

Notes

Use Vagrant for making snapshots. Do not use VirtualBox’s snapshot facility unless you really, really know what you’re doing.

If you cannot connect to a shared folder, make sure that your user is in the vboxsf group.

I am aware that a) this is a mess and b) it will probably be rendered out of date shortly. Email me with improvements and new gists.

If you don’t want to go through the slides, here’s the TL;DR version:

TL;DR User Security

• Use a password manager like LastPass or 1Password (with Dropbox) and use their password generation.
• If no manager available (routers, OS logins, etc), use pass phrases with non-English words or acronyms (see xkcd)
• Assume sites get compromised all the time and you never hear about it. NEVER reuse a password.
• If you’re at a coffee shop or hackerspace, use a public VPN service.
• OAuth / Twitter / Facebook based authentication is putting your auth credentials in their hands.

TL;DR Encryption Security

• Use bcrypt. Bounce up the factor every few years.
• Do not limit password field length. (bcrypt takes up to 55 bytes of input.)
• Run a JS password tester to reject weak passwords.
• Run a password cracker regularly to test your security.
• Suggest to your users that they use passphrases with acronyms, punctuation or LOLspeak.
• Generate random passwords for your users.
• Consider removing password masking.

TL;DR Operational Security

• Use HTTPS for both rendering and submitting login page.
• Show Cain and Abel video to everyone you work with.
• Use HSTS headers with HTTPS.
• Use Synchronizer Token to prevent CSRF attacks (or use a decent web framework).
• Use a captcha / throttle on password attempts.
• Use double validation for registering accounts (register sends email, clicking email link heads back to site).
• Use one time use password reset links.
• Send email notifications on password change attempts.

Extra Credit

• Add Honeypot Logins.
• Use login token IDs with hidden check bits and math invariants that indicate tampering.
• Implement a secret in the session management system to keep state on the client and verify it on server interaction for better session authentication.

OWASP also has cheat sheets which look useful if you’re putting a site together. It still disturbs me how freaking MANUAL so much of this is, but I suppose web frameworks can’t do everything for you. There are some options if you’re on Rails.

It was a surprisingly tough talk to give. At first I was like, ”lol, look at all the companies with crappy security”, but it’s a murky field in general. For example, the XKCD cartoon about passphrases is missing the problem that most people type passphrases in standard English, and only use about two thousand words in general conversation. It may look like there’s more entropy generated, but if your attackers know that your customers use passphrases, you may have just made their jobs much easier.

Also, brute force cracking is surprisingly effective. MD5 and the SHA-* algorithms are inappropriate because GPUs chew through them very quickly, but the newer FPGA chips can do a reasonable implementation of bcrypt in hardware. It’s an issue that computers are fast, but a bigger problem is that they just keep getting faster.

The biggest thing has to be to not let your users pick crappy passwords. Even if you have bcrypt with all the factors, if your users are entering “12345” as the password, it’s not going to make a difference.

Exporting from Typo to Disqus turned out to be a much bigger deal than I anticipated: the initial Typo -> Disqus export didn’t work because the Disqus gem would drop comments on the floor, and the various WXP export scripts I found on the web either used XML::Builder (which didn’t format / escape things correctly) or just plain had bugs in them.

On top of that, WXR is actually not valid XML itself (there’s no default namespace defined) so even if you do it correctly, it won’t validate. You just have to check it matches the import format exactly. So here’s my version.

Although there were a number of people afterwards who were like “too unrealistic” and I was like “yeah, this works better for interviews and college placement but whatchagonnado.”

I really need to write this up as an essay, as I think presentation objects vs domain objects are a much bigger detail than we realize.

Part of what makes this so fascinating to me is that you can actually see the algorithm that tells us “hey, we should do more of this.” That’s a huge step in knowing our blind spots.

After that (and I really wish I’d videoed everyone’s faces), it was onto the presentation itself, which is about how the Bronies beat down 4chan.

I was told my talks were too technical and depressing. WHO’S LAUGHING NOW, HUH. Oh right, EVERYONE.

It has kittens.

Updated: Now with slides!

The first thing I've been thinking about is the people who say "You're doing Agile Wrong."

There's always been a dichotomy for me between the theory of Agile, and the practice.  It's a common problem with any dream; it's always cleaner, brighter, simpler, better than the reality.  Reality is messy.  It is imprecise.  It is never seen directly, always filtered through recollections to make each participant the protagonist of their own play.  

If you try pair programming, then you're going to find that "you should never pair program 100% of the time."  Or that "you should only pair program between people with equal skill sets."  Or that "you should practice pair programming ping pong".  There will always be a special case. There will always be something that works for you that doesn't work for someone else.   There will always be something that doesn't work for you that works for someone else.

Not only do we do Agile "wrong", but we will always do Agile "wrong".  We won't ever do anything "right" – we will do imperfect jobs, come home to imperfect relationships, have imperfect children and live imperfect lives.   This is what happens when you measure yourself against an ideal.

But why believe in Agile then, if the only way you can do it is wrong?  Something that came to mind about that statement.  

If you're trying to do Agile, and it's not working for you… then you're doing it wrong.

Another way of phrasing that statement is that Agile is Doing It Right.  

In fact, almost by definition, Agile is Doing It Right.

"Agile teams produce a continuous stream of value, at a sustainable pace, while adapting to the changing needs of business." – Elizabeth Hendrickson.

"Agile development uses feedback to make constant adjustments in a highly collaborative environment." – Practices of an Agile Developer.

"Agile has no definition. […] There's no standards board, there's no test, there's no approved workbook, there's no checklist.  […] It's based on three things: 1) principles not practices, 2) attention to people, and 3) always be adapting." – Daniel Markham.

Three definitions of Agile.  Nothing about practices, or even methodology.  What they agree on is a feedback cycle that can respond to changing input and produce useful output.  

It's Dorner's model of problem solving.  Or Deming's PDCA cycle.  Or the Military's OODA cycle.   Or the Scientific Method.  Or Kaizen.  It's continous process improvement, in all its forms.

If you're following a "best practice" and that "best practice" isn't working for you, then it's not a case of "You're Doing Agile Wrong."  You're doing something that isn't providing a benefit for you.   By following that "best practice", you're not doing Agile at all. Agile is the ability to plan something new, throw out something old, and challenge preconceived beliefs.

That's the core of Agile for me: it's not about Wrong or Right.  It's the idea of saying "We can do better."  And then doing it.

The book’s been a favorite of mine for years, not just for the set up, but for the detailed, unsparing look it provides on how human beings fail to get things right.  Too often in psychology, there’s an emphasis on either seeing how people feel about a situation, or how well or how poorly they perform at a given task.  Dorner goes further, and tries to understand not just how, but why they fail.

The Slides

The Setup

The setup was simple.  Dorner set up a computer simulation of an African village called Tanaland.  This book was written in 1990, and so Sim City was not widely known, but it’s the same concept.  The players were given dictatorial powers, given the goal to “improve the wellbeing of the people” and had six opportunities over 10 years to review (and possibly change) their policies.

The Experiment

Given the tools the players had at hand, they went to improving what they could.  They improved the food supply (using artifical fertilizer) and increased medical care.  There were more children and fewer deaths, and lif expectancy was higher.  For the first three sessions, everything went well.  But unknown to the players, they’d set up an unsustainable situation.

Famine typically broke out in the 88th month.  The agarian population dropped dramatically, below what they had been initially.  Sheep, goats and cows died off in their herds, and the land was left barren by the end.  Given a free hand, most players engineered a wasteland.

One player, by the end of the simulation, had a stable population and had significantly better quality of life for the villagers.  Failure was the rule, but somehow he had found an exception.

The Breakdown

The litany of possible errors was a long one, and so immediately recognisable that it's hard to suppress a wince of empathy on reading.

The players who did badly tended not to ask "why" things happened.  They tended to jump from one subject to another, switching aimlessly, without focus.  They proposed hypotheses without testing them.  If they did test their hypotheses, they did so on an adhoc basis, testing success cases without testing possible failure cases.  In some cases they had tunnel vision: focussing on irrelevancies at the expence of the larger picture.  In other cases, they attempted to "delegate" intractable problems to the villagers themselves or refused to deal with the issue at all.  Finally, and most tellingly, most players dealt with the problems that they saw "on the spot" without thinking of the larger, longer term problems that they were setting up with that immediate short term solution.

These results were not a surprise.  They were just what Dorner's team was looking for.  Where many scientists would have looked at the successes and determined the optimal "working strategy" – Dorner was just as interested in the range of failures in the experiment.   Dorner's team had specifically designed the simulation so that most people would fail at it, precisely aiming at the weak points of human decision making.  

Not all players failed in the same way.  Even amongst the players who failed the same way, many players had different reasons for their particular mode of failure.  And yet, there were strong commonalities among the failing players, both in their reactions to incipient failure, and in their attempts at recovery.

The Reasons

The reason why most people failed was that they did not understand the nature of Tanaland.  Despite being a simulation, Tanaland was no game, and Dorner's team programmed in as accurate a simulation of an African village as the hardware would allow.  The watertable under the village had a limited amount of water available.  The population grew at an exponential rate given the available food and healthcare.  Even the topsoil was modelled accurately, so that overgrazing caused by massive herds would erode the topsoil over time.  All of this data was available to the players – had they thought to look.  But most players didn't.  The experiment ended in three predictable failure modes: either the cattle starved and died, or the groundwater was exhausted, or the population exceeded the available food.  Far from being a bundle of independent subsystems, all of Tanaland was deeply intertwingled.

The Weaknesses

The deeper reason why Tanaland was so successful at bamboozling players is partly due to the incredible success of the human brain's pattern recognition system.  Human beings are capable of driving in heavy traffic, understanding language, and recognizing patterns in almost random data, feats far beyond most computers.  But there are some problems which defeat human intuition.  

Linear extrapolation. Human beings have a tendency to assume change itself is static.  Even when shown exponential growth, we're not good at internalizing that knowledge.  This may be why calculus is so hard for many people, because we don't think about the rate of growth itself growing.

Delayed Feedback. Human beings tend to assume that an action will yield a response immediately, or not at all.  This is the way that we interact with the world on a daily basis, and we can become very confused when there's a significant delay in the system's response.  Even when we recognize intellectually that a change is "in the pipeline", we may struggle against the instinct to do more and oversteer rather than correctly "sitting on our hands."

Contradicting goals. Part of the failure of players was inherent in the vague goals that they had.  What, exactly does "improve the wellbeing of the people" really mean?  Does it mean providing the best quality of life to all the villagers?  Growing the village as a whole to be more prosperous?  In many cases, the top level goal ended up being broken down into goals that conflicted with each other.  In other cases, players tried to find concrete problems to fix.  One player, deciding that the village needed irrigation, set out building an irrigation system and quickly became fixated on that one problem, becoming "addicted" to his experience of flow.  In such cases,  players were unable to clearly form goals at all.

Priorities. Even when the players had clearly defined goals, they had another problem to contend with: they would be stymied by cases where actions which furthered one goal would thwart another.  The complex interdependencies in the system did not allow for a full optimization of every variable, and players would either flail uselessly or be paralysed by their inability to cover every base.

Information overload. In many cases, the players used too little information to know how to make the best decisions.  However, some players had the opposite problem; given access to all data of a complex system, they tried to see the entire system at once.  These players found themselves paralysed by complexity, and unable to interpret the results of the data.  Interestingly, the problem the players had was not that they did not see the correct chart.  They were literally unable to recognize the charts and the changes in data as relevant – having looked at all the data available, their abilities to see a pattern was exhausted well before they stumbled on the correct chart.

Reductive Hypotheses. By far the worst problem that players had, above all others, was that the first hypotheses they formed about the system were not changed in response to the data.  If anything, the players were apt to be the most sure in their beliefs when the hypotheses were completely wrong.  Part of this came from uncertainty and cognitive dissonance.  Uncertainty produced fear and doubt.  Asserting the hypothesis helped quell this fear and doubt.  Over time, the players learned that the more they believed in the hypothesis, the better they felt.  This reduced their ability to develop new hypotheses, as they were already "wedded" to their existing ideas.

The Successes

There were also commonalities amongst the players who did well.  The players who did well were the ones who could tolerate uncertainty.  They defined clear goals and priorities.  They made many small decisions in different areas, and followed up on the expected vs actual results of most, if not all of those decisions.  They kept an eye on the overall processes of the system, and did not succumb to flow experiences.

Adding to this, Dorner's team ran an experiment with two groups of fifteen players each.  One group was drawn from the student population.  The other group was made of senior managers from large industrial and commercial firms.  The managers did significantly better than the students on every possible metric; given several different challenges, they responded appropriately to each one.  Dorner's team was unable to determine if this was innate talent or the benefit of years of experience.

A Decision Making Model

So what is the right thing to do when faced with a complex situation? Dorner presents a possible schema for problem solving, intended more as a helpful aid than as a representation of how people actually solve problems.   

1. Formulation of Goals - deciding what it is that needed fixing, and putting priorities on those goals.
2. Formulation of Models - determining the internal workings of the system.
3. Prediction and Extrapolation - determining the eventual output of the system.
4. Planning of Actions; decision making, and execution of actions - feeding input into the system.
5. Review of effects of actions and revision of strategy - determining the expected model vs the actual model.

Interestingly, this is very close to the Plan/Do/Check/Act cycle proposed by Deming – it assumes incomplete knowledge of a complex system and tries to improve understanding of the underlying model through repeated iterations of the cycle.

Dorner also notes that being simply told of a decision making strategy did no good at all to players; when given instruction on dealing with complex systems, the players thought that they had been helped and were better able to discuss their failures with better terminology… but their actual performance was the same as the control group.  What really helped players, overall, was repeated exposure to complex systems.  Showing was not enough; they had to experience their own reactions and build up their tolerance to decision making in the face of uncertainty and emotional stress.