Sean Hess

Practical Haskell - Building a JSON API

2015-08-19T00:00:00+00:00

Practical Haskell - Building a JSON API

This is part of a tutorial series intended to introduce Haskell by coding things that work. In this article we will be building a simple JSON API using Scotty and Aeson.

Before you Begin

This article assumes you have a project set up with stack, that you know how to import a 3rd party module, how to write a simple IO program, and how to write a simple function.

We recommend you work through all 3 previous tutorials. In Getting Started we teach you how to build an run Haskell projects, in Importing Code we show you how to import other modules, and in Using Monads we teach you how to use do notation.

Installing Scotty

Scotty is a haskell library that lets you make JSON APIs. It’s a lot like Express or Sinatra. You define routes and it makes a web server that will respond to HTTP requests.

Use the info from Importing Code to install the scotty package, then run stack build to get stack to install it into your project. If this is the first time you’ve added a dependency, it might take a while, but next time it’ll be faster.

$ stack build
...
Completed all 59 actions. 

A simple web server

Let’s start with a simple IO program. Replace src/Main.hs with the following

module Main where

main = do
  putStrLn "Starting Server..."

Before we can use it, we need to import scotty. Add this line to the top:

module Main where

import Web.Scotty

main = do
  ...

Now, let’s use the scotty function in main to start a scotty server:

module Main where

import Web.Scotty

main = do
    putStrLn "Starting Server..."
    scotty 3000 $ do
        get "/hello" $ do
            text "hello world!"

Finally we need a GHC feature called OverloadedStrings that let’s use string literals for other things (like routes, or text).

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Web.Scotty

main = do
    putStrLn "Starting Server..."
    scotty 3000 $ do
        get "/hello" $ do
            text "hello world!"

Let’s run this just like last time. Fire up stack ghci, load our code, and run main

$ stack ghci
Prelude> :load Main
*Main> main
Starting Server...
Setting phasers to stun... (port 3000) (ctrl-c to quit)

We can test it using our web browser: navigate to http://localhost:3000/hello and you should see the result:

hello world!

What’s with the nested do blocks?

Remember from Using Monads that each do block can only contain one type of action. main always has IO actions, so both putStrLn and scotty must return an IO action.

scotty :: Port -> ScottyM () -> IO ()

But it looks like in order to return an IO action, we need to pass a ScottyM action as the second parameter. Our program is equivalent to this:

main = do
  putStrLn "Starting Server..."
  scotty 3000 someScottyMAction

That nested do block after the port number is just a ScottyM action. We could have done this instead:

routes :: ScottyM ()
routes = do
  get "/hello" $ do
    text "hello world!"

main = do
  putStrLn "Starting Server..."
  scotty 3000 routes

The same thing goes for the handlers. Look at the type of get:

get :: RoutePattern -> ActionM () -> ScottyM ()

Same thing again, we pass a ActionM () as the last parameter. We could have written routes like this:

routes :: ScottyM ()
routes = do
  get "/hello" hello

hello :: ActionM ()
hello = do
  text "hello world!"

More Features: Route Parameters

We can add route parameters with :xxx in the url, just like Sinatra and express. To read it in our handler, we use the param function. Edit the /hello route to be this:

get "/hello/:name" $ do
    name <- param "name"
    text ("hello " <> name <> "!")

To get this to work we also need to import <>, which concatenates things that aren’t strings. Add this to the top:

import Data.Monoid ((<>))

Let’s test! Go over to GHCI and type Ctrl-C to stop main. Then reload with :r and type main again. Check it out: http://localhost:3000/hello/bob. Try out a few different names for that last parameter

hello bob!

If you get stuck, be sure to check out the source

Define Types for your API

We can make data types that describe our API. Here we have a User object with two fields. Add these to src/Main.hs at the top level.

data User = User { userId :: Int, userName :: String } deriving (Show)

We can define some hard-coded users

bob :: User
bob = User { userId = 1, userName = "bob" }

jenny :: User
jenny = User { userId = 2, userName = "jenny" }

Or a list of users

allUsers :: [User]
allUsers = [bob, jenny]

Let’s see if it works! Reload in GHCI.

*Main> :r
*Main> bob
User { userId = 1, userName = "bob" }

Let’s do JSON with Aeson

The Aeson library lets you serialize data objects to JSON and vice versa. Let’s install it the same way we did with scotty. Add it to build-depends and run stack build again. Then import it in your code:

import Data.Aeson (FromJSON, ToJSON)

You can tell Aeson how to convert your objects to JSON manually, but it’s easier to use some fancy GHC features. One of them, called Generics, lets you automatically do things based on the data type. Add these to the top:

{-# LANGUAGE DeriveGeneric #-}
...
import GHC.Generics

Now we can have our data type “derive” Generic

data User = User { userId :: Int, userName :: String } deriving (Show, Generic)

Then add this below to make any User JSON serializable.

instance ToJSON User
instance FromJSON User

Now User can be encoded or decoded. Let’s see if it works. Reload in GHCI and test:

*Main> :r
*Main> import Data.Aeson (encode)
*Main Data.Aeson> encode bob
"{\"userName\":\"bob\",\"userId\":1}"

Return Users from our API

Let’s add a new route /users that will return a list of users. We’ll use scotty’s json function instead of text. It will automatically call encode for us.

get "/users" $ do
  json allUsers

Let’s test it in the browser: http://localhost:3000/users

[{"userName":"bob","userId":1},{"userName":"jenny","userId":2}]

Or we can return all users with a given id. First, let’s make a pure function that returns true if an id matches. Put this at the top level

matchesId :: Int -> User -> Bool
matchesId id user = userId user == id

Now add a new route that uses that function:

get "/users/:id" $ do
  id <- param "id"
  json (filter (matchesId id) allUsers)

Test it out: http://localhost:3000/users/1

[{"userName":"bob","userId":1}]

The complete program

Here’s what src/Main.hs should look like when you’re done. Check out the full source here

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}

module Main where

import Data.Monoid ((<>))
import Data.Aeson (FromJSON, ToJSON)
import GHC.Generics
import Web.Scotty

data User = User { userId :: Int, userName :: String } deriving (Show, Generic)
instance ToJSON User
instance FromJSON User

bob :: User
bob = User { userId = 1, userName = "bob" }

jenny :: User
jenny = User { userId = 2, userName = "jenny" }

allUsers :: [User]
allUsers = [bob, jenny]

matchesId :: Int -> User -> Bool
matchesId id user = userId user == id

main = do
  putStrLn "Starting Server..."
  scotty 3000 $ do
    get "/hello/:name" $ do
        name <- param "name"
        text ("hello " <> name <> "!")

    get "/users" $ do
      json allUsers

    get "/users/:id" $ do
      id <- param "id"
      json (filter (matchesId id) allUsers)

Playing around

Check out the Scotty docs to see what else you can do. Remember, anything that returns ScottyM a can be used in the routes do block, and anything that returns an ActionM a can be used in the handlers.

Assignment

Create a route that accepts a User via POST, and prints it back out.

Answers

What’s next

Feel free to suggest another tutorial in the comments!

Practical Haskell - Using Monads

2015-08-18T00:00:00+00:00

Practical Haskell - Using Monads

This is part of a tutorial series intended to introduce Haskell by coding things that work. In this article we will teach you how to use Monads without panicking.

Before you Begin

This article assumes a basic understanding of Haskell Types. You should know how to add a type declaration to a simple function, and understand the difference between String, Maybe String, and [String]. Consider reading Chapter 8 of Learn You a Haskell: Making Our Own Types and Typeclasses. You’ll probably need to have read Chapters 2 and 3 first.

Please work through Getting Started so you know how to run code.

Don’t Panic

Monads are not scary. You’ve been using them since the beginning of this tutorial. Whenever we make an IO function, and use do, we’re using monads. Let’s take a look at a simple example:

main = do
  name <- getLine
  putStrLn ("Hello, " ++ name)

You probably have a pretty good feel for what this is doing. Trust that feeling! There will be plenty of time to get the theory down. Today we’re just going to learn how to use them practically.

Functions that return Actions

putStrLn is a function that prints out a message to the console, right? Not quite. Let’s look at its type. What does it return?

putStrLn :: String -> IO ()

It’s a function that accepts a string, and returns an action to print something out to the console. That’s an important distinction. putStrLn doesn’t actually print anything, it returns an instruction to print something. Let’s call those instructions actions.

What good are actions?

What can we do with actions? They are values, like String or Bool, just fancier. Think of them as Todo list items. We can mix them together, and combine them into larger actions. When you use do syntax, that’s exactly what you’re doing:

main :: IO ()
main = do
  putStrLn "Go shopping"
  putStrLn "Take a nap"
  putStrLn "Learn Haskell"

Look at the type: IO () means that main is one IO action. That do block is combining 3 actions into one.

Since the whole point of do is to combine actions, you can leave it out if there’s only one. These are both equivalent:

main = do
  putStrLn "Hello"

main = putStrLn "Hello"

Calling the function doesn’t perform the action

What will it print out if I do this?

main = do
  let action = putStrLn "Hello World"
  return ()

Calling putStrLn does not print anything, it returns a value of type IO () which can be performed later. In the above, we are never executing it.

Anything at the same indentation level (except let) in a do-block is expected to be an action of the same type, and will be executed when the parent is.

-- what will this do?
main = do
  let action = putStrLn "Hello World"
  action
  action
  action
  return ()

We can see this from the type. When we call putStrLn with one argument, it returns IO (), or an IO action, all ready to be used.

putStrLn :: String -> IO ()

action :: IO ()
action = putStrLn "Hello World"

Actions can have results

Actions can yield something when they are performed. Let’s look at the type of getLine. What does it return?

getLine :: IO String

It doesn’t return a String, it returns an action, right? Then what does that String mean? It means that when the action is performed, it will yield a String. Let’s call that the result of the action.

You can use result from an action with the <- operator.

main = do
  name <- getLine
  putStrLn ("Hello " ++ name)

name is no longer an action. Its type is just String, not IO String like getLine was. It’s the difference between a plan to go to the grocery store (getLine) and the groceries you picked by actually going (name)

A do-block always yields whatever its last action does. If we put return last, we can make our do-block yield any simple value.

sayHello :: IO String
sayHello = do
  name <- getLine
  putStrLn ("Hello " ++ name)
  return name

Different types of Actions

Within a given do-block, all actions must be the same type. So for an IO block like the one in main, every line has to have the type IO a. You can’t put regular values there (without let), because they aren’t actions.

main :: IO ()
main = do
  -- good. putStrLn "Hello" is of type IO ()
  putStrLn "Hello"

  -- error! String is not an IO a!
  "whatever"

Remember that functions like putStrLn aren’t IO actions, they return IO actions. So this won’t work:

main = do

  -- error! putStrLn is a function with type String -> IO ()
  putStrLn

  -- good. If we give it one argument, it's now an IO action
  putStrLn "Hello"

Let’s look at some examples of different kinds of monads, each with their own actions. To use the Maybe monad, each step must be of type Maybe a. You can’t mix different kinds of actions in the same do block.

doesNotWork :: Maybe Int
doesNotWork = do

  -- these work, because they are all Maybe a
  Just 6
  Nothing

  -- error! this is type IO (), not Maybe a!
  putStrLn "Hello"

  return 5

main :: IO ()
main = do
  -- error! this is type Maybe Int, not IO a
  Just 6

Monads decide how to combine actions

Each type of Monad combines actions differently. We already looked at IO. It performs the actions in order with IO side effects. Maybe, on the other hand, specifies that if any step is Nothing, the whole thing stops and exits with a Nothing.

beCareful :: Maybe Int
beCareful = do
  Just 6
  Nothing
  return 5

The do-block here always yields Nothing, because of that Nothing in the second to last step.

Time to code

Add beCareful and sayHello to src/Main.hs. Then head over to GHCI and try them out.

$ stack ghci
Prelude> :load Main

*Main> beCareful
Nothing

*Main> sayHello
woot
Hello woot
"woot"

Now try mixing them up. Add an IO action into beCareful.

beCareful :: Maybe Int
beCareful = do
  Just 6
  putStrLn "oops"
  Nothing
  return 5

What happens when we reload GHCI?

*Main> :r
[1 of 1] Compiling Main             ( src/Main.hs, interpreted )

src/Main.hs:11:3:
    Couldn't match expected type ‘Maybe a0’ with actual type ‘IO ()’
    In a stmt of a 'do' block: putStrLn "oops"
    In the expression:
      do { Just 6;
          putStrLn "oops";
          Nothing;
          return 5 }
Failed, modules loaded: none.

Hopefully that error message makes sense to you now.

Other monads in the wild

The Parsec libary uses Parser actions to define a grammer for parsing. Here’s one to parse an IPv4 address. It combines actions by matching them against an input. If the input doesn’t match any of the steps, it exits and tells its parent it didn’t match.

data IP = IP Word8 Word8 Word8 Word8 deriving Show

parseIP :: Parser IP
parseIP = do
  d1 <- decimal
  char '.'
  d2 <- decimal
  char '.'
  d3 <- decimal
  char '.'
  d4 <- decimal
  return $ IP d1 d2 d3 d4

Can you guess what the type of decimal and char are?

char :: Char -> Parser Char
decimal :: Parser Word8

The Scotty library uses the ScottyM monad to define a web API. Here each action is combined to create an API description which is later matched against incoming HTTP requests. The first ones are given priority when matching.

routes :: ScottyM ()
routes = do
  get "/"      (text "homepage!")
  get "/hello" (text "hello")

Don’t sweat the little stuff

So there you go. Don’t worry about it. You can use do notation, monads, and write programs. You don’t have to understand exactly what monads are doing under the hood to have an intuition for how to use them. Later when you learn the theory, it’ll be much easier if you’ve been using them.

Assignment

Build a program that asks the user for a message and a number on the command line, and print out that message N times. Use the control functions from Control.Monad, like mapM, or replicateM

Read Input and Output from Learn You a Haskell. Implement the unix cat command, which takes a command-line argument as a path to a file and prints out its contents. Refer to Getting Started to see how to use stack exec.

Answers

What’s next

In the next tutorial, Build a JSON API, we code a real webserver that responds to JSON requests over HTTP.

Practical Haskell - Importing Code

2015-08-17T00:00:00+00:00

Practical Haskell - Importing Code

This article is part of a tutorial series intended to introduce Haskell by coding things that work.

In this article we’re going to see how to include other code from Haskell’s core libraries and from the web.

Before You Begin

In Getting Started we showed you how to get everything installed and run some code. Before getting started make sure you follow the directions to get stack installed and to set up a project

Importing other code

Haskell’s core library, called base, contains many useful functions. Many of the most common functions are in the Prelude, and are automatically imported in your code. putStrLn is one of these. A few others are +: the addition operator, and show: which converts something to a String.

printNumbers = do
  putStrLn (show (3+4)) 

Only the Prelude module is imported automatically. If you want to use any other module, you need to import it. Imports belong at the top of your program. Let’s import readFile from the module System.IO and write a program that prints out our stack config file

import System.IO (readFile)

printConfig = do
  contents <- readFile "stack.yaml"
  putStrLn contents

Go ahead and add both of the above functions to src/Main.hs, which you created in Getting Started. Load them into GHCI and let’s try it out.

$ stack ghci
Configuring GHCi with the following packages: my-project
GHCi, version 7.10.2: http://www.haskell.org/ghc/  :? for help

Prelude> :load Main
[1 of 1] Compiling Main             ( src/Main.hs, interpreted )
Ok, modules loaded: Main.

*Main> printNumbers
7

*Main> printConfig
flags: {}
packages:
- '.'
extra-deps: []
resolver: lts-3.1

You can add them to your main program if you like. Edit src/Main.hs and add them to the main function.

main = do
  putStrLn (greet "bobby")
  putStrLn (greet "World")
  printNumbers
  printConfig

If you get stuck, check out the full source.

Finding 3rd party code

Base is pretty cool, but the magic really happens when you import code that doesn’t come standard. Haskell has a wealth of 3rd party modules. Try googling for something, like “Haskell Time”. The first hit points us to the time library.

On that page you can click to the homepage, or to the documentation for a particular module. Let’s click into Data.Time.Clock. There’s a function in there called getCurrentTime that gets the system clock time. Let’s try to use it.

Importing 3rd party code

Before we can use getCurrentTime we need to add the time package to our project. Open your .cabal file and add time to the build-depends field

name:                my-project
version:             0.1.0.0
...

executable my-project
  ...
  build-depends:       base,
                       time

Head over to your terminal and run stack build to get stack to install it into your project.

$ stack build
...

Now we can use getCurrentTime in a program. Add this to src/Main.hs

import Data.Time (getCurrentTime)

printTime = do
  time <- getCurrentTime
  putStrLn (show time)

Now restart ghci, reload it with main, and see what happens!

$ stack ghci
Configuring GHCi with the following packages: my-project
GHCi, version 7.10.2: http://www.haskell.org/ghc/  :? for help

Prelude> :load 
[1 of 1] Compiling Main             ( src/Main.hs, interpreted )
Ok, modules loaded: Main.

*Main> printTime
2015-08-17 18:41:55.068122 UTC

You can add this to your main function if you want, as before. In src/Main.hs

main = do
  putStrLn (greet "bobby")
  putStrLn (greet "World")
  printTime

Source Code

Stackage and Hackage

Hackage stores all versions of all haskell packages. But sometimes these are too bleeding edge. An author might publish a new version that doesn’t work with another dependency in your project. Historically it has been very annoying to deal with this.

Stack fixes this problem by fetching our code from Stackage, which hosts snapshots of common haskell packages known to work well together. Take a look at the resolver field of stack.yml. That lts-3.1 is one snapshot of haskell packages.

flags: {}
packages:
- '.'
extra-deps: []
resolver: lts-3.1

So everything in lts-3.1 is knows to work together. You can see all the available packages in lts-3.1 right here: https://www.stackage.org/lts-3.1

What if it’s not on stackage?

If you need the bleeding edge, or need to depend on a package not on stackage, you can add them to your extra-deps field in stack.yaml, and tell it exactly which version to use. You still need to add them to your .cabal file too.

For example, I use RethinkDB for a database. The driver I use is here on hackage, but it isn’t in LTS Haskell 3.1. If I want to use it, I add it like this:

Stack.yaml

flags: {}
packages:
- '.'
extra-deps:
- rethinkdb-2.0.0.0
resolver: lts-3.1

my-project.cabal

name:                my-project
version:             0.1.0.0
...

executable my-project
  ...
  build-depends:       base,
                       rethinkdb

Now anyone who clones my repository can just run stack build and get exactly the same dependencies as me.

Assignment

Find a 3rd party module for JSON from LTS Haskell or Google. Click through to the “Module documentation” to find a function to encode JSON. Write a program to JSON serialize the following list and print it out. Use print instead of putStrLn.

numbers :: [Int]
numbers = [1,2,3,4]

Answer

What’s next

In the next tutorial, Using Monads, we show how to use monads without panicking

Practical Haskell - Editor Setup with Stack

2015-08-05T00:00:00+00:00

Practical Haskell - Editor Setup with Stack

In Getting Started with Stack we showed you how to install GHC and stack, set up a project, and write a little code.

We can use GHCI without editor integration for a pretty good workflow, but configuring your text editor to underline errors can be very helpful. This article will show you how to get it all working with stack.

The bleeding edge

There are two main editor plugins: ghc-mod and hdevtools. ghc-mod is great, ~~but at the time of this writing they don’t support stack, so we can’t use them with the project setup we introduced in Getting Started.~~ But hdevtools works great!

Update: ghc-mod has been updated to work with stack. For atom users, I’d now recommend using ide-haskell, which should have instructions to install all dependencies. Then refer to this article: Using ghc-mod with stack. Basically you install ghc-mod locally for each project:

cd my-project
stack install ghc-mod

Installing GHC and Hdevtools globally

Normally, we want to use stack in a project folder with a stack.yaml file and a .cabal file so we can install specific dependencies to each project. But our text editor runs globally, so we need to install hdevtools globally instead.

First, head into a global folder that isn’t a haskell project and run stack setup. This will install ghc for you.

$ cd ~
$ stack setup
Run from outside a project, using implicit global config
Using resolver: lts-3.1 from global config file: /Users/seanhess/.stack/global/stack.yaml
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec

Then install hdevtools

$ stack install hdevtools
Run from outside a project, using implicit global config
Using resolver: lts-3.1 from global config file: /Users/seanhess/.stack/global/stack.yaml
NOTE: the install command is functionally equivalent to 'build --copy-bins'
syb-0.5.1: download
...
Completed all 5 actions.
Copying from /Users/seanhess/.stack/snapshots/x86_64-osx/lts-3.1/7.10.2/bin/hdevtools to /Users/seanhess/.local/bin/hdevtools

Copied executables to /Users/seanhess/.local/bin/:
- hdevtools

Adding it to your PATH

We need both hdevtools and ghc to be available on our PATH environment variable for it to work with our text editor. We can use stack path to have stack tell us where it puts binaries:

$ cd ~
$ stack path
Run from outside a project, using implicit global config
Using resolver: lts-3.1 from global config file: /Users/seanhess/.stack/global/stack.yaml
global-stack-root: /Users/seanhess/.stack
...
ghc-paths: /Users/seanhess/.stack/programs/x86_64-osx
local-bin-path: /Users/seanhess/.local/bin
...

ghc is located in a folder under GHC-PATHS/ghc-VERSION/bin, like this on my system:

~/.stack/programs/x86_64-osx/ghc-7.10.2/bin/ghc

hdevtools is located in the local-bin-path folder:

~/.local/bin

Let’s add both to our PATH. On OSX, open ~/.bash_profile and add something like this:

export PATH=~/.stack/programs/x86_64-osx/ghc-7.10.2/bin/ghc:~/.local/bin:$PATH

Testing to see if it works

Let’s open new terminal window to get our PATH to reload and type the following:

$ hdevtools --version
hdevtools: version 0.1.2.1 (ghc-7.10.2-x86_64-darwin, cabal-1.22.4.0)

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.2

Now let’s make sure hdevtools works in our project. Go to a haskell project and use the hdevtools check command. This is what your editor will be doing behind the scenes.

$ cd ~/my-haskell-project
$ hdevtools check src/Main.hs

Getting a Text Editor Plugin

Now that hdevtools is working on the command-line we can install an editor plugin. The hdevtools homepage has a list of available editor plugins, including Vim, Emacs, and Atom. Follow the instructions to get set up!

You should be able to get errors highlighting on save, and see what the type is for different parts of code. Here’s what it looks like in my editor as I edit code and hit save.

Limitations

Hdevtools doesn’t provide any autocomplete information.

~~You can’t use Atom’s Haskell-IDE until ghc-mod gets up to speed. The error highlighting provided by the other plugins are the most useful feature though.~~

GHC-mod now supports stack. Refer to the instructions at the beginning of the article for atom users.

Assignment

Play around with the various features your editor provides on the project you created in Getting Started

Next up, let’s learn how to import code

Practical Haskell - Getting Started with Stack

2015-08-04T00:00:00+00:00

Practical Haskell - Getting Started with Stack

Haskell is famous for having a steep learning curve. As a web developer we’re used to clear tutorials that we can understand and complete within an hour or two. Haskell introduces many new concepts not found in other languages, but we can learn it faster by spending as much time coding as we do reading.

This is the first of a tutorial series intended to introduce Haskell by coding things that work.

In this article we will show you how to get Haskell installed, how to set up a new project, and run your code.

Tools and Names

GHC is the compiler for Haskell. It takes Haskell source code and turns it into an executable.

Cabal is the package description format. You’ll have a file called my-project.cabal with information about your project. There’s an executable called cabal too, but we are going to use stack instead.

Stack is a package manager. It reads my-project.cabal and stack.yaml to link in third party code. It will install GHC for you too.

Installing Stack

The only thing you need to download is Stack. It will install everything else for you.

Download Stack

Then, follow the instructions on the download page for your operating system. Here’s what I did on a mac:

Unzip the file by double clicking it

Move it to /usr/local/bin

$ mv stack-0.1.3.1-x86_64-osx /usr/local/bin/stack

Give it executable permissions
```
$ chmod +x /usr/local/bin/stack
```

Alternatively on OSX you can install via homebrew:

brew update
brew install haskell-stack

Check to make sure it is working

$ stack --version
Version 0.1.3.1, Git revision 908b04205e6f436d4a5f420b1c6c646ed2b804d7

Setting up a new project

We’re going to need a few files to get our first project going: some haskell source code, a stack.yaml and a .cabal file. We can create these files by hand, but stack has a template feature we can use instead. Let’s call our project “my-project” and use the simple template.

$ stack new my-project simple
...

This creates a directory named my-project. Let’s see what’s inside:

LICENSE
Setup.hs
stack.yaml
my-project.cabal
src/
  Main.hs

The stack.yaml config file tells stack which version of GHC and your dependencies to use.

flags: {}
packages:
- '.'
extra-deps: []
resolver: lts-3.1

We use the my-project.cabal config file to store settings like the project name, and license. In the next article, Importing Code, we’ll edit this file to add dependencies.

name:                my-project
version:             0.1.0.0
synopsis:            Simple project template from stack
description:         Please see README.md
homepage:            http://github.com/githubuser/my-project#readme
license:             BSD3
license-file:        LICENSE
author:              Sean Hess
maintainer:          seanhess@gmail.com
copyright:           2010 Author Here
category:            Web
build-type:          Simple
cabal-version:       >=1.10

executable my-project
  hs-source-dirs:      src
  main-is:             Main.hs
  default-language:    Haskell2010
  build-depends:       base >= 4.7 && < 5

Last but not least, we have some source code. src/Main.hs is the main module for our program. This is where the Haskell happens.

module Main where

main :: IO ()
main = do
  putStrLn "hello world"

Installing GHC

Stack will install the correct version of GHC for our project. This is cool because everyone working on your project will be on the same version. Let’s give it a shot: run this in your project folder

$ stack setup

Which results in:

Downloaded lts-3.1 build plan.    
Caching build plan
Fetched package index.
Populated index cache.
Downloaded ghc-7.10.2.
Installed GHC.
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec

Run the Code!

Now we’re ready to run some code! Let’s use stack to fire up GHCI: the Haskell REPL.

$ stack ghci
Configuring GHCi with the following packages: my-project
GHCi, version 7.10.2: http://www.haskell.org/ghc/  :? for help
Prelude> 

There are other great tutorials that will teach you how to use ghci, but here are a few examples:

Prelude> 2 + 15
17

Prelude> 5 == 5
True

We can Use the :load command to load our code. Note that you can tab-complete module names.

Prelude> :load Main
[1 of 1] Compiling Main             ( src/Main.hs, interpreted )
Ok, modules loaded: Main.

Then we can run our program by typing main

Main> main
hello world

Making Changes

We can use ghci to test our changes as we go. Let’s make a greet function! Add this to src/Main.hs

greet name = "Hello " ++ name ++ "!"

Now go back to ghci and type :reload or :r

Main> :r
[1 of 1] Compiling Main   (src/Main.hs, interpreted)
Ok, modules loaded: Main.

We can test greet without it being used in main

Main> greet "bobby"
"Hello bobby!"

Let’s add it to our main program! Edit src/Main.hs

main = do
  putStrLn (greet "bobby")
  putStrLn (greet "World")

Reload again with :r and then run main

Main> :r
Main> main
Hello bobby!
Hello World!

Building an Executable

When you are ready to ship, you can build an executable with stack build

$ stack build

It will tell you where the executable is, but it’s easier to run it with stack exec. It will run anything in your main function.

$ stack exec my-project
Hello bobby!
Hello World!

Other Resources

Learn You a Haskell for Great Good - Good (free) introductory Haskell book.
Prelude Documentation - All the functions that come built in
Complete source code for this tutorial
Configure your text editor to underline errors for you. Check out Editor Setup

Assignment

Read chapter 3 of Learn You a Haskell and add a type declaration to greet
Use the getLine function to read a name from the command-line, and print out a greeting to that name. Will require using the Prelude Documentation, and probably some googling.

Answers

What’s Next

In the next article, Importing Code, we show you how to use other built-in modules and 3rd party code.

Swift - Why Annoying is Good

2014-09-08T00:00:00+00:00

Swift - Why Annoying is Good

Last weekend I attented Startup Weekend, and I finally got a chance to try Swift. We made a simple app called Wolf Pack for parents who want to let their kids run around the neighborhood like savages.

Swift is awesome, but I occasionally found myself butting heads with the type system. I’m a huge fan of type systems so why was Swift getting in my way?

Types vs Convenience: An eternal war

This isn’t the first time that strict typing has slowed me down. A year ago at a Startup Weekend I boldly tried to get over my Haskell-novice-coder’s block. I joined a team with simple requirements and tried making a REST api. After making about a hundred of these in Node.JS, and some good exposure to Haskell, it should have been easy.

I got it printing data from the database, but my pace started to slow right away. Things I never thought about in node required my full attention. This was especially true when it came to handling JSON and the database: the edges of my program.

With node you can just take the client’s POST and throw it in the database. You can take the result of a database query and dump it out. Easy. In Haskell I found myself spending lots of time writing code that translated JSON data to my object, to the database, and back again.

“This is supposed to be automatic! I shouldn’t have to write this code at all!”

I gave up after the first day.

Swift is like Haskell

When I saw the Swift spec, I was excited to see how many ideas they had incorporated from Haskell. Swift has a much stricter type system than Objective-C. One new feature is Optional Typing. You can decide whether a variable is ever allowed to be null. To specify this, you put a question mark after the variable declaration.

// user can be nil
var user:User? 
user = nil
user = User("Bob")

A variable without the question mark cannot ever be null. You have to provide a value during initialization

var user:User = User("Bob")

This is only one simple example of many ways in which Swift is more strict than Objective-C

Strict is annoying

Having to deal with optional types in swift is annoying. If you’re used to the old way of doing things, it seems like a step backwards. The following won’t compile:

// No! user doesn't have a value!
var user:User
userLabel.text = user.firstName 

The first thing you might realize is that you can code like a “normal” person if you throw in gratuitous punctuation. Question marks make it work the old way:

// compiles and the label is blank
var user:User?
userLabel.text = user?.firstName

Or use exclamation points to pretend it isn’t null

// also compiles, but will crash on the 2nd line.
var user:User!
userLabel.text = user.firstName

Why doesn’t the program just let me code? The second example will crash if it runs! In what universe is this better than just leaving the field blank if user is nil?

Data Integrity: Not having to worry

Prior to swift, you had two ways to handle this kind of situation.

You could be careful: make sure your data is all correct going in, check for nils everywhere your program might break, and leave comments with your assumptions.

Or you could be lazy: just assume that everything is correct, and code as if it is.

The only thing that Swift’s strict type system does is enforce your choices. You can make a type optional, and deal with the chance that it might be nil every time you use it, or you can mark it non-optional and force whoever sets the variable to get it right.

With Swift, your type system is enforcing your assumptions. This is a powerful idea. You don’t have to leave a comment. You don’t have to update your wiki. It’s a part of the code itself.

An exercise in Laziness: Back to JSON

In Wolf Pack, we wanted to sync our Core Data store with the REST API. This meant we had to call the API, parse each JSON object into an NSManagedObject, and save them. Here’s some of the relevant code from our User model.

class func syncREST() {
    let url = NSURL(string: "http://wolfpack-api.herokuapp.com/users")
    loadRESTObjects(url) { json in
        let id = json["_id"].string!
        var user = self.fetchOrCreate(id)
        user.updateFromJSON(json)
    }
}

And below is where we map the JSON to the fields. This is almost the most compact way to express this: Take the firstName field from the JSON and set it to our firstName property. Throw in some gratuitous punctuation to get it to compile:

func updateFromJSON(json:JSONValue) {
    self.firstName = json["firstName"].string!
    self.imageUrl = json["imageUrl"].string!
    // etc
}

But what this code also says is: “Crash if you don’t find firstName in the JSON object”. Is this what we really want? ¹

This isn’t a theoretical question. This project requires an imageUrl when we go to display an avatar image, or it crashes. The API didn’t guarantee an imageUrl (the back-end guys were adding the images one at a time).

By cheating with the exclamation points, we’ve swept the decision of how to handle bad data under the rug. It’s like pretending a house is clean by throwing a blanket over everything. Saving this kind of complexity for later is Technical Debt: it’s what makes you afraid to edit a code base later.

Here’s what we really ended up wanting in this case: if imageUrl is missing, set it to a default:

if let value = json["imageUrl"].string {
    self.imageUrl = value
} else {
    self.imageUrl = "http://example.com/empty-avatar.png"
}

Swift encourages you to make these kinds of decisions up front.

Put a fence around your code

We are still left with the decision of where to be strict: should we mark types as Optional or force them to be clean earlier in the process? Should we map our data to strictly typed classes or should we use Dictionaries throughout our program?

In the medium/long-term, it’s best to deal with unknowns as early as possible. Once you have everything nicely typed, knowing you’ve handled missing fields, etc, the rest of your code gets much easier to write.

Think of putting up a fence at the boundaries of your code: enforce the rules when you send or read JSON, or when you get input from a user. That way you get it over with and never have to check again. It makes everything “inside the fence” easy to understand and easy to trust.

Embrace the strictness

So, coming back to that API, it’s more complicated than “Node is easier to work with than Haskell”. Node encourages me to take a very complex and error-prone process (JSON parsing), and sweep it under the rug for another day. It trades long-term maintenance for short-term gains.

Trusting POST data enough to save it straight into your database is crazy. Clients can put anything they want in that object. Sometimes crazy assumptions are ok. If you’re coding a prototype over the weekend, you don’t need to worry about tomorrow. But usually we shouldn’t be optimizing for the first week of a project, because it only lasts a week. The majority of projects take months or years.

Strictness forces us to pay a price up front to minimize complexity in the future. That’s why it’s worth it. Type systems do this for us, and the more expressive and strict a type system is, the more it can help you.

Discuss on Hacker News

Using ! isn’t always bad. Sometimes you want to make assumptions about your code. For example, I pretty much always mark IBOutlet variables as !, because if they aren’t set I probably forgot to attach them, and I want the program to crash. But this isn’t cheating, it’s the intended behavior. ↩

AngularJS Directive Design Made Easy

2013-10-14T00:00:00+00:00

AngularJS Directive Design Made Easy

AngularJS directives are cool

AngularJS is a web application framework that makes creating complicated web applications much simpler. One of its best features is the ability to create directives, or reusable web components. It gives you the ability to create new HTML tags and attributes, which can dynamically display content in response to data changes, as well as update the data when appropriate.

They’re a big productivity booster because they let you wrap up a complicated interaction with the DOM in a nice, reusable package.

Making directives is confusing at first

It doesn’t take long to realize that directives are useful, and the ones that are bundled with AngularJS are well designed, but making directives can feel overwhelming at first. The Angular team has done a good job making directives extremely powerful and flexible, but all that power comes with some complexity.

Specifically, it’s difficult to understand how to create a directive that responds to data changes, updates data, responds to events, or exposes events. Basically it boils down to this:

How do I talk to a directive?

This article aims to explain and simplify some of the most common problems you will run in to when creating directives.

Directive design principles

Directives make our lives easier when you can reuse them without needing to read or edit the source code. Then we can forget how they work, and just remember what they do.

If you’re coming from a view-centric framework, like Backbone, you may be tempted to separate your application into view-like directive chunks. For example, if you want to display a list of users, you might create a directive that reads $scope.users and prints them all out:

<user-list/>

The user-list directive works. I mean, look how DRY it is! However, contrast it with ng-repeat, which handles only the repetition. Which one could be used in more places? What if you need to display users differently in two places?

A good directive only does one job

ng-repeat is better than user-list because it does only one job: It only does the repetition part, so you can reuse it many more situations. It’s job is easy to understand. Instead of making one directive that solves everything, split it up into several focused directives and glue them together.

A good directive is not application specific

Directives are more widely useful the fewer assumptions they make about your application. A directive that allows the user to say which property to observe, like ng-model is more useful than one that assumes that $scope.users exists. As a general rule, if your directive could be useful in a completely different application, it’s more likely to be well designed and useful even if you never publish it.

That’s enough theory for today. Let’s dive in to some specific examples of common ways you can interact with directives.

How to display bindings

The first thing to learn is how to make a directive that respects a binding: the ones with double curly braces. For example, let’s make a directive that displays a photo and a caption.

The first step in any directive design is to choose the names of the attributes that will make up your interface. I’ve chosen to use photo-src for the image src, and caption for the text. Be careful not to use names that other directives use, like ng-src unless you know how they work.

Secondly, decide if you want to support only attributes and class names, or elements too. In this case we decide we want photo to be an element.

<photo photo-src="{{photo.url}}" 
         caption="Taken on: {{photo.date}}"/>

Note that I did not give the directive the whole photo object. It’s better design to allow the directive to work with any data structure.

To read a binding, use attrs.$observe. This will call your callback any time the binding changes. We then use element to make changes to the DOM.

app.directive('photo', function() {
    return {
        // required to make it work as an element
        restrict: 'E',

        // replace <photo> with this html
        template: '<figure><img/><figcaption/></figure>',
        replace: true,

        // observe and manipulate the DOM
        link: function($scope, element, attrs) {
            attrs.$observe('caption', function(value) {
                element.find('figcaption').text(value)
            })

            // attribute names change to camel case
            attrs.$observe('photoSrc', function(value) {
                element.find('img').attr('src', value)
            })
        }
    }
}
})

Alternatively, if your component has its own template, you can do all of this with an isolate scope.

app.directive('photo', function() {
    return {
        restrict: 'E',
        templateUrl: 'photo.html',
        replace: true,
        // pass these two names from attrs into the template scope
        scope: {
            caption: '@',
            photoSrc: '@'
        }
    }
})

<!-- photo.html -->
<figure>
    <img ng-src="{{photoSrc}}"/>
    <figcaption>{{caption}}</figcaption>
</figure>

How to read and write data

Some directives need to write data too, like ng-model.

Let’s make a button toggle directive. This directive will automatically set its toggle state based on some boolean in the scope, and when clicked, it will set the boolean.

When passing data this way, you don’t use curly braces, you use an “Expression”. An Expression is any JS code that would run if it were on the scope. Use expressions whenever you need to write data, or when passing in an Object or Array into the directive instead of a string.

<!-- no double curly braces here -->
<button toggle="preferences.showDetails">Show Details</button>

First we use = on the scope: settings to make scope.toggle available within our directive. Anywhere in our directive, scope.toggle reads and writes to whatever the user set in the attribute.

app.directive('toggle', function() {
    return {
        scope: {
            toggle: '=',
        },
        link: function($scope, element, attrs) {

Next we use scope.$watch, which calls your function whenever the expression changes. We’ll add and remove the active css class whenever it changes.

            $scope.$watch("toggle", function(value) {
                element.toggleClass('active', value) 
            })

Finally, let’s listen to the jQuery click event and update the scope. We need to use scope.$apply any time we respond to changes from outside of Angular.

            element.click(function() {
                $scope.$apply(function() {
                    $scope.toggle = !$scope.toggle
                })
            })
        }
    }
})

Working Demo

How to expose events

Sometimes you want to allow a controller to respond to events from within a directive, like ng-click. Let’s make a scroll directive, that can call a function whenever a user scrolls that element. In addition, let’s expose the scroll offset too.

<textarea scroll="onScroll(offset)">...</textarea>

Similar to the toggle button, we map whatever function they specify in the attribute to scroll in our directive’s scope.

app.directive('scroll', function() {
    return {
        scope: {
            scroll: "&"
        },
        link: function($scope, element, attrs) {

We’ll use jQuery’s scroll event to get what we need. We still need to call scope.$apply here, because even though it calls the handler either way, the handler on the controller might set data.

            element.scroll(function() {
                $scope.apply(function() {
                    var offset = element.scrollTop()
                    $scope.scroll({offset:offset})
                })
            })
        }
    }
})

Notice that we don’t pass the offset in as the first parameter, we pass a hash of available parameters, and make them available to the expression onScroll(offset) that they passed in to the attribute. This is much more flexible than passing parameters directly, because they can pass other scope variables into their functions, like the current item in an ng-repeat.

Working Demo

How to have HTML content

Directives can have html content by default, but the minute you specify a template the content is replaced by the template.

Let’s make a modal component: a popup window with a close button, and we would like to set the body as html.

<modal>
  <p>Some contents</p>
  <p>Put whatever you want in here</p>
</modal>

Our modal is more than just one element though. When we make the template, we include everything we need, then we put a special ng-transclude directive in the div that is supposed to take back over and get all the contents.

<div class="modal">
    <header>
        <button>Close</button>
        <h2>Modal</h2>
    </header>
    <div class="body" ng-transclude></div>
</div>

Wiring things up is pretty simple. Just set transclude: true to get this to work:

app.directive('modal', function() {
    return {
        restrict: 'E',
        templateUrl: 'modal.html',
        replace: true,
        transclude: true,
    }
})

You can combine this with any of the other techniques in this article to make something more complicated.

How to respond to events

Sometimes you might want to call a function on your directive, in response to an event in your scope. For example, you might want to close the open modal if the user hits the escape key.

This is almost always an indication that you are stuck on events, when you should be thinking about data flow. Controllers don’t just contain data, they hold view state too. It’s totally fine to have a windowShown boolean on your controller, and use ng-show or pass a boolean into your directive as described above.

There are cases where it does make sense to use $scope.$on in a directive, but for beginners, try to think about the problem in terms of changing state instead. Things get much easier in Angular if you focus on data and state instead of events.

More Information

There is a lot more to directives. This article doesn’t nearly cover everything they can do. Please visit the directive documentation page for more information.

Discuss on Hacker News

Learn from Haskell - Functional, Reusable JavaScript

2012-02-20T00:00:00+00:00

Learn from Haskell - Functional, Reusable JavaScript

Learn You a Haskell: For Great Good?

For the last couple months I have been learning Haskell. Because there are so many unfamiliar concepts, it feels like learning to program all over again. At i.TV, we write a lot of JavaScript (node.js and front end). While many functional/haskell paradigms don’t translate, there are a few techniques that JS can benefit from.

There are Haskell library functions for everything. At first I thought this was just because it was mature, but then I noticed that these functions could be applied to a wider variety of problems than in other languages. This makes them more useful, as you are less likely to have to write your own solution to a common problem.

These functions are composable¹: They are focused on solving one problem without making any assumptions about your code, so you can mix and match the ones you need to solve bigger problems.

Higher Order Functions

Many of the most reusable functions in Haskell are higher order functions — they either take a function as an argument, or return a function. This makes them inherently flexible. Here’s an inflexible function: it counts the number of items in an array that match a value.

// inflexible
function countMatching(array, value) {
    var counted = 0
    for (var i = 0; i < array.length; i++) {
        if (array[i] == value) 
            counted++
    }
    return counted
}

// == 2
countMatching([1,3,3,4,5], 3) 

It’s inflexible because you can only use this function to count the number of items exactly matching a value.

Below is a more flexible version, which takes a function instead of a value. We can use it for any kind of matching, and any kind of object.

// more flexible
function count(array, matching) {
    var counted = 0
    for (var i = 0; i < array.length; i++) {
        if (matching(array[i]))
            counted++
    }
    return counted
}

// == 2, same as first example
count([1,3,3,4,5], function(num) {
    return (num == 3)
})

// == 2, now we can use our functions for ANY kind of items or match test!
count([{name:"bob"}, {name:"henry"}, {name:"jon"}], function(obj) {
    return (obj.name.length < 4)
})

Since higher order functions are more flexible, you’re less likely to have to write them. Once you write one, you can use it in many different situations.

Reusable Matching Functions

You probably noticed that count is more verbose than countMatching. In addition, while count is reusable, the matching functions² are not. While this seems fine for these simple cases, it’s very likely we’ll want more complicated matching functions. These matching functions could be used for all kinds of things, not just counting, so creating or finding them will save us time and bugs in the long run.

Let’s define some reusable matching functions to clean this up. == isn’t a function. Would it help if we had a function eq that did the same thing?

function eq(a, b) {
    return (a == b)
}

count([1,3,3,4,5], function(num) {
    return eq(3, num)
})

We’ve made a step forward: We are using a library function to match instead of custom code. If eq were more complicated, we would test it and use it elsewhere.

This doesn’t help the verbosity, though, because count takes a function with one parameter, the item, and eq takes two parameters. We still had to define our own anonymous function. Let’s try to reduce the verbosity a little bit.

function makeEq(a) {
    // countMatchingWith wants a function that takes 
    // only 1 argument, just like the one we're returning
    return function(b) { 
        return eq(a, b)
    }
}

// now it's only on one line!
count([1,3,3,4,5], makeEq(3))

We’re generating a function that is compatible with count (one argument, the item, and returns true or false). It’s as if count is calling eq(3, item). We created a function that calls eq with the first argument frozen at 3. This is called partial function application.

Partial Application

Partial Function Application is to create a function that calls another function with some of the arguments pre-set, so that it can be called by something else, like count, that expects fewer arguments. We’ve done this with makeEq, but we don’t want to create makeX versions of all of our functions. Let’s come up with a way to do this for any function.

function applyFirst(f, a) {
    return function(b) {
        return f(a, b)
    }
}

count([1,3,3,4,5], applyFirst(eq, 3))

Now we don’t need a makeEq function. We can use any 2-argument library function the same way. With partial application, defining functional versions of even simple things like == makes sense because we can use them in higher-order functions more easily.

What about functions with more than 2 arguments? This version³ lets us apply as many arguments as we want, and the higher order function can add one argument of its own.

function apply(f) {
    var args = Array.prototype.slice.call(arguments, 1)
    return function(x) {
        return f.apply(null, args.concat(x))
    }
}

function propertyEquals(propertyName, value, obj) {
    return (obj[propertyName] == value)
}

count([{name:"bob"},{name:"john"}], apply(propertyEquals, "name", "bob")) // == 1

We applied 2 arguments, “name” and “bob”, and count provides the last one to complete the call. Partial function application lets us take generic functions, like eq, and use them other generic higher order functions, like count, to solve specific problems.

Partial Application with ES5 Map and Filter

There are some great higher order functions built in to ES5, and underscore has many more.

Array.map lets you convert an array of things into other things.

var usersById = {"u1":{name:"bob"}, "u2":{name:"john"}}
var user = {name:"sean", friendIds: ["u1", "u2"]}

// == ["bob", "john"]
function friendsNames(usersById, user) {
    return user.friendIds.map(function(id) {
        return usersById[id].name
    })
}

We can make a reusable mapping function, just like we made reusable matching functions earlier. Let’s make one called lookup.

function lookup(obj, key) {
    return obj[key]
}

// == [{name:"bob"}, {name:"john"}]
function friends(usersById, user) {
    return user.friendIds.map(apply(lookup, usersById))
}

This is close, but we wanted the names, not the friend objects themselves. If we had a version of lookup with the arguments reversed, we could map over it a second time to get the names.

function lookupFlipped(key, obj) {
    return lookup(obj, key)
}

// == ["bob", "john"]
function friendsNames(usersById, user) {
    return friends(usersById, user)
            .map(apply(lookupFlipped, "name"))
}

But I don’t want to define lookupFlipped, that’s dumb. Instead, let’s make a function that applies arguments to the right instead of the left, so can use lookup instead.

function applyr(f) {
    var args = Array.prototype.slice.call(arguments, 1)
    return function(x) {
        return f.apply(null, [x].concat(args))
    }
}

// == ["bob", "john"]
function friendsNames(usersById, user) {
    return friends(usersById, user)
            .map(applyr(lookup, "name")) // we can use normal lookup!
}

applyr(lookup, "name") creates a function to be called with one argument, the object, and returns the object’s name. We no longer need to flip anything: we can apply arguments to either side of the function.

Now let’s look at Array.filter, which filters an array, given a matching function.

// this equals [1,3,3]
[1,3,3,4,5].filter(function(num) {
    return (num < 4)
})

Let’s use a reusable matching function lt (less than) instead of defining an anonymous one.

function lt(a, b) {
    return (a < b)
}

// filter by: lt(x, 4)
[1,3,3,4,5].filter(applyr(lt, 4))

It might seem silly to have created lt at all, but we can partially apply it to create a concise matching function, and if it were any more complicated we’d benefit from reuse.

Partial Application requires defining a bunch of functional versions of common things, like lt, but that’s the point. You can use partially applied lt for both count and Array.filter. They’re reusable and composable.

Function Composition

In the previous example, we looped through the array twice, once to get the users, and again to get the names. It would be more efficient to loop through once, and do both mappings at the same time.

function friendsNames(usersById, user) {
    return user.friendIds.map(function(id) {
        var friend = lookup(usersById, id)
        return lookup(friend, "name")
    })
}

We are taking the result of the first lookup, and passing it into the second lookup. Function composition means to chain multiple functions into a new, single function, with each step passing its result to the next.

Let’s create a higher-order function to do this for us, then we can rewrite friendsNames with a single mapping function. Note that the functions happen in right-to-left order, just as they would if you were writing f(g(x)).

function compose(f, g) {
    return function(x) {
        return f(g(x))
    }
}

function friendsNames(usersById, user) {
    return user.friendIds.map(compose(applyr(lookup, "name"), apply(lookup, usersById)))
}

This only loops through the array once, and creates a single mapping function exactly like the first example.

We weren’t able to use the friends function we created, because while it contains the logic for how to get a friend, it also contains the mapping. The friends function is less reusable because it does too much — It’s too specific. What if, instead, we created a friend function that mapped only one friend, and a name function that returns the name of something?

var friend = lookup // lookup happens to have the signature we want. 
var name = applyr(lookup, "name")

function friendsNames(usersById, user) {
    // this line is now more semantic. 
    return user.friends.map(compose(name, apply(friend, usersById)))
}

Instead of defining a friends function, which does both the conversion and the iteration, we define friend, which does the conversion, and we already have map which does the iteration. friends is more reusable than friends because it is less specific, and can be used in more situations.

See here for more information on function composition in JavaScript.

Functional and Focused Makes a Clean Codebase

I find myself writing a lot of JavaScript logic from scratch. This isn’t just slower than using existing tools, it’s more bug-prone and harder to read and maintain. With higher order functions and partial function application, we can create reusable tools that focus on exactly the part of a problem they are trying to solve.

Over time, instead of a project increasing in complexity becoming more and more coupled, we benefit from a growing library of useful tools that can be tested independently of each other, resulting in a healthier, more stable code base.

Discuss on Hacker News

Composable in the generic sense. This doesn’t refer to either function or object composition, just the idea that you work with small tools to build large ones. ↩
“Matching functions” are called predicates, but I’m trying to avoid introducing new terminology. ↩
See here for more general implementations of apply. ↩

You Only Wish MongoDB Wasn't Relational

2012-02-01T00:00:00+00:00

You Only Wish MongoDB Wasn’t Relational

MongoDB = Get Stuff Done

Update: Changed blog example to use a normal belongs-to relationship.

Update: Added examples of $slice and $elemMatch to show why they don’t work

When choosing the stack for our TV guide service, we became interested in NoSQL dbs because we anticipated needing to scale horizontally. We evaluated several and settled on MongoDB. The main reason was that MongoDB got out of the way and let us get work done. You can read a little more about our production setup here.

So when you read that MongoDB is a document store, you might get the wonderful idea to store your relationships in a big document. Since mongo lets you reach into objects, you can query against them, right?

Several times, we’ve excitedly begun a schema this way, only to be forced to pull the nested documents out into their own collection. I’ll show you why, and why it’s not a big deal.

Let’s make a blog!

Here’s an example. You just discovered that Mongo lets you store nested documents. You set out to make a blog.

db.posts.save({ title: "My First Blog Post"
              , content: "Here is my super long post ..." 
              , created: 1328118062598
              })

What’s a blog without comments? Hey, mongo is awesome! Let’s nest the comments. Multiple collections are for RDBMS dinoasaurs.

{ title: "My First Blog Post"
, content: "Here is my super long post ..." 
, comments: [ { text: "This post sucks!"
              , name: "seanhess"
              , created: 1328118162000 }
            , { text: "I know! I wish it were longer"
              , name: "bob"
              , created: 1328118262000 } 
            ] 
}

This works nicely with the key/value idea that you should denormalize your data. You can avoid joins if all the data you need for a call is in one document. It’s easy to treat these nested documents as if they were top-level. As long as your blog post displays the post and all comments, this works great:

function postWithAllComments(id) {
    return db.posts.findOne({_id: id})
}

function addComment(postId, comment) {
    comment.created = Date.now()
    db.posts.update({_id: postId}, {$push: {'comments': comment}})
}

When Nested Documents Become a Problem

The minute you need just a little control over querying against those comments, you’re stuck. While you can use $slice to limit comments to a certain number/offset, what if you want to display all comments made by a certain user? You’re first tempted to try this:

// bad idea
function commentsByUser(username) {
    // $slice can't help you here
    return db.posts.find({"comments.name": username}, {comments: 1}).toArray()
}

Which seems to work, except it returns all the comments for each post they commented on. There’s no way to get only the matched comment back. If there are 1000 comments per post, and the user commented on 3 posts, you’ll have to wade through 3000 comments.

Another thing that I need to do a lot is get a range of documents. Perhaps we want to see all comments made yesterday. You have the same problem: you get all the comments for each post.

// bad idea
function commentsBetweenTimes(start, end) {
    return db.posts.find( {"comments.created": {$gte: start, $lt: end}}
                        , {comments: 1}
                        ).toArray()
}

Collections are Cheap

It turns out it isn’t hard to put the comments in their own collection. If you restructure your posts to look like this, everything falls into place:

// db.posts
{ _id: "4f297e550b3e6d9e2b7aa58e"
, title: "My First Blog Post"
, content: "Here is my super long post ..." 
}

// db.comments
{ postId: "4f297e550b3e6d9e2b7aa58e"
, text: "This post sucks!"
, name: "seanhess"
, created: 1328118162000 }

Your client doesn’t have to be aware of the change, just add the comments with a second query. Now the comment-specific queries return only comments.

function postWithAllComments(id) {
    var post = db.posts.findOne({_id: id})
    post.comments = db.comments.find({postId: id}).toArray()
    return post
}

function addComment(postId, comment) {
    comment.created = Date.now()
    comment.postId = postId
    db.comments.save(comment)
}

function commentsByUser(username) {
    return db.comments.find({"name": username}).toArray()
}

function commentsBetweenTimes(start, end) {
    return db.comments.find({"created": {$gte: start, $lt: end}).toArray()
}

While our postWithAllComments function requires 2 queries, it’s very fast. Everything comes out of memory, and from an index. The other functions are as fast or faster and are easy to write.

Many-to-Many Relationships

It’s impossible to model many-to-many relationships without storing them in a separate collection, unless you copy the data to every document. This is fine for tags, but can be unrealistic if your data set is large. It also means that like our blog you can’t query the nested documents themselves. Here’s a post with many-to-many tags.

db.posts.save({ title: "My Post", tags: ["awesome", "incredible"] })
db.posts.find({ tags: "awesome" }) // works!

Let’s say you want to record users that are attending events. You could store this data a variety of ways, but it’s very likely that you’ll want to get both the users attending an event, and the events a user is attending. It doesn’t make sense to put either users under event, or events under users.

Even though you’re forced to be relational, a HUGE advantage over SQL is that you don’t have to have a separate joining collection. We can store the relationships inside the documents. Put the relationship on both if you want to query both directions.

var party = { _id: "chessparty"
            , name: "Chess Party!"
            , attendees: ["seanhess", "bob"] }
var user = { _id: "seanhess", name: "Sean Hess", events: ["chessparty"]}
db.events.save(party)
db.users.save(user)

db.events.find({_id: {$in: user.events}}) // events for user
db.users.find({_id: {$in: party.attendees}}) // users for event

Denormalization might be important

So, that said, sometimes you need your queries to be really fast. If your list of users only needs to show their names, you can nest their names along with their ids, allowing you to display your page without a join. You’re not storing the users within events, your copying some of the fields you need to events, so you can skip the $in query. This is called denormalization, and the usual warnings about early optimization apply.

{ _id: "chessparty"
, name: "Chess Party!"
, attendees: [ {_id:"seanhess", name:"Sean Hess"}
             , {_id:"bob", name:"Bob Somebody"} ] 
}

Don’t fight the Mongo

MongoDB just lets you get stuff done. Don’t become a NoSQL or Document Store purist, just write code that works. It’s the mongo way. It’s easy to store relationships in a separate collection, and the joins are pretty cheap if you don’t split up your data too much. Don’t be overly tempted to store everything in a nested document, because at least in my experience, you end up needing to query against them sooner rather than later.

What about X?

$slice doesn’t solve the problem, because you can only give it offsets, which is great for paging, but doesn’t return the comments you matched on.

db.posts.save({comments:[{num:1, name:"sean"}, {num:2, name:"bob"}]})
db.posts.find()
// { "comments" : [ { "num" : 1, "name" : "sean" }, { "num" : 2, "name" : "bob" } ] }
db.posts.find({"comments.name":"bob"}, {comments:{$slice: 1}})
// { "comments" : [ { "num" : 1, "name" : "sean" } ] }
// I want the 2nd comment, not this one ^^

$elemMatch doesn’t work either, it was intended to help you match nested documents more accurately, but still doesn’t give you only the matched comments

db.posts.find({comments: {$elemMatch: {name:"bob"}}})
// { "comments" : [ { "num" : 1, "name" : "sean" }, { "num" : 2, "name" : "bob" } ] }

Discuss on Hacker News

Bad Optimization is like firing clay

2011-12-15T00:00:00+00:00

Bad Optimization is like firing clay

When “optimized” doesn’t mean better

To Optimize literally means to find the best possible solution. It’s hard to see how that could be bad, since it means to make something better. However, another definition is to make something better for a specific purpose, at the expense of others. In that case, sometimes it’s not the right choice.

Optimization is only bad when you trade flexibility for performance.

Flexibility here means simple, maintainable code, or just time. Get rid of either and it’ll be difficult to change your mind. Plan ahead, and do obvious things in the name of performance. As long as you don’t spend too much time or make things complicated, you can’t go wrong.

But pay attention when you come to a trade-off — where you must make something more complicated or spend extra time. You probably want to wait.

I have to learn this again?

For our new product (a backbone app), our CEO suggested that we support up to 10,000 items on a page at once. We were able to make it perform well by assuming that each item had a static height. This allowed us to calculate the exact position of each item on the screen without drawing it. We coudl then load and unload items as the user scrolled. After a few days of work, we acheived great performance, even for 10,000 items.

Then, our designer showed us higher-fidelity designs. The items needed dynamic heights.

We went back to work, and I was surprised by how long it took to get the items to display correctly. It took nearly as much time to support dynamic heights as it would to write simple layout code from scratch.

In exchange for large scale performance, we traded away our initial time, as well as the ability to work with dynamic heights. Instead, we could have made it perform well for 100 items from the beginning, and we would have known about the dynamic heights before optimizing it for more.

Clay is either flexible, or strong

Clay will stay flexible indefinitely¹. You can work something into a shape, and come back later and change the shape. At some point you can fire the clay, making it rigid and exchanging its flexibility for increased strength.

If you are making a model out of clay, there’s a clear advantage to firing it. It makes the model stronger. A stronger model is “better” — it can be handled without fear of deforming it.

Most of the time, though, a model is strong enough without firing. You might need it to be strong eventually, but for right now it serves its purpose well — to model something. If you keep it unfired, you can add more detail, change its shape, even change its purpose without starting over.

These are all improvements you can make without sacrificing anything. It’s just better. But to make it stronger, you have to trade away its flexibility.

Don’t make tradeoffs until you have to

Software needs to perform, but only well enough. No one will notice if you shave 50ms off of a 300ms page load. You usually have a fairly good idea of what constitutes acceptable performance.

On the other hand, you usually have no idea what changes you might have to make. Design requirements change all the time. Sometimes, you don’t even know the best way to make something until you’ve tried building it once.

In every case it’s easier to make major investments of time and complication in the name of performance after your code has stopped changing, simply because you don’t have to undo anything.

But what if you delay for so long that you forget to go back and optimize? This is actually the best thing that could happen. Frequently we don’t need as much performance as we think. If we do, it’ll come up on its own. If we don’t, our forgotten, simple, and stable code can perform its function in peace.

Optimize — Make improvements freely, but delay trading time or simplicity

As it turns out, we didn’t need our optimization. There’s no way the current UX can support 10,000 items, so making our code complicated to achieve that level of scale was a mistake. Thinking clearly about the difference between improvements (changing the figure) and trade-offs (firing the clay) will help us wait until the right time.

This is more true of polymer clay than ceramic, which must be kept wet to stay flexible. ↩