Flying Machine Studios

How Clojure Babies are Made: What Leiningen Is

2013-04-07T14:23:00Z

"What the hell is Leiningen?" is a question you've probably overheard many times in your day-to-day life. You've probably even asked it yourself. Up until now we've described specific capabilities that Leiningen has and how those capabilities are implemented. It can build and run your app, it has a trampoline, and so forth.

But it's time to take a step back and get a high-level understanding of what Leiningen is. It's time to stare deeply into Leiningen's eyes and say "I see you," like Jake Sully in that Avatar documentary. We'll do this by giving an overview of the non-coding related tasks you need to accomplish when building software. Next, we'll describe how Leiningen helps you accomplish these tasks and compare it to similar tools in Ruby.

This post isn't as nitty-gritty as the previous posts in the Clojure Babies series, but it will help lay the groundwork for an upcoming post on packaging. Additionally, I hope it will clarify what a programming language artifact ecosystem is. This concept is often overlooked when teaching a programming language, and when it is covered it's not covered in a systematic way. Together, noble-chinned reader, we will remedy that situation. For our generation and all generations to come.

Programming Language Artifact Ecosystems

In order to become proficient at a language, you need to know much more than just its syntax and semantics. You need to familiarize yourself with the entire programming language ecosystem, which is comprised of everything you need in order to build working software in that language. It can be broken down into at least the following sub-ecosystems:

The documentation ecosystem
The developer community
The development environment ecosystem (editor support)
The artifact ecosystem

Right now we only care about the artifact ecosystem. For our purposes, a programming artifact is a library or executable. Ruby gems, shell scripts, Java jars, shared libraries, and "HAL9000.exe" are all programming artifacts.

An artifact ecosystem is the set of tools and services that allow you to do the following with regard to artifacts:

Retrieve them from repositories
Incorporate them in your own project, (possibly) resolving conflicts
Build them
Publish them to repositories
Run them

Tools are often layered on top of each other, one tool smoothing out the warts of the tools it wraps. For example, the following tools (and more) are part of the Ruby artifact ecosystem:

Ruby Gems provides a package specification, the means to incorporate gems in your project, and the means to build and publish gems
rubygems.org is a central repo for gems
Bundler provides a layer on top of Ruby Gems, providing dependency resolution and gem retrieval
Jeweler is one of many tools for easing the process of creating gemspecs and building gems.

Other languages have their own tools. Java has Maven, PHP has Pear or whatever. Artifact management is a common need across languages.

In previous Clojure Baby posts, we've seen that we can use Leiningen to build and run Clojure programs. It turns out that Leiningen also handles the remaining tasks - retrieving packages, incorporating them in your project, and publishing them. It's truly the Swiss Army Bazooka (I'm going to keep repeating that phrase until it catches on) of the Clojure artifact ecosystem.

But why is it that in Ruby you need an entire constellation of tools, while in Clojure you only need one?

Leiningen Is a Task Runner with Clojure Tasks Built In

Leiningen is able to handle so many responsibilities because it is, at heart, a task runner. It just happens to come with an excellent set of built-in tasks for handling Clojure artifacts. (Incidentally, this is probably where Leiningen's name came from. "Leiningen Versus the Ants" is a short story where the protagonist fights ants. Ant is a Java build tool that evidently is unpleasant to use for Clojure builds.) By comparison, Ruby's Rake is also a task runner used by many of Ruby's artifact tools, but Rake provides no built-in tasks for working with Ruby artifacts.

"Task runner" is a little bit ambiguous, so let's break it down. Ultimately, all Leiningen tasks are just Clojure functions. However, in previous posts we've seen how fun it is to try and run Clojure functions from the command line. In case you need a short refresher: it's not fun at all!

Leiningen allows the Clojure party to remain fun by serving as an adapter between the CLI and Clojure. It takes care of the plumbing required for you to run a Clojure function. Whether the function is provided by your project, by Leiningen's built-in tasks, or by a Leiningen plugin, Leiningen does everything necessary to get the function to run. In a way, Leiningen's like an attentive butler who quietly and competently takes care of all your chores so that you can focus on your true passions, like knitting sweaters for gerbils or whatever.

This manner of executing code was foreign to me when I first came to Clojure. At that time I had mostly coded in Ruby and JavaScript, and I had a decent amount of experience in Objective C. Those languages represent two different paradigms of code execution.

Ruby and Javascript, being scripting languages, don't require compilation and execute statements as they're encountered. Objective C requires compilation and always starts by executing a main method. With Leiningen, Clojure has achieved an amalgamation of the two paradigms by allowing you to easily run arbitrary functions with a compiled language.

The End

Hopefully, this article has given you a firmer grasp of what Leiningen is. The idea that Leiningen is a task runner with a powerful set of built-in tasks designed to aid Clojure artifact management should help you organize your disparate chunks of Leiningen knowledge.

In the next article, we'll add another chunk of Leiningen khnowledge by examining the way Leiningen retrieves artifacts from repositories and incorporates them in your project.

Goodbye for now!

Shout Outs

Thanks to Pat Shaughnessy and technomancy for reviewing this article. technomancy provided the line "Leiningen is an adapter between the CLI and Clojure", which really helped!

How Clojure Babies are Made: Leiningen's Trampoline

2013-03-23T14:23:00Z

In the last "How Clojure Babies are Made" article I hinted that Leiningen avoids waste by using a trampoline. In this installment of this awkwardly-named Clojure series, you'll learn what the trampoline does and how it does it.

The Rationale Behind the Trampoline

I know what you're thinking, wise-hearted reader. "Why would anyone need to provide a rationale for a trampoline? That's like trying to give a reason for rainbows, or a newborn's laughter, or Michael Jackson's Thriller."

Allow me to explain myself. lein trampoline does indeed give you a feeling of weightless freedom, just not in the way that you're used to.

See, whenever you use Leiningen to run code from your project, you end up with two Java processes running. Each process loads a separate instance of the JVM. We saw this in the previous article in the output of ps | grep lein. The first process is for Leiningen itself, and it's responsible for setting up everything necessary for your project code to run. The second process is where your code actually executes. If you were to run lein -h, you would only start one Java process, as none of your project code would need to be executed.

Leiningen starts a separate process for your project in order to enforce isolation. This is because Leiningen is a true gentleman who does not allow his namespaces and dependencies (like, say, a completely different version of Clojure) to interfere with your meticulously hand-crafted, artisinal program.

However, like a doting father, the Leiningen process continues to stay open for the entire duration of your program's execution. If you have a long-running process, like a web server for your Justin Bieber fan site, then Leiningen stays open the whole time consuming memory that could be put to use in compositing images of the Biebs with hearts all over them.

This is where the trampoline comes into play. It allows the Leiningen process to exit completely before your project's process starts. Now, instead of two JVM's running, you only have one.

I think the name "trampoline" was chosen to evoke an image of Leiningen providing a launching point for your program. However, I choose to envision the trampoline process as follows:

Leiningen takes your program, which is embodied as one of those cymbal-clanging monkeys with unsettling eyes, and winds it up.
Leiningen gingerly places the wind-up monkey which is your program on the floor of a cozy Hobbit hole.
Leiningen steps outside and mounts a gigantic trampoline in Bilbo's front yard.
Leiningen takes two warm-up bounces and then, with a mighty "Hyup!", rockets himself into the sky, his mustaches flapping in the wind. He grows smaller and smaller until, with a bright sparkle, he disappears entirely.
Your cymbal-clanging web server starts up, allowing users to profess their undying love for teenage pop stars.

How lein trampoline Works

Though you don't really need to understand how lein trampoline works in order to use it, I think it's pretty cool. Below I walk you through it step by step, with relevant code. We'll be using the project under leiningen/lein-build of the make-a-clojure-baby github repo.

Run lein trampoline run from the command line. If you're on a linux machine, this executes a bash script. This script is probably at ~/bin/lein on your system.

The bash script sets the TRAMPOLINE_FILE environment variable to a path. Later in this process, Leiningen will write a command to this file. Here's the part of the script that sets the environment variable:

if ([ "$LEIN_FAST_TRAMPOLINE" != "" ] || [ -r .lein-fast-trampoline ]) &&
    [ -r project.clj ]; then
    INPUTS="$@ $(cat project.clj) $LEIN_VERSION $(cat "$LEIN_HOME/profiles.clj")"
    INPUT_CHECKSUM=$(echo $INPUTS | shasum - | cut -f 1 -d " ")
    # Just don't change :target-path in project.clj, mkay?
    TRAMPOLINE_FILE="target/trampolines/$INPUT_CHECKSUM"
else
    TRAMPOLINE_FILE="/tmp/lein-trampoline-$$"
    trap "rm -f $TRAMPOLINE_FILE" EXIT
fi

The first Java process starts:

/usr/bin/java \
  -client -XX:+TieredCompilation  \
  -Xbootclasspath/a:/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \
  -Dfile.encoding=UTF-8 \
  -Dmaven.wagon.http.ssl.easy=false \
  -Dleiningen.original.pwd=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build \
  -Dleiningen.script=/Users/daniel/bin/lein \
  -classpath :/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \
  clojure.main \ `# clojure.main is the entry point` \
  -m leiningen.core.main \
  trampoline run 1

This causes Java to execute the -main method in clojure.main, which in turn loads leiningen.core.main and executes its -main function.

leiningen.core.main/-main applies the trampoline task:

(defn ^:higher-order trampoline
  "Run a task without nesting the project's JVM inside Leiningen's.

Calculates the Clojure code to run in the project's process for the
given task and allows Leiningen's own JVM process to exit before
running it rather than launching a subprocess of Leiningen's JVM.

Use this to save memory or to work around stdin issues."
  [project task-name & args]
  (when (= :leiningen (:eval-in project))
    (main/info "Warning: trampoline has no effect with :eval-in-leiningen."))
  (binding [*trampoline?* true]
    (main/apply-task (main/lookup-alias task-name project)
                     (-> (assoc project :eval-in :trampoline)
                         (vary-meta update-in [:without-profiles] assoc
                                    :eval-in :trampoline))
                     args))
  (if (seq @eval/trampoline-forms)
    (write-trampoline project @eval/trampoline-forms @eval/trampoline-profiles)
    (main/abort task-name "did not run any project code for trampolining.")))

trampoline calls leiningen.core.main/apply-task but with a twist: it passes that function an updated project configuration, setting :eval-in to :trampoline. You can see this is the snippet above.
Eventually, leiningen.core.eval/eval-in-project gets applied. The cool thing about this function is that it then calls leiningen.core.eval/eval-in, which is a multi-method. It has different definitions for :subprocess, :trampoline, :nrepl, and a few more. This is one of the first times I've seen defmethod "in the wild" and it really tickled my pfeffernuesse. Definitely check it out on github.

Since we updated our project configuration in the last step so that :eval-in is :trampoline, the :trampoline method gets matched:
```
(defmethod eval-in :trampoline [project form]
  (swap! trampoline-forms conj form)
  (swap! trampoline-profiles conj (select-keys project
                                               [:dependencies :source-paths
                                                :resource-paths :test-paths])))
```
This updates the trampoline-forms and trampoline-profiles atoms within the leiningen.core.eval namespace.
The trampoline function shown in step 5 above continues executing:
```
  (if (seq @eval/trampoline-forms)
    (write-trampoline project @eval/trampoline-forms @eval/trampoline-profiles)
    (main/abort task-name "did not run any project code for trampolining.")))
```
write-trampoline writes out the entire Java command necessary to finally run our project's main function. It writes this command to the path in the TRAMPOLINE_FILE environment variable set by the bash script in step 2 above.

The Leiningen process exits and the bash process from step 2 continues. It checks for the existence of TRAMPOLINE_FILE, and since it exists, it essentially evaluates the command in it, kicking off the Java process which will run your code:

if [ -r "$TRAMPOLINE_FILE" ] && [ "$LEIN_TRAMPOLINE_WARMUP" = "" ]; then
    TRAMPOLINE="$(cat $TRAMPOLINE_FILE)"
    if [ "$INPUT_CHECKSUM" = "" ]; then
        rm $TRAMPOLINE_FILE
    fi
    exec sh -c "exec $TRAMPOLINE"
else
    exit $EXIT_CODE
fi

It's a bit of circuitous route, but Leiningen is not one to shy away from hard work!

The End

I hope you found this article interesting! The following topics are next on my list of things to write about:

How Leiningen manages dependencies
How to distribute a full application
How to distribute a library

So long for now, and may the lambda be with you!

How Clojure Babies Are Made: Understanding lein run

2013-03-18T14:23:00Z

Leiningen reminds me a lot of Major Alex Louis Armstrong from Fullmetal Alchemist:

They are both manly
They are both artistic
Sometimes you don't know what the f* they're saying (but it's amazing anyway)
Mustaches
Sparkles

Though the Strong Arm Alchemist will forever remain inscrutable, we do have some hope in understanding Leiningen better. In this post, we'll peek under Leiningen's lederhosen so that we can begin to learn precisely what it does and how to make effective use of it in developing, testing, running, and deploying Clojure applications.

Here's what we'll learn:

How to build and run a Clojure program without Leiningen
What happens when you run lein run
Basic Leiningen architecture
What happens when you run a Leiningen task

In the next post, we'll learn:

How Leiningen avoids waste with a trampoline
How Leiningen manages dependencies
How to distribute a full application
How to distribute a library
Other stuff Leiningen does

This post builds on How Clojure Babies Are Made: Compiling and Running a Java Program, so make sure you understand everything in that post first. It doesn't have any pictures of sparkly, sensitive, muscle-bound men though so if that's what you're hoping for you're out of luck. We're here to learn Clojure, dammit, not ogle cartoons!

Brief Overview of Leiningen

Leiningen is the Swiss Army Bazooka of Clojure development. It handles:

Project compilation. Your Clojure has to get converted into Java bytecode somehow, and Leiningen automates the process.
Dependency management. Similar to Ruby's bundler and gemfiles, Leiningen automates the process of resolving and downloading the Java jars your project depends on
Running tasks. Superficially similar to Ruby's Rake, except that Rake is not very declarative. This could have its own blog post.
Deployment. Helps with creating Java jars which can be executed and/or incorporated in other projects. Similar to Ruby gems.

How to Build and Run a Clojure Program Without Leiningen

By understanding how to build and run a Clojure program manually, you'll better understand what Leiningen does to automate the process. We'll be working with the files under leiningen/manual-build which you can get from the make a clojure baby repo. All path references will be relative to make-a-clojure-baby/leiningen/manual-build.

The program we're compiling will make us fluent in the ways of German Love:

(ns learn-a-language.important-phrases
  (:gen-class))

;; It's time for some German love! 
(def german
  [["I love you." "Ich liebe dich."]
   ["You make me so happy!" "Du machst mich so glucklich!"]
   ["I miss you." "Ich vermisse dich./Du fehlst mir."]
   ["Pass me the mustard." "Gib mir den Senf."]   
   ["Kiss me!" "Kuss mich!"]])

(defn -main
  [which]
  (let [phrases (get german (Integer. which))]
    (println "English: " (first phrases))
    (println "German: " (second phrases))))

One important thing to note here is that we included the :gen-class directive in the ns declaration. This tells Clojure to generate a named Java class when it compiles the namespace. Remember that Java requires a public main method within a public class to serve as the entry point for the JVM. Using :gen-class allows us to use learn-a-language.important-phrases/-main as the entry point.

Let's go ahead and compile. First, start up a Clojure repl (note that the git repo includes the 1.5.1 release of Clojure as clojure.jar):

java -cp .:clojure.jar clojure.main

Notice that we specifed the classpath with -cp .:clojure.jar. This does two things:

It allows us to execute the main method in clojure.main similarly to what we saw at the end of the last post
It adds the current directory to the classpath so that when you actually start loading Clojure files and compiling namespaces, the Clojure repl can find them. To see what I mean, try starting the repl with java -jar clojure.jar and then running the code below.

You should now have a Clojure repl running in your terminal. Copy and paste the following lines into it:

(load "learn_a_language/important_phrases")
(compile 'learn-a-language.important-phrases)

The first line reads the specified file. The second actually compiles the learn-a-language.important-phrases namespace, generating a boatload of Java class files in the classes/learn_a_language directory:

$ ls classes/learn_a_language/
important_phrases$_main.class
important_phrases$fn__19.class
important_phrases$fn__48.class
important_phrases$loading__4910__auto__.class
important_phrases.class
important_phrases__init.class

(I won't go into into detail about the purposes of the various class files, but you can start to learn more about that in clojure's compilation documentation.)

After going through the above steps, you might have a question on your mind grapes: "Where did the classes directory come from?"

Oh, gentle-hearted reader. Your dedication to learning has touched my heart. I shall answer your query: Clojure places compiled files under *compile-path*, which is classes by default. Therefore, classes must exist and be accessible from the classpath. You'll notice that there's a classes directory in the git repo with a .gitkeep file in it. Never change, dear, inquisitive reader. Never!

Now that we've compiled our Clojure program, let's get it running:

# Notice that you have to provide a numerical argument
$ java -cp classes:clojure.jar learn_a_language/important_phrases 0
English:  I love you.
German:  Ich liebe dich.

$ java -cp classes:clojure.jar learn_a_language/important_phrases 3
English:  Pass me the mustard.
German:  Gib mir den Senf.

Success! You are now ready to express your love for Leiningen in its native tongue. I highly recommend you use this program to come up with tweets to send to @technomancy to express your appreciation.

But before you do that, notice the classpath. We need to include both the classes directory, because that's where the learn_a_language/important_phrases class files live, and clojure.jar, because the class files generated by compile need to be able to access Clojure class files.

To sum up:

# load clojure repl
$ java -cp .:clojure.jar clojure.main
# in Clojure repl
user=> (load "learn_a_language/important_phrases")
user=> (compile 'learn-a-language.important-phrases)
# back to command line
java -cp classes:clojure.jar learn_a_language/important_phrases 0

I hope this brief foray into the world of manually building and running a Clojure program has been educational. You can see how it would be burdensome to go through his process over and over again while developing a program. Let's finally bring in our pal Leiningen to automate this process.

How Leiningen Compiles and Runs a Basic Clojure Program

Let's build the "learn a language" program with Leiningen. We have a very basic project at make-a-clojure-baby/leiningen/lein-build. Under that the directory, the file src/learn_a_language/important_phrases.clj is the same as the one listed above.

lein run

Lein automates the build + run process with lein run. Go ahead and try that now:

$ lein run 2
Compiling learn-a-language.important-phrases
English:  I miss you.
German:  Ich vermisse dich./Du fehlst mir.

You can probably guess what's happening at this point, at least to a degree. Leiningen is compiling important_phrases.clj resulting in a number of Java .class files. We can, in fact, see these class files:

$ ls target/classes/learn_a_language
important_phrases$_main.class
important_phrases$loading__4784__auto__.class
important_phrases__init.class

Leiningen then somehow constructs a classpath such that both Clojure and the "important phrases" classes are accessible by Java. Finally, the -main function is executed.

I know what you're thinking at this point, noble reader. You're thinking that the "somehow" in "somehow constructs a classpath" is a particularly un-manly, un-artistic, un-mustached, un-sparkly word, unbefitting an article on Leiningen. And you are absolutely right. To avoid your wrath, let's dig into Leiningen's source code so that we can understand what's going on with complete clarity.

Walking Through "lein run"

To get an idea of where to start, let's run lein run 1 again and then run ps | grep lein. The output has been broken up to make more sense (apologies for the goofy highlighting):

# you can actually copy and paste the part after the PID to try this
# in your terminal
8420 /usr/bin/java \
  -client -XX:+TieredCompilation  \
  -Xbootclasspath/a:/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \
  -Dfile.encoding=UTF-8 \
  -Dmaven.wagon.http.ssl.easy=false \
  -Dleiningen.original.pwd=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build \
  -Dleiningen.script=/Users/daniel/bin/lein \
  -classpath :/Users/daniel/.lein/self-installs/leiningen-2.0.0-standalone.jar \
  clojure.main \ `# clojure.main is the entry point` \
  -m leiningen.core.main \
  run 1

8432 /usr/bin/java \
  `# your classpath will be different` \
  -classpath \
    /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/test:\
    /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/src:\
    /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/dev-resources:\
    /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/resources:\
    /Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/target/classes:\
    /Users/daniel/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
  -XX:+TieredCompilation \
  -Dclojure.compile.path=/Users/daniel/projects/web_sites/make-a-clojure-baby/leiningen/lein-build/target/classes \
  -Dlearn-a-language.version=0.1.0-SNAPSHOT \
  -Dfile.encoding=UTF-8 \
  -Dclojure.debug=false \
  clojure.main \ `# clojure.main is the entry point` \
  `# this next part specifies code to be evaluated by Clojure` \
  -e (do \
      (try \
       (clojure.core/require 'learn-a-language.important-phrases) \
       (catch java.io.FileNotFoundException ___6081__auto__)) \
      (set! *warn-on-reflection* nil) \
      (clojure.core/let \
       [v__6079__auto__ \
        (clojure.core/resolve 'learn-a-language.important-phrases/-main)] \
       (if \
        (clojure.core/ifn? v__6079__auto__) \
        (v__6079__auto__ "1") \
        (clojure.lang.Reflector/invokeStaticMethod \
         "learn-a-language.important-phrases" \
         "main" \
         (clojure.core/into-array \
          [(clojure.core/into-array java.lang.String '("1"))])))))

There are two things happening here. When you first run lein run, then the process with PID 8420 starts. There are a lot of configuration variables that we don't necessarily need to care about. What's essentially happening is we're saying:

Start up the JVM with the leiningen standalone jar on the classpath
Use clojure.main as the Java entry point
Pass -m leiningen.core.main run 1 as arguments to clojure.main

That last step specifies the Clojure entry point, as opposed to the Java entry point. Clojure uses it to load the leiningen.core.main namespace and then execute the -main function within it. leiningen.core.main/-main receives the arguments run 1.

We can view Leiningen's leiningen.core.main/-main function on github:

(defn -main
  "Command-line entry point."
  [& raw-args]
  (try
    (user/init)
    (let [project (project/init-project
                   (if (.exists (io/file "project.clj"))
                     (project/read)
                     (assoc (project/make (:user (user/profiles)))
                       :eval-in :leiningen :prep-tasks [])))
          [task-name args] (task-args raw-args project)]
      (when (:min-lein-version project) (verify-min-version project))
      (configure-http)
      (warn-chaining task-name args)
      (apply-task task-name project args))
    (catch Exception e
      (if (or *debug* (not (:exit-code (ex-data e))))
        (.printStackTrace e)
        (when-not (:suppress-msg (ex-data e))
          (println (.getMessage e))))
      (exit (:exit-code (ex-data e) 1))))
  (exit 0))

Just as we would suspect from the ps output, this is the command line entry point for lein. One of the first things this function does is figure out your project configuration from your project.clj file. Here's ours:

(defproject learn-a-language "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]]
  :main learn-a-language.important-phrases)

The most important line, for our purposes, is

:main learn-a-language.important-phrases

This tells Leiningen what function to execute when you run lein run. If you specify a namespace without a function, as we do above, then Leiningen defaults to using the -main function.

Next, if you look about 2/3 of the way down in the leiningen.core.main/-main function, you'll see the apply-task function being called. This calls resolve-task which eventually resolves to leiningen.run, which you can also see on github.

This is pretty cool — run is just another task from Leiningen's point of view. Wait... did I just say "cool"? I meant MANLY and ARTISTIC and SPARKLY. But yeah, it looks like basic leiningen architecture includes the leiningen.core collection of namespaces, which handle task resolution and application, and plain ol' leiningen, which appears to be mostly a collection of default tasks.

Leiningen uses this same mechanism to execute any function in your Clojure project as a task. Bodacious! I recommend checking out the code for the other Leiningen tasks. You'll see how they do more than just require and run a single function, but the task-running infrastructure remains the same.

Anyway, once the run task has been resolved, it is executed and the result is the second process we saw in the ps | grep lein output above, the process with PID 8432. A sub-process is created to enforce isolation between Leiningen and your project. I won't go into how the command for the sub-process gets constructed, as you can figure that all out from leiningen/run. What's more important is what it actually does. The command loads Clojure and tells it to evaluate the following:

(do
  (try
    (clojure.core/require 'learn-a-language.important-phrases)
    (catch java.io.FileNotFoundException ___6081__auto__))
  (set! *warn-on-reflection* nil)
  (clojure.core/let
      [v__6079__auto__
       (clojure.core/resolve 'learn-a-language.important-phrases/-main)]
    (if
        (clojure.core/ifn? v__6079__auto__)
      (v__6079__auto__ "1")
      (clojure.lang.Reflector/invokeStaticMethod
       "learn-a-language.important-phrases"
       "main"
       (clojure.core/into-array
        [(clojure.core/into-array java.lang.String '("1"))])))))

You can see how the end result is that learn-a-language.important-phrases/-main gets executed with the argument "1". One interesting thing to note about this approach is that the :gen-class directive isn't actually needed. In the manual build at the beginning of this article, we needed :gen-class because we were specifying learn-a-language.important-phrases as the JVM entry point. When we use lein run, the entry point is clojure.main. Once Clojure is loaded, we use it to evaluate some code which loads the learn-a-language.important-phrases namespace and then calls the -main function.

Wrapping Things Up

So now we know how Leiningen compiles and runs a basic Clojure program! Can you feel your mustache growing? Can you feel your artistry blooming? Are you feeling just a smidge more sparklier? I sure hope so!

Here's everything we covered:

Building and running a Clojure program manually:

Load the Clojure repl
Load your Clojure code (make sure it includes :gen-class)
Compile your Clojure code. By default code gets put in classes directory
Run your code, making sure the classpath includes the classes directory and clojure.jar

# load clojure repl
$ java -cp .:clojure.jar clojure.main
# in Clojure repl
user=> (load "learn_a_language/important_phrases")
user=> (compile 'learn-a-language.important-phrases)
# back to command line
java -cp classes:clojure.jar learn_a_language/important_phrases 0

Building and running a Clojure program with Leiningen:

Run lein run
Magic!
- Start Java with clojure.main as the entry point
- Execute the leiningen.core/-main function
  - Setup project config with project.clj
  - Resolve the task to be run
- Run the task
  - Automagically set the classpath
  - Use clojure.main as the Java entry point
  - Construct some Clojure code to evaluate so that the project's main function gets executed. This happens in a sub-process to enforce isolcation from Leiningen. Check out Leiningen's eval-in-project function for more info.

In the next article we'll cover:

How Leiningen avoids waste with a trampoline
How Leiningen manages dependencies
How to distribute a full application
How to distribute a library
Manliness

I hope you enjoyed this article. Please do leave me any feedback - I'm always looking for ways to improve my writing!

How Clojure Babies Are Made: Compiling and Running a Java Program

2013-03-13T04:23:00Z

If you're at all like me, the moment you got your first Clojure program running you belted out, "SOOO MUUUUUUUCH POOOOOOOWEEEEEEEER!" and thrust one or both fists into the air. Then, fifteen minutes later, you found yourself muttering things like "What's a maven?" and "Classpath what now?" and "What is Leiningen actually doing, anyway? Sure, those are the most handsome mustaches I ever laid eyes on, but can I really trust them?"

This post is here to help you out. In it, I will describe how Java compiles and runs programs, and how this relates to Clojure. In an upcoming post, I will also describe how Leiningen makes this process way easier. Here's an outline of what's to come:

Java
- Compilation
  - javac
  - Class lookup rules & classpath
  - imports
  - packages
  - JAR files
- Running
  - Class file
  - classpath
  - Executable JAR
Leiningen
- Running an app without Leiningen
- classpath management
- Incorporating Clojure automatically
- Dependency managemenet

(Update: Part 2 of this series is now available!)

Java

Java is going to rule the universe. The sooner you accept that, the better. If you don't have the JDK installed installed, you might want to do that now. To check, just try running java and javac in your terminal. By the way - this post assumes you're somewhat familiar with the terminal. It also assumes you're familiar with object-oriented programming.

Compiling and Running a Basic Program

This section will lay a foundation for your understanding of Java. It doesn't address Clojure directly, but the knowledge you gain will be useful in your Clojuring.

Let's put our game faces on and start with a basic program. Visit the github repo for the examples in this post and clone that baby for all you're worth.

In the folder ShiverMeTimbers, you should find an article named ShiverMeTimbers.java. Here are its contents:

public class ShiverMeTimbers
{
    public static void main(String[] args)
    {
        System.out.println("Shiver me timbers!!!");
    }
}

Once you've recovered from your swoon, cd to ShiverMeTimbers and run javac ShiverMeTimbers.java. If you typed everything correctly and you're pure of heart, you should now see a file named ShiverMeTimbers.class.

$ ls
ShiverMeTimbers.class ShiverMeTimbers.java

You've just compiled your first Java program, son! Now run it with java ShiverMeTimbers. You should see:

Shiver me timbers!!!

Which I'm pretty sure is like the Lord's prayer, but for pirates. Anyway, what's happening here is you used the Java compiler, javac, to create a Java class file, ShiverMeTimbers.class. This file is packed with oodles of Java bytecode which the Java Virtual Machine executes when running a program.

When you ran java ShiverMeTimbers, the JVM first looked on your classpath for a class named ShiverMeTimbers. In Java, you're only allowed to have one public class per file and the filename and class name must be the same. This is how java knows to try looking in ShiverMeTimbers.class for the ShiverMeTimbers class's bytecode.

Also, by default, the classpath includes the directory ., or the current directory. Try running java -classpath /tmp ShiverMeTimbers and you will get an error, even though ShiverMeTimbers.class is right there in the same directory.

After java found the bytecode for the ShiverMeTimbers class, it executed that class's main method. Java's kind of like C that way, in that whenever you say "run something, and use this class as your entry point", it always will run that class's main method. Which means that that method has to be public, as you can see above. (Here's more info on Java's access modifiers if you're curious.)

Key Ideas

javac compiles java source code to bytecode which the JVM can execute
java searches the classpath for the class you specified and executes its main method. That method must be public.

In the next section you'll learn about handling program code that's spread over more than one file. If you don't remove your socks now, they're liable to get knocked off!

Packages and Imports

In this section, you'll learn about how Java handles programs which are spread over more than one file. You'll also learn how to use Java libraries. Once again, we'll look at both compiling and running a program. This section has direct implications for Clojure programming, so pay attention!

Let's start with a couple definitions:

package: Similar to Clojure's namespaces, packages serve two purposes. They provide code organization, just like clojure namespaces. They also enforce access rules, which we don't really care about. The directory that a Java file lives in must mirror the package it belongs to. If a file has the line package com.shapemaster in it, then it must be located at com/shapemaster somewhere on your classpath. More about classpath later.
import: Java allows you to import classes, which basically means that you can refer to them without using their namespace prefix. So, if you have a class in com.shapemaster named Square, you could write import com.shapemaster.Square; or import com.shapemaster.*; at the top of a .java file so that you can use Square in your code instead of com.shapemaster.Square. Code example below.

Now let's see package and import in action. Here they are as used by the files in make-a-clojure-baby/ContrivedPackageExample. Pay attention to the comments, as they explain a lot of what's going on

Contents of make-a-clojure-baby/ContrivedPackageExample/Conversation.java:

public class Conversation
{    
    public static void main(String[] args)
    {
        // The "ns1" prefix is necessary because ShyGhost belongs to
        // the "ns1" package, and we haven't imported the classes in
        // that package
        ns1.ShyGhost shyGhost = new ns1.ShyGhost();
        shyGhost.talk();
    }
}

Contents of make-a-clojure-baby/ContrivedPackageExample/ns1/ShyGhost.java:

// The classes defined in this file belong to the "ns1" package.
// Notice that this file is in the "ns1" directory.
package ns1;

// This basically means, "I hate typing the namespace prefix all the
// time so please allow me to not do that"
import ns2.*;

public class ShyGhost
{
    public void talk() {
        // the shy ghost can't speak for himself and has to get
        // someone else to do it for him

        // Notice that even though SuperManlyIguana belongs to the ns2
        // namespace, we don't have to use the ns2 prefix
        SuperManlyIguana smi = new SuperManlyIguana();
        smi.talk();
    }
}

Contents of make-a-clojure-baby/ContrivedPackageExample/ns2/SuperManlyIguana.java:

// The classes defined in this file belong to the "ns2" package
package ns2;

public class SuperManlyIguana
{
    public void talk()
    {
        System.out.println("Why hello there");
    }
}

You can run all the above code with the following:

cd make-a-clojure-baby/ContrivedPackageExample
javac Conversation.java
java Conversation

Can you guess what this outputs? I bet you can!

Anyway, so far we've established the relationship between importing, packages, and directory structure: Packages organize code and require a matching directory structure. Importing classes allows you to "de-namespace" them.

One piece that's missing, which I alluded to above, is the role of the classpath. Try the following:

cd make-a-clojure-baby/ContrivedPackageExample/ns1
javac ../Conversation.java

Boom! The Java compiler just told you to hang your head in shame, and maybe weep a little:

../Conversation.java:13: error: package ns1 does not exist
        ns1.ShyGhost shyGhost = new ns1.ShyGhost();
           ^
../Conversation.java:13: error: package ns1 does not exist
        ns1.ShyGhost shyGhost = new ns1.ShyGhost();
                                       ^

It thinks that the ns1 package doesn't exist. But that's stupid, right? I mean, you're in the ns1 directory and everything!

What's happening here is that the java compiler is looking for ./ns1/ShyGhost which it can't find because you're already in the ns1 directory. This is because the default classpath includes '.'. Without changing directories, try running javac -classpath ../ ../Conversation.java

Et voila! It works! So let's amend our understanding of the relationship between importing, packages, and the directory structures: when you're compiling a Java program, Java searches your classpath for packages.

Guess what: the same things happens when you're running a Java program, too. Run the following:

cd make-a-clojure-baby/ContrivedPackageExample
mkdir hide
mv ns1 hide
java Conversation

Another explosion! Now try:

java -classpath .:hide Converstaion

Success!

I hope this clarifies the relationship between your directory structure, the classpath, packages, and importing.

Key Ideas

Packages provide a way for you to organize classes. Your directory structure must reflect your package names
Imports allow you to be lazy and not prefix a class name with its package
javac and java search the classpath for packages

JAR Files

JAR, or Java ARchive, files allow you to bundle all your .class files into one single file. Run the following:

cd make-a-clojure-baby/ContrivedPackageExample
jar cvfe contrived.jar Conversation *.class ns*/*.class
java -jar contrived.jar

It works, just like before. You bundled all the class files into contrived.jar with the file patterns *.class and ns*/*.class. You also indicated that the Conversation class is the "entry point" with the e flag. The "entry point" is the class which contains the main method which should be executed when the JAR as a whole is run.

Behind the scenes, java knows which class is the entry point because when you create a jar, the file META-INF/MANIFEST.MF automaticaally gets generated and added. When you add the e flag, the following line gets added:

Main-Class: Conversation

By the way, you might be wondering why Java isn't throwing any exceptions like "can't find package". The reason is that the JAR file maintains the directory structure. You can see its contents with:

jar tf contrived.jar

You'll see that the directory structure is maintained.

One final point: a JAR file is really just a Zip file with a ".jar" extension. You can treat it just the same as any other zip file.

Pulling it All Together: clojure.jar

Let's pull all of this together with some Clojure! Download the 1.5.1 stable release and unzip it. Then cd to the directory that gets created and run

java -jar clojure-1.5.1.jar

You should get that most soothing of sights, the Clojure REPL. So, how did it actually start up? Let's have a look at META-INF/MANIFEST.MF in the jar file:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: hudson
Build-Jdk: 1.6.0_20
Main-Class: clojure.main

It looks like clojure.main is specified as the entry point. Where does this class come from? Well, have a look at clojure/main.java on github:

/**
 *   Copyright (c) Rich Hickey. All rights reserved.
 *   The use and distribution terms for this software are covered by the
 *   Eclipse Public License 1.0 (http://opensource.org/licenses/eclipse-1.0.php)
 *   which can be found in the file epl-v10.html at the root of this distribution.
 *   By using this software in any fashion, you are agreeing to be bound by
 *   the terms of this license.
 *   You must not remove this notice, or any other, from this software.
 **/

package clojure;

import clojure.lang.Symbol;
import clojure.lang.Var;
import clojure.lang.RT;

public class main{

final static private Symbol CLOJURE_MAIN = Symbol.intern("clojure.main");
final static private Var REQUIRE = RT.var("clojure.core", "require");
final static private Var LEGACY_REPL = RT.var("clojure.main", "legacy-repl");
final static private Var LEGACY_SCRIPT = RT.var("clojure.main", "legacy-script");
final static private Var MAIN = RT.var("clojure.main", "main");

public static void legacy_repl(String[] args) {
    REQUIRE.invoke(CLOJURE_MAIN);
    LEGACY_REPL.invoke(RT.seq(args));
}

public static void legacy_script(String[] args) {
    REQUIRE.invoke(CLOJURE_MAIN);
    LEGACY_SCRIPT.invoke(RT.seq(args));
}

public static void main(String[] args) {
    REQUIRE.invoke(CLOJURE_MAIN);
    MAIN.applyTo(RT.seq(args));
}
}

As you can see, the class main is defined. It belongs to the package clojure and defines a public static main method. This is all Java needs to run a program.

I hope this has helped clarify Java and how it relates to Clojure! In my next post, I'll dig in to Leiningen. Fun for everybody!!!

Notes From Tapestry 2013

2013-02-28T05:00:00Z

Yesterday I had the immense pleasure of joining about 100 other designers, programmers, educators, data scientists, journalists and storytellers in a day-long conference "designed to advance interactive online storytelling with data."

Below is a list of links to many of the presentation examples followed by notes on the presentations. You might like the way it's presented on github better - my blog isn't ideal for this kind of content.

If you'd like to make any contributions or corrections, please leave a comment, tweet me @nonrecursive, or submit a pull request :)

Resources

13 point, the studio of Jonathan corum, an information designer and science graphics editor at The New York Times.
Long jump olympic shadow video
nolan reimold strike zone pitches video (I can't find this one)
Flea power - how flees jump
512 paths to the white house
where 50k guns in chicago came from
A Chicago Divided by Killings A New York Times analysis of homicides and census data in Chicago compared areas near murders to those that were not. Residents living near homicides in the last 12 years were much more likely to be black, earn less money and lack a college degree.
The Guantanamo Docket: A History of the Detainee Population
Methadone and the Politics of Pain
Glamour Beasts: The dark side of elephant captivity
Interactive: The tragic family tree of Thonglaw one of the interactive storytelling pieces associated with Glamour Beasts above
Driving Shifts Into Reverse unconventional mapping of time as a function of the x and y axes
Driving Safety, in Fits and Starts same ideas as above
Snow Fall: The Avalanche at Tunnel Creek WOW
The Invention of Hugo Cabret the book that inspired "Snow Fall"
"twenty eleven" no longer sure what this refers to
"with olga" not sure what this refers to either
Chris Ware comic artist
Stephen Few leading thinker in data viz
Monstrous Costs, Nigel Holmes controversial (?) chart
US job loss bar chart, bush administration vs. obama administration clever to make the bars point downward - association downward with worse
the jobless rate for people like you
the grimace project - combine facial expressions
The Influencing Machine recommended (?) by Scott McCloud
www.thersa.org
The Infinite Canvas
His Face All Red
Bongcheon-Dong Ghost Creepy-ass Korean horror comic
Bloom Like an Artist
Weathercraft: A Frank Comic

Keynote by Jonathan Corum

When we tell stories we adjust them based on audience - can't do that with a graphic
Have an audience
- He partitions his audience into reader viewer listener user
Focus on the people rather than the mechanism/content delivery vehicle
A colleague think s of designing for bart simpsons vs lisa simpsons
- quick overview (Bart) vs ability to deep dive (Lisa)
three types of people (science graphics)
- high school science student
- busy commuter - how to keep them interested instead of going to their phone, playing a game
- his grandmother - does it pull together as a cohesive visual whole?
whole goal is to design for someone else
- tensions
  - oversimplification vs. overwhelming detail
  - explanation vs. decoration
  - storytelling vs interactivity - narrative vs exploration
- don't be your own audience
show ideas + evidence
- understand, translate, display, explain
- find the central idea. find one idea to use as the basis for your graphic
respect the reader - help them through the story
- allow for multiple entry points - compartmentalized graphical sections
- interactive tools to allow reader to pace themselves through
- use disparate scales to give context. Example, mixing longjump distances with free-throw line
add meaningful annotations
- close proximity between graphics and labels
- don't make people go back and forth between graphics and labels
- another way to provide context
- example: annotate each step in a sequence. flea jumping
show change
- motion
- show large scale, small scale what's happening each step
- change in form is another kind of change
reduce complexity and opportunities for confusion
- adding interface can be adding complexity
reduce tedium
- interact with data, not the interface
- strip out tedious activities - usability
visualization is not explanation
- dont let technology drive
- add enough information beyond your visualization to explain a pattern in data
- or structure your visualization to reveal and explain patterns
reveal patterns
- layer multiple data sets
respect the data
- show what's unique about it
if your visualization can apply to something completely different, you might not be telling the unique story. detainees vs cups of tea
edit - throw things away.
- throw as much away as possible but actually tell a story
- If you're spending most of your time editing you know you're on the right track
apply common sense vigorously

Showing is Not Explaining, Pat Hanrahan

trying to explain Euclid's algorithm for Greatest Common Divisor
- algorithm animation / explanation
problems with animation
- motion is fleeting and transient
- cannot simultaneously attend to multiple animations
- ... more
Pat showed the animation of the algorithm, but it didn't really explain how the algorim worked

I had trouble thinking of what to note for this one, would greatly appreciate contributions here.

Choosing the Right Visual Story, Cheryl Phillips

Side note: This seemed mostly aimed at journalists.

What's the story?
- data without a theme is just a bunch of data - not a story
- who what when where why how - oldies but goodies
- interview your data. think of it as the man on the street. keep asking it questions
avoid "notebook dump"
- don't put every last detail in the story
use the nutgraf (theme) to help define a strong visualization
data is more than numbers -- what little stories make up the larger whole which can be visualized?
- example: methadone the politics of pain
- example: family tree of songlaw

29, Nigel Holmes

My sparse notes here don't really do the talk justice, probably because I was enthralled with the presentation. Especially Nigel attempting the long jump - I doubt any of us will forget seeing that!

29 is not interesting in itself, but interesting in context
you understand something when you see it next to something you already something understand
context is the key to understanding

The Art of Honest Theft: Evolution of a connected scatterplot, Hannah Fairfield

How graphics influence each other
if you move away from plotting time against the horizontal you can reveal interesting trends

See Driving Shifts Into Reverse and Driving Safety, in Fits and Starts

what's next?
- one technique: associate ancillary content (animations) with scroll
so that extra information shows up in a way that it's tied to what the reader is reading at that moment (this idea shows up a lot during the conf) * focusing on immersive content * it's important to carve out time, even just 10%, to play

The Why Axis, Bryan Connor

nick felt (?) was inspiration
is a critic on the why axis, but doesn't mean that in a negative way
"the finished piece frequently acts as a seductive screen that distracts us from the higher level of investigation"
move past being psychics into being an investigator: as a critic, move from guessing to asking
once you know the objective of the visualization you're able to judge whether it succeeded or failed
designers: provide retros. be accessible to critics

Visual storytelling in the Age of Data, Robert Kosara

academics don't get the idea of presenting data, communicating data. it's just an afterthought
argues that stylizing charts is quite useful. helps to tell the story. emotional impact?
- example: monstrous data by Nigel Holmes
there's a danger to telling stories
- can lead you down the wrong path
example: driving an electric car in the parking lot until the battery runs down
story telling potential of charts

Story Depth	Tells a Story
Facts	Narrative
Information Scent	Focus
Audience	Author

storytelling affordances (I love the idea of storytelling affordances)
- the form which lends itself to storytelling
- what are they?
  - reading direction, left to right
  - in the famous napoleon chart, the area gets thinner
  - follow along a line, like following a journey on a map uses the driving safety in fits and starts article as example
  - animations
  - direction - the bush admin vs. obama admin us job loss bar chart effective way of walking you through developments
narrative ties facts together
- provides causality
- walks you through a story
facts - story depth
focus - tells a story
- kind of the natural enemy of more data
- you must be selective in presenting data for it to be a story
information scent - story depth
- hints used to guide people, indicate that there's more data. example: the jobless rate for people like you
- present a lot of information, but focus only on one bit. provide other data in a less visually prominent manner
author - tells a story
audience - story depth
"we're at the cusp of something amazing and powerful that goes way beyond what's out there right now"

comics and visual communication, scott mccloud

in a 10" cube box of air is information
- wifi, cell, radio
- until there's something to decode that air, it really is just empty
we both receive meaning and create it on the fly
- the artist gives a hint of life, and the audience will meet them half way
cartoonists
- simplification creating a kind of human calligraphy
- write with pictures
human calligraphy takes many forms
- body language
- facial expressions
there are six primary emotional expressions
- anger
- disgust
- happiness
- surprise
- sadness
- fear
- the six primary combine
  - anger + happiness = cruelty
  - you can paint on the face the variety of emotions
the grimace project
- apparently useful for kids on the autism spectrum
secret ingredient of the look of love is sadness
all pictures are words
all pictures speak
all pictures have something to say
something happens in lower grades - teach kids to write and to draw
- at a certain point we teach kids words can be used for
  - lists
  - poems
  - express themselves
but pictures - we try not to be too specific
this overlooks the fact that pictures have a multiplicity of uses
- can transmit messages, emotions
as a result we have a society that divorces pictures from... specificity?
- we have downstream pictures - advertising
- don't have as much going upstream
comics are a way to send messages upstream - can go out to a mass audience but retain their subjectivity
will eisner believed comics can teach
- the influencing machine
tone is important
- readiness.gov spawned great parodies
now people are empowered to combine words and pictures anyway they want to
- example: historical event facebook pages
we need to be vigilant of cognitive load time
- the speed at which individual parts load is the speed at which we can convey complex info
- the rsanimate series, www.thersa.org
synchronization of data/visuals with content
- "i'm not going to even allow you to think about anything other than what I'm talking about right now"
- order of presentation matters
- "if I don't need to think it, I don't need to see it"
- "the quicker the parts the richer the whole" cognitive load
"form and content must never apologize for each other"
the grammar of comics is putting one picture after another
- creating a flow of time between images
- as we move through space we move through time
- any two images, we'll find a story, we'll find a narrative
- cartoonists are finding the poetry in the gaps between frames
as storytellers, you want your audience to lose themselves in the story
- all narrative art forms are based on extremely simple principles
- in comics: space = time
"each successive technology would appropriate the previous technology as its content"
grumpy old man rant about form factor
- we see the world as rectangles in landscape mode
- what possibilities open up when you consider the monitor to be a window?
- lots of cool examples of comics taking advantage of the possibilities inherent in browsers
responsive web design
- separating content from presentation
- a noble impulse
- but at the same time there are art forms which are fixed. comics is one of them
the form matters!
- in comics, the spatial relationship is part of the artistic intent
when you have violent shifts in the landscape, the only thing you can do is hold on to basic principles
- in comics, people losing themselves in the story

Many thanks to the hosts for putting together a great conference. I'm looking forward to next year's!

Datomic for Five Year Olds

2013-02-23T14:23:00Z

If you're interested in Datomic, you may have been deterred from checking it out because you lack the enthusiasm necessary to slog through 3 keynotes, 8 interviews, and innumerable walls of text in order to get a basic idea of what the hell it is and whether you should use it.

Well, good news! I've done all that for you! How could I resist your winning smile, you charmer you? After you're done with this article, you will have a solid conceptual grasp of the three key ways that Datomic is unique. You will also understand key Datomic terms. This will make it much easier to actually get your hands dirty with Datomic if you decide to investigate it further.

Overview of Datomic's Three Righteous Pillars of Databasing

Datomic differs from existing solutions in its information model, its architecture and its programmability. Below is a quick overview. We'll cover each of these righteous pillars of databasing in much more detail.

	Relational DB	Schemaless DB	Datomic
Information Model	The unit of information is an entity in a relation and you alter it in place, forgetting previous values	The unit of information is a schemaless document and you update it in place, forgetting previous values	The unit of information is a fact comprised of an entity, attributes, values, and time. You accrete the assertion and retraction of facts instead of updating in place.
Architecture	Database is a monolithic system responsible for querying, ACID compliance, and storage	Same as RDBMS	Separate querying, transacting, and storage in such a way that you get ACID compliance, read scalability, and more power within the application using Datomic
Programmability	Applications abstract SQL by performing string manipulation	Interact with proprietary data structures which are more programming-friendly than SQL but less powerful	Datalog, completely uses data structures and has as much power as SQL

First Righteous Pillar of Databasing: Information Model

A database's information model is defined by its rules regarding the way entities and attributes relate to each other — for lack of a better term, its schema system.

Yea, the choice between a relational and schemaless database is probably your primary concern in choosing a database because of its influence on how you write and grow your programs. You'll see that Datomic's schema system isn't as rigid as the relational model but affords more power than a schemaless database.

Additionally, a database's information model is defined by its approach to time. Datomic's approach to time is different from most, if not all, existing databases.

Below are the schema systems for relational dbs, schemaless dbs, and datomic, followed by a comparison of the way the information models handle time.

Relational Schemas

You're probably already familiar with the relational model. Here are key definitions:

Entity	An entity is tuple within a relation. It is comprised of a fixed set of attributes. In practice, an entity is a row in a table.
Attribute	A name + a data type. A column in a table. Attributes do not exist outside of relations. Attributes in different relations are always logically distinct, even if they hold the same type of data.

Probably the most important fact about the relational model is that it's rigid. Every entity must belong to a rigid relation. There's no way to add or remove attributes except by altering the structure of the relation, which in turn alters every other entity in that relation.

Schemaless

Schemaless (NoSQL) databases were created to address the rigidity of relational databases. They offer a few facilities for organization - for example, collections in MongoDB - but forego any structure for entities. This lack of structure comes at a cost: it limits your query power and forces data integrity concerns into your application.

Entity	For our purposes, a document. A document has no restrictions on what attributes it can have.
Attribute	A name. Attributes can hold values of any type, and an attribute belonging to one entity is logically distinct from all other attributes, even those belonging to other entities within the same collection. Relationships between attributes are enforced within the client application.

Datomic Schemas

In Datomic, a schema defines a core set of attributes which effectively act as data types. An entity can possess any attribute without restriction. In this way, entities have more structure than a schemaless database but are more flexible than in relational databases. Additionally, you retain the query power of the relational model without having to handle data integrity concerns in your client applications.

Entity	A map of attribute/value pairs. Entities have no fixed shape; they can be comprised of any attributes defined in the schema.
Attribute	Name + data type + cardinality. Attributes themselves can be thought of as data types. They differ from attributes in the relational model in that they exist independently of rigid tables

Time

Datomic's notion of time might be unfamiliar for those unacquainted with Rich Hickey's thoughts on time and how it relates to identity, value, and state. To get up to speed, check out my article, "The Unofficial Guide to Rich Hickey's Brain"

In both relational and schemaless databases, there's no built-in notion of time. It's always "now" - the database doesn't hold on to the past. Whenever you update or delete a row, you have no way of retrieving previous states for that row.

This is the source of many headaches for us. If you've ever used or any code that requires multiple trips to the database in order to perform one unit of work, you know what I mean. In between each trip, you have no way of knowing whether the database has been been altered. You can end up writing all kinds of hacks to avoid race conditions and still not end up with a bulletproof system.

In addition, it's virtually impossible to query past states. You might not think this is really matters because you're used to the limitation, but being able to easily query past states is very powerful.

In Datomic, time is a first-class concept. All changes to the database are associated with a transaction, and each transaction has a timestamp. Rather than updating an entity in place and "forgetting" its previous state, datomic "accretes" states. In this way, you're able to retrieve all previous states for an entity.

Another way of thinking about it is that the database is an identity which we superimpose on a time-ordered collection of values. In this line of thinking, a "value" is the database as whole - its collection of entities and attributes. When you run a transaction which creates, updates, or deletes entities, you create a new database value. You're always able to say, "I want to work with the value of the database at this point in time."

Datomic's Information Model Summarized

In Datomic, the unit of information is a fact. A fact is comprised of an entity, an attribute, a value, and time. If you were to say "The princess", that would not be a fact. If you were to say, "The princess's favorite condiment", that would not be a fact. A fact would be "The princess's favorite condiment is mustard," that would be very close to a fact. "The princess's favorite condiment is mustard as of 10pm on February 10th." is a fact.

Facts don't go away. If the princess's tastes change so that she prefers sriracha, it's still useful to know that in the past she preferred mustard. More importantly, new facts don't obliterate old facts. In Datomic, all prior facts are available. Also, in Datomic, facts are called "datoms" and not "facts" because it's more exciting that way.

For more, check out The Datomic Information Model, an article by Datomic's creator, Rich Hickey.

Second Righteous Pillar of Databasing: Architecture

Existing databases couple the following into one monolithic product:

Reading
Writing
Storage

Datomic decouples these capabilities. All writing is handled by a single "transactor", ensuring ACID compliance. This is the only point of similarity to existing solutions.

Side note: the diagrams at the Datomic Architectural Overview page are quite awful. Besides being ugly, it is difficult to tell what the visual components signify. What do the arrows mean? What does the dashed border mean? WTF is "Comm"? Please, please fix these. OK, rant over.

Your application communicates with datomic through the Peer Library, a component which exists within your applicaiton. Using the Peer Library makes your application a Peer. I think the Datomic guys were trying to avoid the term "client" here, but a Peer is essentially a client. What makes a Peer diffeerent from the usual database client, however, is that querying happens primarily in the Peer. By the way, when I say "querying" I mean "read querying."

Instead of sending a query "over there" to the database server, which runs the query and gives the client results, the Peer Library is responsible for pulling the "value of the database" into the Peer so that querying happens within your actual application. If you were to create a script out of this action-packed drama, it would look like:

Peer: Peer Library! Find me all redheads who know how to play the ukelele!
Peer Library: Yes yes! Right away sir! Database - give me your data!
(Peer Library retrieves the database and performs query on it)
Peer Library: Here are all ukelele-playing redheads!
Peer: Hooraaaaaaaay!

The most novel thing here is "retrieves the database". In Datomic, you have the concept of the "value of the database" - the database as a whole, as a data structure. The Peer Library is responsible for pulling as much of that data structure into memory as necessary to satisfy your queries.

In this way, your queries run largely in your application rather than in a central database server, allowing for easy read scalability. It gives more power to your application rather than keeping all the power in a central server, where you have to worry about bottlenecks.

Finally, Datomic does not implement its own storage solution but instead relies on storage as a service, allowing you to choose which storage backend you use. Right now, the following backends are supported:

The filesystem
DynamoDB
Riak
Couchbase
Infinispan
SQL database

Third Righteous Pillar of Databasing: Programmability

In Datomic, everything is data! Your schema is data! Queries are data! Transactions are data! This is great because it's easier to manipulate data structures than it is to perform crazy string concatenation like you do with SQL. Check out these example queries.

Resources

The End

That's it for now! If you have any suggestions or corrections please let me know.

The Unofficial Guide to Rich Hickey's Brain

2012-12-06T05:00:00Z

Part of my excitement in learning Clojure has been being exposed to Rich Hickey's thoughts on programming. Rich Hickey has a clear, consistent way of viewing fundamental programming concepts that I think any programmer would benefit from. Every time I watch one of his talks, I feel like someone has gone in and organized my brain.

In this article (and more to come (possibly)), I begin my attempt to catalog Mr. Hickey's unique viewpoint. Eventually, I would like to produce a concise summary of his ideas. My hope is that this will provide an easily-scannable reference to Clojurists and an accessible introduction to non-Clojurists.

What follows is derived from Rich Hickey's talk, "Are we there yet?"

Introduction

Today's OOP languages - Ruby, Java, Python, etc. - are fundamentally flawed. They introduce accidental complexity by building on an inaccurate model of reality. Where they have explicit definitions for the following concepts, the definitions are wrong:

Value
State
Identity
Time
Behavior

Below, we'll contrast the OOP viewpoint on each of these topics with the Functional Programming viewpoint. But first, we'll compare the models of reality which underly OOP and FP. It's this underlying difference which gives rise to their different approach to the topics above.

Metaphysics, Programming, and You: Comparing OOP and FP

When talking about metaphysics things tend to get a little fuzzy, but hopefully this will all make sense.

As the ever-trusty Wikipedia explains, metaphysics attempts to answer two basic questions in the broadest possible terms:

What is there?
What is it like?

Rich Hickey explains the difference between OOP and FP metaphysics by contrasting their definitions of what a river is.

Object Oriented Programming

In OOP metaphysics, a river is something which actually exists in the world. I know, I know, I can hear what you're saying: "Uh... yeah? So?" But believe me, the accuracy of that statement has caused many a sleepless night for philosophers.

The wrinkle is that the river is always changing. Its water never ceases to flow. In OOP terms, we would say that it has mutable state, and that its state is ever fluctuating.

The fact that the state of the River Object and that Objects in general are never stable doesn't stop us from nevertheless treating them as the fundamental building blocks of programs. In fact, this is seen as an advantage of OOP - it doesn't matter how the state changes, you can still interact with a stable interface and all will work as it should. An object can change, but to all observers it is still considered the same object.

This conforms to our intuitive sense of the world. The positions of the electrons of the coffee in my mug matters naught; the coffee still interacts with my taste buds in the way I expect.

Finally, in OOP, objects do things. They act on each other. Again, this conforms to our intuitive sense of the world where change is the result of objects acting upon each other. A Person object pushes on a Door object and enters a House object.

Functional Programming

In FP metaphysics, we would say that we never step in the same river twice. What we see as a discrete thing which actually exists in the world independent of its mutations is in reality a succession of discrete, unchanging things.

The "river" is not a thing in and of itself; it's a concept that we superimpose on a succession of related phenomena. This concept is very useful - I won't belabor that point - but it is just a concept.

What really exists are atoms (in the sense of atomic, unchanging, stable entities) and processes. The river is not a stable object; rather, it is a succession of related atoms which are generated by some kind of process.

These atoms don't act upon each other and they can't be changed. They can't do anything. Change is not the result of one object acting on another; change is the result of a process being applied to an unchangeable atom. To say that something has changed is to say, "Oh, there's a new atom in this stream of atoms." It's like saying that HEAD points to a new commit in your git repo.

OK! Enough with metaphysics. Now let's describe the more immediately useful topics, starting with Value.

Value

It's obvious that numbers like 3 and 6 and 42 are values. Numbers are stable, unchanging.

It should also be obvious that OO languages have "no proper notion" of values in this sense. As Rich Hickey points out, you can create a class whose instances are composed of immutable components, but there is no high-level concept of immutable value implemented as a first class construct within the class.

This is one of the main causes of headaches when doing OOP. How many times have you pulled your hair out trying to figure out how an object's attribute got changed? The fact is, in OO languages there is no built in mechanism to ensure that the object you're dealing with is stable.

This is the big reason why concurrent programming is so difficult. Even in single-threaded programs this is a problem, and it's one of the reasons why we develop sprawling test suites. You can't be sure if a method call on your Toupee object is somehow going to cause a change in your HipsterGlasses object.

By contrast, in FP languages emphasis is placed on working with immutable values. Since these values can't change, a whole class of problems simply disappears.

Identity

In the video, Mr. Hickey says:

The biggest problem we have is we've conflated two things. We've said the idea that I attach to this thing that lasts over time is the thing that lasts over time.

In FP, identity is essentially a name we give to a sequence of related atoms. "River" refers to the sequence R1, R2, R3, etc, produced by the river process. This is directly analogous to HEAD in git - it's just a name which is used to refer to actual values. In OO, there really isn't such a distinction.

Or as the man himself says,

Identity is a putative entity we associate with a series of causally related values (states) over time. It's a label, a construct we use to collect a time series.

State

In OO, there is no real clear definition of state. Maybe it's, "the values of all the attributes within an object right now." And it has to be "right now", because there's no language-supported way of holding on to the past.

This becomes clearer if you contrast it with the notion of identity in FP. In the Hickeysian universe, a State is a specific value for an identity at a point in time. (For me, this definition really clarified my own thoughts.)

Time

There's no real notion of time in OO. Everything's just "right now". This is part of what causes problems in concurrent programs.

By contrast, in the FP worldview we've been exploring, time is well-defined. Time is a by-product of ordering states within an identity.

(By the way, I'd really like to write more here, and would appreciate any suggestions.)

Behavior

Finally, behavior. I don't have too much to write here - I might need to watch some more talks - but I'll include Mr. Hickey's thought-provoking observation:

There is no behavior. When you get hit by lightning, who's behaving?

This may be what inspired Steve Yegge's post, Execution in the Kingdom of Nouns.

The End

Well, that's it for now. I hope you've found this post useful. If you have any suggestions, I'd love to hear them!

My Failed(?) Social Web Experiment Still Haunts Me

2012-12-01T05:00:00Z

What will people post if they can't include links and can't use a nickname?

This was the question I was looking to answer back in 2007 I launched nobodynotes.com. It was a very simple web site, consisting mainly of a text field and all the posts which people had entered in the text field. You can see it, unstyled, thanks to the wayback machine.

Initially, the posts were interesting to me, but probably unremarkable to anyone else. Bits of poetry, parts of song, random curses. But then something unexpected happened. More and more of the posts were written in Persian.

Persian!? How did that happen? When I looked at Google Analytics, it showed that the majority of my visits were from Iran, and that most of my visitors were referred from Yahoo Mail. This was not something that I had anticipated. It seemed like the site was going viral in Iran, with people emailing it to each other specifically in Yahoo Mail. Going through the site, I noticed that some of the posts referenced others; each post had a number, and visitors would specify the number of the post they were replying to.

But why? Why all this activity from Iranians? I wanted to know, but sadly had no direct way of finding out. After all, everyone was anonymous. Not knowing what else, to do, I tried using Google Translate to see what people were saying. One post in particular stood out. It's probably irretrievable now, but a young man essentially wrote, "Help, I am a gay man living in Iran. I cannot tell anyone, or I will be killed. Please, someone help me."

To this day, that post haunts me. Who was this young man? Was he able to find help? Where is he today, and is he OK?

I doubt I'll ever know the answer. I eventually shut down the site as traffic slowed to a trickle. But pieces of it remain. It was connected to a twitter account, where you can still read fragments of those posts by anonymous Iranians. And there's the wayback machine, which offers larger slices of conversation. And there's Google Translate, offering me its demented assistance in trying to understand what was happening a world away:

Staring at the horizon, leaving the logo side of life, rectified love sitting in contemplation, madness, stood an innocent face, a tired face , relying on the fact, sad face smiling at what an idea! that madness pounding deep into each side. .

A Detailed Look at a Small DCI Refactoring in Ruby

2012-11-30T15:12:00Z

UPDATE: this isn’t actually DCI! Whoops!

In this post I go over a small refactoring to clean up some code in Whoops by implementing the DCI pattern. I’ll cover the actual code changes and include my usual hand-wringing about what could be done better.

This refactoring was inspired by Jim Gay’s book Clean Ruby. Jim Does an excellent job of concisely explaining what the DCI pattern (data, context, interaction) is, why it’s useful, and how you can implement it in Ruby. This post isn’t a review of Clean Ruby, but I will say that it’s worth the money – it’s actually rekindled my enthusiasm for Ruby!

Background

Whoops is a free, open-source Rails engine for logging. It’s similar to Airbrake, Errbit, and Exceptional except that it’s not limited to exceptions – you can log any event.

The Original Code and Its Defects

The code that got refactored is concerned with processing new events. You can see the entry point to this use case at lines 18-28 here:

whoops-dci-refactoring/event.rb

class Whoops::Event
  include Mongoid::Document
  include FieldNames
  
  belongs_to :event_group, :class_name => "Whoops::EventGroup", :index=>true
  
  field :details
  field :keywords, :type => String
  field :message, :type => String
  field :event_time, :type => DateTime

  index([[:event_group_id,Mongo::ASCENDING],[:event_time, Mongo::DESCENDING]])

  validates_presence_of :message  
  
  before_save :set_keywords, :sanitize_details
  
  def self.record(params)
    params = params.with_indifferent_access
    
    event_group_params                    = params.slice(*Whoops::EventGroup.field_names)
    event_group_params[:last_recorded_at] = params[:event_time]
    event_group_params
    event_group = Whoops::EventGroup.handle_new_event(event_group_params)
    
    event_params = params.slice(*Whoops::Event.field_names)
    event_group.events.create(event_params)
  end 
  
  def self.search(query)
    conditions = Whoops::MongoidSearchParser.new(query).conditions
    where(conditions)
  end
  
  def set_keywords
    keywords_array = []
    keywords_array << self.message
    add_details_to_keywords(keywords_array)
    self.keywords = keywords_array.join(" ")
  end
  
  def sanitize_details
    if details.is_a? Hash
      sanitized_details = {}
      details.each do |key, value|
        if key =~ /\./
          key = key.gsub(/\./, "_")
        end
        
        if value.is_a? Hash
          child_keys = all_keys([value])
          if child_keys.any?{ |child_key| child_key =~ /\./ } 
            value = value.to_s
          end
        end
        
        sanitized_details[key] = value
      end
      
      self.details = sanitized_details
    end
  end
  
  def all_keys(values)
    keys = []
    values.each do |value|
      if value.is_a? Hash
        keys |= value.keys
        keys |= all_keys(value.values)
      end
    end
    keys
  end
    
  private

  
  def add_details_to_keywords(keywords_array)
    flattened = details.to_a.flatten
    flattened -= details.keys if details.respond_to?(:keys)
    
    until flattened.empty?
      non_hash = flattened.select{ |i| !i.is_a?(Hash) }
      keywords_array.replace(keywords_array | non_hash)
      flattened -= non_hash
      
      flattened.collect! do |i|
        i.to_a.flatten - i.keys
      end.flatten!
    end
  end
end

The above record method calls handle_new_event in Whoops::EventGroup (lines 21-41):

whoops-dci-refactoring/event_group.rb

class Whoops::EventGroup
  # notifier responsible for creating identifier from notice details
  include Mongoid::Document
  include FieldNames
  
  [
    :service,
    :environment,
    :event_type,
    :message,
    :event_group_identifier,
    :logging_strategy_name
  ].each do |string_field|
    field string_field, :type => String
  end
  
  field :last_recorded_at, :type => DateTime
  field :archived, :type => Boolean, :default => false
  field :event_count, :type => Integer, :default => 0

  class << self
    def handle_new_event(params)
      identifying_params = params.slice(*Whoops::EventGroup.identifying_fields)
      event_group = Whoops::EventGroup.first(:conditions => identifying_params)
      
      if event_group
        event_group.attributes = params
      else
        event_group = Whoops::EventGroup.new(params)
      end
      
      if event_group.valid?
        event_group.send_notifications
        event_group.archived = false
        event_group.event_count += 1
        event_group.save
      end

      event_group
    end
  end
  
  has_many :events, :class_name => "Whoops::Event"
  
  validates_presence_of :event_group_identifier, :event_type, :service, :message
  
  def self.identifying_fields
    field_names - ["message", "last_recorded_at"]
  end
  
  # @return sorted set of all applicable namespaces
  def self.services
    all.distinct(:service).sort
  end

  def should_send_notifications?
    (archived || new_record) && Rails.application.config.whoops_sender
  end
  
  def send_notifications
    return unless should_send_notifications?
    matcher = Whoops::NotificationSubscription::Matcher.new(self)
    Whoops::NotificationMailer.event_notification(self, matcher.matching_emails).deliver unless matcher.matching_emails.empty?
  end
end

I’ve included the full source of the files to show why this code doesn’t belong. See, one of the main ideas in Jim’s book is that code becomes unmaintainable when we start overburdening classes with methods that are only needed in specific contexts, or use cases. Additionally, we scatter all the code needed in one context across multiple classes and files rather than keeping it all in one place, creating a cognitive burden.

Violating The Single Responsibility Principle

In this case, the context is “record an event.” The code we’ve added to the Whoops::Event class begins to overburden the class by giving it a new responsibility and requiring that it “know” more about the outside world. Here’s what the class is already responsible for, along with what we’re adding:

CRUD events
Massage event data before persistence
NEW Prepare event group data (lines 21-23)
NEW Initiate event group handling of new event (line 24)
NEW Figure out how to separate event-relevant data from event params (line 27)

In this case, the impact isn’t that high because we still don’t even have 100 LOC for the entire class. But by continuing to shove code into a class just because a class’s data or existing behavior is somehow involved, we transform our classes from usable, focused abstractions into code warehouses. And I really do mean warehouse. You have to start using signage in the form of comments or mixins to find your way around. “Aaah yes, here we are, the email notification section!” or “Oh shit! it looks like someone put the methods for handling user registration at different ends of the file. Let’s put those two guys together. There, what a tidy warehouse!”

The Whoops::EventGroup class was becoming a code warehouse. Here are the new responsibilities brought on by handle_new_event

CRUD event groups
validate event groups
NEW send a notification on new events
NEW figure out whether a notification should be sent (lines 56-58)

This notification code is most definitely not the responsibility of a Whoops::EventGroup. But here it is, because hey, fat model skinny controller.

Spreading Out Related Code

The above two files also show how related code gets spread out. In this case, to understand the full behavior of “handle new event” you have to start in the Whoops::Event file, then go to the Whoops::EventGroup file, then return back to the Whoops::Event file.

Burdening Your Mind Grapes

(For more on mind grapes, see #6 in this cracked article)

As anyone who’s dealt with this kind of code can attest, organizing your code this way adds cognitive load. By violating the single responsibility principle, we make it more difficult for ourselves to find the methods we’re interested in. The class’s name starts to lose its meaning as its conceptual category becomes broader and broader until it becomes useless as an organizational unit. It’s like trying to find the latest teen vampire romance novel at a bookstore only to find it in the computer section because the sexy, misunderstood male vampire love interest has an iphone.

And everyone knows that having to navigate multiple files in order to understand one use case is a huge pain in the ass. I mean, that’s pretty much why Chris Granger is making Light Table, right?

The Refactoring

This github diff shows all the changes to DCI-ify the code.

As you can see, our Whoops::Event and Whoops::EventGroup classes are no longer violating the single responsibility principle. Now when you dig into those classes you no longer have to wade through code that’s only tangentially related to the real purpose of those closses. We’re no longer burdening our classes with knowledge that they shouldn’t have, reducing coupling.

Additionally, all of the code related to handling a new event is now in one place, the Whoops::NewEvent class. This makes it much easier to understand completely everything that happens when a new event is handled. Rejoice!

I’m not sure how much more to say about the benefits of this refactoring. It feels a lot nicer to me, and it un-does the problems that existed earlier. Hopefully that’s evident, but if it’s not then please leave a comment or tweet me!

The Promised Hand-Wringing

There are still some things that I’m not completely happy with in Whoops::NewEvent.

First, it basically looks one gigantic method that’s been split up into tiny methods just so that I could name groups of related code. I don’t know that there’s much value in that.

Second, lines 13-15 look like they need some sort of explanation. It’s especially not obvious why I’m setting archived to false.

Last, I’m not completely happy with the name NewEvent. If you were new to the project you’d probably ask yourself, “Uh… how is that not Whoops::Event.new?” I also considered “Whoops::NewEventHandler” but I was afraid that would summon Steve Yegge and he’d make me read one of his 10,000 word essays.

I’d love to get some thoughts on this! Whether or not this is a reasonable DCI refactoring, I think that DCI is a great tool and something OOP folks should investigate.

Look, Ma! No batteries! A Clojure Web App Without Noir

2012-11-10T18:23:00Z

Awhile back I released OMG! SMACKDOWN!!!, which still holds the title for dumbest Clojure app in existence. To make up for that ludicrostrosity, I’ve created Grateful Place. This time I around I didn’t use Noir, instead going closer to the metal with Compojure and Enlive. The experience was a lot easier than I anticipated, perhaps easier than using Noir. This post will cover some of the more interesting bits of code:

Templating heaven with Enlive and Middleman
Using macros to enforce full stack consistency
Roll-your-own validations
Pagination is more fun with macros
Using fierce mustaches to run tasks
Fun utilities

You can find all source code on
github. Also, I’m going to be at the Clojure Conj and would love do meet folks. If you’d like to chat, please do tweet me (@nonrecursive) or find me at the conj!

App Overview

Grateful Place is a social app with only a few models: Users, Comments, Posts, and Likes. Nothing too fancy. If you’re interested in why I made it, here’s a post I wrote about gratitude.

Templating heaven with Enlive and Middleman

Enlive is the bee’s knees

I can say without reservation that Enlive is the best templating library I’ve ever used. I absolutely love it! If you try out one thing from this post, try out the workflow I describe here. Enlive is completely different from Hiccup, ERB, HAML and the like and it’s worth giving it a shot.

If you’re not familiar with Enlive, it’s a selector-based templating library. In the same way that CSS allows you to separate structure and presentation, Enlive lets you separate structure and view logic. This is huge! Unless you’re the kind of person who still likes to use font and bold tags in your HTML, you should give Enlive a try.

Middleman/Enlive Workflow

Using Enlive allowed me to do all my design work using the Ruby library Middleman. This allowed me to build stand-alone web pages so that I could see how they looked without worrying about any view logic. For example, here’s the example home page I put together:

clojure/gratefulplace-dissected/v1/assets/generator/source/index.html.haml

---
title: Grateful Place
---
#posts
  .post
    .content
      %p Here is some content.
    %footer
      .date Jan 03, 2012
      .author
        by
        %a{:href => ""} Joe Schmoe
      .favorite
        %a{:href => "#"} &#9733;
        .status Like this
      %a.comments{:href => ""} 35 Comments
      

  .post
    .content
      %p Here is some content, too.
    %footer
      .date Jan 03, 2012
      .author
        by
        %a{:href => ""} Joe Schmoe
      .favorite
        %a{:href => "#"} &#9733;
        .status Like this
      %a.comments{:href => ""} 35 Comments

  .pagination
    %a.previous{:href => "#"}
      Previous  
    %span.page-link.current
      1
    %a.page-link{:href => "#"}
      2
    %a.page-link{:href => "#"}
      3
    %a.next{:href => "#"}
      Next

You might have noticed a few weird things. First, this is written using HAML. Yep, HAML. I prefer HAML over HTML because it’s more concise and the consistent structure allows me to visually scan it more easily. And since my HTML templates and view logic are completely separate, I can write my designs in HAML, preview them with Middleman, and then generate an HTML file for Enlive to work with. Here’s my process in full:

Build on every change by cd’ing to assets/generator/ and running fswatch . ./g. This runs a bash script which generates the HTML, CSS, and Javascript files and moves them to the proper directories.
Start up Middleman by cd’ing to assets/generator/ and running middleman
View site at http://localhost:4567 and make changes to my heart’s content
When I’m happy with the changes, create or recompile a Clojure file which makes use of the HTML file with Enlive. It’s necessary to recompile the files so that Enlive uses the updated HTML. That doesn’t happen automatically.

Another thing you might have noticed is that in the HAML file I’m showing two example posts. With Enlive, you actually have to add some logic to remove one of the post divs:

clojure/gratefulplace-dissected/v1/src/gratefulplace/views/posts.clj

(defpage all "index.html"
  [posts current-auth record-count page max-pages]
  ;; don't show the second post as it's just an example
  [[:.post (h/nth-of-type 2)]] nil

Some code to reduce boilerplate

Finally, here’s some code I put together to reduce boilerplate:

clojure/gratefulplace-dissected/v1/src/gratefulplace/views/common.clj

(defonce template-dir "gratefulplace/templates/")

(h/defsnippet nav (str template-dir "index.html") [:nav]
  [logged-in]
  [:.auth] (if logged-in
             (h/do-> (h/content "Log Out")
                     (h/set-attr :href "/logout"))
             (h/do-> (h/content "Log In")
                     (h/set-attr :href "/login"))))

(h/deftemplate layout (str template-dir "index.html")
  [html]
  [:html] (h/substitute html)
  [:nav] (h/substitute (nav (current-authentication)))

  [:nav :ul.secondary :#logged-in :a]
  (if-let [username (:username (current-authentication))]
    (h/do->
     (h/content username)
     (h/set-attr :href (str "/users/" username))))
  
  [:nav :ul.secondary :#logged-in :span]
  (when (current-authentication)
    (h/content "Logged in as")))

;; Need to come up with better name
;; Bundles together some defsnippet commonalities for user with the
;; layout template
;;
;; TODO destructuring doesn't work in argnames
(defmacro defpage
  [name file [& argnames] & body]
  `(do
     (h/defsnippet ~(symbol (str name "*")) (str template-dir ~file) [:html]
       [~@argnames]
       ~@body)
     (defn ~name
       [{:keys [~@argnames]}]
       (layout (~(symbol (str name "*")) ~@argnames)))))

The ultimate purpose of this code is the defpage macro. This helps me in a few ways. It lets me easily write view code for each page without having to worry about the surrounding layout. For example, before writing the above code I had to include the navigation transformations in every page, which quickly got annoying.

The defpage macro also bundles up the template-dir variable, saving me some keystrokes and some duplication. Finally, it establishes a consistent way of interacting with the controller, a topic I’ll explore more in the next section. Before that, here’s an example of defpage in action:

clojure/gratefulplace-dissected/v1/src/gratefulplace/views/posts.clj

(defpage show "posts/show.html"
  [post comments current-auth]
  [:.post :.author :a]   (linked-username post)
  [:.post :.date]        (h/content (created-on post))
  [:.post :.content]     (md-content post)

  [:.post :.edit]        (keep-when (can-modify-record? post))
  [:.post :.edit :a]     (set-path post post-edit-path)

  

  [:.post :.moderate]    (keep-when (moderator? (:username current-auth)))
  [:.post :.moderate :a] (h/do->
                          (set-path post post-path)
                          (h/content (if (:hidden post) "unhide" "hide")))
  
  [:#post_id]            (h/set-attr :value (:id post))
  [:.favorite]           (favorite current-auth post)

Thanks to defpage, my view code contains only logic specific to that view. I love it!

Using macros to enforce full-stack consistency

Creating a route→controller interface

When I was putting the app together, each of my compojure routes was a unique and beautiful snowflake:

  (GET  "/users/new" []
        (users/show-new))
  
  (POST "/users" {params :params}
        (users/create! params))

  (GET  "/users/:username" [username]
        (users/show username))

As you can see, each route captures variables in a completely different way. At first I thought this made sense; I was passing exactly the data that was needed to each controller.

However, this quickly became a pain in the ass. For example, if I wanted to make a request param available to a view, I had to modify functions all up and down the stack: in the routes, in the controller function, and in the view function. I ended up making all routes send the full request object to their functions:

clojure/gratefulplace-dissected/v1/src/gratefulplace/middleware/routes.clj

(defmacro route
  [method path & handlers]
  `(~method ~path req#
            (->> req#
                ~@handlers)))

(defroutes routes
  (compojure.route/files "/" {:root "public"})

  ;; posts
  (route GET "/" posts/all)
  (route GET "/posts" posts/all)
  (route GET "/posts/new" posts/show-new)
  (route POST "/posts" posts/create! (friend/authorize #{:user}))
  (route GET "/posts/:id" posts/show)
  (route GET "/posts/:id/edit" posts/edit)
  (route POST "/posts/:id" posts/update)

The route macro is very handy. It saves me from typing and (for the most part) from burning precious brain cells on thinking, and it enforces the “send the full request” constraint by not allowing me to deviate. Of course, I can still deviate, and this is useful if you’re using a library:

clojure/gratefulplace-dissected/v1/src/gratefulplace/middleware/routes.clj

56
57
58

(friend/logout
 (ANY "/logout" []
      (ring.util.response/redirect "/")))

If I want to get to the params, I just use a let; no big deal:

clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/posts.clj

38
39
40

(defn show
  [req]
  (let [id (get-in req [:params :id])

Creating a controller→view interface

In the same way that it was useful to create a route→controller interface, it became useful to create a controller→view interface. That was accomplished with this macro:

clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/common.clj

(defmacro view
  "provides defaults for the map provided to view functions and allows
  you to provide additional key value pairs. Assumes that a variable
  named req exists"

  [view-fn & keys]
  `(let [x# {:current-auth (friend/current-authentication)
             :errors {}
             :params (:params ~'req)
             :req ~'req}]
     (~view-fn (into x# (map vec (partition 2 ~(vec keys)))))))

This also helped with one weird problem I ran into with cemerick’s friend library where (friend/current-authentication) returns nil from within an anonymous function (and there are many anonymous functions in my view code). I could circumvent this problem by assigning doing something like (let [current-auth (friend/current-authentication)]). But it seemed to make more sense to just always pass that data along when calling view functions, as you can see in the macro above.

The defpage macro also helped in establishing this interface in that it allowed me to destructure the arguments sent by the controller. For example, if a controller had

(view posts/show {:post post})

I could create the corresponding view with

(defpage show [post current-authentication])

And I wouldn’t even have to worry about the order of the arguments. It really reduced some “incidental complexity” (and I’m almost certainly using that term incorrectly).

Roll-your-own validations

It wasn’t too difficult to write my own way of doing validations, and I’m satisfied with it for now. Here’s a description of the data structure, with an example following:

Validation: a combination of a key and “validation check groups”
Validation check group: first element is an error message and the rest are “validation checks”. If any validation check returns false then the error message is added to a list of error messages for the given key.
Validation check: a boolean function to apply to one of a record’s values. The value corresponds to the validation key

Here’s an example:

(validate
 {:username "joebob", :password "hey"}
 {:password
   ["Your password must be at least 4 characters long"
    #(>= (count %) 4)]})

This validation would fail because the value "hey" for the :password is not at least four characters long. Here’s the main validation code:

clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/common.clj

(defmacro if-valid
  [to-validate validations errors-name & then-else]
  `(let [to-validate# ~to-validate
         validations# ~validations
         ~errors-name (validate to-validate# validations#)]
     (if (empty? ~errors-name)
       ~(first then-else)
       ~(second then-else))))

(defn error-messages-for
  "return a vector of error messages or nil if no errors
validation-check-groups is a seq of alternating messages and
validation checks"
  [value validation-check-groups]
  (filter
   identity
   (map
    #(when (not ((last %) value)) (first %))
    (partition 2 validation-check-groups))))

(defn validate
  "returns a map of errors"
  [to-validate validations]
  (let [validations (vec validations)]
    (loop [errors {} v validations]
      (if-let [validation (first v)]
        (let [[fieldname validation-check-groups] validation
              value (get to-validate fieldname)
              error-messages (error-messages-for value validation-check-groups)]
          (if (empty? error-messages)
            (recur errors (rest v))
            (recur (assoc errors fieldname error-messages) (rest v))))
        errors))))

if-valid wasn’t strictly necessary but I made it and ended up liking it. It’s a small improvement, but a handy one.

Pagination is more fun with macros

It was a bit difficult for me to pull the pagination code together. I tried many approaches, and I’m not 100% satisfied with what I have, but it’s better than nothing! The basic approach was to add a couple Korma clauses to a base set of clauses. Here are the base clauses, along with a macro that allows me to add more clauses willy-nilly:

clojure/gratefulplace-dissected/v1/src/gratefulplace/models/post.clj

(def base
  (->
   (select* e/post)
   (with e/user
         (fields :username))
   (with e/comment
         (aggregate (count :*) :count)
         (where {:hidden false}))
   (with e/favorite
         (aggregate (count :*) :count))
   (order :created_on :DESC)))

(defmacro all
  [& clauses]
  `(-> base
       ~@clauses
       select))

Here’s the pagination macro:

clojure/gratefulplace-dissected/v1/src/gratefulplace/models/helpers.clj

;; TODO make better!
(defmacro paginate
  ([page num-per-page query]
     `(~@query
       (limit ~num-per-page)
       (offset (dec ~page)))))

And here’s the whole thing in action:

clojure/gratefulplace-dissected/v1/src/gratefulplace/controllers/posts.clj

(defn all
  [req]
  (let [current-auth (friend/current-authentication)
        per-page 20
        page (str->int (or (get-in req [:params :page] 1)))
        conditions (with-visibility
                     current-auth
                     {:moderator true
                      :logged-in (or {:hidden false}
                                     {:user_id [= (:id current-auth)]})
                      :not-logged-in {:hidden false}})
        record-count (post/record-count (where conditions))]
    (view
     view/all
     :posts (paginate page per-page (post/all (where conditions)))
     :record-count record-count
     :page page
     :max-pages (ceil (/ record-count per-page)))))

I definitely think there’s room for improvement here and would love suggestions.

Using fierce mustaches to run tasks

So it turns out that you can use leiningen to run tasks, kind of like Rake. As far as I know, it’s not as powerful in that you can’t specify dependencies, but it’s still better than nothing! Here’s a file I wrote to allow me to rebuild and re-seed by db:

clojure/gratefulplace-dissected/v1/src/tasks/db.clj

(ns tasks.db
  (:refer-clojure :exclude [alter drop complement
                            bigint boolean char double float time])
  (:require [gratefulplace.models.user :as user])
  (:use (lobos core connectivity)))

(defn rebuild
  []
  (rollback :all)
  (migrate))

(defn seed
  []
  (println (user/create! {:username     "higginbotham"
                          :email        "daniel@flyingmachinestudios.com"
                          :display_name "higginbotham"
                          :password     "password"})))
(defn -main
  [task-name]
  (condp = task-name
    "rebuild" (rebuild)
    "seed" (seed)))

I ran lein run -m tasks.db rebuild and lein run -m tasks.db seed.

Fun utilities

Here’s some stuff that was fun to write:

clojure/gratefulplace-dissected/v1/src/gratefulplace/utils.clj

(ns gratefulplace.utils)

(defn str->int
  ([str]
     (Integer.  str))

  ([m & keys]
     (reduce
      (fn [m k]
        (if-let [val (k m)]
          (assoc m k (str->int val))
          m))
      m keys)))

(defn deserialize
  [m & ks]
  (reduce #(assoc % %2 (read-string (%2 %))) m ks))

(defmacro self-unless-fn
  [self fn otherwise]
  `(let [self# ~self]
     (if (~fn self#)
       ~otherwise
       self#)))

The End

And that’s it! I hope you’ve found this useful. I’d love any feedback, so please do leave a comment or tweet me or email me. And like I mentioned above – it’d be great to meet some folks at the Conj next week!

Loving Without Fear

2012-11-08T01:02:00Z

I was sitting at the Cambridge Public Library, and for the first time in months - years? - I felt at peace. In my journal I had just written what's become a kind of mantra for me: "Love without fear."

At the time I was going through a very difficult breakup. My relationship of 6 years had been slowly and painfully dissolving, in part because of my own inability to end it decisively.

Part of my difficulty what was that I had a lot of guilt over wanting to end the relationship because the person I was with had a long-term illness and I was supporting her both financially and as a kind of house helper or ersatz nurse. I had watched her suffer terribly.

I'm not sure how to convey what that was like. How do you share the experience of watching someone you love get a spinal tap? Of wheeling them out of the hospital and to the hotel next door, trying to do it quickly but smoothly because every bump causes excruciating pain? Or how about cleaning the vomit out of a mixing bowl because antibiotics are causing her nausea? Or giving the antibiotics shots yourself, night after night? Or knowing that there was no end in sight? How could I leave someone to face that on her own?

After six years, I was in a state of chronic exhaustion and stress. Often I would start my day by sitting at my desk for a couple hours and just staring, not really able to do anything. I knew that something had to change, but I felt too guilty to leave.

And, to be completely honest, I was scared about what it might say about me if I left her. Would it mean that I was a callous, heartless person? More, I was scared to live without her. We'd been together for 6 years, and the life I knew was familiar and provided comfort in a way. In the state I was in, I did not feel confident that I could make a large life change. Michael Singer said it well:

You will get to a point in your growth where you understand that if you protect yourself, you will never be free. It’s that simple. Because you’re scared, you have locked yourself within your house and pulled down all the shades. Now it’s dark and you want to feel the sunlight, but you can’t. It’s impossible. If you close and protect yourself, you are locking this scared, insecure person within your heart. You will never be free that way.

-- The Untethered Soul, by Michael Singer

But those words, "love without fear", helped give me strength. Beneath the fear, beneath the raw ache of being a perpetual witness to suffering, I could still feel the beating heart of love. I felt it there, ready, waiting for me without judgment. I felt the desire to open up again, to build honest and meaningful relationships with others. I felt like I had nothing to fear from embracing every moment of life whole-heartedly.

The months that followed that day at the library were messy and painful in many ways. But despite many mistakes, I knew that basically I was on the right path, that my heart was opening up more and more and I was learning to love myself and be myself without giving in to my fears of what other people might think or what vague disaster might befall me.

Now, more than two and a half years later, I can say without reservation that I took the right path. My life has not been problem-free and I don't ever expect it to be. In embracing love, though, I've learned that freedom and joy can be found in every moment, even when problems do arise.

My life has improved as well. Actually, that's a major understatement. My life is freaking awesome right now. About a month ago, I got engaged to the girl I love. A few days ago, I moved out of a drafty, gross old apartment into a beautiful new house that's close enough to my brother and mom to allow me to visit them much more often. And my latest project, something I put together with the idea of lifting up other people, is almost ready to ship.

And through it all, I keep telling myself: Love without fear. Love without fear. Love without fear.

Daily Gratitude

2012-10-17T01:02:00Z

Around the beginning of the year I read about the benefits of cultivating a daily practice of gratitude. Making a point of being grateful trains you to think more positively and just makes you feel better.

I've personally found it to be a rewarding, easy practice. Every morning I think about three things that I'm grateful for as soon as I wake up, and it's a fantastic way to start the day. And over the past six months, I've found myself feeling grateful for my life much more often.

The fact is, I have a great life. I do what I love, I'm with the woman I love, and I'm financially secure. Now I appreciate my great situation much more than I did six months ago.

And if there's something in my life I want to change, practicing gratitude has helped to give me the energy to change it. Instead of feeling overwhelmed and anxious, I feel like I ultimately have nothing to worry about.

The best part about it is that it's freaking easy and only takes a few minutes, but the benefits are huge and accumulate over time. It's such a powerful practice that I truly think everyone should try it. Here are a couple suggestions for getting started:

What you don't lack

It's easy to take our everyday lives for granted. If you have a toothache, you are grateful when it's gone. But you quickly adjust to your new situation and forget that your tooth is better.

But when you look closely, you can see that our everyday lives are full of small miracles like these. Do you have food? Do you have your vision? Your hearing? None of these are guaranteed to anyone.

As Thich Nhat Hanh put it:

For me, to be alive is a miracle. It is the greatest of all miracles. To feel that you are alive and are breathing in is to perform a miracle - one that you can perform at any time.

Or think about this. There's a condition called "alien hand syndrome" where those who suffer from it can't control one of their hands. They try to button up their shirt with one hand, and the other hand unbuttons it. They move in to hug a loved one, only to have the alien hand punch the person. Not having alien hand syndrome is something to be grateful for.

Your past

Is there anyone in your past that helped you when you needed it?

Did you have good friends? A family that loved you?

Even if you are feeling a lack in your life now, you can be grateful for what you've had in the past. Even if it's difficult, trying feeling grateful for people or events in your past.

These two sources - your past and what you don't lack - can be very fruitful sources of gratefulness.

Love is Power

2012-09-21T15:55:00Z

"You can't park like that, you dumb bitch!" A man, not ten feet away, had just shouted this at my then-girlfriend Su.

It was the middle of winter in New England, and the streets were narrow with large snow banks piled on each side. Earlier, Su had called people all day trying to get someone to remove the snow -- to no avail.

That night I was helping her get groceries out of the car, parked temporarily in front of the house, and the snow bank made it hard for other cars to pass. She laughingly commented that "now people can't get through!" Besides being unable to finding someone to plow the snow, a lot of other stuff had gone wrong that day, and the best we could do was laugh.

And then that man starting shouting at us from inside his truck. We took it in stride, but then the guy rolled down his window. He just wouldn't stop. He parked up the street, got out and continued shouting, even after we told him we were parked there because Su's disabled and that we'd call the police if he didn't stop.

When he finally left I was furious. This was right before heading to aikido practice, so I thought to myself, "What's an aikido way to deal with this?" If you don't know what aikido is, it's pretty much a martial art for hippies. Here's a quote from its founder:

Aikido is love. You make this great great love of the universe your heart and then you must make your own mission the protection and love of all things.

-- O'Sensei

In aikido you train to extend love to all things - including angry jerks yelling at you for no reason. Including people who are attacking you, whether physically or verbally or indirectly.

So I took a step back, and tried to see it from Angry Jerk's perspective. Clearly, he was distressed. For someone to blow up uncontrollably like that they have to be in some kind of emotional turmoil. And that freed something up inside me.

Thinking about Angry Jerk with love and compassion opened up enough space for me to realize that, actually, I was completely OK. Sure, this guy had made me scared, but right then I was physically safe and unharmed. What I found was that extending love and compassion towards the people attacking you forces you to be OK. You have to be full in order to give, and the act of giving creates fullness

I went to AJ's house. When I knocked on the door, he answered it, and I said something like, "I know that the way we parked made it hard to drive -- in fact, we were about to move the car when you came by. We're sorry it made it hard for you to drive."

We went back and forth a little, and the guy said he thought Su was laughing at him. When I told him she was laughing at the situation, not at him, he started saying he was sorry over and over, sorry for being a jerk, and I should tell Su he apologized.

In this situation, many people would feel like they had to "set this jackass straight". They'd feel the need to assert their manhood or otherwise angrily retaliate to demand respect.

But think about it - how likely are you to respect someone who's blowing up at you? Fear them, maybe. Respect them, no. Besides that, you risk escalating the situation to the point where it does become physical. And finally, you end up losing sight of the fact that another human is in distress. You diminish your capacity for compassion.

We never invited the guy over for dinner or anything, but I think the situation was resolved pretty happily. It was pretty awesome to be able to confront the guy in a non-threatening, non-accusatory way, and to have him end up apologizing.

There have been many other times during my life where I've employed this technique, and it has never failed me. It empowers me to protect myself and in the process lift up others.

Releasing Foreman

2012-09-09T23:55:00Z

My first ever Mac app is out. Get it while it's hot!

On Learning Skills (A Response to Move Your Feet)

2012-08-27T16:23:00Z

Matt Swanson wrote an inspiring blog post on learning new skills and developing new habits. Below is additional information that could help you learn a new programming skill or develop a new programming habit, along with additional resources. Here's the summary:

Programming is a skill
- You need practice as well knowledge
- Learning can be seen as a process of bridging your gaps
- Performing a new skill requires you to develop new habits, and the rules for forming habits apply
There are many things you can do to successfully create a new habit
- Social support
- Environmental support

Programming is a skill

Programming is a skill, which means that learning programming languages and techniques is not as simple as accumulating facts. No written guide to programming, no matter how thorough, will ever be sufficient because there are countless details which have to be left out, details which you can only assimilate while engaging in the practice of programming.

No one would suggest that you could become a world-class singer by merely going through a book on the subject. With these physical skills, it's obvious that learning involves doing something other than just absorbing knowledge. I think that, with programming, its nature as a skill is often overlooked because it's knowledge work. The key thing to keep in mind is that you must actually do programming in order to learn languages and techniques.

In Matt's case, he was also trying to develop a new habit - using TDD. Many aspects of programming are like this. Perhaps you want to start documenting your code more or you want to start doing daily standup with your team or whatever. These are all habits, some involving just you and some involving your team or organization. Recognizing them as habits will allow you to apply techniques for successfully developing a new habit.

One habit that you can start developing now is to identify your learning gaps (This list, by the way, is pulled in part from Design for How People Learn):

Knowledge
- What information do you need to be successful?
- Where kind you find this information?
Skill
- What will you need to practice to develop the desired proficiency?
- How can you make opportunities for practice?
Motivation
- How do you actually feel about learning the skill?
- Do you feel resistant to changing?
- More on motivation
Environment
- Do you have all the tools you need?
- Are they within easy reach? For example - if you want to run like Matt, are your shoes in your closet or right by your bed so that you can put them on first thing in the morning?

Programming is comprised of habits

I'm not a great programmer, I'm a pretty good programmer with great habits

-- Kent Beck

There's a mountain of text out there on how to form habits. Here's my contribution to that mountain!

Get Social Support

Enlist your coworkers. Go to hackfests. Changing your habits is way, way easier when you have other people supporting you.

Environmental Support

Change your environment to make your new habit easy to do and antagonistic habits hard to do. As I mentioned above - put your running shoes by your bed.

When it comes to programming - change your environment to support your new habits. When I started learning Clojure, I invested a lot of effort up front in making it painless to actually run Clojure code because I knew that otherwise it would be too easy to just tell myself "meh, I'll get to this later. Now, time to learn angular.js!" or something like that.

Develop Motivation

I have a little essay on this topic.

Finally, "Change Anything" is probably the best book I've read on developing new habits.

I hope you find this helpful!

Three Dev Tools You Probably Already Know

2012-08-26T23:55:00Z

You probably already know about the following tools, but I found it useful to be reminded of them:repl

Partitioning

Partitioning is effective as a strategy to combat complexity and scale when two conditions are true: first, the divided parts must be sufficiently small that a person can now solve them; second, it must be possible to reason about how the parts assemble into a whole. Parts that are encapsulated are easier to reason about, because you need to track fewer details when composing the parts into a solution. You can forget, at least temporarily, about the details inside the other parts. This allows the developer to more easily reason about how the parts will interact with each other.

Knowledge

Software developers use knowledge of prior problems to help them solve current ones. This knowledge can be implicit know-how or explicitly written down. It can be specific, as in which components work well with others, or general, as in techniques for optimizing a database table layout. It comes in many forms, including books, lectures, pattern descriptions, source code, design documents, or sketches on a whiteboard.

Abstraction

Abstraction can effectively combat complexity and scale because it shrinks problems, and smaller problems are easier to reason about. If you are driving from New York to Los Angeles, you can simplify the navigation problem by considering only highways. By hiding details (excluding the option of driving across fields or parking lots), you have shrunken the number of options to consider, making the problem easier to reason about.

From Just Enough Software Architecture: A Risk-Driven Approach by George H. Fairbanks