mcrammm.com

Mailfeed In Retrospect

Sat, 15 Apr 2017 13:21:00 -0700

It’s been a few weeks since I announced on /r/Clojure I was finished working on Mailfeed, a service that emails you whenever an RSS feed updates. A comment on that thread suggested that I share a little about the technologies I used and some of the pitfalls I hit during development.

There’s a lot to talk about, so I thought I’d just drop a nice big list of all the major tech, services and libraries, then move on to some of the more interesting challenges.

Major Tech used:

Clojure(script)
Postgres
Nginx

Services:

Stathat
Rollbar
Papertrail
Mailgun
Digital Ocean

Libraries:

Ring
Compojure
Enlive (for emails)
Hiccup (for everything else)
HugSQL
Ragtime
Environ
Quartzite
Buddy
Component
clj-time
clj-stripe
clj-http
clj-rollcage

The meat of the application is in the worker and the web apps. Both of these are written entirely in Clojure with a little Clojurescript on one page in the web app. I chose Postgres as a database simply because I already knew it really well.

RSS Processing Woes

The first iteration of Mailfeed was basically the same app that exists today with the worker existing as a background thread that was spun up whenever the app booted. I designed the system using Component early on, and although I had a couple of small trip-ups along the way, I’m very glad that I chose it.

At that time the worker was very un-optimized, generated a lot of errors, and died from time to time. Rebooting it meant rebooting the whole application and this made the engineer in me cry. The worker is the core of the application; it does a lot of heavy lifting and is the main reason anyone would ever want to actually use the service.

In my first spike I tried to convert the worker to using core.async, but it started to feel like the wrong solution pretty quick. The library is more about communication between two processes and what I wanted was a completely independent process that would:

Query the database at a regular interval
Get any feeds that needed to be updated
Process new entries for these feeds
Send any relevant emails to subscribers.

On my next spike I sat down and wrote a “Scheduler” component in Quartzite, abstracting the task of feed refreshing into it’s own job. I peppered in some error handling to ensure that when an error occurred, the job could clean up and continue. I made sure to fire the error off to Rollbar as I still wanted visibility on them.

Everything worked but the process was still really slow. At the time I had around 30 feeds that I wanted to refresh every 5 minutes. On average, the job was completing in about 40 seconds with an occasional spike up into 2 minutes. This was technically still within my acceptable time frame, but still felt a little extreme for such a small number of feeds. I added in some monitoring, sending the processing times of a few different functions to StatHat, and left it for a few days to get a good baseline. I blocked off my next weekend to investigate and see what improvements could be made.

When I sat down on Saturday morning I poked around and determined the job was spending most of its time waiting for a response from a few different sites. Everything else was pretty speedy. Since network latency was completely outside of my ability to correct, I decided to look at parallelizing the feed fetching/processing task. I went back to core.async very briefly again, before remembering about pmap. I made the switch from:

(dorun (map update-feed (get-feeds)))

To:

(dorun (pmap update-feed (get-feeds)))

After a lot more testing to make sure I hadn’t missed anything, I found that I had brought the average total processing time for the job down to 2 seconds.

Woo hoo!

I decided to call that “mission accomplished” and closed my computer to go make breakfast. Even now with almost 3 times the number of feeds, the mean time for the job to complete is 3.113 seconds.

The Monolith with Two Doors

At this point I was really happy with how the worker was performing, but still felt like it made sense to completely separate the worker from the web app in production. I wanted them to be able to exist on their own boxes so that I could fine tune each individually.

I mentioned previously that I was using component to try and keep the logical pieces of Mailfeed as separate as possible. With Component you create “systems” which is basically just a way of describing all your components and the components they depend on to do their job. What I had was two systems that shared a lot of the same components, but whose primary functions were quite different.

So that’s what I created. The “Web System” contained the all the routes, the web handler, the database and the mailer, while the “Worker System” contained the scheduler, the database, the mailer and the feed parser. When I go to deploy, I build two JARs in Boot:

(deftask build-web []
 (comp (aot :namespace '#{mailfeed.web.core})
  (pom)
  (uber)
  (jar :main 'mailfeed.web.core
       :file "mailfeed-web.jar")
  (target :dir #{"target/web"})))

(deftask build-worker []
 (comp (aot :namespace '#{mailfeed.worker.core})
  (pom)
  (uber)
  (jar :main 'mailfeed.worker.core
       :file "mailfeed-worker.jar")
  (target :dir #{"target/worker"})))

Overall I’m actually really happy with this approach. The only thing I have to be careful of is database migrations. If I make a change for one part of the application, I need to make sure everything is backwards compatible, since there is no mechanism for keeping both the worker and the web app in-sync. I do one deploy, and then the other. This is mostly-OK, since it’s generally a good idea to write a migration to add a new thing, then, if you need to, write another migration to remove the old thing once you’ve verified it’s not being used anymore.

This leads me into my next topic; one that I’m not very excited to share:

Deployment

Right now Mailfeed is deployed to Digital Ocean. Everything is deployed with a couple of shell scripts and some preconfigured “droplets” that I set up by hand then took snapshots of.

The above sentence should have made you go:

Yeah… I’m right there with you. Dumb! Dumb. Baaad Mike.

But it works! I ran a failure-scenario a couple of months ago and I managed to get everything up and running again within 30 minutes. That’s not … great, but considering how mission-critical the application is, it’s not bad either. And if I had paying customers then I would spend the time to rework everything through Ansible.

If I was starting again today, then Ansible would definitely be technology I would consider to provision these servers for me. I’ve used in the past and had very a very good experience with it. Daniel Higginbotham has also recently released a book “Deploying Your First Clojure App …From the Shadows“ that I wish had existed before I started Mailfeed.

Monitoring

I’ve written a bit about monitoring, but in summary I:

Track of errors with Rollbar
Send some key stats with Stathat
Monitor hardware with Digital Ocean Monitoring
Logs with Papertrail

I experimented a bit with Riemann early on but found it was a little over my head at the time. It certainly seems capable of tracking most of the above, without the limitations imposed by having to stick to the Free account tiers.

What’s the lesson?

I was really happy with using Clojure and I didn’t have any huge surprises during development. But every story should have a lesson! I think the big lessons I learned weren’t really related to the actual development process, but more to the things that need to go along around it. If I had to pick something I needed to improve it would be to do a little more up front market research and planning instead of just jumping in and building something I and a few friends of mine wanted.

Hopefully some of that was insightful. If you’re reading this and are curious about something specific, please don’t hesitate to reach out on Twitter.

Integration Testing with Clojure and Postgres

Wed, 14 Dec 2016 14:33:00 -0600

If you’re writing a non-trivial application that will run in production, it’s usually a good idea to have some automated way to make sure that all the pieces are working together correctly. On a lot of projects this is going to mean integration tests. How granular these tests become will depend on your level of paranoia and how critical those integration points are to your application as a whole.

In this post I’m going to use an example from a previous article and add some integration tests to ensure that we’re creating and updating accounts correctly.

It’s worth noting that my opinion on integration tests is that they should act as smoke tests, except in some extreme cases. If you find yourself testing complex business logic and needing to integrate with the database to do so, then chances are you’re doing something wrong. Integration tests should not be a replacement for QA or to compensate for bad design. But the world isn’t perfect, and sometimes a convoluted, slow running set of integration tests is the best you can do.

Alright, enough postulating. Let’s move on. Here is the namespace that we’ll be targeting for our tests:

(ns postgres-example.entities.accounts
  (:require [clj-time.jdbc]

            [postgres-example.sql :as sql]
            [postgres-example.components.postgres])
  (:import [postgres_example.components.postgres Postgres]))


(defprotocol AccountOps
  (by-id [this id])
  (create! [this status])
  (set-opened! [this account])
  (set-closed! [this account]))

(defn sql->account [sql-entity]
  (when (:id sql-entity)
    #:account {:id (:id sql-entity)
               :status (:status sql-entity)
               :created-at (:created_at sql-entity)
               :updated-at (:updated_at sql-entity)}))

(def opened-status "open")
(def closed-status "closed")

(extend-protocol AccountOps
  Postgres
  (by-id [store id]
    (-> (sql/account-by-id (:uri store) {:id id})
        sql->account))

  (create! [store status]
    (let [result (sql/insert-account! (:uri store) {:status status})]
      (by-id store (:id result))))

  (set-opened! [store account]
    (sql/update-account! (:uri store) {:id (:account/id account)
                                       :status opened-status})
    (by-id store (:account/id account)))

  (set-closed! [store account]
    (sql/update-account! (:uri store) {:id (:account/id account)
                                       :status closed-status})
    (by-id store (:account/id account))))

This namespace’s sole responsibility is to provide a touchpoint for the rest of our app to where we’re storing our accounts data. This is where we go when we need to fetch or update something in our database. The reason we defined the AccountOps protocol is that we may want to extend these operations over a different store, like an AtomStore, when we move on to writing tests for other pieces of the system.

I think it’s worth mentioning that I feel like there could be a good fit for clojure.spec here. I’ll probably explore this in a future post.

To start we’ll need some way to actually run our tests, both from the REPL and outside if it. For outside the REPL, we can just use lein test. For inside though, we’re going to add a test method to dev/user.clj that uses the awesome Eftest to find and run our tests.

; ... truncated ...
(defn test []
  (let [path "test/postgres_example/integration"]
    (eftest/run-tests (eftest/find-tests path))))

Note that I had to make some other changes here as well to ensure that we have a separate test database loaded up and migrated to the same version we’re developing against. For the full list of changes to this file, see this commit.

Let’s make sure this is working with a dummy test. Create a file at test/postgres_example/integration/entities/accounts.clj and add the following content:

(ns postgres-example.integration.entities.accounts
  (:require [clojure.test :refer :all]))

(deftest foo-test
  (testing "our setup"
    (is (= 1 2))))

Running (test) at the REPL should display a failure. If it didn’t, then you should stop here and figure out why. If the test failed successfully, we can move on to writing something a little more useful. We’re going to write this test first, then figure out some of the missing pieces in a minute.

(ns postgres-example.integration.entities.accounts
  (:require [clojure.test :refer :all]
            [postgres-example.entities.accounts :refer :all]))

(deftest create!-test
  (testing "create! creates and returns an account"
    (let [account (create! store "open")]
      (is (not (nil? (:account/id account))))
      (is (= "open" (:account/status account))))))

Pretty easy right? All we’re doing with this test is ensuring that the result of calling create! returns a map that has an :account/id set, and was assigned the correct status. But as I said, we’re missing a couple of things. First, we haven’t defined what store is in this context. Second, we should be cleaning up any data we create once the test is completed.

To handle both of these problem we’re going to create a test-helpers namespace that our tests can reference to get a copy of the store (that we’ll point at our test database), and we’ll create a fixture that will execute some code to clean up any test data:

(ns postgres-example.test-helpers
  (:require [clojure.java.jdbc :as jdbc]
            [environ.core :refer [env]]
            [postgres-example.components.postgres :as postgres]))

(def ^:dynamic store nil)

(def test-db-uri (str (:database-url env) "_test"))

(defn db-transaction-fixture [f]
  (jdbc/with-db-transaction [conn test-db-uri]
    (jdbc/db-set-rollback-only! conn)
    (binding [store (postgres/build conn)]
      (f))))

From the top down, we create a dynamic var for store that we’ll re-bind to a new connection for every test. That connection will happen to be a database transaction that we’ll instruct to rollback when it’s complete, instead of simply comitting.

Credit to this post by Eric Normand. Prior to this I had been using an atom instead of a dynamic var and was pulling my hair out trying to get my tests to run without hitting concurrency issues. Changing it to a dynamic var and leveraging binding made things quite a bit nicer. (and quite a bit faster too)

We’ll need to require this namespace in our test, and tell our tests to use this db-transaction-fixture fixture:

(ns postgres-example.integration.entities.accounts
  (:require [clojure.test :refer :all]
            [postgres-example.test-helpers :refer [store db-transaction-fixture]]
            [postgres-example.entities.accounts :refer :all]))

(use-fixtures :each db-transaction-fixture)

(deftest create!-test
  (testing "create! creates and returns an account"
    (let [account (create! store "open")]
      (is (not (nil? (:account/id account))))
      (is (= "open" (:account/status account))))))

Running (test) at the repl should be successful now. Let’s fill out the rest of our tests. I’m going to include the whole thing since it’s so short:

(ns postgres-example.integration.entities.accounts
  (:require [clojure.test :refer :all]
            [postgres-example.test-helpers :refer [store db-transaction-fixture]]
            [postgres-example.entities.accounts :refer :all]))

(use-fixtures :each db-transaction-fixture)

(deftest create!-test
  (testing "create! creates and returns an account"
    (let [account (create! store "open")]
      (is (not (nil? (:account/id account))))
      (is (= "open" (:account/status account))))))

(deftest by-id-test
  (testing "by-id returns the correct account by id"
    (let [account (create! store "open")]
      (is (= account
             (by-id store (:account/id account)))))))

(deftest set-opened!-test
  (testing "set-opened! sets an account's status to opened-status"
    (let [account (create! store "closed")]
      (set-opened! store account)
      (is (= opened-status
             (:account/status (by-id store (:account/id account))))))))

(deftest set-closed!-test
  (testing "set-closed! sets an account's status to closed-status"
    (let [account (create! store "open")]
      (set-closed! store account)
      (is (= closed-status
             (:account/status (by-id store (:account/id account))))))))

These 4 tests run in about 0.022 seconds on my machine. If you check your local postgres database, you should (hopefully) see that your accounts table is empty:

$ psql -U postgres_example postgres_example_test

postgres_example_test=# select * from accounts;
┌────┬────────┬────────────┬────────────┐
│ id │ status │ created_at │ updated_at │
├────┼────────┼────────────┼────────────┤
└────┴────────┴────────────┴────────────┘
(0 rows)

Time: 1.330 ms

The id column is an auto-incrementing sequence though, so you should still see that changing:

postgres_example_test=# select currval('accounts_id_seq'::regclass);
┌─────────┐
│ currval │
├─────────┤
│      47 │
└─────────┘
(1 row)

Time: 1.910 ms

I’ll reiterate that integration tests should be used sparingly, and only in critical places where two or more things are interacting together. This pattern is the same one I apply to all Clojure projects that interact with Postgres.

Hopefully this has been helpful to someone :). If you notice any errors in this post, please let me know.

Off the ground with Clojure and Postgres

Mon, 12 Dec 2016 20:10:00 -0600

I’ve been writing a few apps in my spare time, most notably Mailfeed, and I’ve developed a simple pattern whenever I need to pull data out of the database. This pattern could be be applied to any database you’re interacting with, but in this case I’ll be showing how I tend to do it with Postrges.

I should say that none of this is groundbreaking stuff. If you’re an experienced developer then you’ll probably be saying “duh” a lot, but if your playing with Clojure and are struggling to come up with a good structure on how to do this kind of thing, then maybe this is something you could apply.

This is going to be pretty quick. Lets say you’re tracking user accounts with a status. We’d like to be able to do the following:

(accounts/by-id store 1)
; => nil

(accounts/create! store "open")
; => #:account{:id 1,
;              :status "open",
;              :created-at "<some-instant-in-time>",
;              :updated-at "<some-instant-in-time>"}

(accounts/set-closed! store (accounts/by-id store 1))
; => #:account{:id 1,
;              :status "closed",
;              :created-at "<some-instant-in-time>",
;              :updated-at "<some-instant-in-time>"}

Note that the resulting representation of accounts and invoices is a namespaced map, which is new to Clojure 1.9. It’s exactly the same as:

{:account/id 1
 :account/status "closed"
 :account/created-at "<some-instant-in-time>"
 :account/updated-at "<some-instant-in-time>"}

Let’s tackle this top-down by defining a protocol for the operations we’re performing.

(ns my-project.entities.accounts)

(defprotocol AccountOps
  (by-id [this id])
  (create! [this status])
  (set-opened! [this account])
  (set-closed! [this account]))

Okay that was easy, but what the heck is this going to be in the context of the final implementations of these methods? At this point it doesn’t really matter. We could define a new record called AtomStore and extend our protocol over it, but that isn’t the point of this post. I’ll leave that as an exercise for the reader.

We’re going to jump right in and create a Postgres component that will be passed a connection string to a running postgres instance, with a database already created. I have a full example here that also sets up Ragtime to ensure the necessary schema exists.

(ns my-project.components.postgres)

(defrecord Postgres [uri])

(defn build [uri]
  (->Postgres uri))

At this point we could switch back to our accounts namespace and extend the AccountOps protocol over it, but we still need some way of actually querying our database. For that we’re going to use HugSQL which will will let us define our queries in raw sql.

Let’s start with writing a query to look up an account by an id. Open a new file at resources/sql/accounts.sql and add the following content:

-- :name account-by-id :? :1
-- :doc Get an account by id
SELECT *
FROM accounts
WHERE id = :id

HugSQL will parse this file and define a new function called account-by-id in whatever namespace we load it in. The :? marks it as a query and the :1 will cause it to only return 1 result.

Now we’ll create a namespace to define this function in:

(ns my-project.sql
  (:require [hugsql.core :as hugsql]))

(hugsql/def-db-fns "sql/accounts.sql")

After loading this namespace, we’ll then have a function we can call to load an account by an id:

(require '[my-project.sql :as sql])
(sql/account-by-id "your-database-uri" {:id 123})
; => nil

Hurray! It worked… kinda. Let’s define a way to create a new account with an initial status:

-- :name insert-account! :<! :1
-- :doc Inserts an account and returns the id
INSERT INTO accounts (status)
VALUES (:status)
RETURNING id

You’ll have to reload your REPL if you’re following along at one. This will define a new method called insert-account! and return the id of the row that was just inserted. Now you can do the following:

(require '[my-project.sql :as sql])
(sql/insert-account! "your-database-uri" {:status "open"})
; => {:id 1}
(sql/account-by-id "your-database-uri" {:id 1})
; => {:id 1, :status "open", :created_at #inst "2016-12-12T00:00:00.000000000-00:00", :updated_at #inst "2016-12-12T00:00:00.000000000-00:00"}

Your database uri should look something like postgresql://postgres_example:secret@localhost:5432/postgres_example, assuming you’ve created a user postgres_example with the password secret, and a dabaase with the same name. This dosen’t have to be a connection string, but it’s the most straightforward way of specifying the connection details that I’ve encountered so far.

Switch back to our accounts namespace and use these functions in our AccountOps protocol:

(ns my-project.entities.accounts
  (:require [my-project.components.postgres]
            [my-project.sql :as sql])
  (:import [my_project.components.postgres Postgres]))

(defprotocol AccountOps
  (by-id [this id])
  (create! [this status])
  (set-opened! [this account])
  (set-closed! [this account]))

(extend-protocol AccountOps
  Postgres
  (by-id [this id]
    (sql/account-by-id (:db-spec this) {:id id}))

  (create! [this status]
    (let [result (sql/insert-account! (:db-spec this) {:status status})]
      (by-id this (:id result)))))

Note that I haven’t implemented the set-closed! or set-opened! protocols yet. We’ll get to them in a minute.

Because this example is a little contrived, the solution here seems almost too straightforward. The only interesting piece is that create! passes it’s result immediately to by-id for re-fetching. This is a design decision I’m making; your needs may vary.

We’re missing something though. Remember our example at the beginning of this article returned us a namespaced map, but we’re getting back just a regular one. To do this we’re going to pass every result of sql/account-by-id through a function sql->account:

(ns my-project.entities.accounts
  (:require [my-project.components.postgres]
            [my-project.sql :as sql])
  (:import [my_project.components.postgres Postgres]))

(defprotocol AccountOps
  (by-id [this id])
  (create! [this status])
  (set-opened! [this account])
  (set-closed! [this account]))

(defn sql->account [sql-entity]
  (when (:id sql-entity)
    #:account{:id         (:id sql-entity)
              :status     (:status sql-entity)
              :created-at (:created_at sql-entity)
              :updated-at (:updated_at sql-entity)}))

(extend-protocol AccountOps
  Postgres
  (by-id [this id]
    (-> (sql/account-by-id (:db-spec this) {:id id})
        sql->account))

  (create! [this status]
    (let [result (sql/insert-account! (:db-spec this) {:status status})]
      (by-id this (:id result)))))

It’s usually a good idea to insulate your code from outside dependencies like the database. Here we’re taking the raw result returned to us from HugSQL and mapping it to our own internal representation of it. This also gives us a place to manipulate the data to suite our needs as it comes out of the database.

There is one more thing I would recommend doing at this point, and it would be to require clj-time.jdbc in our accounts namespace:

(ns my-project.entities.accounts
  (:require [clj-time.jdbc]

            [my-project.components.postgres]
            [my-project.sql :as sql])
  (:import [my_project.components.postgres Postgres]))

The clj-time library is great on it’s own, and including this namespace will ensure that as the JDBC library pulls dates out of the database, that they’re mapped to JodaTime instances.

Now we’re finally ready to give these a try:

(accounts/by-id store 1)
; => nil

(accounts/create! store "open")
; => #:account{:id 1,
;              :status "open",
;              :created-at #object[org.joda.time.DateTime 0x17dffb5 "2016-12-12T00:00:00.000Z"],
;              :updated-at #object[org.joda.time.DateTime 0x7e0ac645 "2016-12-12T00:00:00.000Z"]}

(accounts/by-id store 1)
; => #:account{:id 1,
;              :status "open",
;              :created-at #object[org.joda.time.DateTime 0x17dffb5 "2016-12-12T00:00:00.000Z"],
;              :updated-at #object[org.joda.time.DateTime 0x7e0ac645 "2016-12-12T00:00:00.000Z"]}

Success! The last thing we’ll do is implement our set-* functions.

;; ========================================
;; in my-project.entities.accounts

(def opened-status "open")
(def closed-status "closed")

(extend-protocol AccountOps
  Postgres
  ;; ... truncated ...
  (set-closed! [store account]
    (sql/update-account! (:uri store) {:id (:account/id account)
                                       :status closed-status})
    (by-id store (:account/id account))))
  (set-open! [store account]
    (sql/update-account! (:uri store) {:id (:account/id account)
                                       :status opened-status})
    (by-id store (:account/id account))))

-- ========================================
-- in resources/sql/accounts.sql

-- :name update-account! :< :1
-- :doc Updates an account by id
UPDATE accounts
SET status = :status,
    updated_at = now()
WHERE id = :id
RETURNING id

And let’s try them out:

(def my-account (accounts/by-id store 1))

(:account/status my-account)
; => "open"

(accounts/set-closed! store my-account)
; => #:account{:id 1,
;              :status "closed",
;              :created-at #object[org.joda.time.DateTime 0x17dffb5 "2016-12-12T00:00:00.000Z"],
;              :updated-at #object[org.joda.time.DateTime 0x7e0ac645 "2016-12-12T00:00:00.000Z"]}

;; Note that `my-account` hasen't changed
(:account/status my-account)
; => "open"

(accounts/set-opened! store my-account)
; => #:account{:id 1,
;              :status "open",
;              :created-at #object[org.joda.time.DateTime 0x17dffb5 "2016-12-12T00:00:00.000Z"],
;              :updated-at #object[org.joda.time.DateTime 0x7e0ac645 "2016-12-12T00:00:00.000Z"]}

And bam! That’s it.

As I said before, this example is a little small and contrived, but I’ve found it to be a good jumping off point for most projects to start with.

If you notice any errors in this post, please let me know.

Datomic Setup

Wed, 22 Jul 2015 20:10:00 -0600

I’ve recently been exploring Datomic more seriously and have found myself jumping through the same hoops as I have in the past just to get things up and running. I’ve also encountered slight deficiencies in the documentation that I’ve had to re-investigate since the exploratory project I created was deleted quite a while ago.

I’m tired of retracing my same steps over and over again so I thought I’d create a quick post with some of the basic steps to get setup and using Datomic in a Clojure application.

This is just going to cover the basics. Datomic does some crazy things I haven’t had a chance to try yet, like using rules, querying the database at a particular instant in time, or getting a list of changes to an entity.

Installing Datomic

You do not need to install Datomic to get started, you can use the in-memory database.

Go here: https://my.datomic.com/downloads/free and find the latest version.

Then add [com.datomic/datomic-free "<the-latest-version>"] to your Leiningen project.

Component Setup

This is pretty easy, but you should have something like this:

(ns my-project.components.datomic
  (:require [com.stuartsierra.component :as component]
            [datomic.api :as datomic]))

(defrecord DatomicComponent [uri conn]
  component/Lifecycle
  (start [this]
    (if (:conn this)
      this
      (do
        (assoc this :conn (datomic/connect uri)))))
  (stop [this]
    (assoc this :conn nil)))

Schema

Schema should be ideally be kept in an EDN file and loaded on demand:

(def schema
  (delay
    (read-string
      (slurp (io/resource "my_project/schema.edn"))))

(defn create-schema [conn]
  (datomic/transact conn @schema))

Here is what your schema might look like:

; resources/my_project/schema.edn

[
  {:db/id #db/id[:db.part/db]
   :db/ident :cake/name
   :db/valueType :db.type/string
   :db/cardinality :db.cardinality/one
   :db/fulltext true
   :db/doc "The name of a cake"
   :db.install/_attribute :db.part/db}

  {:db/id #db/id[:db.part/db]
   :db/ident :cake/owner
   :db/valueType :db.type/ref
   :db/cardinality :db.cardinality/one
   :db/doc "The owner of a cake"
   :db.install/_attribute :db.part/db}

  {:db/id #db/id[:db.part/db]
   :db/ident :user/email
   :db/unique :db.unique/value
   :db/valueType :db.type/string
   :db/cardinality :db.cardinality/one
   :db/doc "Email address of a user"
   :db.install/_attribute :db.part/db}

  {:db/id #db/id[:db.part/db]
   :db/ident :user/phone-numbers
   :db/valueType :db.type/string
   :db/cardinality :db.cardinality/many
   :db/doc "Contact numbers for a user"
   :db.install/_attribute :db.part/db}
]

Information on defining your schema and all the options available is documented here.

Seed Data

Like your schema, any seed data should be kept in a separate file:

(def seed-data
  (delay
    (read-string
      (slurp (io/resource "my_project/seed.edn"))))

(defn seed-db [conn]
  (datomic/transact conn @seed-data))

Here is what your seed data might look like:

; resources/my_project/seed.edn
[
  ;; Users
  {:db/id #db/id[:db.part/user -1000001]
   :user/email "sally@test.com"
   :user/password "supersecret"
   :user/phones ["8469481047", "9471038596"]}

  {:db/id #db/id[:db.part/user -1000002]
   :user/email "bob@test.com"
   :user/password "secret"
   :user/phones ["1234567890", "0987654321"]}

  ;; Cakes
  {:db/id #db/id[:db.part/user]
   :cake/owner #db/id [:db.part/user -1000001]
   :cake/name "Carrot"}
  {:db/id #db/id[:db.part/user]
   :cake/owner #db/id [:db.part/user -1000001]
   :cake/name "Cheese"}
  {:db/id #db/id[:db.part/user]
   :cake/owner #db/id [:db.part/user -1000002]
   :cake/name "Carrot"}
]

Queries & Updates

The operations to be performed on an entity should be confined to it’s own namespace:

(ns my-project.users
  (:require [datomic.api :as datomic]
            [my-project.component.datomic])
  ; Note the change from using a dash to an underscore
  (:import [my_project.component.datomic DatomicComponent]))

(defprotocol UserOps
  (all [this])
  (by-email [this email])
  (save! [this user])

(extend-type DatomicComponent
  UserOps
  (all [this]
    (datomic/q '[:find [(pull ?user [*]) ...]
                 :where [?user :user/email]]
               (datomic/db (:conn this))))
  (by-email [this email]
    (datomic/q '[:find [(pull ?user [*])]
                 :in $ ?email
                 :where [?user :user/email ?email]]
               (datomic/db (:conn this))
               email))
  (save! [this user]
    (datomic/transact (:conn this) user)

The [(pull ?user [*]) ...] is an example of Datomic’s pull syntax. This basically says “after all ?users, bring in all of their attributes. Be careful when using the wildcard * as this will recursively pull any component attributes.

Traversing refs forwards and backwards

It’s possible to pull in refs by specifying them in the pull pattern. If you wanted cakes with their owners:

(datomic/q '[:find [(pull ?cake [* {:cake/owner [*]}]) ...]
             :where [?cake :cake/owner]]
           db)

If however you wanted the reverse, users and their cakes:

(datomic/q '[:find [(pull ?user [* {:cake/_owner [*]}]) ...]
             :where [?user :user/email]]
           db)

Recursive (graph) queries

This is one I haven’t found a good real world use case for yet, but it is possible. Read the following if you’re looking at doing these kinds of queries:

http://docs.datomic.com/query.html#rules

http://hashrocket.com/blog/posts/using-datomic-as-a-graph-database

Rack and Ring Basics

Wed, 15 Apr 2015 18:04:47 -0600

In the Clojure world, when you want to write a web app, you will almost certainly be using Ring. Ring is directly inspired by Rack, the defacto webserver abstraction for frameworks in Ruby.

The two are conceptually very similar, but there are some slight differences (aside from language) that might be interesting to highlight. I won’t be going too in depth with this post, and will focus instead on the basics of using Rack and Ring. I thought I would just get that out of the way before your expectations got too high.

It’s worth noting that most apps won’t actually use either Rack or Ring directly like we will here. It would be a lot faster and safer to use something like Sinatra or Rails for Rack, and Compojure or Luminus for Ring.

If you want to see the final versions of these examples in their full forms please click:

The Minimal First Step

Let’s start with Rack, since it’s obviously the more popular of the two. I’m going to assume that you can read and get Rack installed if you don’t already have it. If you can’t read, then I have nothing to worry about because this will all look like gobbledygook to you. I hope ASCII art will serve as an appropriate apology :-).

The first step in getting either application off the ground is to create a entry point. In the case of Rack this is will be an object with a call method. This method needs to return the basic structure required for a Rack response, which is an array containing the response status, headers and body:

    # my_rack_app.rb
    class MyRackApp
      def call(env)
        ['200', {'Content-Type' => 'text/html'}, ['Hello World']]
      end
    end

Note how simple this is. This is just a regular old ruby object with a method that returns a triplet. It knows nothing about Rack or anything else that might be using it.

Unfortunately this does absolutely nothing, which is pretty boring. To boot this app up, we’ll add a config.ru to the current directory:

# config.ru
require 'rack'
require_relative 'my_rack_app'

run MyRackApp.new

We can now run rackup from our current directory and, navigating to http://localhost:9292, we should see our “Hello World” response.

Getting our Ring app off the ground requires a little bit more setup, but we’ll be using Leiningen to do most of the trivial stuff for us.

First let’s create a new project with lein new app my-ring-app. Add [ring "1.3.2"] to the list of dependencies in project.clj. You might need to run lein deps to download the Ring library if you don’t already have it.

Ring is similar to Rack in that we need to give it a function that returns some standard response. Ring expects a map instead of an array:

    (ns my-ring-app.app)

    (defn handler [request]
      {:status 200
       :headers {"Content-Type" "text/html"}
       :body "Hello World"})

Calling the function “handler” above is simply the convention in Clojure-land. Like the Rack example, we need some way to boot this app up. Some examples will mash this boot process together with the code we wrote above but I prefer to keep this separate. Plus it makes it similar to our Rack example so win-win, right?

    (ns my-ring-app.core
      (:require [ring.adapter.jetty :refer [run-jetty]]
                [my-ring-app.app :as app)
      (:gen-class))

    (defn -main [& args]
      (run-jetty app/handler {:port 3000}))

Now we can just do lein run from our project root. Navigating to http://localhost:3000 should display a very familiar page.

Middleware

Middleware is essentially a series a steps a request has to go through in order to generate a response. Once a response is generated, it will return the response back up through any middleware in reverse order.

We’re going to introduce some middleware to both of our applications that translates “Hello” to it’s French counterpart “Bonjour”. Why French? Well I’m Canadian, and if I don’t provide some kind of French content, then the CRTC might come and yell at me for not being fair.

I also don’t know how to say “Hello” in any other language off the top of my head and I’m too lazy to do any more research so we are going to stick with “Bonjour”.

In Rack our middleware is going to be another ruby object, except that we will actually be doing something with that env parameter we saw in our earlier example. You will notice that our middleware bears some resemblance to our running MyRackApp

    # hello_translator.rb
    class HelloTranslator
      def initialize(app)
        @app = app
      end

      def translate_hello(str)
        str.gsub(/Hello/, 'Bonjour')
      end

      def call(env)
        status, headers, body = @app.call(env)
        body.map! { |str| translate_hello(str) }
        [status, headers, body]
      end
    end

First we initialize the middleware with our running application. Calling @app.call(env) simply passes the request down the stack. If there was another piece of middleware beneath us, then it would be the next reciever of our env parameter. This would keep going until a response is generated, which in our case will be from MyRackApp that we defined earlier. Our return value needs to be the same status, headers and body array in order for Rack to be able to serve the response.

If you ran rackup right now, then you wouldn’t see any change. This is because we actually need to tell Rack about this is middleware. To accomplish this we need to require and use it in our config.ru:

    # config.ru
    require 'rack'
    require_relative 'my_rack_app'
    require_relative 'hello_translator'

    use HelloTranslator
    run MyRackApp.new

Running rackup and refreshing the previous page will now show a lovely greeting in the famously romantic French language.

Again Ring is very similar. The biggest difference is that the middleware is just a function instead of an object:

    (ns my-ring-app.middleware
      (:require [clojure.string :as s]))

    (defn translate-hello [body]
     (s/replace body "Hello" "Bonjour"))

    (defn wrap-hello-translator [handler]
      (fn [request]
        (let [response (handler request)]
          (update-in response [:body] translate-hello))))

Again, prefixing our middleware with wrap is convention. As before we will need to require and use this middleware in the namespace responsible for booting the app:

    (ns my-ring-app.core
      (:require [ring.adapter.jetty :refer [run-jetty]]
                [my-ring-app.app :as app]
                [my-ring-app.middleware :refer [wrap-hello-translator]])
      (:gen-class))

    (defn -main [& args]
      (run-jetty (-> app/handler
                     wrap-hello-translator)
                 {:port 3000}))

Dealing With Query Params

I was originally going to end the post here with an amazingly concise, beautifully written conclusion to tie a nice little bow over everything. Instead I’m going to take this comparison one step further and really phone in the whole “wrapping things up” section that is structurally necessary.

I want to show something that is much more common than our mostly pointless “Hello” to “Bonjour” translation app that we’ve written. I’m going to show you how Ring and Rack each let you access any query parameters that may have come along with the request.

Let’s say that we want to let the request specify who we want to say “Bonjour” to, and “World” if nothing is provided. The request can tell us what name to use by simply providing a name parameter.

With the Rack example, we aren’t going to touch anything except for our base application:

    # my_rack_app.rb
    class MyRackApp
      def call(env)
        request = Rack::Request.new(env)
        subject = request.params.fetch('name', 'World')
        body = ["Hello #{subject}"]

        ['200', {'Content-Type' => 'text/html'}, body]
      end
    end

The change here is that we wrap the env parameter in a Rack::Request object, then attempt to fetch the 'name' parameter. If one isn’t provided, then we default to 'World'. Our response is essentially the same.

Opening http://localhost:9292?name=Pierre, we should see “Bonjour Pierre”. If we omit the name parameter entirely, then we should see “Bonjour World” as before.

Ring’s approach is slightly different. Instead of wrapping the request in another object, there is a set of very common default middleware that we can choose to include. One of these is wrap-params, which will add a :params key to the request and take any parameters out of the query string and put into a map at that key.

I’ve included full-versions of each file here to keep things easy to follow. Additions to my-ring-app.core are noted:

    (ns my-ring-app.core
      (:require [ring.adapter.jetty :refer [run-jetty]]
                [ring.middleware.params :refer [wrap-params]] ; NEW
                [my-ring-app.app :as app]
                [my-ring-app.middleware :refer [wrap-hello-translator]])
      (:gen-class))

    (defn -main [& args]
      (run-jetty (-> app/handler
                     wrap-params ; NEW
                     wrap-hello-translator) {:port 3000}))

    (ns my-ring-app.app)

    (defn handler [request]
      (let [subject (get-in request [:params "name"] "World")]
        {:status 200
         :headers  {"Content-Type" "text/html"}
         :body (format "Hello %s" subject)}))

The difference between the two examples is the fact that the query string processing is handled in middleware with Ring, while with Rack I wrapped it. I tried to find a similar piece of middleware for Rack but after an hour of clicking around I came up short. There is some stuff in rack-contrib that comes close, but not close enough. I’m sure one exists and I’ll update this post if I can find something analogous to Ring’s wrap-params.

I hope this been at least a little bit enlightening. I found the basics of Ring and Rack to be so similiar that I couldn’t help but write a post about the two. The differences in application structure and methodologies become more pronounced the more complicated your stack becomes, but this is usually due to stuff built on top of them.

In future posts I hope to show something a bit more useful built with Ring.

Composing Functions

Sat, 11 Apr 2015 18:04:47 -0600

Let’s look at the different flavors of function composition in Clojure.

Comp

There is a function in clojure.core called comp that takes a set of functions and returns a function that is a composition of those functions:

    (defn doubler [x]
      (* x 2))

    (defn incrementer [x]
      (+ x 1))

    (def doubler-and-incrementer (comp incrementer doubler))

    (map doubler-and-incrementer [1 2 3 4]) ; => [3 5 7 9]

clojure.core/inc exists, but I recreate it here to be explicit

Note that when composing functions with comp, they are applied right to left to their arguments:

    (def incrementer-and-doubler (comp doubler incrementer))

    (map incrementer-and-doubler [1 2 3 4]) ; => [4 6 8 10]

Partial

Another function in clojure.core is partial that takes a function and some arguments to that function and returns a new function that will accept the remaining number of arguments. This can be useful when combined with comp and writing a custom reducer:

    (def incrementer-and-doubler (comp (partial map doubler)
                                       (partial map incrementer)))

    (incrementer-and-doubler [1 2 3 4]) ; => [4 6 8 10]

Clojure 1.7 will introduce transducers, which will likely become the idiomatic way of accomplishing this same task.

Threading Macros

A more popular method of composition are the thread-first and thread-last macros.

The thread-first macro evaluates the first expression then passes the result into first argument of the next form, and the result of that into the first argument of the next form and so on.

In other words, this:

    (doubler (incrementer 2))

Is the same as this:

    (-> 2 incrementer doubler)

Thread-last works the same way, but it passes the result of each expression into the last element of each successive form.

    (->> [1 2 3 4]
         (map incrementer)
         (map doubler))

Is the same as:

    (map doubler (map incrementer [1 2 3 4]))

Om Comparison

Sat, 01 Feb 2014 18:04:47 -0600

In my last post I built a simple text manipulation widget with React. I recommend reading through that post first, before this one. As promised, I’ve built the same widget in Om, a ClojureScript library that sits on top of React.

If you want to follow along, you’ll need to install Leiningen and run:

    lein new mies-om om-intro

cd into the new directory and make your project.clj look like the following:

    (defproject om-intro "0.1.0-SNAPSHOT"
      :description "FIXME: write this!"
      :url "http://example.com/FIXME"

      :dependencies [[org.clojure/clojure "1.5.1"]
                     [org.clojure/clojurescript "0.0-2138"]
                     [org.clojure/core.async "0.1.267.0-0d7780-alpha"]
                     [om "0.3.1"]
                     [com.facebook/react "0.8.0.1"]]

      :plugins [[lein-cljsbuild "1.0.1"]]

      :source-paths ["src"]

      :cljsbuild {
        :builds [{:id "dev"
                  :source-paths ["src"]
                  :compiler {
                    :output-to "om_intro.js"
                    :output-dir "out"
                    :optimizations :none
                    :source-map true}}]})

You will also want to update your index.html look like this:

    <html>
        <body>
            <div id="app"></div>
            <script src="http://fb.me/react-0.8.0.js"></script>
            <script src="out/goog/base.js" type="text/javascript"></script>
            <script src="om_intro.js" type="text/javascript"></script>
            <script type="text/javascript">goog.require("om_intro.core");</script>
        </body>
    </html>

Get any missing dependencies with lein deps, then build the project with lein cljsbuild once dev. Open index.html in a browser and you should see the bare-bones example that comes with this template.

For the rest of this tutorial, I recommend running lein cljsbuild auto dev in a separate terminal. The first time the project gets built takes a second or two, but after the JVM has warmed up, it takes just milliseconds.

The snippets above are for a development build of the project. The final example I link to at the end of this post contains a release build, that generates a single JavaScript file.

With the setup out of the way we can start rebuilding this widget.

    (ns om-intro.core
      (:require [om.core :as om :include-macros true]
                [om.dom :as dom :include-macros true]))

    (def app-state (atom {:text "Some Text"}))

    (defn my-widget [app owner]
      (reify
        om/IRender
        (render [this]
          (dom/div nil (:text app)))))

    (om/root
      app-state
      my-widget
      (. js/document (getElementById "app")))

This is analogous to the first example in the React version; all we’re doing is defining a component that renders a div containing the value of :text from our application state.

There are already a differences though. First, we’ve moved all of our state into an atom. Components are given cursors into this application state that they can use to read/update.

Second, our my-widget component is returning a reified object that satisfies the om/IRender interface. The render method simply returns the aforementioned div.

You should see something like this:

Like our first example in the React version, this is pretty boring. Let’s add in the text input.

We’re going to be using core.async at the edges of our components, wherever our users will be interacting with the various inputs we’ll eventually have.

Change the namespace declaration to the following:

    (ns om-intro.core
      (:require-macros [cljs.core.async.macros :refer [go]])
      (:require [om.core :as om :include-macros true]
                [om.dom :as dom :include-macros true]
                [cljs.core.async :refer [put! chan <!]]))

Then we’ll update the widget. We’re going to walk through this step-by-step in a minute, but here is what it should look like:

    (defn my-widget [app owner]
      (reify
        om/IInitState
        (init-state [this]
          {:comm {:string (chan)}})

        om/IWillMount
        (will-mount [this]
          (let [{:keys [string]} (om/get-state owner :comm)]
            (go (while true
                  (let [value (<! string)]
                    (om/transact! app :text (fn [_] value)))))))

        om/IRenderState
        (render-state [this {:keys [comm]}]
          (dom/div nil
                   (dom/input #js {:type "text"
                                   :ref "text"
                                   :value (:text app)
                                   :onChange #(put!
                                                (:string comm)
                                                (-> (om/get-node owner "text")
                                                    .-value))})

                   (dom/div nil (:text app))))))

We’ve changed our widget to satisfy a few more Om interfaces that take advantage of the React life cycles.

The first is om/IInitState which sets up some initial, local state for the component. Here we are creating a map with a channel assigned to the :string key. init-state is called once on a component.

In om/IWillMount, we setup a go loop that blocks on the channel assigned to :string earlier, then sets the :text attribute in our application state to the value we get off of that channel. Once it’s done it goes back to waiting on the channel.

If you’re new to Clojure, then the destructuring we do in the let binding can be a little confusing. The gist of what we’re doing is creating a local string variable for our go block that is based on a key in the map returned by (om/get-state owner :comm). In other words, it takes the map we created earlier and creates a local variable that is assigned the value of the :string key.

We use om/transact! here since updating an atom needs to occur within a transaction. We could have also used swap! here to modify the atom manually.

will-mount is called once, before the component is mounted into the DOM.

Finally, we’ve changed om/IRender to om/IRenderState. Every component needs to satisfy one of these interfaces, but not both. The difference between the two is that IRenderState is passed the component state as it’s second argument. We need it so that we can have access to the channel we created earlier.

Finally we create the input:

    (dom/input #js {:type "text"
                    :ref "text"
                    :value (:text app)
                    :onChange #(put!
                                (:string comm)
                                (-> (om/get-node owner "text")
                                    .-value))})

The element is actually only taking a single argument, though it looks like two. #js is a reader literal for Clojurscript that transforms the following object into literal JavaScript object. The map that we pass is setting some attributes on the component. In this case, we want a text input that contains the value of the :text key from our application state. We assign it the ref text so that we can refer to it from the onChange callback via om/get-node.

This callback is really simple, and is one of the reasons why core.async is so attractive. All it does is take the value of the text node and put it onto the string channel.

If you’ve been following along, then you should see the following:

The next step is to add in the text-size slider. First, let’s add the size to our application state:

    (def app-state (atom {:text "Some Text"
                          :size 15}))

Next we’ll create another channel for manipulating this size:

    om/IInitState
    (init-state [this]
      {:comm {:string (chan)
              :size (chan)}})

We’ll create another go block to update :size whenever we get a value off of this channel:

    om/IWillMount
    (will-mount [this]
      (let [{:keys [string size] :as comm} (om/get-state owner :comm)]
        (go (while true
              (let [value (<! string)]
                (om/transact! app :text (fn [_] value)))))
        (go (while true
              (let [value (<! size)]
                (om/transact! app :size (fn [_] value)))))))

And then we’ll add the input. Since we’re getting the value off in the input in a similar way as before, I created a small helper to do this. I would place this function at the top of your source file, underneath the atom:

    (defn get-value [owner ref]
      (-> (om/get-node owner ref)
          .-value))

    (dom/div nil
            (dom/input #js {:type "range"
                            :min 10
                            :max 50
                            :step 0.2
                            :ref "size"
                            :value (:size app)
                            :onChange #(put!
                                        (:size comm)
                                        (get-value owner "size"))})
            (dom/label nil (str (:size app) "px")))

Note that you may want to update the text input as well.

Finally, we want to modify our div to have it’s font-size restyled whenever this changes. Right now it looks like this:

    (dom/div nil (:text app))

Change it to this:

    (dom/div #js {:style #js {:font-size (str (:size app) "px")}}
          (:text app))

Again, #js turns the following object into a JavaScript object. It’s shallow, so we need to do it twice to set :style correctly.

You should see this now:

Now for the color sliders. First, we’ll add in the new state:

    (def app-state (atom {:text "Some Text"
                          :size 15
                          :colors {:red 0
                                   :green 0
                                   :blue 0}}))

As in the React widget, we’ll create a more general color-slider:

    (defn color-slider [colors owner {:keys [label onChange color-key]}]
      (reify
        om/IRenderState
        (render-state [this {:keys [comm]}]
          (dom/div nil
                   (dom/input #js {:type "range"
                                   :min 0
                                   :max 255
                                   :step 1
                                   :ref "color"
                                   :value (color-key colors)
                                   :onChange #(onChange color-key owner)})
                   (dom/label nil (str label ": " (color-key colors)))))))

The important bit here is extra map of attributes we’ll be passing to this component. We’re going to give it a label, a color key to pull from the application state, and an onChange function.

Next we’ll create a channel for the changing colors:

    om/IInitState
    (init-state [this]
      {:comm {:string (chan)
              :size (chan)
              :colors (chan)}})

And a go block:

    (go (while true
          (let [[c value] (<! colors)]
            (om/update! app assoc-in [:colors c] value))))))

This looks slightly different than the previous go blocks because we’re dealing with a map of colors in the application state instead of a straight value.

    (let [putfn (fn [k o]
                 (put! (:colors comm) [k (get-value o "color")]))]
        (apply dom/div nil
             (map (fn [[label color-key]]
                    (om/build color-slider
                              (:colors app)
                              {:opts {:label label
                                      :color-key color-key
                                      :onChange putfn}}))
                  [["Red" :red] ["Green" :green] ["Blue" :blue]])))

Next we’ll add the inputs right below the text size slider. We use some high level functions here to avoid having to write three calls to om/build.

Finally we can modify the div to re-color our text:

    (let [size (:size app)
          text (:text app)
          {:keys [red green blue]} (:colors app)]
     (dom/div #js {:style #js {:font-size (str size "px")
                               :color (str "rgb(" red "," green "," blue ")")}}
              text))))))

Here is the final product, for the second time:

The full source for this example can be found here.

Om is still very new, and changing rapidly. If you’re interested, then I recommend running through the Tutorial in LightTable.

React Intro

Sun, 26 Jan 2014 18:04:47 -0600

React is a new-ish library from Facebook/Instagram that is designed to make building user interfaces easy.

I thought it would be fun to build a basic example in React, and then contrast it with the same example in Om.

First let’s setup the page where our example will live. We’re going to want a container for the React root component, which we’ll give the id app, and we’ll include the React library and the JSX transformer.

    <html>
        <body>
            <div id='app'></div>
            <script src="http://fb.me/react-0.8.0.js"></script>
            <script src="http://fb.me/JSXTransformer-0.8.0.js"></script>
            <script type='text/jsx'>
                /**
                * @jsx React.DOM
                */
                // Your code goes here
            </script>
        </body>
    </html>

Note the @jsx React.DOM in the block comment at the top of the script tag where our code will go. This is required for the JSX transformer to be able to process elements in our components from this:

    <a href="http://google.com">Click Me</a>

Into this:

    React.DOM.a( {href:"http://google.com"}, "Click Me")

JSX is completely optional, but it is a little bit easier to visualize the arrangement of nodes, so I’ll be using it for this example.

Let’s start writing some code. Our first goal is to create a text input where any entered text is rendered out to some node.

First we’ll create our root component and pass it some initial state:

    var VariableText = React.createClass({
        render: function() {
            return (
                <p>{this.props.defaultText}</p>
            );
        }
    });
    React.renderComponent(
        <VariableText defaultText="Some text" />,
        document.getElementById('app')
    );

What we’re doing here is create a component called VariableText that knows how to render itself. render simply creates a <p> tag and inserts defaultText inside of it. The second argument to renderComponent is simply explaining where to insert our component on our page.

This is great, but kinda boring. Let’s create our text input and change the content of the <p> tag when we type into it. This next snippet will be a little bit bigger, but all we really want to do is create a TextInput component that renders an input tag and responds to changes. We’ll also want to pass it our default text to set the input’s initial value.

    var TextInput = React.createClass({
        handleChange: function() {
            var text = this.refs.text.getDOMNode().value;
            this.props.onTextChanged(text);
            return false;
        },
        render: function() {
            return (
                <div>
                    <input
                        ref="text"
                        onChange={this.handleChange}
                        type='text'
                        value={this.props.inputValue}
                    />
                </div>
            );
        }
    });

    var VariableText = React.createClass({
        getInitialState: function() {
            return {
                text: this.props.defaultText
           }
        },
        handleTextChange: function(text) {
            this.setState({text: text});
        },
        render: function() {
            return (
                <div>
                    <TextInput
                        onTextChanged={this.handleTextChange}
                        inputValue={this.state.text}
                    />
                    <p>{this.state.text}</p>
                </div>
            );
        }
    });

We’ve had to change our VariableText component since we’ll need to introduce some mutable state, the value of the text input, into our app. The getInitialState function is called once, before the component is mounted. The return value of this function will set the initial value of the state attribute for this component. Here we’re setting it to the default text property. VariableText is going to be our root component, and will coordinate any state changes based on events triggered on its children.

Our first child component is TextInput. As mentioned before this component renders an input tag and calls the component’s handleChange function whenever the value changes. handleChange gets the value from the input via the refs attribute. When we create components in render, we can attach a special property ref to it that lets us refer back to it later. Once we get the value, we execute a callback that was assigned in our root component. This is the typical way a child communicates with its parent, and we’ll see this pattern occur throughout the rest of the example.

If you’re following along on your own, this is what you should have so far:

Now let’s introduce a slider to change the font size of our output text. We’ll create a FontSizeSlider component that again renders an input tag.

    var FontSizeSlider = React.createClass({
        handleChange: function() {
            var value = this.refs.slider.getDOMNode().value;
            this.props.onSliderChanged(value);
            return false;
        },
        render: function() {
            return (
                <div>
                    <input
                        ref="slider"
                        onChange={this.handleChange}
                        value={this.props.size}
                        type='range'
                        min="9"
                        max="50"
                        step="0.2" 
                    />
                    <span> Font Size: {this.props.size} </span>
                </div>
            );
        }
    });

This should look very similar to the TextInput component. Here is what changed on VariableText:

    getInitialState: function() {
        return {
            size: this.props.defaultSize,
            text: this.props.defaultText,
        }
    },
    handleFontSizeChange: function(size) {
        this.setState({size: size});
        this.refs.outputText.getDOMNode().style.fontSize = size + "px";
    },
    render: function() {
        return (
            <div>
                <TextInput
                    onTextChanged={this.handleTextChange}
                    inputValue={this.state.text}
                />
                <FontSizeSlider
                    onSliderChanged={this.handleFontSizeChange}
                    size={this.state.size}
                />
                <p ref='outputText'>{this.state.text}</p>
            </div>
        );
    }

Again this should look similar to the previous change. All we’ve done is add another child component and react to it changing it the same way as TextInput.

The defaultSize prop will also need to be passed into the root component:

    <VariableText defaultText="Some text" defaultSize={15}/>,

You should now see something like the following:

Let’s create a few more sliders and see how React gives you the ability to reduce code duplication. These sliders will control the red, green, and blue levels of our output text.

Since these sliders will respond to change similarly to our FontSizeSlider component, we’ll create a mixin:

    var HandleSliderChangeMixin = {
        handleChange: function() {
            var value = this.refs.slider.getDOMNode().value;
            this.props.onSliderChanged(value);
            return false;
        }
    };

Each of these color sliders will be pretty similar. They will all range from 0 to 255, and have a label to display its value. Let’s create a generic component first:

    var FontColorSlider = React.createClass({
        mixins: [HandleSliderChangeMixin],
        render: function() {
            return (
                <div>
                    <input
                        ref="slider"
                        onChange={this.handleChange}
                        value={this.props.value}
                        type='range'
                        min="0"
                        max="255"
                        step="1"
                    />
                    <label>{this.props.label}: {this.props.value}</label>
                </div>
            );
        }
    });

Note that we use the HandleSliderChangeMixin here. We’ll also want to update FontSizeSlider to use this mixin.

Next we’ll create a component that will create a FontColorSlider for each color value and communicate to our root component with the new values.

    var FontColorSliders = React.createClass({
        handleChange: function(r, g, b) {
            this.props.onColorsChanged({r: r, g: g, b: b});
        },
        handleRedChange: function(value) {
            this.handleChange(value, this.props.colors.g, this.props.colors.b);
        },
        handleGreenChange: function(value) {
            this.handleChange(this.props.colors.r, value, this.props.colors.b);
        },
        handleBlueChange: function(value) {
            this.handleChange(this.props.colors.r, this.props.colors.g, value);
        },
        render: function() {
            return (
                <div>
                    <FontColorSlider
                        onSliderChanged={this.handleRedChange}
                        label="Red"
                        value={this.props.colors.r}
                    />
                    <FontColorSlider
                        onSliderChanged={this.handleGreenChange}
                        label="Green"
                        value={this.props.colors.g}
                    />
                    <FontColorSlider
                        onSliderChanged={this.handleBlueChange}
                        label="Blue"
                        value={this.props.colors.b}
                    />
                </div>
            );
        }
    });

Here are the changes to VariableText that we’ll need to make:

    getInitialState: function() {
        return {
            size: this.props.defaultSize,
            text: this.props.defaultText,
            colors: {r:0, g:0, b:0}
        }
    },
    colorsToRGB: function(colors) {
        return "rgb(" + colors.r + "," + colors.g + "," + colors.b + ")";
    },
    handleColorsChange: function(colors) {
        this.setState({colors: colors});
        this.refs.outputText.getDOMNode().style.color = this.colorsToRGB(colors);
    }

Don’t forget to add our new component to the render method:

    <FontColorSliders
        onColorsChanged={this.handleColorsChange}
        colors={this.state.colors}
    />

Here is the final product:

The full source for this example can be found here.

Most of this should be pretty straight forward. Again, the goal here was to create something simple to get your feet wet with. There are a few opportunities to make things even simpler, but I’ll leave that as an exercise for the reader.

If you plan on exploring more of React, then I recommend going through their tutorial, and checking out the different life cycle methods.

In an upcoming post, I’ll be creating a similar app in Om.

Leiningen Templates

Mon, 06 Jan 2014 18:04:47 -0600

Getting started with Clojurescript can be tough if you’ve never done it before, and once you do you might find yourself running through the same steps to get new projects into a structure that makes sense. In either case, the solution is to use a Leiningen template. My goal here is to show how easy it can be to create one of your own. Most templates are built for Clojure, but we’ll be creating one for Clojurescript. On top of that we’ll get ourselves started off with the new Om library.

Before beginning I want to make it clear that this isn’t meant as an introduction to Clojurescript, Om, or the framework it acts as an interface to (React). This is meant to help you through creating a Leiningen template that will get you going quickly on a new project. If you want a bare-bones template for a new Clojurescript project, then I recommend mies.

Using Leiningen you can create new templates with a default project structure through the new task: lein new <template> <name>.

Let’s create one together. Since we’re creating one for the Om library, we’ll call it ‘Hum’:

    lein new template hum

To install this template locally, cd into it and run lein install. Let’s see what a project based off this bare template looks like. Run lein new hum my-app. You should get a directory tree with one file like this: my-app/src/my_app/foo.clj.

The contents of this file should simply be (def my-app :foo).

Open up src/leiningen/new/hum.clj and look at the hum function:

    (defn hum
      "FIXME: write documentation"
      [name]
      (let [data {:name name
                  :sanitized (name-to-path name)}]
        (main/info "Generating fresh 'lein new' hum project.")
        (->files data
                 ["src/{{site.lcbs}}sanitized{{site.rcbs}}/foo.clj" (render "foo.clj" data)])))

Let’s go through this line by line. This function takes a name then creates a map, data, that contains this name and a sanitized version of it. name-to-path simply takes a name and first replaces any dashes with underscores, then any periods with directory separators. So a name like lawrence-of.arabia becomes lawrence_of/arabia. For us, hum will remain unchanged.

The main/info line is simply printing some info to the user.

->files is where the action happens. This function takes a map containing at least the key name and a variable number of “paths”. Each path is a vector containing a string representing the path to a file, and some content for that file.

To understand the render call a little better, we need to look at the definition right above this function:

    (def render (renderer "hum"))

All this does is create a renderer that will process one of our templates and replace the appropriate variables. This templating is based off of mustache which you’ll see in a moment. For now, all you need to know is that render will take the name of a file in src/leiningen/new/hum and replaces certain areas that we specify with our supplied data. If we don’t supply any data then the file remains unchanged.

Keeping with our example, using this line:

    ["src/{{site.lcbs}}sanitized{{site.rcbs}}/foo.clj" (render "foo.clj" data)]

Will create a file at src/hum/foo.clj with the content of the file foo.clj. Let’s open that file up and see this templating in action:

    (def {{site.lcbs}}name{{site.rcbs}} :foo)

Remember that the :name key in data is set the unsanitized value we supply.

Now lets start making some changes. Create a file at src/leiningen/new/hum/project.clj and give it the following content:

    (defproject {{site.lcbs}}name{{site.rcbs}} "0.1.0-SNAPSHOT"
      :description "FIXME: write this!"
      :url "http://example.com/FIXME"

      :dependencies [[org.clojure/clojure "1.5.1"]
                     [org.clojure/clojurescript "0.0-2138"]
                     [om "0.1.4"]]

      :plugins [[lein-cljsbuild "1.0.1"]]

      :cljsbuild {
        :builds [{:id "dev"
                  :source-paths ["src"]
                  :compiler {
                    :output-to "{{site.lcbs}}sanitized{{site.rcbs}}.js"
                    :output-dir "out"
                    :optimizations :none
                    :source-map true
                    :externs ["om/externs/react.js"]}}]})

This is doing a bunch of stuff, but is a reasonably standard starting point. The main differences is that I’ve included the Om library as a dependency.

Now add this to the list of processed files in hum.clj:

    (->files data
             ["project.clj" (render "project.clj" data)]
             ["src/{{site.lcbs}}sanitized{{site.rcbs}}/foo.clj" (render "foo.clj" data)])))

To test it out, run lein new hum my-app. Remember to remove the previous test first, and make sure you are in the root of the template project. Open up my-app/project.clj and marvel at the magic.

There are few more things we need to add to finish this off. First let’s get rid of foo.clj by renaming it to a Clojurescript file, like core.cljs. Change the content of the file to this:

    (ns {{site.lcbs}}name{{site.rcbs}}.core
      (:require [om.core :as om :include-macros true]
                [om.dom :as dom :include-macros true]))

    (enable-console-print!)

    (def app-state
      (atom {:clicks 0}))

    (defn button [data owner]
      (om/component
        (dom/div nil
                 (dom/button
                   #js {:onClick #(om/transact! data :clicks inc)}
                   "Click Me")
                 (dom/span #js {} (:clicks data)))))

    (defn my-app [app owner]
      (reify
        om/IRender
        (render [_]
          (om/build button app {}))))

    (om/root app-state my-app js/document.body)

Again the intent here is not to introduce you to Om, or Clojurescript. All you need to know for now is that this creates a button and counter that tracks how many times the button has been clicked.

Your list of processed files should look something like this:

    ["project.clj" (render "project.clj" data)]
    ["src/{{site.lcbs}}sanitized{{site.rcbs}}/core.cljs" (render "core.cljs" data)]

Let’s add in one more file, index.html:

    <html>
        <body>
            <script src="http://fb.me/react-0.8.0.js"></script>
            <script src="out/goog/base.js" type="text/javascript"></script>
            <script src="{{site.lcbs}}sanitized{{site.rcbs}}.js" type="text/javascript"></script>
            <script type="text/javascript">goog.require("{{site.lcbs}}sanitized{{site.rcbs}}.core");</script>
        </body>
    </html>

And to the list of processed files:

    ["index.html" (render "index.html" data)]
    ["project.clj" (render "project.clj" data)]
    ["src/{{site.lcbs}}sanitized{{site.rcbs}}/core.cljs" (render "core.cljs" data)]

The full hum function should look similar to this:

    (defn hum [name]
      (let [data {:name name
                  :sanitized (name-to-path name)}]
        (main/info "Generating a new Om project using the Hum template")
        (->files data
                 ["index.html" (render "index.html" data)]
                 ["project.clj" (render "project.clj" data)]
                 ["src/{{site.lcbs}}sanitized{{site.rcbs}}/core.cljs" (render "core.cljs" data)])))

Now we’re done. Run lein new hum my-app again and cd into it. Run lein deps to get any dependencies, then lein cljsbuild once dev. Open up index.html and voila! You should see something like the following:

If you’re looking for more about creating templates, then I recommend the excellent Leiningen docs.

The full source for this template can be found here.

Specificity in ClojureScript

Thu, 02 Jan 2014 18:04:47 -0600

This commit was pushed up to Clojurescript core recently, adding a new macro specify. This means we can now have instance-level implementations of protocols on specific values:

    (ns example1)
    (enable-console-print!)

    (defprotocol Listable
      (render [items]))


    (def elements
     (specify ["one" "two" "three"]
       Listable
       (render [items]
         (str
           "<ul>"
           (apply str (map #(str "<li>" % "</li>") items))
           "</ul>"))))

    (println (satisfies? Listable elements))
    ; true

    (println (satisfies? Listable ["one" "two" "three"])) ; false
    ; false

    (println (render elements))
    ; <ul><li>one</li><li>two</li><li>three</li></ul>

    (println elements)
    ; [one two three]

    (println (= elements ["one" "two" "three"]))
    ; true
    (println (identical? elements ["one" "two" "three"]))
    ; false

This is pretty important feature for Clojurescript. extend-type is still powerful, but there is the odd time where you want to make a specific value conform to an interface. Because we’re on Javascript, we can get this level of modularity without a huge performance cost.

Note that this relies on version 0.0-2138 of Clojurescript.

This change was authored by the indomitable David Nolen whose latest open source offering in a Clojurescript interface over Facebook’s React.

Mon, 01 Jan 0001 00:00:00 +0000