Andrew Meredith

How to Grow as a Software Engineer - 5 Tips

2021-10-27T00:00:00+00:00

So you got a job as a software engineer. What now? There are 1,000 technical skills that you can dig into. There is always a new framework to learn. Your company probably has a large codebase that you can get familiar with. You need to know Git, the command line… the list goes on. In many ways, how you approach career growth is more important than the specific things you learn. In this post, we’ll look at 5 tips for helping you grow as a software engineer. We will not be talking about specific technologies since those change all the time. Instead, we’ll look at the mindset that will best help you advance in your software engineering career:

Set Goals
Ask Questions
Let Yourself Get Stuck
Follow Trends
Don’t Follow Trends

1. Set Goals

If you know what you want to achieve, there is always a way to plan how to get there. Without clear goals, it is all too easy to spend hours on things that, in retrospect, did not serve to help you in your career. Reasonable goals guide us in determining what to do and what to say “no” to. That means that a goal must be specific enough to serve as a compass guiding your journey but general enough that it is not just another task on your to-do list. A good rule of thumb is that a goal should be something that you cannot complete over a weekend, and it should also not be something that will take more than a year.

Goals should also be actionable. Just like a good software feature spec, a goal should have criteria that you can use to determine whether you have met it. For example, if you want to learn Vue, your goal should not be “Learn Vue.” Try something like “Learn Vue well enough to rewrite [PROJECT X] in it without referring to documentation” or “Learn Vue so that I can write a blog post comparing it to React.” With more concrete goals and completeness criteria, you will find that it is easier to decide what to work on next.

Also, set aside some time - maybe just five or ten minutes per week - to think back over what you have accomplished and to refocus if necessary. Keep a document where you write down the progress you have made, the things you have learned, and then return to it when you feel discouraged. I promise that it will be motivating to remember how far you have come already!

2. Ask Questions

Whether working on a team or as an individual developer, one of the most helpful practices that you can foster is asking questions - even if you already have a good idea of the answer! By asking someone a question (in person or online), you may get a direct answer, and that is great. However, you may learn another way to think about your problem. Even the exercise of formulating a compelling question can help the way you think about it.

Another good reason to ask questions is to fight imposter syndrome. Most of us struggle with feeling like we are not adequate to solve the problem in front of us. The best way to fight imposter syndrome is by being honest with yourself and others about what you know and what you don’t know (yet). By asking questions, you tell yourself and everyone around you that you want to learn and that you can learn.

3. Let Yourself Get Stuck

Asking questions is great, but sometimes the best thing you can do is struggle through a complex problem. While you should not try to solve every problem from scratch, you should occasionally take the opportunity to use what you already know to figure out something that you don’t. As you advance in your career, you will find that you sometimes face problems that nobody on your team or StackOverflow has solved before. Developing the problem-solving skills to overcome these situations can be one of the most valuable tools you will ever put in your toolbox.

The feeling of accomplishment when you are stuck for a while and finally figure out how to move on is incredibly motivating, and it will give you a nice boost of confidence. If you have been working on one problem for a while without making progress, go back to the previous tip and ask some questions. There is no shame in getting help, and some issues really are incredibly tough. Challenge yourself, but never drive yourself to the point where you feel defeated. Remember that when a problem feels too hard, it is not because you are not good enough to solve it. It is because it requires specific knowledge you haven’t gained yet, so cut yourself some slack, do some research, ask questions, and move on.

If you are the note-taking type, then it can be helpful to chronicle how you wrestled with the problem. Even if you never return to these notes, the process of writing them will help you remember what worked and what did not the next time you encounter a similar problem.

4. Follow Trends

Career growth follows some sort of progression. You start at some point, and you grow toward some future version of yourself. To best direct the goals you set (remember the first tip!) to get to the version of yourself that you want to be, you need to be strategic. Want to be a kick-ass front-end engineer? Learning React and Tailwind are probably better uses of your time than learning jQuery. Want to get into cloud-native development? Go, Kubernetes, and service mesh should be the things that you focus on.

Try to dedicate a little time every week to keep up with what is new and exciting in the communities that interest you. This can be difficult because there is a lot of noise to filter for every bit of signal. At first, it may be difficult to tell what is hype and what is a legitimate trend. You will develop the skills over time to predict whether some new technology is going to have a lasting impact. Still, at least initially, you may want to be among the early adopters or early majority for a tool or technique rather than try to be among the innovators. It feels good to be on the bleeding edge, but it feels better to look back with confidence in what you have chosen to invest your time in.

Finally, your interests will probably change over time, and that is a good thing! Don’t let dedication to something you used to enjoy prevent you from pivoting and pursuing something else. Sometimes it can even be advantageous to shift directions in your career because you will be a more well-rounded individual and bring a unique perspective that not everyone else will have. Personally, I have learned more from engineers who started doing something else - from QA to quantum physics research - than from people who have spent their entire career in a single track. Your career is your own.

5. Don’t Follow Trends

Wait, didn’t I just say to not follow trends? Paying attention to the trends helps you decide what goals to set for yourself, but (from someone who has been there) chasing every new technology leads to burn-out. Try to have just a couple of goals at any point, and don’t spend time on things that don’t further those goals. Ultimately, when you force yourself to prioritize what is most important, you have to just wave at some trends as they pass by. Especially in fast-moving ecosystems, there will always be more you want to discover than you will be able to. Remember that even the developers you look up to are not experts in everything, and they likely know surprisingly little about most of the tech headlines we see.

You can measure your skillset by depth and breadth. Now, it may be perfectly legitimate to have a goal like “Improve the breadth of my front-end knowledge by reading 1 book on each of the 4 most popular frameworks,” but do not try to become an expert in all four frameworks simultaneously. Make sure that you are clear on when you are working for depth and when you are working for breadth.

Conclusion

I hope that this article helps you focus your time and energy on the things that help you get to where you want to be. Whatever direction you want to take your career, whatever the specific things you decide to invest yourself in, remember to cultivate the habit of being mindful about where you are going and purposeful about how you get there.

Sorting a Dependency Graph in Go

2021-10-26T00:00:00+00:00

Recently, I was thinking about how many of the nontrivial problems that I run into with software engineering boil down to a few simple problems. Just look at any book on algorithms, and the majority of them will be some variation on sorting or searching collections. Google exists because the question of “what documents contain these phrases?” is a genuinely hard problem to solve (okay, that is vastly simplifying the enormous scope of Google’s product, but the basic idea still holds).

What is Topological Sorting?

One of these common problems that I have run into again and again over the course of my career is topologically sorting the nodes of a dependency graph. In other words, given some directed acyclic graph - think software packages that can depend on other software packages or tasks within a large company project - sort them such that no item in the list depends on anything that comes later in the list. Let’s imagine that we are making a cake, and before we can get started, we need some ingredients. Let’s simplify things and say we just need eggs and flour. Well, to have eggs, we need chickens (believe me, I’m resisting the urge to make a joke here), and to have flour, we need grain. Chickens also need grain for feed, and grain needs soil and water to grow in. If we consider the graph that expresses all of these dependencies, it looks something like this:

One possible topological order of this graph is:

[]string{"soil", "water", "grain", "chickens", "flour", "eggs", "cake"}

However, there are other possible sort orders that maintain topological ordering:

[]string{"water", "soil", "grain", "flour", "chickens", "eggs", "cake"}

We could also put the flour after the eggs, since the only thing that depends on eggs is cake. Since we can rearrange the items, we could also fulfill some of them in parallel while maintaining that no item appears before anything that depends on it. For example, by adding a level of nesting, we can indicate that everything within an inner slice is independent of anything else in that slice:

[][]string{
    {"soil", "water"},
    {"grain"},
    {"chickens", "flour"},
    {"eggs"},
    {"cake"},
}

From this graph, we get a nice “execution plan” for how to prepare the dependencies for a cake. First, we need to find some soil and water. Next, we grow grain. Then, we simultaneously raise some chickens and make flour. Next, we collect the eggs. Then finally, we can make our cake! That may seem like a lot of work for petit fours, but good things take time.

Building a Dependency Graph

Now that we understand what we are trying to do, let’s think about how to write some code that is able to build this sort of dependency list. We will certainly need to keep track of the elements themselves, and we will need to keep track of what depends on what. In order to make both “What depends on X?” and “What does X depend on?” efficient, we will track the dependency relationship in both directions.

We have enough of an idea of what we need to start writing some code:

// A node in this graph is just a string, so a nodeset is a map whose
// keys are the nodes that are present.
type nodeset map[string]struct{}

// depmap tracks the nodes that have some dependency relationship to
// some other node, represented by the key of the map.
type depmap map[string]nodeset

type Graph struct {
	nodes nodeset

	// Maintain dependency relationships in both directions. These
	// data structures are the edges of the graph.

	// `dependencies` tracks child -> parents.
	dependencies depmap
	// `dependents` tracks parent -> children.
	dependents depmap
	// Keep track of the nodes of the graph themselves.
}

func New() *Graph {
	return &Graph{
		dependencies: make(depmap),
		dependents:   make(depmap),
		nodes:        make(nodeset),
	}
}

This data structure should suit our purposes, since it holds all of the information we need: nodes, “depends-on” edges, and “depended-on-by” edges. Let’s think now about creating the API for adding new dependency relationships to the graph. All we need is a method for declaring that some node depends on another, like so: graph.DependOn("flour", "grain"). There are a couple of cases that we want to explicitly disallow. First, a node cannot depend on itself, and second, if flour depends on grain, then grain must not depend on flour, otherwise we would create an endless dependency cycle. With that, let’s write the Graph.DependOn() method.

func (g *Graph) DependOn(child, parent string) error {
	if child == parent {
		return errors.New("self-referential dependencies not allowed")
	}

	// The Graph.DependsOn() method doesn't exist yet.
	// We'll write it next.
	if g.DependsOn(parent, child) {
		return errors.New("circular dependencies not allowed")
	}

	// Add nodes.
	g.nodes[parent] = struct{}{}
	g.nodes[child] = struct{}{}

	// Add edges.
	addNodeToNodeset(g.dependents, parent, child)
	addNodeToNodeset(g.dependencies, child, parent)

	return nil
}

func addNodeToNodeset(dm depmap, key, node string) {
	nodes, ok := dm[key]
	if !ok {
		nodes = make(nodeset)
		dm[key] = nodes
	}
	nodes[node] = struct{}{}
}

This will effectively add a dependency relationship to our graph once we implement Graph.DependsOn(). We could very easily tell whether a node depends on some other node directly, but we also want to know whether there is a transitive dependency. For example, since flour depends on grain, and grain depends on soil, flour depends on soil as well. This will require us to get the direct dependencies of a node, then for each of those dependencies, get its dependencies and so on until we stop discovering new dependencies. In computer science terms, we are computing a fixpoint to find the transitive closure of the “DependsOn” relation on our graph.

func (g *Graph) DependsOn(child, parent string) bool {
	deps := g.Dependencies(child)
	_, ok := deps[parent]
	return ok
}

func (g *Graph) Dependencies(child string) nodeset {
	if _, ok := g.nodes[root]; !ok {
		return nil
	}
	
	out := make(nodeset)
	searchNext := []string{root}
	for len(searchNext) > 0 {
		// List of new nodes from this layer of the dependency graph. This is
		// assigned to `searchNext` at the end of the outer "discovery" loop.
		discovered := []string{}
		for _, node := range searchNext {
			// For each node to discover, find the next nodes.
			for nextNode := range nextFn(node) {
				// If we have not seen the node before, add it to the output as well
				// as the list of nodes to traverse in the next iteration.
				if _, ok := out[nextNode]; !ok {
					out[nextNode] = struct{}{}
					discovered = append(discovered, nextNode)
				}
			}
		}
		searchNext = discovered
	}
	
	return out
}

Sorting The Graph

Now that we have a graph data structure, we can think about how to get the nodes out in a topological ordering. If we can discover the leaf nodes - that is, nodes that themselves have no dependencies on other nodes - then we can repeatedly get the leaves and remove them from the graph until the graph is empty. On the first iteration we will find the elements that are independent, then on each subsequent iteration, we will find the nodes that only depended on elements that have already been removed. The end result will be a slice of independent “layers” of nodes that are sorted topologically.

Getting the leaves of the graph is simple. We just need to find that nodes that have no entry in dependencies. This means that they do not depend on any other nodes.

func (g *Graph) Leaves() []string {
	leaves := make([]string, 0)

	for node := range g.nodes {
		if _, ok := g.dependencies[node]; !ok {
			leaves = append(leaves, node)
		}
	}

	return leaves
}

The final piece of the puzzle is actually computing the topologically sorted layers of the graph. This is also the most complex piece. The general strategy that we will follow is to iteratively collect the leaves and remove them from the graph until the graph is empty. Since we will be mutating the graph, we want to make a clone of it so that the original graph is still intact after we perform the sort, so we’ll go ahead and implement that clone:

func copyNodeset(s nodeset) nodeset {
	out := make(nodeset, len(s))
	for k, v := range s {
		out[k] = v
	}
	return out
}

func copyDepmap(m depmap) depmap {
	out := make(depmap, len(m))
	for k, v := range m {
		out[k] = copyNodeset(v)
	}
	return out
}

func (g *Graph) clone() *Graph {
	return &Graph{
		dependencies: copyDepmap(g.dependencies),
		dependents:   copyDepmap(g.dependents),
		nodes:        copyNodeset(g.nodes),
	}
}

We also need to be able to remove a node and all edges from a graph. Removing the node is simple, as is removing the outbound edges from each node. However, the fact that we keep track of every edge in both directions means that we have to do a little extra work to remove the inbound records. The strategy that we will use to remove all edges is as follows:

Find node A’s entry in dependents. This gives us the set of nodes that depend on A.
For each of these nodes, find the entry in dependencies. Remove A from that nodeset.
Remove node A’s entry in dependents.
Perform the inverse operation, looking up node A’s entry in dependencies, etc.

With the help of a small utility that allows us to remove an node from a depmap entry, we can write the method that completely removes a node from the graph.

func removeFromDepmap(dm depmap, key, node string) {
	nodes := dm[key]
	if len(nodes) == 1 {
		// The only element in the nodeset must be `node`, so we
		// can delete the entry entirely.
		delete(dm, key)
	} else {
		// Otherwise, remove the single node from the nodeset.
		delete(nodes, node)
	}
}

func (g *Graph) remove(node string) {
	// Remove edges from things that depend on `node`.
	for dependent := range g.dependents[node] {
		removeFromDepmap(g.dependencies, dependent, node)
	}
	delete(g.dependents, node)

	// Remove all edges from node to the things it depends on.
	for dependency := range g.dependencies[node] {
		removeFromDepmap(g.dependents, dependency, node)
	}
	delete(g.dependencies, node)

	// Finally, remove the node itself.
	delete(g.nodes, node)
}

Finally, we can implement Graph.TopoSortedLayers():

func (g *Graph) TopoSortedLayers() [][]string {
	layers := [][]string{}

	// Copy the graph
	shrinkingGraph := g.clone()
	for {
		leaves := shrinkingGraph.Leaves()
		if len(leaves) == 0 {
			break
		}

		layers = append(layers, leaves)
		for _, leafNode := range leaves {
			shrinkingGraph.remove(leafNode)
		}
	}

	return layers
}

This method clearly outlines our strategy for topologically sorting the graph:

Clone the graph so that we can mutate it.
Repeatedly collect the leaves of the graph into a “layer” of output.
Remove each layer once it is collected.
When the graph is empty, return the collected layers.

Now we can go back to our original cake-making problem to make sure that our graph solves it for us:

package main

import (
	"fmt"
	"strings"

	"github.com/kendru/darwin/go/depgraph"
)

func main() {
	g := depgraph.New()
	g.DependOn("cake", "eggs")
	g.DependOn("cake", "flour")
	g.DependOn("eggs", "chickens")
	g.DependOn("flour", "grain")
	g.DependOn("chickens", "grain")
	g.DependOn("grain", "soil")
	g.DependOn("grain", "water")
	g.DependOn("chickens", "water")

	for i, layer := range g.TopoSortedLayers() {
		fmt.Printf("%d: %s\n", i, strings.Join(layer, ", "))
	}
	// Output:
	// 0: soil, water
	// 1: grain
	// 2: flour, chickens
	// 3: eggs
	// 4: cake
}

All of that work was not exactly a piece of cake, but now we have a dependency graph that can be used to topologically sort just about anything. You can find the full code for this post on GitHub. There are some notable limitations to this implementation, and I would like to challenge you to improve it so that it can:

Store nodes that are not simple strings
Allow nodes and edges/dependency information to be added separately
Produce string output for debugging

I hope you enjoyed this tutorial! Please let me know what you would like me to cover next.

Looping

2019-09-16T00:00:00+00:00

Making Choices

2019-09-14T00:00:00+00:00

Using Variables and Values

2019-09-13T00:00:00+00:00

Capstone 1 - Weather Forecasting App

2019-09-12T00:00:00+00:00

REPL Crash Course

2019-09-11T00:00:00+00:00

Receiving Rapid Feedback with Figwheel

2019-09-10T00:00:00+00:00

Bootstrapping a ClojureScript Project

2019-09-09T00:00:00+00:00

Expressions and Evaluation

2019-09-08T00:00:00+00:00

Building Blocks

2019-09-05T00:00:00+00:00

ClojureScript in the JavaScript Ecosystem

2019-09-04T00:00:00+00:00

Why ClojureScript Matters

2019-09-01T00:00:00+00:00

Rate Limiting in Javascript with a Token Bucket

2018-12-28T00:00:00+00:00

Recently I was looking into options to add rate limiting to a specific endpoint in an application at work. Most endpoints are only exposed internally, and we are careful to not make more requests than the system can handle. However, in one case, the endpoint is open to our customers, and it runs some pretty intensive database operations, so we wanted to limit the rate at which clients can make requests. This functionality is available in pretty much every API gateway out there as well as in a lot of reverse proxies. In our case, application updates are easier to make than config updates, so we opted for a simple solution that we could deploy as part of our Node.js app.

Enter the Token Bucket.

A token bucket is an algorithm that allows tokens to be accumulated over time at a specific rate. These tokens can then be “redeemed” to execute some action. If there are no tokens available, the action cannot be taken. Imagine that we have a bucket that holds some number of balls, say 100. When there are fewer than 100 balls in the bucket, a machine will automatically refill the bucket at a rate of 1 ball per second until it is full again. We can take as many balls as we want as quickly as we want, but once the bucket is empty, we have to wait for it to start filling up again before we can take more.

If we use a token bucket to rate limit an API, then it allows us to set a request rate (the rate at which tokens are added to the bucket) with the ability to burst above this rate for a short period (until we have drained the capacity of the bucket). Let’s take a first pass at implementing a token bucket.

Initial TokenBucket Implementation

class TokenBucket {

    constructor(capacity, fillPerSecond) {
        this.capacity = capacity;
        this.tokens = capacity;
        setInterval(() => this.addToken(), 1000 / fillPerSecond);
    }

    addToken() {
        if (this.tokens < this.capacity) {
            this.tokens += 1;
        }
    }

    take() {
        if (this.tokens > 0) {
            this.tokens -= 1;
            return true;
        }

        return false;
    }
}

We could then use this in a Node.js/express application to limit the number of requests made to a particular endpoint:

Rate Limiting with TokenBucket

const express = require('express');
const app = express();

function limitRequests(perSecond, maxBurst) {
    const bucket = new TokenBucket(maxBurst, perSecond);

    // Return an Express middleware function
    return function limitRequestsMiddleware(req, res, next) {
        if (bucket.take()) {
            next();
        } else {
            res.status(429).send('Rate limit exceeded');
        }
    }
}


app.get('/',
    limitRequests(5, 10), // Apply rate limiting middleware
    (req, res) => {
        res.send('Hello from the rate limited API');
    }
);

app.listen(3000, () => console.log('Server is running'));

In this example, the / endpoint is restricted to serving 5 requests per second across all clients. If we wanted to have a per-client limit , then we could keep a map of IP address (or API keys) to token bucket, creating a new token bucket every time we encounter a new client, as in the following example:

Rate Limiting by IP

function limitRequests(perSecond, maxBurst) {
    const buckets = new Map();

    // Return an Express middleware function
    return function limitRequestsMiddleware(req, res, next) {
        if (!buckets.has(req.ip)) {
            buckets.set(req.ip, new TokenBucket(maxBurst, perSecond));
        }

        const bucketForIP = buckets.get(req.ip);
        if (bucketForIP.take()) {
            next();
        } else {
            res.status(429).send('Client rate limit exceeded');
        }
    }
}

Using this approach, we would need to be careful, since a large number of distinct IPs could create quite a bit of overhead in terms of both memory and timers to refill buckets. In practice, we would probably want to remove the token buckets after some time, and we would also want to defer adding tokens until they are requested, which will eliminate the need for JavaScript timers. Here is our new timer-free TokenBucket implementation:

Timer-Free TokenBucket

class TokenBucket {
    constructor(capacity, fillPerSecond) {
        this.capacity = capacity;
        this.fillPerSecond = fillPerSecond;

        this.lastFilled = Math.floor(Date.now() / 1000);
        this.tokens = capacity;
    }

    take() {
        // Calculate how many tokens (if any) should have been added since the last request
        this.refill();

        if (this.tokens > 0) {
            this.tokens -= 1;
            return true;
        }

        return false;
    }

    refill() {
        const now = Math.floor(Date.now() / 1000);
        const rate = (now - this.lastFilled) / this.fillPerSecond;

        this.tokens = Math.min(this.capacity, this.tokens + Math.floor(rate * this.capacity));
        this.lastFilled = now;
    }
}

This implementation should be have the same, but it only does work when take() is called, which should be more efficient in most cases.

Please leave a comment to let me know if this post was useful!

Beginning a Database

2018-09-10T00:00:00+00:00

For years, my dream project has been to build a database from scratch. For most of my career, the most challenging questions have revolved around how to ingest, store, and query data in a performant way, and while I have found a number of really great solutions, there usually has not been a single product that can meet all of the requirements.

For instance, at my current job, we receive potentially thousands of updates per second, but we need to run ad-hoc analytical queries over terabytes of data spanning many months. At a high level, we ingest the data into MySQL, and at a certain point, we guarantee that no more data will be written to particular rows, and we have a complex ETL process that moves them to Google BigQuery. On the application side, we have to take user queries and distribute them across both data sources and merge them on the way back. A lot of engineering effort went into this system, and it works remarkably well, but in the process, we ended up essentially building a distributed query engine, a query optimizer, and a high-level storage manager. These are all normally pieces of a database management system, but in our case, they were ad-hoc components that we added to our system as necessary. In this case, the problem that we are solving is needing both a write-heavy OLTP system and a performant OLAP system, which do not normally go together.

I have been doing some reach into database systems, and I think that the research and technology are finally at a point where it is feasible to build a database that has acceptable performance for both transactional and analytics workloads. Between advances in algorithms and datastructures for indexing data, the low cost of DRAM, SSDs, and now, persistent memory, and the understanding of the properties of distributed systems, I believe that we can now start to develop database systems that act as both transactional engines and data warehouses. Particularly, I believe that in-memory (or primarily in-memory) databases are the key to unlocking this sort of system.

This is not a novel idea, and there are a number of databases out there that do in fact have a similar design: HyPer, which is now part of Tableau, and Kudu come to mind. There are a couple of other OLAP databases out there that support inserts or streaming ingestion but not updates, such as Druid and Google BigQuery. My understanding is that Google’s internal PowerDrill product is similar to Druid in that it keeps recent data in memory for faster processing, but as far as I know, it does not support OLTP functionality. Additionally, the trade-offs that these databases make render them unsuitable for certain applications. For instance, Druid (which we initially considered at my current workplace) favors accumulation of highly-denormalized event-structured data. That is, in traditional data warehouse terms, each entry is a fact along with all of its dimensions. While Druid’s compression is fantastic, it was not feasible for us to collect all dimensions that we were interested in tracking with every insert. Additionally, Druid is (or at least was at the time I evaluated it) quite painful to set up and operate.

Ultimately, the database that I would like to build has a hybrid NSM (row-oriented) and DSM (column-oriented) storage model that can seamlessly distribute queries against both engines and handles migration of tuples between each storage layout seamlessly. Also, in my experience, most users are more interested in recent data, so the system should keep data in memory for as long as possible but should still be able to process data that does not reside in main memory, preferably in a parallel fashion. However, unlike a traditional disk-based DBMS, it should not have the overhead of a buffer manager to page old data into memory until it is evicted, since queries over old data are more likely to be one-offs and should not evict the “hot” data set. This sort of parallel processing of OLAP queries over old data is now readily available with technologies like fast object storage, cloud computing, and container orchestration, and it does not need to first be loaded into the database’s primary memory, especially since it is so simple to elastically spin up worker processes to execute the queries over old data in parallel. This is where I believe that a lot of in-memory and columnar databases go wrong. They assume that processing is expensive and centralized. On the other side of the coin, inherantly distributed systems like Hive and Spark SQL assume that all data is far away - at least not already indexed and in memory - and that access paterns are not predictable enough to optimize.

My plan is to begin building the components necessary for a complete DBMS in Rust, partly because I do not have a lot of experience with systems programming, and I like having a very picky compiler to help me from doing a lot of stupid things and partly because I believe that it would be much quicker for me to build a large-scale system in Rust than in C or C++. I plan to write about the components as I build them, including the query parser, planner, and optimizer, MVCC protocols, scheduler, a consensus protocol, distributed hash table, cluster membership and failure detector, storage manager, system catalog, indexes, user interfaces and tooling, and possibly vectorized execution for small-scale paralellism. This is by far the largest project that I have ever taken on, but I hope that through the process I can learn a lot and that the posts to come will be useful to other database developers.

First thoughts on Elixir

2018-02-20T00:00:00+00:00

For the past few days, I have been playing around with Elixir to build an API gateway for WP-Pulsar. While I am far from an Elixir expert (I’d barely even consider myself and Elixir novice yet), I have really started to fall in love with the language!

I tried learning Erlang a few years ago, and while it was an interesting experience, the syntax was a bit of a turn-off, and I could not find a lot of information available on building web applications with Erlang - which is what I was working on at the time. My interests turned more to Clojure and the JVM at the time, and I pretty much forgot about Erlang.

Now that I have decided to learn Elixir, I have to say that it is probably the most fun that I have had programming since I moved from PHP to Ruby. Even in a short time, it has started to change the way that I think about building systems in a functional language.

Syntax

First, there is the syntax. Largely inspired by Ruby, the code is delimited with do … end blocks, like so:

def greet(name) do
  "Hello, #{name}"
end

Apparently, this is just syntactic sugar for a keyword list, which is a list of 2-tuples whose first element is an atom (basically a symbol that simply evaluates to itself), so that same function can be defined as:

def greet(name), [{:do, "Hello, #{name}"}]

Sweet! That’s definitely uglier, but it hints at the fact that code is almost data. What looked like a concrete syntactic construce is just a nested data structure. I have not had a chance to work with Elixir’s macro system yet, but I am guessing that this sort of almost-homoiconic structure is what enables these macros.

For the most part, the syntax reminds me of Ruby, but I can remember a number of times thinking, “Oh, I remember that from Erlang”, such as pattern matching a list: [ head | tail ] = my_list. Also, the pipeline operator (|>) is very similar to Clojure’s threading macro (->). Overall, the language has a pretty friendly syntax, and it felt familiar to my, having worked with both Ruby and some functional languages in the past.

The Best Parts

The nice syntax meant that there was a low barrier to entry, but it’s not what is winning me over. The things that make Elixir stand out are its advanced pattern matching, functions with multiple parameter lists and bodies, the actor model, and OTP.

Pattern matching

First off, Elixir has an if/else construct, but I haven’t used it. Instead, I am relying on pattern matching and functions with multiple parameter lists and bodies. For example, I might write something like the following in JavaScript:

const jobsUrl = (type) => {
  if (type === 'db') {
    return apiUrl('db-jobs');
  }
  return apiUrl('file-jobs');
};

In Elixir, I would just write a function definition for each possible branch:

def jobs_url(:file), do: api_url "/file-jobs"
def jobs_url(:db),   do: api_url "/db-jobs"

There is one definition of the function when it is called with :file and another when it is called with :db. In my opinion, this is much clearer and helps avoid incidental complexity. You can get much more advanced with pattern matching and even add guards to restrict the match to some narrower criteria. For example, the following code checks some HTTP response and determines whether it was successful.

# Succeeded with a normal status code
def resp_status({:ok, %{status_code: status_code}}) when status_code in 200..299 do
  :ok
end

# Succeeded, but not with a status we expected
def resp_status({:ok, _}) do
  :error
end

# Failed with an error
def resp_status({:error, _}) do
  :error
end

I am still getting used to thinking in terms of pattern matching rather than imperative branching, but so far it has made for much more maintainable code (i.e. I can still understand what I wrote the following day).

Actor model and OTP

First off, I have not had a chance to do a ton of concurrent programming with Elixir, so most of my exposure to processes and OTP have been through trivial examples like spinnng up a bunch of processes and passing messages around in a ring. Still, even with toy examples, I’m beginning to see the power that Elixir’s (and Erlang’s) actor model could bring to an application like the API gateway/reverse proxy I am building.

In Elixir, we run things concurrently by running them in their own process and exchanging messages back and forth. Each process gets its own heap (which is garbage-collected independently) and scheduled by the Erlang VM. However, unlike OS processes, these virtual processes are super lightweight (I just created 1,000,000 of them as a test on my desktop without any problems). I am finding that I am thinking of my code in terms of a society and imagining the processes as little people who each have some role in moving that society to some goal. Maybe that’s a common way to feel when encountering this model of programing, but it has been a log of fun.

On top of the actor-based concurrency model that is enabled by lightweight processes, Elixir can take advantage of the Erlang ecosystem, including OTP. As far as I can tell OTP is both a library and a set of conventions for building concurrent, scalable applications. It includes GenServer, which is a behaviour that takes care of some of the boilerplate for defining processes that act as servers and respond to certain types of requests. Being this new to Elixir/Erlang/OTP, it seems like a behaviour is essentially a convention or contract that a process must follow that provides a well-known interface for other processes to use. I’m sure I’m missing subtleties here, and I could be downright wrong, but that is my current take on it.

OTP also has the concept of supervisors, which are processes that are solely responsible for monitoring other processes and deciding what to do when a process under their watch exits. One of the cool things is that supervisors can manage other supervisors, so you can form a hierarchy of supervised processes. Like I mentioned before, this makes the application feel much more organic and like an analogy for a human society.

I hear that there is much more to OTP, including a key/value store and a fully-fleged database, but I’m still trying to get the basics down.

Hopefully this post helped convey some of the things that make Elixir a worthwhile language to learn. I hope to be able to post some more focused, in-depth thoughts as I dig deeper into the language!

Key Takeaways:

The Ruby-like syntax makes Elixir easy to learn
Pattern matching is really cool and eliminates the need for most conditionals
Similarly, splitting a function into multiple bodies is a clean way to refactor code
OTP is huge but amazing
Processes are really, really, really cheap

Functional Data Modeling

2017-09-18T00:00:00+00:00

This tutorial has moved to the book site for Learn ClojureScript:

Lesson 19: Mastering Data with Maps and Vectors

20 Things (that I want to learn)

2016-10-20T00:00:00+00:00

For the past 2 years or so, I have been writing predominantly JavaScript. While I have mixed feelings on JavaScript as a language, the ecosystem is both awesome and utterly terrifying. There are so many options for every aspect of development and tooling that the choices can be paralyzing. The glut of options for, well, everything has caused me to focus more on deciding which technologies I really want to learn and for which ones I can be content in my ignorance.

In the spirit of being intentional about what to learn, I have compiled a list of 20 technologies that interest me. My plan is to learn at least enough about each one to carry on an intelligent conversaion about it and eventually do a deep dive on the ones that I enjoy the most. I plan to share my experience in this learning process as I go, so I’ll be posting to the blog as a work through each item.

The 20 things I would like to learn are - in no particular order:

Containers without docker
ARM assembly
Building a columnar database
Blockchain
Rust Programming Language
Linux filesystems
How an OS Works
strace
Raft consensus protocol
Device drivers
Type inferrence
Go language
Neural networks
TCP
Distributed filesystems and storage
Etherium Smart Contracts
Falcor
BSD Jails
GPU Programming
Algorithmic trading

Let me know if there are any topics that you’d like me to cover first or in more depth, just let me know in the comments. I hope that this learning journey is interesting/helpful!

Securing the service | RESTful Clojure, Part 4

2015-03-13T00:00:00+00:00

Drum roll, please… After more than a year, I finally have some time to pick up this blog series on creating RESTful data APIs with Clojure. It is amazing to me how much the Clojure ecosystem has changed (for the better) even in the past year. One of these changes has been the release of the the Buddy security library Okay, it has been around for over a year, but in the past few months or so, Buddy has become the de-facto library for Clojure web app security.

Originally, I had planned on using Friend - which I still think is a great library - but Buddy is far more flexible and jives well with the Clojure community’s focus of building amazing things from small, composable parts. Buddy consists of modules for crypto, auth/auth, password hashing, and message signing, which may be used together or a la carte. For our shopping list API, we will use the auth and password hashing functionality of Buddy. We will only be scratching the surface of both web security in general and the Buddy library specifically, but I have listed some Resources at the end of this article that are definitely worth a read/watch.

Security Architecture

There are a number of options for securing a web service or data API. One of the main considerations is whether the API will be consumed by other back-end services or by front-end applications. In our case, the API will be consumed by a JavaScript application directly, so we will provide a mechanism that will be simple for that sort of client to consume.

We will introduce two different user types - :user and :admin - who will each have different permissions in the system. Admins should be able to do everything that regular users can as well as a few more privileged actions. Users will authenticate with the system by providing an auth token in an Authorization HTTP header, which the server will verify to determine whether the auth token is valid and which user it is associated with.

Note that for an application with higher security requirements, this approach would probably not be sufficient because if a hacker got the auth token, they could have access to the system without even knowing the identity of the user that they were impersonating. However, as long as the initial token request as well as every single request containing the token happens over HTTPS, we don’t need to be too worried.

If a logged-out client requests a resource that requires authentication, or if a logged-in client an requests a resource that they are not authorized for, an HTTP 401 Unauthorized response should be returned. Eventually, we would probably want to return the 401 only for unauthenticated requests and use an HTTP 403 Forbidden instead for requests that to not have adequate authorization, but to make things simple for this tutorial, we’ll stick with returning the 401 in both cases.

Code, glorious code

With those high-level requirements outlined, let’s start turning them into tests. The previous tutorials were pretty code-heavy, but by now I assume that you know how the project is organized, so I’ll only include the most pertinent portions of code. The full source code is available on the companion repo.

First, we’ll implement password hashing for users. We want to be able to validate a user’s password without storing their password in the database in plain text. Buddy’s hashers module is perfect for this.

Passwords for users

; test/restful_clojure/users_test.clj
; ...
(deftest authorize-users
  (let [user (users/create {:name "Sly" :email "sly@falilystone.com" :password "s3cr3t"})
        user-id (:id user)]
    (testing "Accepts the correct password"
      (is (users/password-matches? user-id "s3cr3t")))

    (testing "Rejects incorrect passwords"
      (is (not (users/password-matches? user-id "not_my_password"))))))
; ...

Before we can write the code to get these tests passing, we need to create a database migration adding a password_digest column to the users table. We also need to add the buddy-hashers dependency to our project.clj. Once those steps are complete, we’re ready to update the restful-clojure.models.users namespace.

; src/restful_clojure/models/users.clj
; ...
(defn find-by [field value]
  (some-> (select* e/users)
          (where {field value})
          (limit 1)
          select
          first
          (dissoc :password_digest)))
; ...
(defn create [user]
  (-> (insert* e/users)
      (values (-> user
                  (assoc :password_digest (hashers/encrypt (:password user)))
                  (dissoc :password)))
      insert
      (dissoc :password_digest)))
; ...
(defn password-matches?
  "Check to see if the password given matches the digest of the user's saved password"
  [id password]
  (some-> (select* e/users)
            (fields :password_digest)
            (where {:id id})
            select
            first
            :password_digest
            (->> (hashers/check password))))

Most of this is pretty standard stuff for a web app, but notice that we are taking care that the password is hashed and stored in the database but that even the hash is never exposed to the client. When you run the tests now, you’ll also notice that they are much slower. This is due to the fact that the bcrypt algorithm that Buddy uses by default to hash passwords is slow. Slowness is a good thing when it comes to password hashing because the slower a hash algorithm is, the less effective it renders brute-force attacks.

Auth tokens

The next order of business is allowing users to supply their user id and password and get an auth token that is valid for some specific amount of time. This time I’ll skip the tests because they are not very interesting.

We’ll need to create a new migration that will create an auth_tokens table:

CREATE TABLE auth_tokens (
    id VARCHAR(36) PRIMARY KEY,
    user_id INTEGER REFERENCES users(id) ON DELETE CASCADE,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX ON auth_tokens (id, created_at DESC);

While we’re at it, let’s go ahead and create another migration adding a level column to the users table to identify a user a being a “user” or “admin”.

ALTER TABLE users ADD COLUMN level VARCHAR(12) NOT NULL default 'user';

We will bundle both authorization and authentication together into one “auth” namespace. I’ll show you the full code below, and then we’ll walk through what it does. This is probably the longest code sample in the tutorial, but it contains most of the internals of auth, so try to follow along.

; src/restful_clojure/auth.clj
(ns restful-clojure.auth
  (:use korma.core)
  (:require [restful-clojure.entities :as e]
            [restful-clojure.models.users :as users]
            [buddy.auth.backends.token :refer [token-backend]]
            [buddy.auth.accessrules :refer [success error]]
            [buddy.auth :refer [authenticated?]]
            [crypto.random :refer [base64]]))

(defn gen-session-id [] (base64 32))

(defn make-token!
  "Creates an auth token in the database for the given user and puts it in the database"
  [user-id]
  (let [token (gen-session-id)]
    (insert e/auth-tokens
      (values {:id token
               :user_id user-id}))
    token))

(defn authenticate-token
  "Validates a token, returning the id of the associated user when valid and nil otherwise"
  [req token]
  (let [sql (str "SELECT user_id "
                 "FROM auth_tokens "
                 "WHERE id = ? "
                 "AND created_at > current_timestamp - interval '6 hours'")]
    (some-> (exec-raw [sql [token]] :results)
            first
            :user_id
            users/find-by-id)))

(defn unauthorized-handler [req msg]
  {:status 401
   :body {:status :error
          :message (or msg "User not authorized")}})

;; Looks for an "Authorization" header with a value of "Token XXX"
;; where "XXX" is some valid token.
(def auth-backend (token-backend {:authfn authenticate-token
                                  :unauthorized-handler unauthorized-handler}))

;; Map of actions to the set of user types authorized to perform that action
(def permissions
  {"manage-lists"    #{:restful-clojure.models.users/user}
   "manage-products" #{:restful-clojure.models.users/admin}
   "manage-users"    #{:restful-clojure.models.users/admin}})

;;; Below are the handlers that Buddy will use for various authorization
;;; requirements the authenticated-user function determines whether a session
;;; token has been resolved to a valid user session, and the other functions
;;; take some argument and _return_ a handler that determines whether the
;;; user is authorized for some particular scenario. See handler.clj for usage.

(defn authenticated-user [req]
  (if (authenticated? req)
    true
    (error "User must be authenticated")))

;; Assumes that a check for authorization has already been performed
(defn user-can
  "Given a particular action that the authenticated user desires to perform,
  return a handler determining if their user level is authorized to perform
  that action."
  [action]
  (fn [req]
    (let [user-level (get-in req [:identity :level])
          required-levels (get permissions action #{})]
      (if (some #(isa? user-level %) required-levels)
        (success)
        (error (str "User of level " (name user-level) " is not authorized for action " (name action)))))))

(defn user-isa
  "Return a handler that determines whenther the authenticated user is of a
  specific level OR any derived level."
  [level]
  (fn [req]
    (if (isa? (get-in req [:identity :level]) level)
      (success)
      (error (str "User is not a(n) " (name level))))))

(defn user-has-id
  "Return a handler that determines whether the authenticated user has a given ID.
  This is useful, for example, to determine if the user is the owner of the requested
  resource."
  [id]
  (fn [req]
    (if (= id (get-in req [:identity :id]))
      (success)
      (error (str "User does not have id given")))))

First, we create a utility function for generating session identifiers, which are cryptographically strong random bits that are base-64 encoded. While it would be easier to use something like a UUID here, that would create a guessable session identifier, making it much easier for a hacker to exploit the system. I am using James Reeve’s crypto-random library here, but any strong random generator is okay here (even pulling bytes off of /dev/urandom).

Next, the make-token! function simple creates a new session in the database associated with the given user id.

We then create a Buddy token-based authentication backend, that will hook into our Ring middleware and look for a “Authorization” HTTP header, extracting the token from that. If a valid token is found, Buddy will associate the returned user in the Ring request map, which we can use to either require that the user is logged-in or that they have a specific user level. The Buddy middleware will call the authenticate-token function with the Ring request map and the token found in the “Authorization” header and will expect a user if the token was valid and nil otherwise.

Finally, we create a simple permission structure such that users can only manage lists (create, add/remove products, etc.), and admins can manage products and users in addition to everything that users can do. In order to create the user hierarchy such that admins will inherit all user privileges, we will create an ad-hoc hierarchy in the users namespace. We also need to make a few changes additional changes to accommodate the addition of user levels. Most of the relevant code from the users namespace is below.

; src/restful_clojure/models/users.clj
; ...
(def user-levels
  {"user" ::user
   "admin" ::admin})
(derive ::admin ::user)

(defn- with-kw-level [user]
  (assoc user :level
              (get user-levels (:level user) ::user)))

(defn- with-str-level [user]
  (assoc user :level (if-let [level (:level user)]
                       (name level)
                       "user")))
; ...
(defn find-by [field value]
  (some-> (select* e/users)
          (where {field value})
          (limit 1)
          select
          first
          (dissoc :password_digest)
          with-kw-level))
; ...
(defn create [user]
  (-> (insert* e/users)
      (values (-> user
                  (assoc :password_digest (hashers/encrypt (:password user)))
                  with-str-level
                  (dissoc :password)))
      insert
      (dissoc :password_digest)
      with-kw-level))
; ...

The two changes here are that we create a hierarchy of ::users and ::admins and that we convert between string representations for storage and keyword representations for programmatic operation. Another minor detail is that we ensure users with no level specified are always stored with the “user” type.

Finally, we make the necessary changes in our handlers to restrict endpoints to logged-in users/users with the appropriate privileges. This is a good time to refactor the handler namespace to extract the business logic for each route to its own function. As the app grows, these will probably move into other namespaces, so for now having the business logic isolated from the routing logic will come in handy. With the routes cleaned up, it is now easier to see how the various authorization rules play out with our application routes.

; src/restful_clojure/models/users.clj
; ...
(defroutes app-routes
  ;; USERS
  (context "/users" []
    (GET "/" [] (-> get-users
                    (restrict {:handler {:and [authenticated-user
                                               (user-can "manage-users")]}
                               :on-error unauthorized-handler})))
    (POST "/" [] create-user)
    (context "/:id" [id]
      (restrict
        (routes
          (GET "/" [] find-user)
          (GET "/lists" [] lists-for-user))
        {:handler {:and [authenticated-user
                         {:or [(user-can "manage-users")
                               (user-has-id (read-string id))]}]}
         :on-error unauthorized-handler}))
    (DELETE "/:id" [id]
        (-> delete-user
            (restrict {:handler {:and [authenticated-user
                                 (user-can "manage-users")]}
                       :on-error unauthorized-handler}))))

  (POST "/sessions" { {:keys [user-id password]} :body}
    (if (users/password-matches? user-id password)
      {:status 201
       :body {:auth-token (make-token! user-id)}}
      {:status 409
       :body {:status "error"
              :message "invalid username or password"}}))

  ;; LISTS
  (context "/lists" []
    (GET "/" []
        (-> get-lists
            (restrict {:handler {:and [authenticated-user
                                       (user-isa :restful-clojure.models.users/admin)]}
                       :on-error unauthorized-handler})))
    (POST "/" [] (-> create-list
                     (restrict {:handler {:and [authenticated-user
                                                (user-can "manage-lists")]}
                                :on-error unauthorized-handler})))
    (context "/:id" [id]
      (let [owner-id (get (lists/find-by-id (read-string id)) :user_id)]
        (restrict
          (routes
            (GET "/" [] find-list)
            (PUT "/" [] update-list)
            (DELETE "/" [] delete-list))
          {:handler {:and [authenticated-user
                           {:or [(user-can "manage-lists")
                                 (user-has-id owner-id)]}]}
           :on-error unauthorized-handler}))))

  ;; PRODUCTS
  (context "/products" []
    (restrict
      (routes
        (GET "/" [] get-products)
        (POST "/" [] create-product))
      {:handler {:and [authenticated-user (user-can "manage-products")]}
       :on-error unauthorized-handler}))

  (route/not-found (response {:message "Page not found"})))

I admit that some of the nesting of routes within contexts can get a little ugly. Many developers prefer to write “flat” routes with a little more duplication, which is admittedly easier to read, but it can become more difficult to maintain with code duplicated between a number of route definitions.

Note, however, how nicely we can express our auth requirements, as in the following example extracted from the routes above:

(restrict
  (routes
    (GET "/" [] find-list)
    (PUT "/" [] update-list)
    (DELETE "/" [] delete-list))
  {:handler {:and [authenticated-user
                   {:or [(user-can "manage-lists")
                         (user-has-id owner-id)]}]}
   :on-error unauthorized-handler}))

This very declaratively expresses that, “For finding, updating, and deleting a specific list, we need the user to be authenticated and either be able to manage lists or be the owner of the specific list”. This, I think, is where Clojure really shines - declarative APIs expressed as data. Did you notice that we lay out our auth rules as some nested maps and vectors? I’ll take that over an imperative auth system any day!

In Conclusion

I have only briefly glossed over a couple of the many security concerns that web apps face. Please know that this is only the tip of the iceberg and that there is much more to building a “bulletproof” web service that what was covered here. That said, the authorization and authentication practices in this tutorial should pretty much cover what you’ll need to build into a typical web service or application of this scale.

If you do not have the code for this tutorial, I’d recommend checking it out from its GitHub repo and playing around with it. If you have been walking through these tutorials from part 1, then you should now have a complete, albeit simple, API server that you can deploy and build a front-end app on!

Next up, we’ll build a simple front-end app to consume our web service, then finally, we will deploy it to a DigitalOcean droplet. Stay tuned!

Resources:

Go To

Part 1: Setup
Part 2: Getting a web server up and running
Part 3: Building out the web service
Part 4: Securing the service

Building out the web service | RESTful Clojure, Part 3

2014-03-01T00:00:00+00:00

Passing our tests

If you recall from the last tutorial, we have a few tests that are currently failing. This is a good thing, because now we know exactly what we need to do to get the tests passing. For reference, here are the tests from last time:

(ns restful-clojure.handler-test
  (:use clojure.test
        ring.mock.request
        restful-clojure.handler))

(deftest test-app
  (testing "users endpoint"
    (let [response (app (request :get "/users"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "lists endpoint"
    (let [response (app (request :get "/lists"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "not-found route"
    (let [response (app (request :get "/bogus-route"))]
      (is (= (:status response) 404)))))

We need to expose /users and /lists as JSON api endpoints, and we also need to have our app send an HTTP 404 response for undefined routes. Since the last test will be the simplest to pass, let’s go ahead and tackle that one first. Open up handler.clj and get rid of the code from the dummy counting routes that we created before, and replace them instead with a handler that only has a not-found route:

(defroutes app-routes
  (route/not-found "{\"message\":\"Page not found\"}"))

(def app app-routes)

If you’re like me, you might be wondering how the not-found route works. Well, since we’re dealing with a Clojure library, we’re in luck! Go ahead and start up a REPL (lein repl from the restful_clojure directory), and we can delve into the code:

(doc compojure.route/not-found)
; => ([body])
; =>   A route that returns a 404 not found response, with its argument as the
; =>   response body.
; => nil
(source compojure.route/not-found)
; => (defn not-found
; =>   "A route that returns a 404 not found response, with its argument as the
; =>   response body."
; =>   [body]
; =>   (wrap-head
; =>     (fn [request]
; =>       (-> (response/render body request)
; =>           (status 404)))))
; => nil

Expecting more? Sorry to disappoint! That is the beauty of lisp - that something like handling 404s can be handled with a 4 line function.

If you run the tests again, you’ll notice that the last one is passing! However, we’re creating a JSON string directly in our code - yuck! Before we move on to implementing the /users and /lists endpoints, let’s introduce some middleware that will create JSON response bodies from Clojure data structures.

Wiring custom Ring middleware

In most applications, there are certain concerns (e.g. logging, authentication, content nedotiation, etc.) that we want to take care of at a high level. In some frameworks, these concerns are taken care of by a Front Controller or plugin. With Ring, however, we can simply wrap our routes in another handler that does… well, pretty much anything we want it to. Can you feel the power? I can feel the power.

Middleware are higher-order functions that take in a handler function and return a handler function. You can modify the request or the response, or you can perform some side-effect like logging the request to a file. Let’s extend our application by adding some extremely basic logging - we’ll log the request map to the server’s console:

; handler.clj
; ...

; Behold, our middleware! Note that it's common to prefix our middleware name
; with "wrap-", since it surrounds any routes an other middleware "inside"
(defn wrap-log-request [handler]
  (fn [req] ; return handler function
    (println req) ; perform logging
    (handler req))) ; pass the request through to the inner handler

; We can attach our middleware directly to the main application handler. All
; requests/responses will be "filtered" through our logging handler.
(def app
  (-> app-routes
    wrap-log-request))

If you fire up the server with lein ring server-headless (or just server if you’re testing it out locally). And hit it with any URL, you should see your request map printed out to the console. This can be helpful for debugging in development but is probably not something that we want in production.

As you can see, writing middleware is not terribly complicated. That doesn’t mean, however, that we should re-invent the wheel just because we can. For our application, we want to convert Clojure data structures to JSON responses (and we probably want to parse JSON request bodies as well). For this we can use ring-json from James Reeves (the maintainer of Ring and a number of other projects). Let’s add it to our project.clj as a dependency:

(defproject restful-clojure "0.1.0-SNAPSHOT"
; ...
:dependencies [
; ... other dependencies ...
		[ring/ring-json "0.2.0"]])

Now over in handler.clj, we can add the middleware provided by ring-json. Here’s what the entire file should look like at this point:

(ns restful-clojure.handler
  (:use compojure.core
        ring.middleware.json)
  (:require [compojure.handler :as handler]
            [ring.util.response :refer [response]]
            [compojure.route :as route]))

; Notice that the body of out not-found route is now a Clojure map.
; The wrap-json-response middleware will take care of converting this to a
; JSON string.
(defroutes app-routes
  (route/not-found
    (response {:message "Page not found"})))

(defn wrap-log-request [handler]
  (fn [req]
    (println req)
    (handler req)))

(def app
  (-> app-routes
    wrap-log-request
    wrap-json-response
    wrap-json-body))

With this middleware in place, we are all set to parse JSON request bodies and serve up JSON responses. Now we are finally ready to implement the core of our application, starting with the tests.

Persisting to a database

In order to persist our data to a database, we’ll add the JDBC driver for PostgreSQL, the Korma library (which provides a nice DSL for querying SQL databases), and Ragtime for database migrations. Let’s add the dependencies to project.clj. I’ll go ahead and display the entire source of the file since we’re adding code in a few places:

(defproject restful-clojure "0.1.0-SNAPSHOT"
  :description "An example RESTful shopping list application back-end written
in Clojure to accompany a tutorial series on kendru.github.io"
  :url "https://github.com/kendru/restful-clojure"
  :license {:name "MIT"
            :url "http://opensource.org/licenses/MIT"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [ring/ring-core "1.2.1"]
                 [ring/ring-jetty-adapter "1.2.1"]
                 [compojure "1.1.6"]
                 [cheshire "5.3.1"]
                 [ring/ring-json "0.2.0"]
                 [korma "0.3.0-RC5"]
                 [org.postgresql/postgresql "9.2-1002-jdbc4"]
                 [ragtime "0.3.4"]]

  :plugins [[lein-ring "0.8.10"]
            [ragtime/ragtime.lein "0.3.6"]]

  :ring {:handler restful-clojure.handler/app
         :nrepl {:start? true
                 :port 9998}}

  ; Have ragtime default to loading the database URL from an environment
  ; variable so that we don't keep production credentials in our
  ; source code. Note that for our dev environment, we set this variable
  ; with Puppet (see default.pp).
  :ragtime {:migrations ragtime.sql.files/migrations
            :database (System/getenv "DB_URL")}

  :profiles
  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                        [ring-mock "0.1.5"]]}
   :test {:ragtime {:database
"jdbc:postgresql://localhost:5432/restful_test?user=restful_test&password=pass_test"}}})

Now we can create the directory to hold our migrations and create our initial migrations.

$ pwd
# /home/andrew/restful-clojure/restful-clojure
$ mkdir migrations
$ NOW=$(date "+%Y-%m-%d-%H%M%S")
$ touch migrations/$NOW-add-initial-tables.up.sql
$ touch migrations/$NOW-add-initial-tables.down.sql

The migrations are in the project repo, but we won’t go into them in detail here. Basically, they create tables for users, lists, products, and product_lists (and a couple other things). If you’re following along with the project, you can just copy the files from the repo.

And with that, we should be able to run the initial migration from our VM to create the users table:

$ lein ragtime migrate
# Applying 2014-03-03-213517-add-initial-tables

And let’s go ahead and migrate the test database as well:

$ lein with-profile test ragtime migrate
# Applying 2014-03-03-213517-add-initial-tables

This will use the comfiguration specified under the :test profile specified in project.clj for migrating the database. We’re set up. We’re migrated. We’re ready to code.

Getting to the good stuff

Let’s begin by writing the tests for our CRUD operations on both users and lists:

; tests/restful_clojure/users_test.clj
(ns restful-clojure.users-test
  (:use clojure.test
        restful-clojure.test-core)
  (:require [restful-clojure.models.users :as users]
            [environ.core :refer [env]]))

; Run each test in an isolated db transaction and rollback
; afterwards
(use-fixtures :each with-rollback)

(deftest create-read-users
  (testing "Create user"
    (let [count-orig (users/count-users)]
      (users/create {:name "Charlie" :email "charlie@example.com"})
      (is (= (inc count-orig) (users/count-users)))))

  (testing "Retrieve user"
    (let [user (users/create {:name "Andrew" :email "me@mytest.com"})
          found-user (users/find-by-id (user :id))]
      (is (= "Andrew" (found-user :name))
      (is (= "me@mytest.com" (found-user :email))))))

  (testing "Find by email"
    (users/create {:name "John Doe" :email "j.doe@ihearttractors.com"})
    (let [user (users/find-by-email "j.doe@ihearttractors.com")]
      (is (= "John Doe" (user :name))))))

(deftest multiple-user-operations
  (testing "Find all users"
    (doseq [i (range 10)]
      (users/create {:name "Test user"
                     :email (str "user." i "@example.com")}))
    (is (= 10 (count (users/find-all))))))

(deftest update-users
  (testing "Modifies existing user"
    (let [user-orig (users/create {:name "Curious George" :email
"i.go.bananas@hotmail.com"})
          user-id (user-orig :id)]
      (users/update-user (assoc user-orig :name "Chiquita Banana"))
      (is (= "Chiquita Banana" (:name (users/find-by-id user-id)))))))

(deftest delete-users
  (testing "Decreases user count"
    (let [user (users/create {:name "Temporary" :email
"ephemerial@shortlived.org"})
          user-count (users/count-users)]
      (users/delete-user user)
      (is (= (dec user-count) (users/count-users)))))

  (testing "Deleted correct user"
    (let [user-keep (users/create {:name "Keep" :email "important@users.net"})
          user-del (users/create {:name "Delete" :email
"irrelevant@users.net"})]
      (users/delete-user user-del)
      (is (= user-keep
             (users/find-by-id (user-keep :id))))
      (is (= nil
             (users/find-by-id (user-del :id)))))))

The basic tests for lists are pretty similar, so I won’t post them here, but they are in the tutorial repository. I have also included a users+lists_test.clj for testing the operations that affect both models. Later in this tutorial we will take a look at the tests and implementation code for dealing with managing the items on a list.

Now the implementation for both users and lists is fairly straightforward. We’re using Korma to simplify database transactions. We can put all of the boilerplace Korma code that we will use to define the entities in our system in a restful-clojure.entities namespace. The models themselves will go in separate namespaces under restful-clojure.models.

; src/restful_clojure/entities.clj
(ns restful-clojure.entities
  (:use korma.core
        restful-clojure.db))

(declare users lists)

(defentity users
  (pk :id)
  (table :users)
  (has-many lists)
  (entity-fields :name :email))

(defentity lists
  (pk :id)
  (table :lists)
  (belongs-to users {:fk :user_id})
  (entity-fields :title))

; src/restful_clojure/models/users.clj
(ns restful-clojure.models.users
  (:use korma.core)
  (:require [restful-clojure.entities :as e]))

(defn find-all []
  (select e/users))

(defn find-by [field value]
  (first
    (select e/users
      (where {field value})
      (limit 1))))

(defn find-by-id [id]
  (find-by :id id))

(defn for-list [listdata]
  (find-by-id (listdata :user_id)))

(defn find-by-email [email]
  (find-by :email email))

(defn create [user]
  (insert e/users
    (values user)))

(defn update-user [user]
  (update e/users
    (set-fields (dissoc user :id))
    (where {:id (user :id)})))

(defn count-users []
  (let [agg (select e/users
              (aggregate (count :*) :cnt))]
    (get-in agg [0 :cnt] 0)))

(defn delete-user [user]
  (delete e/users
    (where {:id (user :id)})))

Again, since the boilerplate code for lists is not too interesting (and it’s very similar to the code for users, I am not posting the entire code here. I’ll only cover what’s different and interesting about lists - adding and removing items.

You know the drill. We start with the tests:

(By the way, I know that unit tests are not the most riveting read, so please do not feel beholden to take in every line. I post the tests here in case you are interested in seeing some examples of ordinary tests in Clojure.)

; test/restful_clojure/lists_test.clj
; ...
(deftest add-products
  (let [user (users/create {:name "Test user" :email "me@mytest.com"})
        my-list (lists/create {:user_id (:id user) :title "My list"})
        pintos (products/create {:title "Pinto Beans"
                                 :description "Yummy beans for burritos"})]
    (testing "Adds product to existing list"
      (let [modified-list (lists/add-product my-list (:id pintos))]
        (is (= [pintos] (:products modified-list)))))

    (testing "Creates new list with products"
      (let [listdata (lists/create {:user_id (:id user)
                                    :title "Most interesting"
                                    :products [pintos]})]
        (is (= [pintos] (:products listdata)))
        (is (= [pintos] (:products (lists/find-by-id (:id listdata)))))))

    (testing "Creates products added with an update"
      (let [listdata (lists/create {:user_id (:id user)
                                    :title "Things to update"
                                    :products [pintos]})
            coffee (products/create {:title "Coffee Beans"
                                     :description "No, not *THAT* Java"})
            updated (lists/update-list (update-in listdata [:products] conj coffee))]
        (is (= [pintos coffee] (:products updated)))
        (is (= [pintos coffee] (:products (lists/find-by-id (:id listdata)))))))))

(deftest remove-products
  (let [user (users/create {:name "Test user" :email "me@mytest.com"})
        kidneys (products/create {:title "Kidney Beans"
                                  :description "Poor Charlie the Unicorn..."})
        limas (products/create {:title "Lima Beans"
                                :description "Yuck!"})
        my-list (lists/create {:user_id (:id user)
                               :title "My list"
                               :products [kidneys limas]})]
   (testing "Does not remove a product from the database entirely"
     (let [fresh-list (lists/create {:user_id (:id user)
                                     :title "My list"
                                     :products [kidneys limas]})]
       (lists/remove-product fresh-list (:id kidneys))
       (is (not (nil? (products/find-by-id (:id kidneys)))))))
    (testing "Removes a product from a list"
      (let [modified-list (lists/remove-product my-list (:id kidneys))]
        (is (= [limas] (:products modified-list)))))

    (testing "Removes products absent from an update"
      (let [coffee (products/create {:title "Coffee Beans"
                                     :description "No, not *THAT* Java"})
            listdata (lists/create {:user_id (:id user)
                                    :title "Things to update"
                                    :products [limas coffee]})
            updated (lists/update-list (assoc listdata :products [coffee]))]
        (is (= [coffee] (:products updated)))
        (is (= [coffee] (:products (lists/find-by-id (:id listdata)))))))))

Although I just pasted all of these tests in at once, keep in mind that when writing code, I try to only write one (failing) test at a time, then write the code to implement it, but it would be distracting to have so many tiny code samples littering the post.

There are a couple of interesting things going on here. First, we want to be able to associate products with a list when it is created and associate/disassociate them when the list is updated. Second, when we remove a product from a list, we want to be sure that we are just breaking the relationship between lists and products, not deleting the product entirely. Now the application code:

; src/restful_clojure/models/lists.clj
; ...
(defn add-product
  "Add a product to a list with an optional status arg"
  ([listdata product-id]
    (add-product listdata product-id "incomplete"))
  ([listdata product-id status]
    (let [sql (str "INSERT INTO lists_products ("
                   "list_id, product_id, status"
                   ") VALUES ("
                   "?, ?, ?::item_status"
                   ")")]
      (exec-raw [sql [(:id listdata) product-id status] :results])
      (find-by-id (:id listdata)))))

(defn remove-product [listdata product-id]
  (delete "lists_products"
    (where {:list_id (:id listdata)
            :product_id product-id}))
   (update-in listdata [:products]
     (fn [products] (remove #(= (:id %) product-id) products))))

(defn- get-product-ids-for
  "Gets a set of all product ids that belong to a particular list"
  [listdata]
  (into #{}
    (map :product_id
      (select "lists_products"
        (fields :product_id)
        (where {:list_id (:id listdata)})))))

(defn update-list [listdata]
  (update e/lists
    (set-fields (dissoc listdata :id :products))
    (where {:id (:id listdata)}))
  (let [existing-product-ids (get-product-ids-for listdata)
        updated-product-ids (->> (:products listdata)
                                 (map :id)
                                 (into #{}))
        to-add (difference updated-product-ids existing-product-ids)
        to-remove (difference existing-product-ids updated-product-ids)]
    (doseq [prod-id to-add]
      (add-product listdata prod-id))
    (doseq [prod-id to-remove]
      (remove-product listdata prod-id))
    (find-by-id (:id listdata))))

Notice that in the add-product function, we build up a query to execute against the database directly. This is because we are taking advantage of Postgres’s Enumerated Types, which do not play nice with Korma. I am slowly becoming convinced that SQL is best written in SQL, so if I were to start this tutorial again, I would probably use something like yesql.

Additionally, when we update a list we compare the set of products that are on the list at update time to the products that were previously on the list, adding the new products, and removing the products that are no longer present.

Coming up next…

Whew. That was a lot of code! If you have made it this far, I applaud you. In the next tutorial, I will cover adding some basic authentication and authorization to the API using buddy, which is a very flexible security library. After that, we’ll create a simple ClojureScript client to consume our API - after all, what good is a service without anything to use it? Finally, we’ll deploy the app to a DigitalOcean server (disclaimer: affiliate link) using nginx as a reverse proxy and SSL terminator.

Go To

Part 1: Setup
Part 2: Getting a web server up and running
Part 3: Building out the web service
Part 4: Securing the service

Getting a web server up and running with Compojure | RESTful Clojure, Part 2

2014-02-19T00:00:00+00:00

This tutorial is the second in a series on building a RESTful web service in Clojure. In the first tutorial, we set up a development environment using Puppet and Vagrant, and we created a skeleton Leiningen project. With all of the prerequisites taken care of, now it’s time to get our hands dirty with some code!

Ring

If you have done any web programming in Clojure before, you have probably used Ring. Ring provides a very thin abstraction over the HTTP request->response cycle. It is similar Ruby’s Rack or the Java servlets specification, allowing you to write applications that respond to HTTP requests without having to deal with the low-level details of sockets, request parsing, etc.

In order to make use of Ring in our application, we need to add it as a dependency in Leiningen’s project.clj.

; project.clj
(defproject restful-clojure "0.1.0-SNAPSHOT"
  ; ...project settings...

  ; The :dependencies key maps to a vector containing all dependencies
  ; necessary for our project. The dependency on Clojure itself should
  ; have already been added by Leiningen. We will add Ring, the Jetty
  ; adapter (so we can start up a web server to serve our application),
  ; and Compojure, which we will use later on in this tutorial.
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [ring/ring-core "1.2.1"]
                 [ring/ring-jetty-adapter "1.2.1"]
                 [compojure "1.1.6"]])

With our dependencies declared, let’s fire up a REPL and build a quick server (Leiningen will download the dependencies before launching the repl):

lein repl

(use 'ring.adapter.jetty)
; => nil
(defn app-handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello from Ring"})
; => #'user/app-handler
(run-jetty app-handler {:port 3000})

That’s it! We now have a Ring application up and running, albeit a pretty boring one. If you visit localhost:3000 in a web browser, you should see the message, “Hello from Ring” displayed.

Ring basics

A Ring application consists of a handler function, which is just a normal Clojure function that takes a request map and returns a response map. In the case of the app-handler function above, we’re binding the request map to the request var and returning a response map with :status, :headers, and :body keys. Let’s go back to the REPL and create a new server that responds by printing out the request map received as a Clojure map:

(use 'ring.adapter.jetty)
; => nil
(defn app-handler [request]
  {:status 200
   :headers {"Content-Type" "text/plain;=us-ascii"}
   :body (str request)})
; => #'user/app-handler
(run-jetty app-handler {:port 3000})

Now when you visit localhost:3000 with a browser (or any other HTTP client), you should see your HTTP request printed as a Clojure map.

As I said above, Ring is a thin abstraction over HTTP. Consider the following HTTP request and response below (with some formatting applied to the response body):

GET /page HTTP/1.1
Host: localhost
Accept: text/plain

HTTP/1.1 200 OK
Date: Sat, 22 Feb 2014 05:21:47 GMT
Content-Type: text/plain;charset=ISO-8859-1
Content-Length: 350
Server: Jetty(7.6.8.v20121106)

{:ssl-client-cert nil,
:remote-addr "0:0:0:0:0:0:0:1",
:scheme :http,
:request-method :get,
:query-string nil,
:content-type nil,
:uri "/page",
:server-name "localhost",
:headers {"accept" "text/plain", "host" "localhost"},
:content-length nil,
:server-port 80,
:character-encoding nil,
:body #<HttpInput org.eclipse.jetty.server.HttpInput@2523a57f>}

You can see how Ring converted our request into a map, which is just a Clojure datatype. We now have all of the power of Clojure to pull apart that request to generate a response.

Okay, so we can create a “Hello World” web server in Clojure. Big deal. Let’s move on to something more useful.

Meet Compojure

Ring is a great library, but it is pretty low-level, and building a web app directly on top of Ring would be only mildly less tedious than herding cats (and probably not nearly as entertaining). One of the beautiful aspects of functional programming in general, and Clojure in particular, is that you can work with very high-level abstractions that are very close to the business domain of whatever application you are writing. Why bother when we can work more abstractly?

Enter Compojure, an excellent library from James Reeves that abstracts away some of the details of Ring and provides us with a simple interface for routing requests. If you have ever used Sinatra in Ruby, Flask in Python, or Slim in PHP, then Compojure will seem pretty familiar.

Before we dive in, there are a couple more options that we should add to project.clj to make our development a little smoother with Ring and Compojure.

; project.clj
(defproject restful-clojure "0.1.0-SNAPSHOT"
  ; ...project settings...

  ; The lein-ring plugin allows us to easily start a development web server
  ; with "lein ring server". It also allows us to package up our application
  ; as a standalone .jar or as a .war for deployment to a servlet contianer
  ; (I know... SO 2005).
  :plugins [[lein-ring "0.8.10"]]

  ; See https://github.com/weavejester/lein-ring#web-server-options for the
  ; various options available for the lein-ring plugin
  :ring {:handler restful-clojure.handler/app
         :nrepl {:start? true
                 :port 9998}}
  :profiles
  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                        [ring-mock "0.1.5"]]}})

Instead of a lengthy introduction, let’s start by playing with some code! Create restful-clojure/src/restful_clojure/handler.clj, and use the code from below to create an application that can count either up or down.

(ns restful-clojure.handler
  (:use compojure.core)
  (:require [compojure.handler :as handler]))

(defn- str-to [num]
  (apply str (interpose ", " (range 1 (inc num)))))

(defn- str-from [num]
  (apply str (interpose ", " (reverse (range 1 (inc num))))))

(defroutes app
  (GET "/count-up/:to" [to] (str-to (Integer. to)))
  (GET "/count-down/:from" [from] (str-from (Integer. from))))

Starting a web server

Now that we have a counting API, let’s get the web server started. Instead of running locally, we’ll start the server on our VM to mimick a more production-like environment. Change into the root of the repository (the directory that contains the Vagrantfile, and log onto your VM:

vagrant up
vagrant ssh
cd /vagrant
lein ring server-headless

If all goes well, you should be able to visit http://192.168.33.10:3000/count-up/10 on your host machine and see that your web server does indeed know how to count. One thing that you may have noticed when you started the server on your VM was the line,

Started nREPL server on port 9998

nREPL allows us to connect to a REPL remotely and interact with our server as it is running. If you want to play around with it, you can connect to the REPL on your VM using a number of tools, including Leiningen:

lein repl :connect 192.168.33.10:9998

(in-ns 'restful-clojure.handler)
(defn str-to [num] "I forgot how to count!")

Now if you reload http://192.168.33.10:3000/count-up/10 in your browser, you should see “I forgot how to count!” Pretty cool, huh? Just remember that in the immortal words of Uncle Ben, “With great power comes great responsibility.”

Test-driven development

I am a firm believer in the value of writing tests first. There are several huge advantages that I can identify in TDD:

When you write the test first, it forces you to think through what the expected functionality is
Tests serve as technical documentation that describe by example what any given portion of code is supposed to do
It’s satisfying to see your code transform a failing test into a passing test.

That said, tests are not an excuse to not think about the code you’re writing. Just because a test passes does not mean that the code under test is elegant or efficient. Tests are just a tool in our toolbox to help us craft solid, reliable applications.

Instead of writing tests for our “dummy” counting application, let’s go ahead and write several tests for the real application that we’ll begin building in the next tutorial. We know that we are going to have a “users” resource as well as a “lists” resource that should both return JSON, so let’s go ahead and write the tests to check that loading “/users” and “/lists” return an HTTP 200 response code with a JSON response body. Remove the existing test in tests/restful_clojure/ and create tests/restful_clojure/handler_test.clj with the code below.

(ns restful-clojure.handler-test
  (:use clojure.test
        ring.mock.request  
        restful-clojure.handler))

(deftest test-app
  (testing "users endpoint"
    (let [response (app (request :get "/users"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "lists endpoint"
    (let [response (app (request :get "/lists"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "not-found route"
    (let [response (app (request :get "/bogus-route"))]
      (is (= (:status response) 404)))))

We’re using the clojure.test framework for testing because it probably has the lowest barrier to entry of any of the Clojure testing frameworks out there. If you’re used to a tool like Rspec and would like something a little more expressive, I’d recommend looking into Midje. Now let’s run those tests and watch them fail.

lein test

lein test restful-clojure.handler-test

lein test :only restful-clojure.handler-test/test-app

FAIL in (test-app) (handler_test.clj:9)
users endpoint
expected: (= (:status response) 200)
  actual: (not (= nil 200))

... MORE OUTPUT ...

Ran 1 tests containing 5 assertions.
5 failures, 0 errors.

Perfect - we now know exactly what we need to implement in order to get these tests to pass. In the next tutorial, we’ll start implementing a few features of our RESTful web service, including authentication and authorization, a data model, and a full RESTful interface for our users and lists resources (all with tests, of course).

I would love to get your feedback on these tutorials! Useful or not useful?

Go To

Part 1: Setup
Part 2: Getting a web server up and running
Part 3: Building out the web service
Part 4: Securing the service

Writing a RESTful web service in Clojure | Part 1: Setup

2014-02-16T00:00:00+00:00

Over the next several posts, we’ll be doing and end-to-end walkthrough of creating a RESTful web service using the Clojure programming language. Unlike a lot of webapp tutorials on the web, this tutorial series will focus on the entire project as a whole, including infrastructure setup using Vagrant and Puppet, testing/CI, and deploying to Heroku.

The project that we will be building is a shopping list application. The basic operation will be fairly simple, but it will allow us to tackle most of the issues that we’ll run into building RESTful web services professionally. At the end of the tutorial series, we’ll build a simple javascript client application so that we can actually enjoy the fruits of our labour. You should be able to extend the application to create an awesome web or mobile app.

All of the source code for this project is available as a GitHub repo. Each part of the project has a corresponding branch that you can checkout for that specific part. For example, for this first tutorial, you can checkout part2. Please fork the project and experiment with it. I’m releasing all of the code under a MIT license, so you can do pretty much anything you want with it.

Project setup

In this first tutorial, we’ll be setting up a development environment with Vagrant and Puppet, and we will set up our initial project with Leiningen. I have seen a number of projects that started out small then grew too big too fast, retaining an ad-hoc build process and a development environment that took days to get set up. For our project, we’ll avoid that technical debt by manging our development environments with Vagrant (using Puppet for provisioning) and managing our build with the excellent Leiningen build tool for Clojure.

Let’s start out by downloading Leiningen and creating a new project. Leiningen is a build tool written in Clojure and designed specifically for Clojure projects. Over the course of this tutorial, we’ll be using it for running tests, controlling database migrations, and packaging our application for deployment. Its plugin system makes it suitable to be a general-purpose task runner, so any process that you find yourself repeating often while developing an app can be easily automated with Leiningen. If you have used ruby’s Rake or a similar tool, you should be able to pick up on Leiningen very quickly. Although we won’t go into Leiningen internals in this tutorial series, I would strongly encourage you to read over the documentation and even dig into the code - it is an awesome tool that I have found invaluable for developing in Clojure. With that brief introduction, let’s dig in!

You’ll need to download the lein script and place it somewhere on your path. Assuming you have a bin subdirectory inside your home directory, the following commands are all that is necessary to get started:

wget https://raw.github.com/technomancy/leiningen/stable/bin/lein -O ~/bin/lein
chmod a+x ~/bin/lein

With that, we can create our project with one command:

lein new restful-clojure

This will create the basic project structure that we will be building our app on. I have removed the LICENSE and README.md files that Leiningen generates and have placed a license and readme in the project root instead. Lein works under the assumption that the project forder that it generates will be the root of a project, but since we’re keeping Vagrant/Puppet configuration in our project as well, our project root will actually be the parent folder to the folder that Leiningen generates. Right now, your project should look something like the following:

├── LICENSE
├── README.md
└── restful-clojure
    ├── doc
    ├── project.clj
    ├── resources
    ├── src
    └── test

Next up, let’s install Vagrant. For this tutorial, I am running 1.4.3, but any 1.4 release should work. Head over to the Vagrant downloads page to find the package for your OS. Vagrant uses VirtualBox to host its VMs, so please download the latest version. Once you have Vagrant and VirtualBox installed, grab the project source from GitHub:

git clone --recursive https://github.com/kendru/restful-clojure
cd restful-clojure
git checkout part1

At this point, you can run vagrant up to download the vagrant box and boot your dev VM. Since the box is about 400MB, go ahead and start Vagrant so that the download will start. In the meantime, let’s take a look at the Vagrantfile:

config.vm.box = "ubuntu-puppetlabs"
config.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210.box"

Here we are instructing Vagrant to download the box from the Puppetlabs URL as “ubuntu-puppetlabs”. If you want to use another box, you can easily specify the box name and URL here.

The next settings we’ll tweak are the network settings. We’ll assign the VM an IP on a private network so that we can treat it almost like a remote machine and test our deployment using the VM. We also allow the machine to access our host machine’s network so that we can take advantage of our host’s internet connection for downloading the packages that we want to install.

config.vm.network :private_network, ip: "192.168.33.10"
config.vm.network :public_network

We also want to be able to access our project from within the VM, so we’ll want to mount our Leiningen project folder within the VM.

config.vm.synced_folder "./restful-clojure", "/vagrant"

Finally, we configure Vagrant to provision our VM using Puppet. We create a puppet folder with manifests and modules subfolders.

mkdir puppet puppet/manifests puppet/modules

For our app, our only dependencies (at least initially) will be a Java runtime and the PostgreSQL database. Instead of manually writing all of the boilerplate Puppet code for setting up Java and Postgres, we’ll use Puppet modules, which we’ll add to our repo as git submodules If you have cloned the project from github, you can get the submodules by executing:

git submodule update --init --recursive

Otherwise, add all of the puppet modules that we need as git submodules within puppet/modules:

git submodule add https://github.com/puppetlabs/puppetlabs-stdlib.git puppet/modules/stdlib
git submodule add https://github.com/puppetlabs/puppetlabs-postgresql.git puppet/modules/postgresql
git submodule add https://github.com/puppetlabs/puppetlabs-concat.git puppet/modules/concat
git submodule add https://github.com/softek/puppet-java7 puppet/modules/java7
git submodule add https://github.com/puppetlabs/puppetlabs-apt.git puppet/modules/apt

Finally, we’ll create a Puppet manifest at puppet/manifests/default.pp and configure Vagrant to use this manifest for provisioning the VM:

Vagrantfile

config.vm.provision "puppet" do |puppet|
	puppet.manifests_path = "puppet/manifests"
	puppet.module_path = "puppet/modules"
	puppet.manifest_file = "default.pp"
end

puppet/manifests/default.pp

exec { 'update_packages':
	command => 'apt-get update',
	path    => '/usr/bin',
}

# Install Leiningen to run tests and build project
exec { 'get_leiningen':
	command => '/usr/bin/wget https://raw.github.com/technomancy/leiningen/stable/bin/lein -O /usr/bin/lein && /bin/chmod a+x /usr/bin/lein',
	unless  => '/usr/bin/which lein &>/dev/null',
}


class { 'postgresql::server': }

postgresql::server::db { 'restful_dev':
	user     => 'restful_dev',
	password => postgresql_password('restful_dev', 'pass_dev'),
}

postgresql::server::db { 'restful_test':
	user     => 'restful_test',
	password => postgresql_password('restful_test', 'pass_test'),
}

include java7

From this point, fire up your VM, and you should have your environment available:

# If you have not yet started your vagrant box
vagrant up
# If you have already run 'vagrant up'
vagrant provision

Congratulations - you now have a functional Leiningen project set up with a shiny new development VM to run it on! In the next tutorial, we’ll take our application from skeleton to basic web server with a few tests to boot.

Go To

Part 1: Setup
Part 2: Getting a web server up and running
Part 3: Building out the web service
Part 4: Securing the service

Simple is better

2014-02-04T00:00:00+00:00

Why simple?

When I’m committing to a project, I have a general rule of thumb that I like to remove more lines of code than I create. Of course this does not always happen, but I get a warm, fuzzy feeling from knowing that I implemented a feature by removing code. Why? Because when there is less code, there is generally less complexity, and when there is less complexity, the code is easier to grok.

Really, the issue is not that smaller code is better, per se, but simpler code is. In fact, the best code for any given problem is the simplest possible code that actually solves the problem. Of course, there are some problems that are inherently complex, and these will require a complex solution. However, even among complex solutions, there are solutions that are simpler than others.

Let’s take a naive example: your task is to write a program that prints the words “Hello World” to the screen? (sarcasm alert) Well, if we need to print some words, then we naturally need an abstraction for words, right? We’ll certainly also need a separate entity for printing the words to the screen. But then, we’ll probably also want to do other things with words than print them to the screen - after all, our next assignment will probably be to write them to a file or a printer. Looks like we need an interface…

Before long, this is what we end up with:

<?php
class MyString
{
   private $str;

   public function __construct($str)
   {
      $this->str = $str;
   }

   public function unpack()
   {
      return $this->str;
   }
}

class MyStringHolder
{
   private $strings = [];

   public function addString(MyString $str)
   {
      $unpacked = $string->unpack();
      if (empty($unpacked)) return;
      $this->strings[] = $string;
   }

   public function getStrings()
   {
      return $this->strings;
   }
}

interface MyStringHolderViewInterface
{
   public function render(MyStringHolder $holder, array $options);
}

class StdOutPrinterStringHolderView implements MyStringHolderViewInterface
{
   public function render(MyStringHolder $holder,
                          array $options = ['delimeter' => ' '])
   {
      $delimeter = isset($options['delimeter']) ? $options['delimeter'] : ' ';
      $finalString = '';
      foreach($holder->getStrings() as $i => $packedString) {
         $finalString .= ($i >= 1) ? $delimeter : '';
         $finalString .= $packedString->unpack();
      }

      $this->doRender($finalString);
   }

   private function doRender($string)
   {
      if (!empty($string)) {
         print($string);
      }
   }
}

$hello = new MyString("Hello");
$world = new MyString("World");
$helloWorldHolder = new MyStringHolder();
$helloWorldHolder->addString($hello);
$helloWorldHolder->addString($world);
$printer = new StdOutPrinterStringHolderView();
$printer->render($helloWorldHolder);

I sincerely hope that you didn’t actually read all of that. For the poor souls who did, how apparent was it from reading the code that it prints “Hello World” to the screen?

Oh, but it doesn’t actually do that - did I metion that there’s a bug in the code? Yep. It actually doesn’t print anything. I don’t know about you, but I don’t really want to track down that bug right now. (If you’re curious where the bug is, look at the parameters for MyStringHolder::addString).

Maybe an implementation like this would eventually be necessary for your project. If that’s the case, only resort to the complex implementation when necessary. You don’t need the additional layers of abstraction today, so leave them out.

What is simple?

Simplicity in software is not easy to define. Short code is not always simple. Pattern-centric code is not always simple. No specific paradigm of code

OO, functional, imperative, etc. - is always simple. However, I believe that the essence of simplicity boils down to this:

Simple code is code that looks like the problem you are trying to solve.

To borrow the “Hello World” example from above, this is the simplest implementation that I could think of:

print "Hello World";

This solution has a verb, “print”, and a noun, “Hello World”. The vocabulary for the solution comes straight from the problem itself. Since this code performs the designated task with the least amount of abstraction and the least additional vocabulary, it is truly simple.

While this example is overly simplistic, and the actual problems that we as software engineers face are vastly more intricate, we should take away the idea that lean, concise code is always preferrable to overly-designed, overly-abstracted code.

Is complex ever okay?

Yes, but only when we choose the simplest possible solution among many complex solutions. Some problem domains contain so many concepts and so many moving pieces that we can only simplify to a certain extent - and this is not a bad thing. Inherent complexity is a fact that we must live with, but that is not an excuse for introducing additional complexity that is not warrented. Again, always start with the most naive solution that actually works. You will probably have to refactor it eventually, but you may not, and you will not be left with a ton of kludge that you never end up using.

Again, this is a very simplistic look at what simple code is and why we should strive for simplicity, but I hope it was helpful!