Jay Fields' Thoughts

The Single Best Thing For My Career

2012-05-31T09:00:00.001-04:00

In 2004 I was working at nelnet - for one of the best bosses I've ever had. I was the lone developer working on NextGen projects, our customers were happy, I was paid well, & I had plenty of vacation - things were good. Truthfully, I felt like I was on top of the world. However, I couldn't help but feel like something was missing. I'd seen a bit of 'how things could be' by reading Refactoring & Extreme Programming Explained. I was 24 years old, and I felt like my current situation wasn't going to allow me to grow much. I'm convinced that if I were older I wouldn't have changed a thing, but I was young, and I dove into the deep-end of consulting.

Like I mentioned, I'd seen a bit of the light before I made the jump. I had read a few good books, I was using Test Driven Development (TDD) and Continuous Integration at work, and I'd seen the (successful) results of my efforts. Before I joined the NextGen team I was working with a larger group of developers, and at one point we were in the middle of delivering a huge release. While everyone else was working weekends I was either not in the office or I was at my desk reading about poker theory. My component of the release worked, 100% of the time. My tests prevented regressions even as the release evolved. Through the many month ordeal, I never once had an issue. I had seen the benefits of TDD, there was no turning back, and I wanted to mature my skills even further.

I joined ThoughtWorks in early 2005. From the moment I walked into the office I felt like I was drinking from a fire-hose. What's firefox? Better than IE, okay then. What's DI? Inject the dependencies so you can use mocks or stubs in the tests, got it. Ruby on Rails, PostgreSQL, I'm on it. Then there's all the other things that aren't technology related, but are still worth knowing: international travel, frequent flyer programs, points credit cards, car rentals, etc etc. In my first year with ThoughtWorks I grew more as a person than I had in the previous 3 years combined.

Obviously, it's not all roses. In my orientation class I remember everyone admitting that they were going to put in 2 years for the experience and get out - I had the same plan. The guy with the most experience made it 6 months, two other guys made it a year, and I stayed for 3. The constant travel can wear on you, but it's worth doing it for as long as you possibly can.

There's three reasons that consulting was like steroids for my career:

diversity of experience
building my network
focus on my craft

Each project of my consulting career was very different: C# in the Travel Industry, VB.net in Insurance, Ruby (& DSLs) in Credit Cards, Ruby on Rails (RoR) in VoIP, RoR in Auctions, RoR in Advertising. Projects often used different operating systems (ms, linux, apple) and databases (mysql, postgres, oracle, sql server). Naturally, I couldn't become an expert in everything I touched, but the exposure gave me views into each world. Additionally, I was able to meet experts from each of those worlds. When I joined ThoughtWorks I was a C# developer carrying a Dell, when I walked out I was carrying a MacBook Air and I was best known in the Ruby space - consulting completely changed the trajectory of my career path.

As I mentioned, I often got the opportunity to meet with various experts. I don't know Oracle well, but if I have a problem, I know I can email Pramod. Even though he never worked there, I met Stu Halloway while working for ThoughtWorks. Stu introduced me to Rich Hickey, which is pretty nice, since I spend the majority of my time working with Clojure these days. I met Martin Fowler and eventually wrote a book with him. My network gets me invited to conferences, allows me to co-organize speakerconf, and when I joined DRW, I brought 13 friends with me. If you believe that "your network determines your net-worth", then the majority of my net-worth is due to what I built while I was consulting.

At speakerconf 2011 I noted that everyone involved in the Software Craftsmanship movement was either a consultant or had very recently left consulting. The question of why this was the case was sent to the twitterverse. Brian Guthrie's response rang very true (it was something along the lines of): as a consultant, our craft is the only thing we own. Consultants hone their craft endlessly. I'm convinced that's the reason that consultants are almost always ahead of the technology curve. As a consultant, to invest in your business is to invest in yourself. If you work for a consultancy you'll not only have the time to focus on your craft, you'll also likely be surrounded by a group of people who have been doing the same - and are happy to show you what they've learned.

These days, I'm no longer at ThoughtWorks. In the end, I couldn't deal with the travel or the business model. However, that doesn't change the fact that it was the single best thing that has ever happened to my career. I would highly recommend consulting to everyone who is truly serious about maturing their software development skills.

note: Not all consultancies seem to require as much travel, Relevance, for example, requires very little travel - from what I hear. I also hear that it's an amazing place to work. It would be near the top of my list if I were to get back into consulting.

Is Productivity Killing Your Creativity?

2012-05-30T09:00:00.000-04:00

I listen to audio books while I work out. I've been known to leave earworms 'Rapid Italian' on while trying to go to sleep. I read books on trains. I answer emails while eating dinner. I clear out my Google Reader while watching TV with my wife. I wanted to learn Ruby and Blackjack perfect strategy, so I wrote a perfect strategy simulator in Ruby. I'm a multi-tasking machine. I am the worlds most productive man, or so I liked to pretend.

About 4 years ago at a conference in São Paulo, Chad Fowler told the audience to delete every feed in their blog reader if they wanted to gain an incredible amount of productivity. I was appalled. I already believe that engineers don't spend nearly enough time staying current, and Chad was telling the audience to spend even less time reading about current events. I had (and have) way too much respect for Chad to call bullshit, but I definitely didn't agree. Then again, I'm aggressive about removing noise from my subscription list, so I chalked up the disagreement to him recommending that the audience remove what were likely very noisy subscription lists anyway - probably a net positive act.

A few years later I found myself on a train from NYC to Greenwich, CT - listening to the Zen and the Art of Motorcycle Maintenance audiobook. The book changed my perspective greatly, and I'll never forget the line: I haven't really had a new idea in years. The following paragraph haunted me as well.

What is in mind is a sort of Chautauqua...that’s the only name I can think of for it...like the traveling tent-show Chautauquas that used to move across America, this America, the one that we are now in, an old-time series of popular talks intended to edify and entertain, improve the mind and bring culture and enlightenment to the ears and thoughts of the hearer. The Chautauquas were pushed aside by faster-paced radio, movies and TV, and it seems to me the change was not entirely an improvement. Perhaps because of these changes the stream of national consciousness moves faster now, and is broader, but it seems to run less deep. The old channels cannot contain it and in its search for new ones there seems to be growing havoc and destruction along its banks. In this Chautauqua I would like not to cut any new channels of consciousness but simply dig deeper into old ones that have become silted in with the debris of thoughts grown stale and platitudes too often repeated. "What’s new?" is an interesting and broadening eternal question, but one which, if pursued exclusively, results only in an endless parade of trivia and fashion, the silt of tomorrow. I would like, instead, to be concerned with the question "What is best?," a question which cuts deeply rather than broadly, a question whose answers tend to move the silt downstream. There are eras of human history in which the channels of thought have been too deeply cut and no change was possible, and nothing new ever happened, and "best" was a matter of dogma, but that is not the situation now. Now the stream of our common consciousness seems to be obliterating its own banks, losing its central direction and purpose, flooding the lowlands, disconnecting and isolating the highlands and to no particular purpose other than the wasteful fulfillment of its own internal momentum. Some channel deepening seems called for.

As I said, Zen and the Art of Motorcycle Maintenance changed my perspective. I began to look back at the last few years of my life, and I felt like my creativity had been stunted. At one point in my life I would stare out the window of a plane for several hours a week pondering whatever technical problem that was troubling me. But, for the last several years I've been being 'productive' by listening to an audiobook or reading something on my iPad. I've been listening, but I haven't been thinking, not deeply.

If you listen to the people around you, they are saying the same things - whether they know it or not. I'm sure at one time you read an article where a CEO swore they were most productive while they were on a plane, unreachable. Just last week a story made the rounds about a company that moved to Hawaii temporarily. In that story they describe how walks on the beach brought a greater understanding, something that probably couldn't have happened while they were all distracted by the day-to-day activities of living in Silicon Valley. And, I now believe that Chad was talking about the same issue - distractions (masked as productive tasks) stealing your creativity.

I'm convinced that my iPhone was the root of my creativity issues. Life is full of 'waiting time' - waiting for the subway, waiting to see your doctor, waiting in the elevator, waiting in line at airport/grocery store/coffee shop, and waiting at the bar to meet your friends. Pre-iPhone I would spend this waiting time pondering anything that was troubling me. Now, I open Safari on my iPhone to see who is the latest injury on the Knicks, who is the lastest football player to sign with FSU, or who's tweeting about what (seems like it's mostly sponsorship requests these days). I don't spend that time thinking about anything, I spend that time reading - reading about things that have very little impact on my life, but seem to always more than fill my waiting time.

At least, that's what I used to do. Now, I've moved anything that can steal my waiting time to the 2nd page on my iPhone. It's no longer taunting me to check twitter, facebook, sports scores, or anything else. My main page allows me to get things done if I need to do them, but it doesn't offer me anything to fill my 'waiting time'. Those apps are just a page away, but not having them staring at me when I instinctually unlock my phone reminds me that I need that time to think, even if it's not deep thinking, I need to 'do' less and 'think' more.

Even this small step has led to better organization in my mind. Now that I'm not 'productive' 100% of the time, I find myself solving issues with more innovation and greater efficiency. The small step and the noticeable improvement have led me to make larger changes - I still multitask as much as I can, but I also set aside time to stare out the window. No agenda, no priorities, just stare and let my mind go wherever it needs to go. After making these changes, I feel better. I have more mental energy to produce innovative solutions at work, and I find that I'm getting things done in a way that leads to greater long-term productivity. My priorities feel right, if you will.

This isn't the kind of thing I usually produce on this blog, but the impact that my changes have made on my life compelled me to share. And, as I already said, if you listen, more and more people are saying the same thing, even if they aren't using the same words. Technology has driven us to greater 'productivity', but often at the cost of deep concentration and thought. Not everyone is okay with that, and more and more people are beginning to push back in their own ways.

Clojure: Freezing Time in expectations

2012-05-29T09:00:00.000-04:00

The current version of expectations (1.4.3) contains support for freezing time within an expectations scenario. I already put this information out in a previous blog entry, and I'm going to use the same examples here.

The following code is a test I was working on at work:

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (first @fills)))

The test builds a PartialFill domain object, and passes it to handle-fill. The handle-fill fn converts the domain object to a map and conj's the new map onto the fills vector (which is an atom). The above test would fail due to the time not being frozen, and the traditional way to deal with this issue was to change the test to use (in ..) and ignore the :time key-value pair.

The following code would have resulted in a passing test:

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33} (in (first @fills))))

The above code is fine, as long as you're not interested in verifying the time; however, things get a lot uglier if you do want to verify time. The following code freezes (joda) time, allowing for time verification:

(scenario
 (DateTimeUtils/setCurrentMillisFixed 100)
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 100} (first @fills))
 (DateTimeUtils/setCurrentMillisSystem))

While the above code does result in a passing test, it would cause unexpected issues if expect ever failed. In expectations a failure throws an exception (to terminate scenario execution) and the time reset would never occur. Even if that wasn't the case, the time related code takes away from the actual focus of the test.

Therefore, expectations has been modified to take a :freeze-time parameter as part of a scenario definition. The :freeze-time parameter can be a string or a boolean - when specifying a string, anything parse-able by Joda will do; if you specify a boolean time will simply be stopped at the moment of execution.

With :freeze-time available we can rewrite the above test in the following way:

(scenario
 :freeze-time "2012-4-30"
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (first @fills)))

The resulting code is easier to work with, while still allowing verification of time.

I believe the domain related code does a better job of demonstrating the point; however, the following examples are available if you'd like something simplified.

(scenario
  :freeze-time true
  (expect (DateTime.) (do (Thread/sleep 10) (DateTime.))))

(scenario
  :freeze-time "2012-4-30TZ"
  (expect (.getMillis (DateTime.)) 1335744000000))

Clojure: expectations, colorized

2012-05-24T09:00:00.000-04:00

The current version of expectations (1.4.3) prints colorized results by default on non-windows boxes.

The following screenshot is an example of the output when no tests are failing:

And, the following screenshot is an example of the output when there are failures or errors:

Colorized output is one of those small things that is easy to de-prioritize, but once it's done you can't figure out why you didn't do it earlier. The code to colorize your results is very simple, and there's even a lib, colorize, if you prefer to simply include a dependency instead.

Of course, if you prefer to stick with non-colorized results that's possible as well - simply set the environment variable EXPECTATIONS_COLORIZE to 'false'.

Killing IntelliJ Launched Processes

2012-05-23T09:00:00.000-04:00

I often use IntelliJ to run applications, and on occasion things go wrong. For example, a thread that wont terminate can cause a running application to become unstoppable via the IntelliJ UI. Usually when this happens I end up running ps aux | grep java and following up with a kill -9 for each process that looks like it might be the one I'm looking for. On good days there's only a few processes; however, things are more complicated when I have several to look through.

Last week I noticed that the command used to launch the process printed in the Console window, and, more importantly, the idea.launcher.port is part of that command: e.g. idea.launcher.port=7538. Assuming the port is unique, or even almost unique it's much easier to ps aux | grep for than java.

sidenum function

2012-05-22T09:00:00.000-04:00

sidenum function

The sidenum function appends a sign to a number based on a side. The name sidenum was chosen due to it's similarities with signum in spelling and behavior (dealing with signs).

Definition

The sidenum function of a side x and a number y is defined as follows:

note: 'bid' can be replaced with 'buy', and 'ask' can be replaced with 'sell' or 'offer'. The string representation or type of side varies by trading system.

Usage

The sidenum function appears in most trading applications, though not always named 'sidenum'. It can be helpful for both position related math and conserving screen real-estate by displaying 2 concepts (side & quantity) as 1 (a signed quantity).

Examples

Java

    public static int sidenum(String side, int quantity) {
        if (side.equals("ask"))
            return -quantity;
        return quantity;
    }

Clojure

    (defn sidenum [side quantity]
      (if (= side "ask")
        (- quantity)
        quantity))

Ruby

    def sidenum(side, quantity)
      if side == "ask"
        -quantity
      else
        quantity
      end
    end

jQuery find function

2012-05-17T08:45:00.000-04:00

I've recently been using Tilava Table on a few different projects. At one point I ran into a situation where I wanted to change all of the visible rows as well as the template, but the row template was no longer found when using a traditional jQuery selection. The solution to my problem was a combination of holding on to the template as well and using jQuery find to select the td elements I was interested in altering.

In Tilava Table you create a template, but that template row isn't displayed within the table. We could use that as an example, but it seems easier to simplify the example to the following code:

<script src="http://code.jquery.com/jquery-1.7.2.min.js">
</script>
<label><input type="checkbox" id="hideMoreData"/>hide more data</label>
<table id="myTable" style="border:1px solid black">
  <tr class="template">
    <td>row data</td><td class="additional-field">more data</td>
  </tr>
</table>
<button id="addButton">Add</button>
<script>
$(document).ready(function () {
  function addRow(template) {
    $('#myTable').append(template.clone().removeClass('template'));
  }
  var template = $(".template");
  $('#myTable').html("");
  $("#addButton").click(function () {
    addRow(template);
  });
  $("#hideMoreData").click(function () {
    $(".additional-field").toggle(!$('#hideMoreData').is(':checked'));
  });
  addRow(template);
  addRow(template);
});
</script>

The executing version of that code can be found below.

hide more data

row data

more data

In the above example we have a table in which we can dynamically add rows by clicking the 'Add' button. We also have a checkbox that determines whether or not the additional data is displayed in our table. Click the add button a few times and show and hide the additional data, just to make sure everything is working as you'd expect.

You may have noticed (either by reading the code, or by playing with the example) that when you hide the additional data it hides what is already in the table; however, if you add additional rows the additional data for the new rows will be shown. This is due to the fact that when you select all td elements with the class 'additional-field' the template is not included in the results. Even though the template is not returned in our 'additional-field' selection, it does still exist and is accessible as the var 'template'. When we clone the template var the cloned row will contain the additional-field td, but it will not be 'correctly' toggled.

One solution would be to append the row and rerun the toggle code, but that wouldn't be as efficient - and, jQuery gives us a better solution anyway: find

find: Get the descendants of each element in the current set of matched elements, filtered by a selector, jQuery object, or element.

The simple solution is to add the following line in the #hideMoreData click handler.

template.find(".additional-field").toggle(!$('#hideMoreData').is(':checked'));

The finished code is below in it's entirety, and the working version of the code can also be found at the bottom.

<label><input type="checkbox" id="hideMoreData2"/>hide more data</label>
<table id="myTable2" style="border:1px solid black">
  <tr class="template2">
    <td>row data</td><td class="additional-field2">more data</td>
  </tr>
</table>
<button id="addButton2">Add</button>
<script>
$(document).ready(function () {
  function addRow(template) {
    $('#myTable2').append(template.clone().removeClass('template2'));
  }
  var template = $(".template2");
  $('#myTable2').html("");
  $("#addButton2").click(function () {
    addRow(template);
  });
  $("#hideMoreData2").click(function () {
    $(".additional-field2").toggle(!$('#hideMoreData2').is(':checked'));
    template.find(".additional-field2").toggle(!$('#hideMoreData2').is(':checked'));
  });
  addRow(template);
  addRow(template);
});
</script>

hide more data

row data

more data

How I Open Source

2012-05-16T15:07:00.000-04:00

It's been recently brought to my attention that I don't view open-source the way that many of my friends do. My attitude has always been:

Here's some code that works well for what I want. If it works well for what you want, great! If not, I'm willing to make changes that improve the library; however, I'm also going to reject any code that causes the library to bloat. Lastly, the library will likely be 'done' in the next few years - at which time I expect it will be mature enough that a stable final release will be possible, or someone will (re)write a superior version. I don't expect an open-source project will ever define what I accomplish in our industry.

It finally occurred to me that others don't think this way when they began telling me that they don't open-source due to -

the code isn't mature enough
they don't want to write documentation
they don't want their time monopolized by feature request emails

I can understand each point of view, but events have occurred in my career that have shaped my (differing) opinion.

I still remember my first open-source project: I didn't tell a soul about it until I had used it in production for over a year and was very confident that I'd addressed the majority of common use cases. It was a .net object relational mapper called NORM, and it was released in 2005. No, you haven't ever heard of it. I polished it for months, and no one cared. After that I never waited to release anything. I now believe that it's highly unlikely that whatever I create will ever gain any traction, so I might as well get it out there, fail quickly, and move on.

No one writes documentation for themselves, they write it for people who they hope will use their software - and very few people people ever gain anything from someone else using their open-source software. That simple equation makes documentation scarce; however, scarce documentation doesn't mean you can't open-source your software, it just means adoption rates will very likely be slowed.

Two years ago I open-sourced expectations with zero documentation, and documentation stayed at zero for at least a year. In that year very few people paid any attention to expectations; however, expectations does fit a sweet spot for some people, and some adoption did occur. Eventually, new adopters began to send pull requests with documentation, and their contributions inspired me to write some documentation of my own. It can be hard to get motivated about providing documentation to theoretical adopters; however, I got my code out there and adoption began, and the motivation came along with those (no longer theoretical) adopters.

If you end up lucky enough to create a project that is widely used, there's no doubt that you'll start getting swamped with email. In the beginning I expect everyone will be overjoyed with their success, and the added workload will be no big deal. However, I imagine over time it begins to feel like a second full-time job, and for what? Developer-fame doesn't get you closer to retirement any faster than watching grass-grow. However, I don't believe that should deter you from putting your work out there. Additionally, I think GitHub has changed the game with respect to moving on. If your project is on GitHub and you decide to call it quits tonight, there will probably be plenty of forks that are more than happy to take your place.

I have no qualms with walking away from projects, as I expect that if the idea is valuable, someone else will be happy to step up and take my place; furthermore, it's more likely that several people will step up and the strongest will survive - which is best for everyone. The best example I've ever seen of this behavior was Capistrano. Jamis Buck famously walked away from Cap in 2009, yet I still know plenty of people using it today without any issue. I firmly believe that if an idea is good, it'll live on even if you've decided you're ready to do something else.

It occurs to me that I might be a bad open-source citizen - releasing way too early and walking away too early as well. If that's the case, then I expect well deserved criticism, but that's just not how I see the world at this point...

Agile Development with Clojure

2012-05-15T07:24:00.000-04:00

If you've ever spent any time learning Rails then you probably read one of the editions of Agile Web Development with Rails, and if you're like me (skeptical & pedantic) then you probably asked yourself: what the hell does Rails have to do with Agile development? At the time, I assumed that Dave and David were merely capitalizing on the buzz around Agile; however, even if that's the case, I think they did manage to highlight one of my favorite aspects to building websites with Rails: The ability to make a change, reload the page and see the results makes you a much more agile programmer - where 'agile' is defined as: Characterized by quickness, lightness, and ease of movement; nimble

It turns out, it's not very hard to get that same productivity advantage in Clojure as well. I would go so far as to say that the ability to change the server while it's running is assumed if you're using emacs+slime; however, what's not often mentioned is that it's also possible (and trivial) to reload your server code (while it's running) even if you're using IntelliJ, scripts, or anything else.

The majority of the servers I'm working on these days have some type of web UI; therefore, I tie my server side code reloading to a page load. Specifically, each time a websocket is opened the server reloads all of the namespaces that I haven't chosen to ignore. The code below can be found in pretty much every Clojure application that I work on.

(defonce ignored-namespaces (atom #{}))

(defn reload-all []
  (doseq [n (remove (comp @ignored-namespaces ns-name) (all-ns))]
    (require (ns-name n) :reload )))

Like I said, when I open a new websocket, I call (reload-all); however, the (reload-all) fn can be called on any event. When discussing this idea internally at DRW, Joe Walnes pointed out that you could also watch the file system and auto-reload on any changes. That's true, and the important take-away is that you can easily become more productive simply by finding the appropriate hook for what you're working on, and using the code above.

The ignored-namespaces are important for not reloading namespaces that don't ever need to be reloaded (user); other times you'll have a namespace that doesn't behave properly if it's reloaded (e.g. I've found a record + protocol issue in the past, so I don't dynamically reload defrecords in general).

The change-reload webpage-test loop is nice for making changes and seeing the results very quickly - and I strongly prefer it to having to stop and start servers to see new functionality.

Follow-up Thoughts on Aligning Business & Programmer Goals

2012-05-09T16:41:00.000-04:00

My recent entry on Aligning Business & Programmer Goals led to an email conversation that I thought might be worth sharing.

From A. Nonymous:

I have an issue with tying bonuses to performance due to, basically, performance being out of the programmer's hands. Where I'm working right now developers are treated as code monkeys: We're there to implement features the business people dream up, and nothing more. How could I provide more value when I work in an environment where
The visual design phase has already happened (no pushback on any design elements taken seriously)
The business development phase has already happened (i.e. the business decides to create a new service tells it's code monkeys: "we need feature x, y, and z")
The individual projects we're working on is out of our control.
I can't figure out how to tie what I do to business value in the short term (or even in the year-long-term), because I don't have the autonomy to work on things that I think would benefit the business. I'm forced to do the work that someone else assigns me. Are the new features/bug fixes creating value? Definitely. How much value? Not a clue. How can the value I bring be measured? The only metric I'm seeing is "ease of implementing the next new feature", but the next feature that touches the code I wrote probably won't be developed for months, if not years.

So, it doesn't seem justified to assign me a bonus based on the value that features create, when I have no control over the features.

My response:

Given your context, I wouldn't want a bonus either. I imagine that many people are in the same situation... probably most people.

2 notes-

The blog entry isn't solely about bonuses, the ending is all about a visible P&L and nothing about a bonus. I hope people don't completely miss that point.
While your position is status quo, I think we should strive to do better as an industry. I don't think it's healthy that we (programmers in general, not necessarily you or me) work for organizations where we know little about the business, don't trust our business counterparts, and are not trusted by our business counterparts. Programmers are often ruthless about measurement and improvement, and wasting that effort on resume driven development or process tweaks is bad for our businesses and our reputations.
It's easy for me to say that the good programmers should quit companies that don't expose a P&L and don't empower their programmers, but I know that personal situations can limit people's choices. At the same time, I feel good about continuing to write about the path that I take, which includes quitting companies that I don't feel are aligned. Even if my ideas can't help you at this point; hopefully, something will come of them in the future.

A. Nonymous followed up with:

I totally get the visible P&L part and agree with you on it. Having more visibility into the inner workings of the business can only help focus effort where it's important. Not every business leader feels that way, unfortunately. I'm currently fighting with my boss about a similar issue. I alluded to it before, but he wants to follow what many consultancies are now calling "Agile":
analyst meets with business team
analyst comes up with features
qa writes acceptance test for features
developer receives/estimates/completes features
At no point in this process does the developer have any contact with the business team.

Do you have any idea how I'd go about convincing upper management that sharing the P&L would be a good idea?

And, my final response:

I've never seen studies on the impact of sharing the P&L. Most of the discussion around this stuff is still in it's infancy as far as I can tell. Brian Goetz has been talking about 'language & framework introduction' for awhile, but I've never heard it combined with P&L.

I would attack your situation in one of the two following ways-

Fully support your boss' plan, but request to be put on a 'research' or 'experimental' team where you interact directly with the business on a new product (assuming that's possible). I've done this successfully. The rest of IT followed the traditional approach that your boss is proposing, but I broke off with 1 PM and we worked directly with the business on technology for new business lines. I never got to see P&L (though, I never asked), but I would get requirements and deliver features on demand and we ended up constantly impressing the business. e.g. "business: How quickly can you deliver this? (I start thinking of my estimate) business: August? me: (shocked, it was March) I was thinking in the next 2 weeks. -- that conversation actually happened, I'll never forget it. If you get this approved, your boss still gets to do what he wants, and you get to try to do better. If you succeed, he can take the credit for approving the experiment - win/win.
Assuming you cannot find a way to interact with the business, focus all your efforts on reducing the delivery cycle. If he's going to force the team to be basically skilled labor, you need to deliver at a rapid enough pace that you fail quickly and pivot toward profitable business goals. You build, they point you in a better direction, you build, they point you in a better direction, on and on. Your 'interaction' with the business is delivering them software, and you will benefit from doing that as often as possible. So, focus on things like choosing the best tools for the job, getting the build time down, reducing the test running time, etc, etc. Anything that's slowing you down from delivering faster, it's in your best interest to remove that, so you can get the business feedback as fast as possible. Even if your boss forced you to only do a roll-out monthly (or worse), the more you have to roll-out, the more feedback you'll get - so removing any thing that takes your time and isn't contributing to feature delivery. Dan Bodart is doing interesting stuff with build times and you should be able to find plenty of advice on speeding up tests - attack that stuff so you deliver as much to the business as possible.

While A. Nonymous may never get to see the P&L, and may never be in a situation where we would want a bonus - there are steps that can be taken to align the business & the programmers.

Clojure: Conditionally Importing

2012-05-07T18:55:00.002-04:00

I recently ran into a test that needed (org.joda.time.DateTime.) to always return the same time - so it could easily be asserted against. This situation is fairly common, so it makes sense to add support to expectations. However, I didn't want to force a joda-time dependency on everyone who wanted to use expectations. Luckily, Clojure gives me the ability to conditionally import dependencies.

The test looked something like the following code snippet.

(scenario
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (in (first @fills))))

note: build is a fn that creates domain objects (I do a lot of Java Interop).

The test builds a PartialFill domain object, and passes it to handle-fill. The handle-fill fn converts the domain object to a map and conj's the new map onto the fills vector (which is an atom).

The build fn creates a PartialFill who's time is (DateTime.), which means you can either not test the time or you need to freeze the time. Joda time makes it easy to freeze time by calling DateTimeUtils/setCurrentMillisFixed, so you could write the test as:

(scenario
 (DateTimeUtils/setCurrentMillisFixed 100)
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 100} (in (first @fills)))
 (DateTimeUtils/setCurrentMillisSystem))

Of course, that would cause issues if expect ever failed, as failure throws an exception and the time reset would never occur. Even if that wasn't the case, the time related code takes away from what I'm really testing - so I set out to create a declarative approach from within expectations.

The syntax for the freezing time is fairly straightforward, and the original test can be written as what follows (with expectations 1.4.3 & up)

(scenario
 :freeze-time "2012-4-30"
 (handle-fill (build PartialFill))
 (expect {:px 10 :size 33 :time 1335758400000} (in (first @fills))))

Writing the code to freeze time with expectations is trivial enough, but it's only relevant if you actually use joda-time. Since not everyone uses joda-time, it seemed like the best option was to make expectations import joda only if you actually use :freeze-time within your code.

The code linked here shows the conditional imports: https://github.com/jaycfields/expectations/blob/f2a8687/src/clojure/expectations/scenarios.clj#L81-L86

Even if you're not familiar with macros it should be obvious that I'm importing and using the joda-time classes conditionally. At macro expansion time I'm checking for the presence of freeze-time, and if it does exist the necessary classes are imported. If freeze-time is never found, the joda-time classes will never be imported, and no joda-time dependency will need to live within your project.

I always assumed this would be possible, but I'd never actually attempted to put this type of code together. I'm definitely happy with how the code turned out - I can freeze time in expectations if I want, but I don't force anyone to use joda.

Eval a Clojure Snippet in Java

2012-04-09T22:14:00.001-04:00

The vast majority of the existing code for my current project is in Java, but the majority of the new code that we write is in Clojure; as a result, I spend quite a bit of time bouncing between Clojure & Java. Recently I was working with some Clojure code that was easily testable (in isolation) within Clojure, but was also executed as part of some higher-level Java integration tests. Within the Clojure tests I used with-redefs to temporarily set the state of a few atoms I cared about; however, setting the same state within the higher-level Java tests turned out to be a bit more of an interesting task.

Our Java integration tests, like all integration tests I've ever run into, look simple enough when looking at the code of the test; however, if you want to understand how the simple test exercises several collaborations you'll need to dig deep into the bowels of the integration test helpers. Integration test helpers are usually a very fragile combination of classes that stub behavior, classes with preloaded fake datasets, and classes that provide event triggering hooks. In my experience you also generally need the High-Level Test Whisperer on hand to answer any questions, and you generally don't want to make any changes without asking for the HLTW's assistance.

Today's experience went exactly as I expect all of my experiences with the functional tests to go:

I added a bit of behavior, and tested everything that I thought was worth testing. With all the new tests passing, I ran the entire suite - 85 errors. The nice thing about errors in functional tests: they generally cause all of the functional tests to break in an area of code that you know you weren't working in. I wasn't that surprised that so many tests had broken, I was adding a bit of code into a core area of our application. However, I wasn't really interested in testing my new behavior via the functional tests, so I quickly looked for the easiest thing I could do to make the functional tests unaware of my change. The solution was simple enough, conceptually, the new code only ran if specific values were set within the atom in my new namespace; therefore, all I needed to do was clear that atom before the functional tests were executed.

Calling Clojure from Java is easy, so I set out to grab the atom from the Clojure and swap! + dissoc the values I cared about. Getting the atom in Java was simple enough, and clojure.lang.Atom has a swap() method, but it takes an IFn. I spent a few minutes looking at passing in a IFn that dissoc'd correctly for me; however, when nothing was painfully obvious I took a step back and considered an easier solution: eval.

As I previously mentioned, this is not only test code, but it's test code that I already expect to take a bit longer to run**. Given that context, eval seemed like an easy choice for solving my issue. The following code isn't pretty, but it got done exactly what I was looking for.

RT.var("clojure.core", "eval").invoke(
  RT.var("clojure.core","read-string").invoke("(swap! my-ns/my-atom dissoc :a-key)"));

I wasn't done yet, as I still needed the HLTW's help on getting everything to play nice together; however, that little Clojure snippet got me 90% of what I needed to get my behavior out of the way of the existing functional tests. I wouldn't recommend doing something like that in prod, but for what I needed it worked perfectly.

** let's not get carried away, the entire suite still runs in 20 secs. That's not crazy fast, but I'm satisfied for now.

When to Break Apart your Application

2012-03-04T12:40:00.000-05:00

At speakerconf Aruba 2012, Stefan Tilkov presented on the topic of building Systems. One aspect of his presentation that really struck a chord with me was around the competing desires while designing and delivering an application. This blog entry is about summarizing that aspect and what I took away from it.**

Stefan noted that early in an application's life, a monolithic application is far easier to deliver; however, as the application grows, what is important begins to change. The slide to the right shows how priorities can change over time.

While starting with a monolithic application is easier to deliver, that application often grows to an unmaintainable size - i.e. the vast majority of the time we boil the frog. Stefan argued that the benefits are so obviously there that for an application of any moderate size it's more efficient to bite the bullet and break things up (where logical) in the beginning.

I'm guessing the vast majority of the applications that Stefan and innoQ encounter probably began knowing that they would grow to significant size and would have benefited from any type of modularization, but my working environment is a bit different. When I start something new for a trader at DRW -

the project may never grow to the unmaintainable size
if it does, it's likely that I'll want things broken up in a way that I couldn't have anticipated when the project began.

Given my context, I couldn't conclude that things should be broken up immediately, but I didn't want to ignore the point that Stefan was making about goals changing over time.

Stefan also had a slide about Modularization (also on the right) that gave numbers and recommendations on how to break things up. This slide reminded me of Matt Deiters' presentation at speakerconf Rome 2011. Matt told us about his experience teaching kids in Africa to program. While teaching programming, Matt observed that giving the children the single rule that they shouldn't have methods longer than 10 lines resulted in the children writing code with SRP, encapsulation, and other positive attributes. Matt argued that beginning with a simple rule that could get you 80% of what you want is likely better than 10 rules that, if followed correctly, could get you 100% of what you want.

I applied Matt's idea to Stefan's talk and suggested that perhaps the easiest thing was to take Stefan's modularization table and strictly adhere to changing things when LOC thresholds are broken. This would allow you to quickly deliver a monolithic application early in the development process, while avoiding boiling the frog in the future - once you cross the agreed upon magic number (LoC) then you aggressively address breaking the application into smaller and independent modules/processes/components/projects/whatever.

This strategy gives a nod to the fact that priorities do change as an application evolves and it's best to evolve the application along with those priorities, but not when it's too late. I plan to try this on my next greenfield application.

** I'm going from memory, so I could be slightly off. Don't hold Stefan accountable for my interpretation of his ideas.

Aligning Business & Programmer Goals

2012-02-28T06:27:00.000-05:00

A theme emerged, last week at speakerconf, around the idea that programmers and businesses still don't seem to be on the same page. I'm not sure if it came up even earlier, but for me it began after I gave a presentation on Lessons Learned Adopting Clojure. The talk is similar to Lessons Learned while Introducing a New Programming Language, but focuses more on the technical aspects of selecting Clojure. Upon talk completion the first question I got was from Brian Goetz, something along the lines of:

(Brian) So, you wanted to use another language and simply introduced it into production?
(me) More or less.
(Brian) Do you think that's the most effective way for organizations to select languages or frameworks?

The first response to the question was the obvious and appropriate one:

That sounds better than a CTO making a decision while drunk and on a golf-course with a vendor. -- Josh Graham

Of course, I agree with that response, but that doesn't mean that my choice was the best answer either. The question is actually something I've been struggling with for awhile. At speakerconf Aruba 2011, Lyle (DRW CTO) gave a presentation on how we do things at DRW. One of Lyle's slides listed each language that we use in production at DRW. There were a lot of languages, probably around 25, and Dave Thomas questioned whether it was a positive or a maintenance nightmare. The answer today is the same as it was then, so far the ROI is positive - but, I couldn't help feeling uneasy about the situation.

I answered somewhat noncommittally, along the lines of: While selecting a language, I chose one that I felt anyone in DRW could easily learn and use. Additionally, I selected a language that aligned with our business goals of developing software quickly and delivering applications that performed at the speeds required by the traders. And, if not the man on the ground, who is better suited for selecting languages to introduce into production?

Brian followed up with:

The point I was trying to get at was: how does this scale? If every developer chooses what is best for them, is that necessarily best for the organization? How do we balance between the business value of increasing individual productivity and the business value of reducing maintenance costs or development risk? If we have to make compromises to achieve better scale, who should make these decisions, and how?

The usual developer argument -- "this is more productive for me, and increasing my productivity is good for the company" is good as far as it goes, but most developers incorrectly assume these interests are 100% aligned when they are in reality probably only merely overlapping.

These are truly great questions, and no one had a great response. The discussion basically died off as we pondered the tough answers to these questions.

At lunch the next day I asked Brian how he thought companies should go about selecting languages, and I think his response was spot-on (again, going from memory, so likely misquoting): I'm not sure, but I sure do see a lot of developers introducing whatever language or framework they want, leaving within a year, and sticking their employer with a big maintenance problem. The same can be said for consultants. They often show up advocating a sexy new language, mostly deliver a new application and then leave the company with an application that can't be maintained by the existing full-time staff.

Someone quickly spoke up, stating that Resume Driven Development has been going on for years. I can't argue with that, but I did have a different perspective.

A few years ago I joined DRW, and a substantial portion of my salary became bonus money. I had previously worked in places where getting bonus money was about as likely as meeting your future wife at a JavaOne, so I was obviously skeptical. Luckily, I had a friend who already worked for DRW; he reassured me that, in finance, having a bonus as part of your compensation was common, and so was it actually getting paid. I took the job, and instantly my performance at work was more directly tied to my income. I'm sure this reality influenced my decision making, and I witnessed the difference in the decisions my colleagues were making. When people spoke about using/upgrading frameworks/libraries, we didn't talk about what was cool, we discussed the risks of breaking things, the outstanding bugs with the framework/lib, and the immediate value that the inclusion would provide.

Given this experience, I advocated to the other speakerconf presenters that bonuses are a potential solution to combatting Resume Driven Development. The other speakerconf presenters gave me the same response I generally get when advocating bonuses for programmers: That works for you and your trading firm, but most people can't directly tie what they do to how the business makes money. I'm always skeptical of this response for 2 reasons: The business must be able to track whether or not their new idea is making them money, or, if the new software isn't targeted to an external customer, then it should be able to be rated by internal customers. Also, shouldn't the CTO or the Director of Software be able to make an informed bonus decision by either directly interacting with the teams, or trusting the feedback of the managers of those teams?

I guess, if you can't tie what you are doing to the value it's providing the company, and you don't trust your managers to give an accurate assessment, building your resume is probably the best use of your time... and the results aren't very surprising... but, I digress.

I gave the same response I've given a few dozen times: I just don't buy it. You should be able to see how many new customers your new website generated, or less cancellations, new reservations, policy efficiency, auction traffic, add-on sales purchased, etc. Each of the projects I've worked on for the past 8 years could generate metrics that I would have been happy to tie to a bonus. In our back-office, it's harder to tie the value of a back office system to an individual trade; however, we can capture metrics on uptime, latency, and any other attribute that we deem important to the internal systems that are customers of the back office systems.

If you want to measure your value and you trust your management, receiving part of your salary as a bonus is completely viable.

Aligning the goals of the business and the programmers came up again when the afternoon presentations began. Joshua Kerievsky is looking to tackle this same issue at Industrial Logic. Joshua's presentation discussed bridging the gap between managers and programmers, which is a more general version of the same conversation we'd been having about language adoption. Bridging this gap is something he's begun focusing on at Industrial Logic. The benefits are obvious for customers, they should receive higher quality and more relevant software. However, it's not just about customers, the programmers will be more effective and should also have higher job satisfaction rates. No one likes being stuck on yet another, doomed to fail, misguided project using outdated technology. Both parties should benefit by unifying the goals of the programmer and the business.

Josh knew my opinion on using bonuses to help align both parties; however, he and I both incorrectly assumed that the developers at Forward also received business performance based bonuses. I believe that they have some things in the works and situations can vary, but Fred George made it clear that in general a bonus is not tied to business focus. That's fairly standard, but what makes Forward unique is that the business works very, very closely with the programmers. I think Fred nicely summed up the situation: Our programmers are gamers, and we show them the P&L. They want to see that number grow, even if it doesn't directly deposit in their account.

Fred's statement rang very true to me. I've had jobs in the past where I didn't even have a bonus, but the ones that I felt most dedicated to were the ones where I was given visibility into the business. If I can't see the outcome of my programming choices to the business, I'll have a much harder time iterating towards success - worse, I won't even know at what level we are failing and I'll lack the motivation that the failure would provide me.

At the end of the day, programmers like to measure and improve. If you give them a P&L and let them know that it's the most important thing, they'll do whatever they can to help. If you don't give them a P&L they'll focus on making other metrics more efficient. This may include using new technologies to inspire more blog entires, spending hours unnecessarily tweaking the development process, or any other way of measuring their "success." The answer is obvious, let the programmers measure success the same way the business does, using the bottom line. What's not always obvious is how to properly share that information; however, all the discussion at speakerconf definitely convinced me that it's worth solving this issue even if it requires a bit of extra effort.

How To Get Invited To speakerconf

2012-02-15T09:30:00.000-05:00

Every year it seems that someone is disappointed (or even insulted) that they didn't receive an invite to speakerconf. If you haven't gotten an invite at this point, contrary to popular belief, it's not because Josh and I hate you. There's actually very little magic that goes into receiving a speakerconf invite, so it probably makes sense to put the formula out there.

I'm not a big fan of receiving spam, and while I consider speakerconf to be a great event, I'm sure some people will consider speakerconf related email to be spam. With that in mind, I generally don't like to email people I don't know out of the blue. Along the same lines, some people are going to politely decline every invite I send their way. The less I know you, the less likely I am to continue inviting you, simply because I don't want to bother you. If you say to me "please keep me in mind in the future," I will. If you don't, I'll likely default on the safe side of not bothering you until you show some interest.

So, if you haven't gotten an invite lately (or ever) it's likely only because I don't want to bother you.

With that out of the way, here's roughly how we select who to invite to speakerconf:
speakerconf needs sponsors to run in the way that Josh and I want to run it. If you've previously attended a speakerconf you can guarantee yourself an invite if your company is willing to become a sponsor. If you've never attended a speakerconf you can also guarantee yourself an invite by getting your company to become a sponsor and getting an endorsement from any one of the speakerconf alumni (list available at http://speakerconf.com/speakers). If you are interested in discussing specifics, drop me an email (jay@jayfields.com).

If you're not interested in sponsoring, your invite depends on space or endorsements. If you've previously attended a speakerconf and you'd like to return, all you need to do is let me know. If we have space at the next event, I'll gladly extend you an invite. If we don't have space, I'll extend you an invite to the following event.

Future speakerconf events will likely have a maximum number of 20 attendees. Approximately 15 spots will be reserved for alumni, and the remaining ~5 spots will be available to anyone who has never attended a speakerconf. If you would like to grab one of those spots, all you need to do is get as many alumni as you can to email me and endorse you. They don't need to send a long email, something as simple as "you should invite Josh Graham to the next speakerconf" will do. Then, approximately 6 months before an upcoming speakerconf I will invite the 5 most alumni endorsed new comers.

That's it. If you want to join us all you need to do is to utilize your network or become a sponsor. Hopefully this view into the invitation process will help avoid any disappointment in the future.

Lessons Learned while Introducing a New Programming Language

2012-01-24T16:27:00.000-05:00

I've used a lot of languages (professionally) over the years: (off the top of my head) Cold Fusion, HTML, Javascript, php, SQL, CSS, ASP(classic & .net), C#, Ruby, Flex, Java, & Clojure. Each language has pros and cons. Being a programmer, it's easiest to discuss the cons - and in general I believe it was best said:

I hate all programming languages - Matt Foemmel

I think it's important to start with this in mind. At some point you're going to hate what you're advocating, so imagine how other people feel about it.

In 2008 I introduced Clojure into a DRW codebase. This blog entry focuses on the adoption lessons I've learned in the past few years.

Language Selection
Introducing a new language to an organization isn't an easy task; if you're going to succeed you're probably going to need to pick a language that satisfies a large technical need and is also considered socially acceptable. I was writing Java 100% of the time when I joined DRW, despite the fact that a large portion of the code I was writing only needed to run in eye-ball time (250ms). Java was absolutely the right choice for the Faster Than Eye-Ball Time code we were writing, but using Java for the other code always made me feel like I was unnecessarily paying the Java verbosity tax.

On occasion I would mention the Java verbosity tax, and I found that my boss was actually interested in looking more closely at JRuby. I think JRuby would have been a win for us, but for me it was more interesting to hear that I had an ally if I wanted to explore non-Java options. If JRuby was on the table, then I knew that any high-level, dynamically typed language would also likely be on the table.

However, before I'd even discovered any JRuby curiosity, I'd already begun looking into Haskell. Generally, software in trading firms needs to run "fast." If I was going to successfully introduce a new language it was going to need to run "almost as fast as Java." I'd heard good things about Haskell's speed, and it also fulfilled another adoption desire of mine:

A language that doesn't affect the way you think about programming, is not worth knowing - Alan Perlis

I believed that if I found a language that was "performant enough", allowed us to deliver at a faster pace, and improved our programming skills then the level of effort required for adoption would be justified.

I toyed with Haskell for a bit, but the adoption path seemed a bit too steep. There's the effort it takes to learn Haskell, but more importantly: we were already on the JVM. If I was going to get any support, I needed something that was easy to sneak into our existing infrastructure. Enter Clojure. Clojure is performant enough, more succinct than Java, and sufficiently different to anything else I'd previously worked with. Clojure is also dynamically typed and high level (like Ruby), so I was hoping to get some support from my boss.

Causing my teammates as little pain as possible was a very large requirement - I believed this was key to gaining adoption. Clojure appeared to be the best choice due to the fact that we were already using:

IntelliJ all day
JUnit to run all of our tests
TeamCity for CI & artifact creation
The JVM on our servers
Yourkit for profiling

Clojure was the language that gave me everything I was looking for and had the easiest adoption path for the rest of the team.

I would have preferred Haskell or OCaml from a learning perspective, but they didn't seem like practical choices - and I doubt I would have had as much success getting them in production. While I need to be an expert in many things Clojure related, I relied on the JVM server settings that someone else has determined to be "the best". If I had chosen Haskell or OCaml I'd have had to become an expert on a much larger scope of topics (e.g. deployment, memory model, libraries, new tools, etc).

I believed then, and I still believe today that Clojure was the best choice given our technical needs and the social context.

Hello World
Introducing a language is a delicate act. There's a bunch of valid concerns that you're going to need to address. I wasn't exactly sure how my teammates were going to react to the introduction of Clojure, so I wrote the original code in my own time at home. I knew that we needed some integration tests for our application; however, no one was actively working on implementing anything. I began by writing them in Java and then wrote Clojure versions as well. I knew enough Clojure that I was able to showcase it's succinctness - something I knew the team would value in integration tests. Additionally, since the tests aren't part of the production running code there were no real speed concerns to consider.

Integration tests are a good place to introduce a new language, but any non-production code will probably be an equally good choice. For example, you could also choose database migration scripts, log file parsers, 3rd party software simulators, or deployment software. As long as you pick something that can fail without too much immediate pain you should be able to easily recover from any adoption issues.

After I had both sets of tests complete I showed both versions to the other developers on the team. I pointed out why I preferred the Clojure versions and asked if they would be willing to give Clojure a chance. I also made commitments that made it hard for them to say no to the experiment.

Your Commitment
I eased my teammates adoption fears by making the following commitments.

If you want to work on the code I'll work with you (if you want me to work with you).
If you don't want to work on the code I'll fix anything that's broken.
If the initial pain of working with a new language becomes unbearable to you, I'll rewrite everything in Java on my own time.

Obviously you'll need to get team buy-in before writing too much in your new language - otherwise you could be setting yourself up for working a lot of nights and weekends.

Tool Support
Chances are your team already has a tool-chain that they are happy with. Whatever that tool-chain is, your new language is going to need to play well within it. For me this meant being able to execute Clojure code within IntelliJ as easily as Java. For the most part the La Clojure plugin does the heavy lifting; however, I did need to write a testing framework that allowed me to run focused tests and seamlessly integrated with our existing JUnit test suite. The main point here is to remove any adoption friction that your team may be feeling. Learning a new language is a reasonable request, but changing the way a team works simply to accommodate an unproven (on that team) language choice is probably too much to ask.

You may also need to make some sacrifices as well. I prefer to write Clojure in emacs; however, I'd rather be writing Clojure(where appropriate) in IntelliJ than writing Java in IntelliJ. During the early/fragile adoption time, you're the one who's going to need to do the majority of the compromising.

Find allies
Chances are you'll have varying levels of interest in your new language. During the early days you should do anything you can to encourage others when they're interested; however, you don't want to push anything on anyone - that's the easiest way to find enemies. Hopefully you'll have a few teammates who are also as excited about a new language as you are - work closely with them on improving both of your skills. You'll want to get as many advocates as you can, otherwise you'll end up looking like the lone team member forcing the team to do something they aren't comfortable with.

It's also inevitable that you'll end up needing more research time, needing tool support, and dealing with more production issues than you originally anticipated. You're going to need a few other people to pick up some of the slack when you find yourself overextended. Even when things are going well you'll find yourself in need of allies to help you support the growing code written in a new language.

Lastly, the worst case scenario is you leaving a team and no one is left on the team that wants to support that code. It's pretty easy to see how a staffing situation can be portrayed as an adoption problem.

Know Everything
Obviously you can't actually know everything, but you're going to need to have an answer or be able to quickly get an answer to anything that comes up. You'll definitely want to read a few books on your new language before putting it in any codebase that your team members also work with or rely on. That's likely not enough though, you'll also need to know where to go if you run into issues. For Clojure this was the IRC channel for immediate answers and the mailing list for less time sensitive issues or issues that required a bit more explanation. If you're really looking to cover your bases you'll want to have some type of relationship with the creator or one of the community leaders.

Once people start adopting the language you introduced they're going to start doing things you didn't anticipate. You're going to need to know the dark corners of the language and the associated corner cases that exist. You'll also need to be an expert on issues such as memory allocation, performance, deployment, tool integration, library support, upgrade schedules, and everything else that is outside of the language's syntax.

The more allies you have, the less you'll need to 'know everything'; however, at the end of the day you're going to need to know as much as possible. If things ever go wrong, all eyes are going to be on you. That's the level of commitment you're making by introducing a language, so you better know what you're getting into.

Get Help
If your company is willing to let you introduce languages then they are probably a fairly supportive organization. Hopefully you'll have a bit of a training budget. See what opportunities you have for bringing in the language creator or the community leaders to work with you or provide training. If you're having issues with anything, having the creator of a language work with you is obviously a huge advantage. However, if things are going well it will probably benefit you to give up your training budget to contribute to a pool that could cover some training for other team mates who are interested in learning from the language's creator (or a community leader). Whether it's for you or interested allies, utilize your companies training budget to encourage adoption.

Be an Advocate, not a Zealot
At the end of the day it's not likely that everyone is going to have a compatible opinion. That's fine. Don't push your opinions on people who aren't interested. On most occasions the 'right' choice is the one that someone is passionate about. You might be passionate about your language, but a teammate of yours might be passionate about the old language. Neither of you needs to be right or wrong. People should work with what they are passionate about, and any attempt to make them work in another way is likely to do more harm than good. People who want to work with the new language will find a way to organize themselves together and people who don't will do the same thing. There's no reason to force adoption.

Originally, I approached the problem as "If I create a clearly superior solution then everyone will want to come along." This definitely turned out not to be the case, thus the blog entry on compatible opinions on software. I learned that one person's "obviously better" is another person's "obviously worse." In the end the team ended up organically dividing into people who worked on the Clojure code and those that didn't. This worked out well for both parties, as the people who wanted to work with Clojure had enough available to them, and the people who didn't weren't forced to.

The divide was more of a practice than an official split. We were all still part of "one team"; however, we tended to work on separate applications that communicated through messaging or not at all. I advocated strongly for a team split based on incompatible opinions and size (we were at 7, and I thought 4 and 3 would be fine), but we never ended up making anything official. Eventually other factors changed and the team shrunk to a size where a split was no longer necessary. I still believe splitting would have been the best solution if the team hadn't reorganized for other reasons.

The End
Introducing a new language is likely a multi-year affair for any moderately sized organization. There's not likely an "end" where your responsibilities go back to what they were before introducing a new language. On the flip-side, you get to use what you consider to be the best tool for the job. Hopefully it's worth it when it's all said and done. Personally, I'm happy with my choice, but I expect to be learning new lessons on this topic for at least the next few years as well.

Convert java.util.Properties to a Clojure Map

2011-12-28T21:15:00.001-05:00

As I previously mentioned, a lot of the work I do involves Clojure & Java interop. This work includes the occasional case of working with a java.util.Properties object from within Clojure. Working with a Properties object isn't a huge deal, but while in Clojure I prefer to use destructuring and the various functions (e.g. update-in, assoc, dissoc, etc) that are designed to work with Clojure maps.

The following example shows how easy it is to convert a Properties object to a Clojure map.

user=> (def prop-obj (doto (java.util.Properties.) (.putAll {"a" 1 "b" 2})))
user=> prop-obj
#<Properties {b=2, a=1}>

user=> (reduce (fn [x [y z]] (assoc x y z)) {} prop-obj)
{"a" 1, "b" 2}

That's fairly easy, but you'll quickly want your keys to be keywords if you plan on destructuring. You can drop in a quick call to keyword to convert the keys; however, you'll probably also want to dasherize the keys to allow for easy destructuring using keys and idiomatic names.

user=> (defn dash-match [[ _ g1 g2]]      
          (str g1 "-" g2))

user=> (defn dasherize [k]
          (-> k
            (clojure.string/replace #"([A-Z]+)([A-Z][a-z])" dash-match)
            (clojure.string/replace #"([a-z\d])([A-Z])" dash-match)
            (clojure.string/lower-case)))#'user/dash-match

user=> (def prop-obj (doto (java.util.Properties.) (.putAll {"FirstName" "Mike" "LastName" "Green"})))
user=> (reduce (fn [x [y z]] (assoc x y z)) {} prop-obj)                                              
{"LastName" "Green", "FirstName" "Mike"}
user=> (reduce (fn [x [y z]] (assoc x (-> y dasherize keyword) z)) {} prop-obj)
{:last-name "Green", :first-name "Mike"}

That looks good, but you might also find yourself working with a properties file that uses dots or dashes to group similar data. For example, you might find the following entry in your properties file (and the resulting Properties object).

person.name=Mike Green
person.age=26
person.sex=male

Loading this into a map works fine; however, it would be nice if the resulting map was nested (to keep common data together, and for easier access using get-in, update-in, etc).

The following code splits on dots and nests the values appropriately.

user=> (def prop-obj (doto (java.util.Properties.) (.putAll {"person.name" "Mike Green" "person.age" "26" "person.sex" "male"})))        
user=> prop-obj                                                                                                                  
#<Properties {person.name=Mike Green, person.age=26, person.sex=male}>
user=> (reduce (fn [x [y z]] (assoc-in x (-> y dasherize (clojure.string/split #"\.")) z)) {} prop-obj)
{"person" {"sex" "male", "age" "26", "name" "Mike Green"}}

user=> (-> (reduce (fn [x [y z]] (assoc-in x (-> y dasherize (clojure.string/split #"\.")) z)) {} prop-obj) clojure.walk/keywordize-keys)
{:person {:sex "male", :age "26", :name "Mike Green"}}

So, not too complicated, but much more complicated than what we started with. It turns out you'll likely want to do other things like parse integers, parse booleans, require keys, and add defaults when a key=value pair isn't specified.

We could go through the effort here of providing all that functionality, but instead I've created a small library that provides all of the above mentioned features: propertea

If you'd like to see the implementation of the above features, you'll just need to look in propertea.core. If you want examples of all of the features of propertea, checkout the tests in github.

Clojure & Java Interop

2011-12-28T06:46:00.000-05:00

About a year ago I got a phone call asking if I wanted to join another team at DRW. The team supports a (primarily) Java application, but the performance requirements would also allow it to be written in a higher level language. I'd been writing Clojure (basically) full-time at that point - so my response was simple: I'd love to join, but I'm going to want to do future development using Clojure.

A year later we still have plenty of Java, but the vast majority of the new code I add is Clojure. One of the big reasons I'm able to use Clojure so freely is the seamless interop with Java.

Execute Clojure from Java
Calling Clojure from Java is as simple as loading the .clj file and invoking a method from that file. I used the same example years ago, but I'll inline it here for simplicity.

; interop/core.clj
(ns interop.core)

(defn print-string [arg]
  (println arg))

// Java calling code
RT.loadResourceScript("interop/core.clj");
RT.var("interop.core", "print-string").invoke("hello world");

note: examples from this blog entry are available in this git repo. The commit with the code from the previous example is available here and I'm running the example from the command line with:

lein jar && java -cp "interop-1.0.0.jar:lib/*" interop.Example

Execute Java from Clojure
At this point we have Java executing some Clojure code, and we also have Clojure using an object that was created in Java. Even though we're in Clojure we can easily call methods on any Java object.

(ns interop.core)

(defn print-string [arg]
  (println arg "is" (.length arg) "characters long"))

commit

The above code (using the length method of a String instance) produces the following output.

hello world is 11 characters long

Calling a Java method and passing in additional arguments is also easy in Clojure.

(ns interop.core)

(defn print-string [arg]
  (println (.replace arg "hello" "goodbye")))

commit

The above code produces the following output.

goodbye world

There are a few other things to know about calling Java from Clojure. The following examples show how to call static methods, use enums, and use inner classes.

(ns interop.core)

(defn print-string [arg]
  ;;; calling a static method
  (println (String/valueOf true))

  ;;; using an enum
  (println (java.util.concurrent.TimeUnit/SECONDS))

  ;;; using a Java nested (inner) class. Note, in Clojure you
  ;;; use a $ instead of a .
  (println (java.util.AbstractMap$SimpleEntry. "key" "val")))

commit

And, the output:

true
#< SECONDS>
#<SimpleEntry key=val>

Create Java objects in Clojure
When working with Clojure you'll likely want to interact with existing Java objects, but you'll probably also want to create new instances of Java objects. You might have noticed the dot at the end of Abstract$SimpleEntry. in the previous example - that's how you instruct Clojure to create an instance of a Java object. The following example shows the dot notation for calling a constructor of the String class.

(ns interop.core)

(defn print-string [arg]
  (println (String. arg)))

commit

At this point our output is back to the original output.

hello world

When creating Java objects it's often beneficial to know which Java interfaces the Clojure data structures implement. The following examples demonstrate how you can create Java objects while passing Clojure datastructures (and functions) as constructor arguments.

(ns interop.core)

(defn print-string [arg]
  ;;; pass a Clojure vector where Java expects a java.util.Collection
  (println (java.util.HashSet. ["1" "2"]))

  ;;; pass a Clojure map where Java expects a java.util.Map
  (println (java.util.LinkedHashMap. {1 "1" 2 "2"}))

  ;;; pass a Clojure function where Java expects a Runnable
  (println (Thread. (fn [] (println "clojure fns are runnables (and callables)")))))

commit

The output shows the constructed Java objects.

#<HashSet [2, 1]>
#<LinkedHashMap {1=1, 2=2}>
#<Thread Thread[Thread-1,5,main]>

Calling constructors in Clojure is very easy, but that's not always an option when creating a Java object. At times you will likely need to create an instance of a Java interface. Clojure provides both proxy and reify for creating instances of Java interfaces. The following example demonstrates the syntax for using either proxy or reify.

(ns interop.core)

(defn proxy-coll []
  (proxy [java.util.Collection] []
    (add [o]
         (println o)
         true)))

(defn reify-coll []
  (reify java.util.Collection
    (add [this o]
         (println o)
         (println this)
         true)))

(defn main []
  (.add (proxy-coll) "this string is printed on proxied.add")
  (.add (reify-coll) "this string is printed on reified.add"))

commit

note, I also changed Example.java (the details are available in the above linked commit). The syntax for proxy and reify are fairly similar, and both offer additional options that are worth looking into. The primary differences between these two simple examples are:

The proxy implementation requires an empty vector where we could specify constructor arguments (if this were an abstract class instead of an interface).
The arg list for all methods of reify will specify the reified instance as the first argument. In our example the Collection.add method only takes one argument, but in our reify we also get the instance of the collection.

You might have also noticed that both implementations of add have "true" at the end - in our example we're hard-coding the return value of add to always return true. The following output is the result of running the current example code.

this string is printed on proxied.add
this string is printed on reified.add
#<core$reify_coll$reify__11 interop.core$reify_coll$reify__11@556917ee>

It's worth reading the docs to determine whether you want proxy or reify; however, if you don't see a clear choice I would opt for reify.

Returning objects from Clojure to Java
Our current Example.java returns something from the call to invoke on the clojure.lang.Var that is returned from RT.var("interop.core", "main"), but we're ignoring it so we have no idea what's returned.* Let's change the code and return something on purpose.

// interop/Example.java
package interop;

import clojure.lang.RT;

public class Example {
    public static void main(String[] args) throws Exception {
        RT.loadResourceScript("interop/core.clj");
        System.out.println(RT.var("interop.core", "main").invoke());
    }
}

; interop/core.clj
(ns interop.core)

(defn main []
  {:a "1" :b "2"})

Running our changes produces the following output.

{:a "1", :b "2"}

commit

At this point we are back in Java land after making a quick trip to Clojure to get a value. Returning most objects will be pretty straightforward; however, at some point you may want to return a Clojure function. This turns out to be fairly easy as well, since Clojure functions are instances of the IFn interface. The following code demonstrates how to return a Clojure function and call it from within Java.

// interop/Example.java
package interop;

import clojure.lang.RT;

public class Example {
    public static void main(String[] args) throws Exception {
        RT.loadResourceScript("interop/core.clj");
        clojure.lang.IFn f = (clojure.lang.IFn) RT.var("interop.core", "main").invoke();
        f.invoke("hello world");
    }
}

// interop/core.clj
(ns interop.core)

(defn main [] println)

commit

The above example returns the println function from interop.core/main and then invokes the println function from within Java. I only chose to pass one argument to invoke; however, the IFn.invoke method has various overrides to allow you to pass several arguments. The above code works, but it can be simplified to the following example.

package interop;

import clojure.lang.RT;

public class Example {
    public static void main(String[] args) throws Exception {
        clojure.lang.IFn f = (clojure.lang.IFn) RT.var("clojure.core", "println");
        f.invoke("hello world");
    }
}

commit

It seems like a fitting end that our final output is the same as our original output.

hello world

*actually, it's the last thing that's returned, or "true" for this specific case.

Clojure: expectations - scenarios

2011-11-19T08:24:00.001-05:00

When I set out to write expectations I wanted to create a simple unit testing framework. I'm happy with what expectations provides for unit testing; however, I also need to write the occasional test that changes values or causes a side effect. There's no way I could go back to clojure.test after enjoying better failure messages, trimmed stack traces, automatic testing running, etc. Thus, expectations.scenarios was born.

Using expectations.scenarios should be fairly natural if you already use expectations. The following example is a simple scenario (which could be a unit test, but we'll start here for simplicity).

(ns example.scenarios
  (:use expectations.scenarios))

(scenario
 (expect nil? nil))

A quick trip to the command line shows us that everything is working as expected.

Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.

As I said above, you could write this test as a unit test. However, expectations.scenarios was created for the cases in which you want to verify a value, make a change, and verify a value again. The following example shows multiple expectations verifying changing values in the same scenario.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 1 @a)
    (swap! a inc)
    (expect 2 @a)))

In expectations (unit tests) you can only have one expect (or given) so failures are captured, but they do not stop execution. However, due to the procedural nature of scenarios, the first failing expect stops execution.

(scenario
  (let [a (atom 0)]
    (swap! a inc)
    (expect 2 @a)
    (println "you'll never see this")))

failure in (scenarios.clj:4) : example.scenarios
           (expect 2 (clojure.core/deref a))
           expected: 2 
                was: 1
           on (scenarios.clj:7)
Ran 1 tests containing 1 assertions in 81 msecs
1 failures, 0 errors.

expectations.scenarios also allows you to easily verify calls to any function. I generally use interaction expects when I need to verify some type of side effect (e.g. logging or message publishing).

(scenario
 (println "1")
 (expect (interaction (println "1"))))

It's important to note the ordering of this scenario. You don't 'setup' an expectation and then call the function. Exactly the opposite is true - you call the function the same way you would in production, then you expect the interaction to have occurred. You may find this jarring if you're used to setting up your mocks ahead of time; However, I think this syntax is the least intrusive - and I think you'll prefer it in the long term.

The above example calls println directly, but your tests are much more likely to look something like this.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (expect (interaction (println "1"))))

Similar to all other mocking frameworks (that I know of) the expect is using an implicit "once" argument. You can also specify :twice and :never if you find yourself needing those interaction tests.

(defn foo [x] (println x))

(scenario
 (foo "1")
 (foo "1")
 (expect (interaction (println "1")) :twice)
 (expect (interaction (identity 1)) :never))

On occasion you may find yourself interested in verifying 2 out of 3 arguments - expectations.scenarios provides the 'anything' var that can be used for arguments you don't care about.

(defn foo [x y z] (println x y z))

(scenario
 (foo "1" 2 :a)
 (expect (interaction (println "1" anything :a))))

That's about all there is to expectations.scenarios, hopefully it fills the gap for tests you want to write that simply can't be done as unit tests.

Clojure: expectations unit testing wrap-up

2011-11-01T06:59:00.003-04:00

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six (this entry)

The previous blog posts on expectations unit testing syntax cover all of the various ways that expectations can be used to write tests and what you can expect when your tests fail. However, there are a few other things worth knowing about expectations.

Stacktraces
expectations aggressively removes lines from the stacktraces. Just like many other aspects of expectations, the focus is on more signal and less noise. Any line in the stacktrace from clojure.core, clojure.lang, clojure.main, and java.lang will be removed. As a result any line appearing in your stacktrace should be relevant to your application or a third-party lib you're using. expectations also removes any duplicates that can occasionally appear when anonymous functions are part of the stacktrace. Again, it's all about improving signal by removing noise. Speaking of noise...

Test Names
You might have noticed that expectations does not require you to create a test name. This is a reflection of my personal opinion that test names are nothing more than comments and shouldn't be required. If you desire test names, feel free to drop a comment above each test. Truthfully, this is probably a better solution anyway, since you can use spaces (instead of dashes) to separate words in a comment. Comments are good when used properly, but they can become noise when they are required. The decision to simply use comments for test names is another example of improving signal by removing noise.

Running Focused Expectations
Sometimes you'll have a file full of expectations, but you only want to run a specific expectation - expectations solves this problem by giving you 'expect-focused'. If you use expect-focused only expectations that are defined using expect-focused will be run.

For example, if you have the following expectations in a file you should see the following results from 'lein expectations'.

(ns sample.test.core
  (:use [expectations]))

(expect zero? 0)
(expect zero? 1)
(expect-focused nil? nil)

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 2 msecs
IGNORED 2 EXPECTATIONS
0 failures, 0 errors.

As you can see, expectations only ran one test - the expect-focused on line 6. If the other tests had been run the test on line 5 would have created a failure. It can be easy to accidentally leave a few expect-focused calls in, so expectations prints the number of ignored expectations in capital letters as a reminder. Focused expectation running is yet another way to remove noise while working through a problem.

Tests Running
If you always use 'lein expectations' to run your tests you'll never even care; however, if you ever want to run individual test files it's important to know that your tests run by default on JVM shutdown. When I'm working with Clojure and Java I usually end up using IntelliJ, and therefore have the ability to easily run individual files. When I switched from clojure.test to expectations I wanted to make test running as simple as possible - so I removed the need to specify (run-all-tests). Of course, if you don't want expectations to run for some reason you can disable this feature by calling (expectations/disable-run-on-shutdown).

JUnit Integration
Lack of JUnit integration was a deal breaker for my team in the early days, so expectations comes with an easy way to run all tests as part of JUnit. If you want all of your tests to run in JUnit all you need to do is implement ExpectationsTestRunner.TestSource. The following example is what I use to run all the tests in expectations with JUnit.

import expectations.junit.ExpectationsTestRunner;
import org.junit.runner.RunWith;

@RunWith(expectations.junit.ExpectationsTestRunner.class)
public class SuccessTest implements ExpectationsTestRunner.TestSource{

    public String testPath() {
        return "test/clojure/success";
    }
}

As you can see from the example above, all you need to do is tell the test runner where to find your Clojure files.

That should be everything you need to know about expectations for unit testing use. If anything is unclear, please drop me a line in the comments.

Clojure: expectations - removing duplication with given

2011-11-01T06:58:00.003-04:00

create some state
verify a bit about the state
alter the state
verify more about the state

Part of the problem is that the assertions that occurred before the altering of the state may or may not be relevant after the alteration. Additionally, if any of the assertions fail you have to stop running the entire test - thus some of your assertions will not be run (and you'll be lacking some information).

expectations takes an alternate route - embracing the idea of multiple assertions by providing a specific syntax that allows multiple verifications and the least amount of duplication.

The following example shows how you can test multiple properties of a Java object using the 'given' syntax.

(given (java.util.ArrayList.)
       (expect
         .size 0
         .isEmpty true))

jfields$ lein expectations
Ran 2 tests containing 2 assertions in 4 msecs
0 failures, 0 errors.

The syntax is simple enough: (given an-object (expect method return-value [method return-value]))
note: [method return-value] may be repeated any number of times.

This syntax allows us to expect return-values from as many methods as we care to verify, but encourages us not to change any state between our various assertions. This syntax also allows us to to run each assertion regardless of the outcome of any previous assertion.

Obviously you could call methods that change the internal state of the object and at that point you're on your own. I definitely wouldn't recommend testing that way. However, as long as you call methods that don't change any state 'given' can help you write succinct tests that verify as many aspects of an object as you need to test.

As usual, I'll show the output for tests that fail using this syntax.

(given (java.util.ArrayList.)
       (expect
         .size 1
         .isEmpty false))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 1 (.size (java.util.ArrayList.)))
           expected: 1 
                was: 0
failure in (core.clj:4) : sample.test.core
           (expect false (.isEmpty (java.util.ArrayList.)))
           expected: false 
                was: true

This specific syntax was created for testing Java objects, but an interesting side effect is that it actually works on any value and you can substitute method calls with any function. For example, you can test a vector or a map using the examples below as a template.

(given [1 2 3]
       (expect
         first 1
         last 3))

(given {:a 2 :b 4}
       (expect 
         :a 2
         :b 4))

jfields$ lein expectations
Ran 4 tests containing 4 assertions in 8 msecs
0 failures, 0 errors.

And, of course, the failures.

(given [1 2 3]
       (expect
         first 2
         last 1))

(given {:a 2 :b 4}
       (expect 
         :a 1
         :b 1))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 2 (first [1 2 3]))
           expected: 2 
                was: 1
failure in (core.clj:4) : sample.test.core
           (expect 1 (last [1 2 3]))
           expected: 1 
                was: 3
failure in (core.clj:9) : sample.test.core
           (expect 1 (:a {:a 2, :b 4}))
           expected: 1 
                was: 2
failure in (core.clj:9) : sample.test.core
           (expect 1 (:b {:a 2, :b 4}))
           expected: 1 
                was: 4
Ran 4 tests containing 4 assertions in 14 msecs
4 failures, 0 errors.

When you want to call methods on a Java object or call functions with the same instance over and over the previous given syntax is really the simplest solution. However, there are times where you want something more flexible.

expectations also has a 'given' syntax that allows you to specify a template - thus reducing code duplication. The following example shows a test that verifies + with various arguments.

(given [x y] (expect 10 (+ x y))
       4 6
       6 4
       12 -2)

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 5 msecs
0 failures, 0 errors.

The syntax for this flavor of given is: (given bindings template-form values-to-be-bound). The template form can be anything you need - just remember to put the expect in there.

Here's another example where we combine given with in to test a few different things. This example shows both the successful and failing versions.

;; successful
(given [x y] (expect x (in y))
       :a #{:a :b}
       {:a :b} {:a :b :c :d})

;; failure
(given [x y] (expect x (in y))
       :c #{:a :b}
       {:a :d} {:a :b :c :d})

lein expectations
failure in (core.clj:8) : sample.test.core
           (expect :c (in #{:a :b}))
           key :c not found in #{:a :b}
failure in (core.clj:8) : sample.test.core
           (expect {:a :d} (in {:a :b, :c :d}))
           expected: {:a :d} 
                 in: {:a :b, :c :d}
           :a expected: :d
                   was: :b
Ran 4 tests containing 4 assertions in 13 msecs
2 failures, 0 errors.

That's basically it for 'given' syntax within expectations. There are times that I use all of the various versions of given; however, there seems to be a connection with using given and interacting with Java objects. If you don't find yourself using Java objects very often then you probably wont have a strong need for given.

Clojure: expectations and Double/NaN

2011-11-01T06:57:00.002-04:00


;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a Double/NaN :b {:c Double/NaN}})

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 Double/NaN})

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 Double/NaN])

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

As you would expect, you can also count on Double/NaN being considered equal even if you are using the 'in' function.

;; allow Double/NaN equality when verifying values are in a map
(expect {:a Double/NaN :b {:c Double/NaN}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality when verifying it is in a set
(expect Double/NaN (in #{1 Double/NaN}))

;; allow Double/NaN equality when verifying it's existence in a list
(expect Double/NaN (in [1 Double/NaN]))

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 32 msecs
0 failures, 0 errors.

For completeness I'll also show the examples of each of these examples failing.

;; allow Double/NaN equality in a map
(expect {:a Double/NaN :b {:c Double/NaN}} {:a nil :b {:c Double/NaN}})

;; allow Double/NaN equality with in fn and map
(expect {:a Double/NaN :b {:c nil}} (in {:a Double/NaN :b {:c Double/NaN} :d "other stuff"}))

;; allow Double/NaN equality in a set
(expect #{1 Double/NaN} #{1 nil})

;; allow Double/NaN equality with in fn and set
(expect Double/NaN (in #{1 nil}))

;; allow Double/NaN equality in a list
(expect [1 Double/NaN] [1 nil])

;; allow Double/NaN equality with in fn and list
(expect Double/NaN (in [1 nil]))


jfields$ lein expectations
failure in (core.clj:5) : sample.test.core
           (expect {:a Double/NaN, :b {:c Double/NaN}}
                   {:a nil, :b {:c Double/NaN}})
           expected: {:a NaN, :b {:c NaN}} 
                was: {:a nil, :b {:c NaN}}
           :a expected: NaN
                   was: nil
failure in (core.clj:8) : sample.test.core
           (expect {:a Double/NaN, :b {:c nil}} (in {:a Double/NaN, :b {:c Double/NaN}, :d "other stuff"}))
           expected: {:a NaN, :b {:c nil}} 
                 in: {:a NaN, :b {:c NaN}, :d "other stuff"}
           :b {:c expected: nil
                       was: NaN
failure in (core.clj:11) : sample.test.core
           (expect #{1 Double/NaN} #{nil 1})
           expected: #{NaN 1} 
                was: #{nil 1}
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:14) : sample.test.core
           (expect Double/NaN (in #{nil 1}))
           key NaN not found in #{nil 1}
failure in (core.clj:17) : sample.test.core
           (expect [1 Double/NaN] [1 nil])
           expected: [1 NaN] 
                was: [1 nil]
           nil are in actual, but not in expected
           NaN are in expected, but not in actual
failure in (core.clj:20) : sample.test.core
           (expect Double/NaN (in [1 nil]))
           value NaN not found in [1 nil]
Ran 6 tests containing 6 assertions in 66 msecs
6 failures, 0 errors.

There always seems to be downsides to using NaN, so I tend to look for the least painful path. Hopefully expectations provides the most pain-free path when your tests end up needing to include NaN.

Clojure: expectations with values in vectors, sets, and maps

2011-11-01T06:56:00.002-04:00

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three (this entry)
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

I've previously written about verifying equality and the various non-equality expectations that are available. This entry will focus on another type of comparison that is allowed in expectations - verifying that an 'expected' value is in an 'actual' value.

A quick recap - expectations generally look like this: (expect expected actual)

verifying an expected value is in an actual value is straightforward and hopefully not a surprising syntax: (expect expected (in actual))

If that's not clear, these examples should make the concept completely clear.

;; expect a k/v pair in a map.
(expect {:foo 1} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :foo (in #{:foo :bar}))

;; expect a val in a list
(expect :foo (in [:foo :bar]))

As you would expect, running these expectations results in 3 passing tests.

jfields$ lein expectations
Ran 3 tests containing 3 assertions in 8 msecs
0 failures, 0 errors.

As usual, I'll show the failures as well.

;; expect a k/v pair in a map.
(expect {:foo 2} (in {:foo 1 :cat 4}))

;; expect a key in a set
(expect :baz (in #{:foo :bar}))

;; expect a val in a list
(expect :baz (in [:foo :bar]))

jfields$ lein expectations
failure in (core.clj:18) : sample.test.core
           (expect {:foo 2} (in {:foo 1, :cat 4}))
           expected: {:foo 2} 
                 in: {:foo 1, :cat 4}
           :foo expected: 2
                     was: 1
failure in (core.clj:21) : sample.test.core
           (expect :baz (in #{:foo :bar}))
           key :baz not found in #{:foo :bar}
failure in (core.clj:24) : sample.test.core
           (expect :baz (in [:foo :bar]))
           value :baz not found in [:foo :bar]

expectations does it's best to provide you with any additional info that might be helpful. In the case of the vector and the set there's not much else that can be said; however, the map failure gives you additional information that can be used to track down the issue.

There's nothing magical going on with 'in' expectations and you could easily do the equivalent with select-keys, contains?, or some, but expectations allows you to get that behavior while keeping your tests succinct.

Clojure: Non-equality expectations

2011-11-01T06:55:00.004-04:00

Clojure Unit Testing with Expectations Part One
Clojure Unit Testing with Expectations Part Two (this entry)
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

In my last blog post I gave examples of how to use expectations to test for equality. This entry will focus on non-equality expectations that are also available.

Regex
expectations allows you to specify that you expect a regex, and if the string matches that regex the expectation passes. The following example shows both the successful and failing expectations that use regexes.


(expect #"in 14" "in 1400 and 92")

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 4 msecs
0 failures, 0 errors.

(expect #"in 14" "in 1300 and 92")

jfields$ lein expectations
failure in (core.clj:17) : sample.test.core
           (expect in 14 in 1300 and 92)
           regex #"in 14" not found in "in 1300 and 92"
Ran 1 tests containing 1 assertions in 5 msecs
1 failures, 0 errors.

As you can see from the previous example, writing an expectation using a regex is syntactically the same as writing an equality expectation - and this is true for all of the non-equality expectations. In expectations there is only one syntax for expect - it's always (expect expected actual).

Testing for a certain type
I basically never write tests that verify the result of a function is a certain type. However, for the once in a blue moon case where that's what I need, expectations allows me to verify that the result of a function call is a certain type simply by using that type as the expected value. The example below shows the successful and failing examples of testing that the actual is an instance of the expected type.

(expect String "in 1300 and 92")

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 6 msecs
0 failures, 0 errors.

(expect Integer "in 1300 and 92")

jfields$ lein expectations
failure in (core.clj:17) : sample.test.core
           (expect Integer in 1300 and 92)
           in 1300 and 92 is not an instance of class java.lang.Integer
Ran 1 tests containing 1 assertions in 5 msecs
1 failures, 0 errors.

Expected Exceptions
Expected exceptions are another test that I rarely write; however, when I find myself in need - expectations has me covered.

(expect ArithmeticException (/ 12 0))

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 6 msecs
0 failures, 0 errors.

(expect ClassCastException (/12 0))

jfields$ lein expectations
failure in (core.clj:19) : sample.test.core
           (expect ClassCastException (/ 12 0))
           (/ 12 0) did not throw ClassCastException
Ran 1 tests containing 1 assertions in 4 msecs
1 failures, 0 errors.

There's another non-equality expectation that I do use fairly often - an expectation where the 'expected' value is a function. The following simple examples demonstrate that if you pass a function as the first argument to expect it will be called with the 'actual' value and it will pass or fail based on what the function returns. (truthy results pass, falsey results fail).

(expect nil? nil)
(expect true? true)
(expect false? true)

jfields$ lein expectations
failure in (core.clj:19) : sample.test.core
           (expect false? true)
           true is not false?
Ran 3 tests containing 3 assertions in 4 msecs
1 failures, 0 errors.

These are the majority of the non-equality expectations; however, there is one remaining non-equality expectation - in. Using 'in' is fairly straightforward, but since it has examples for vectors, sets, and maps I felt it deserved it's own blog post - coming soon.

Clojure: expectations Introduction

2011-11-01T06:54:00.001-04:00

Clojure Unit Testing with Expectations Part One (this entry)
Clojure Unit Testing with Expectations Part Two
Clojure Unit Testing with Expectations Part Three
Clojure Unit Testing with Expectations Part Four
Clojure Unit Testing with Expectations Part Five
Clojure Unit Testing with Expectations Part Six

A bit of history
Over a year ago I blogged that I'd written a testing framework for Clojure - expectations. I wrote expectations to test my production code, but made it open source in case anyone else wanted to give it a shot. I've put zero effort into advertising expectations; however, I've been quietly adding features and expanding it's use on my own projects. At this point it's been stable for quite awhile, and I think it's worth looking at if you're currently using clojure.test.

Getting expectations
Setting up expectations is easy if you use lein. In your project you'll want to add:

:dev-dependencies [[lein-expectations "0.0.1"]
                   [expectations "1.1.0"]]

After adding both dependencies you can do a "lein deps" and then do a "lein expectations" and you should see the following output.

Ran 0 tests containing 0 assertions in 0 msecs
0 failures, 0 errors.

At this point, you're ready to start writing your tests using expectations.

Unit Testing using expectations
expectations is build with the idea that unit tests should contain one assertion per test. A result of this design choice is that expectations has very minimal syntax.

For example, if you want to verify the result of a function call, all you need to do is specify what return value you expect from the function call.

(expect 2 (+ 1 1)

note: you'll want to (:use expectations); however, no other setup is required for using expectations. I created a sample project for this blog post and the entire test file looks like this (at this point):

(ns sample.test.core
  (:use [expectations]))

(expect 2 (+ 1 1))

Again, we use lein to run our expectations.

jfields$ lein expectations
Ran 1 tests containing 1 assertions in 2 msecs
0 failures, 0 errors.

That's the simplest, and most often used expectation - an equality comparison. The equality comparison works across all Clojure types - vectors, sets, maps, etc and any Java instances that return true when given to Clojure's = function.

(ns sample.test.core
  (:use [expectations]))

(expect 2 (+ 1 1))
(expect [1 2] (conj [] 1 2))
(expect #{1 2} (conj #{} 1 2))
(expect {1 2} (assoc {} 1 2))

Running the previous expectations produces similar output as before.

jfields$ lein expectations
Ran 4 tests containing 4 assertions in 26 msecs
0 failures, 0 errors.

Successful equality comparison isn't very exciting; however, expectations really begins to prove it's worth with it's failure messages. When comparing two numbers there's not much additional information that expectations can provide. Therefore, the following output is what you would expect when your expectation fails.

(expect 2 (+ 1 3))

jfields$ lein expectations
failure in (core.clj:4) : sample.test.core
           (expect 2 (+ 1 3))
           expected: 2 
                was: 4

expectations gives you the namespace, file name, and line number along with the expectation you specified, the expected value, and the actual value. Again, nothing surprising. However, when you compare vectors, sets, and maps expectations does a bit of additional work to give you clues on what the problem might be.

The following 3 expectations using vectors will all fail, and expectations provides detailed information on what exactly failed.

(expect [1 2] (conj [] 1))
(expect [1 2] (conj [] 2 1))
(expect [1 2] (conj [] 1 3))

jfields$ lein expectations
failure in (core.clj:5) : sample.test.core
           (expect [1 2] (conj [] 1))
           expected: [1 2] 
                was: [1]
           2 are in expected, but not in actual
           expected is larger than actual
failure in (core.clj:6) : sample.test.core
           (expect [1 2] (conj [] 2 1))
           expected: [1 2] 
                was: [2 1]
           lists appears to contain the same items with different ordering
failure in (core.clj:7) : sample.test.core
           (expect [1 2] (conj [] 1 3))
           expected: [1 2] 
                was: [1 3]
           3 are in actual, but not in expected
           2 are in expected, but not in actual
Ran 3 tests containing 3 assertions in 22 msecs
3 failures, 0 errors.

In these simple examples it's easy to see what the issue is; however, when working with larger lists expectations can save you a lot of time by telling you which specific elements in the list are causing the equality to fail.

Failure reporting on sets looks very similar:

(expect #{1 2} (conj #{} 1))
(expect #{1 2} (conj #{} 1 3))

jfields$ lein expectations
failure in (core.clj:9) : sample.test.core
           (expect #{1 2} (conj #{} 1))
           expected: #{1 2} 
                was: #{1}
           2 are in expected, but not in actual
failure in (core.clj:10) : sample.test.core
           (expect #{1 2} (conj #{} 1 3))
           expected: #{1 2} 
                was: #{1 3}
           3 are in actual, but not in expected
           2 are in expected, but not in actual
Ran 2 tests containing 2 assertions in 15 msecs
2 failures, 0 errors.

expectations does this type of detailed failure reporting for maps as well, and this might be one if the biggest advantages expectations has over clojure.test - especially when dealing with nested maps.

(expect {:one 1 :many {:two 2}}
        (assoc {} :one 2 :many {:three 3}))

jfields$ lein expectations
failure in (core.clj:13) : sample.test.core
           (expect {:one 1, :many {:two 2}} (assoc {} :one 2 :many {:three 3}))
           expected: {:one 1, :many {:two 2}} 
                was: {:many {:three 3}, :one 2}
           :many {:three with val 3 is in actual, but not in expected
           :many {:two with val 2 is in expected, but not in actual
           :one expected: 1
                     was: 2
Ran 1 tests containing 1 assertions in 19 msecs
1 failures, 0 errors.

expectations also provides a bit of additional help when comparing the equality of strings.

(expect "in 1400 and 92" "in 14OO and 92")

jfields$ lein expectations
failure in (core.clj:17) : sample.test.core
           (expect in 1400 and 92 in 14OO and 92)
           expected: "in 1400 and 92" 
                was: "in 14OO and 92" 
            matches: "in 14" 
           diverges: "00 and 92" 
                  &: "OO and 92"
Ran 1 tests containing 1 assertions in 8 msecs
1 failures, 0 errors.

That's basically all you'll need to know for using expectations to equality test. I'll be following up this blog post with more examples of using expectations with regexs, expected exceptions and type checking; however, if you don't want to wait you can take a quick look at the success tests that are found within the framework.