Relevance Blog

Entropy and Evolution of a Codebase

2013-05-21T13:30:00Z

Let's do a thought experiment about change over time. We will start with a "clean" codebase, organized into a bunch of modules. These can be classes, subsystems, namespaces... it doesn't matter for our purposes.

As time goes on, we have to make changes in the code. Suppose the changes are distributed randomly across the codebase. Some modules will get changed a lot more often than others. Empirically, the distribution of changes almost always follows a power law. That is, a few modules get changed very frequently while a decreasing number have an increasing interval between changes.

(In fact, concave shapes resembling a power law are very common in software. We see them in object lifespans, coupling, and the distribution of complexity across modules.)

We started with a clean codebase, but what does that mean? To me, it means that the code is simple, cohesive, and loosely coupled. In short, it is easy to change, and most changes are local.

With every change a developer makes, the module might lose cohesion, gain coupling, or become complex. These are entropy-increasing changes. On the other hand, the developer could split an incohesive module, decouple a coupled one, or factor out complexity. These are entropy-reducing changes.

Even with a very disciplined team, some proportion of changes will be entropy-increasing. We would all like to avoid them, but they happen anyway. I think of these as introducing stress into the codebase. The stress is trying to collapse the codebase into a Ball of Mud. We must eventually relieve that stress through refactoring. At every change, there is a chance that the module transitions from clean to dirty, or from dirty to clean.

If the odds of a clean-to-dirty transition and a dirty-to-clean transition were equal, then we should expect the codebase to reach a kind of thermodynamic equilibrium where dirtiness is conserved, but shuttled around. Sadly, the transition odds are not equal. The definition of clean means that it is easy to change. Dirty code is hard to change. We all intuitively understand that it is much harder to clean up a module than to dirty it in the first place. Therefore, the likelihood of clean-to-dirty transitions is higher than that of the reverse. From this alone, we could expect the fraction of dirty modules to increase over time. It is entropic decay.

There's a secondary effect at play that exacerbates this problem. Because it is easier to change a clean module than a dirty one, we will see some kinds of changes "sliding sideways" from a dirty module to a cleaner one. This is just another way of saying that kludges cluster. A developer who needs to change a dirty module in a difficult way is very likely to make a nearby change instead, especially under time or deadline pressure. That nearby change inevitably dirties the module it lands in, often by adding coupling.

If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you'll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.

Can this entropic decay be prevented?

Yes, but it is not easy. We know of several ways to avoid it.

Surgical anti-entropy. Have people consistently tackle the gnarliest, smelliest parts of the system and bring them back to health.
Ongoing stewardship. Have people nurture their part of the codebase indefinitely. They should have the time, skill, and desire to tend to their garden.
Methodical attacks on suspected entropy strongholds. Use version control records to find the least-frequently-changed modules that are coupled to areas with high change rates. Odds are, the infrequently changed module is really dirty. The frequent changes in nearby modules indicate "sliding away" from the problem. The long time between changes on the module in the middle is a sign of sickness, not stability.
Community spirit. Make sure the whole team knows that every single person is responsible for cleaning modules, fixing broken windows, reducing coupling, and reversing entropy. (This is the XP approach of Collective Code Ownership.)
Disposable code. Discard the entire codebase periodically and recreate it. Do this often enough and you will create smaller and smaller codebases, each isolated from the others. Sam Umbach points out that many effective startups are built on this idea: if the startup fails, you throw the code away; if it succeeds, you throw it away and rewrite.

Some applications live for decades. These require strong anti-entropy measures. Others live for days or weeks. Dirty code is more expensive to maintain, but with such a short lifespan, who cares if these get dirty? Whatever your approach, think about the expected lifespan of the codebase. Be deliberate about your approach to entropy.

Chas Emerick, Mostly Lazy - Podcast Episode 031

2013-05-14T14:00:00Z

Chas Emerick is a well-known and respected member of the Clojure community. His contributions to the Clojure world include the Friend authentication library, the annual State of Clojure Survey, and, of course, his book, to name just a few. On top of that, he's a very insightful and interesting person. So I was thrilled to get the chance to sit down with him and record what I think turned out to be a fascinating conversation. We talked about his book, his business, his secret new project, "100% time", and the weight of the word "should".

As a bonus, although he and I disagree about whose idea this was (I still say it was his), we continued the conversation on Mostly Lazy, his podcast. That episode should be available not too long after this one.

Download the episode.

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Chas Emerick

His blog
@cemerick on Twitter
His GitHub home

Music

Chas chose "Whispering Wind" by Moby as the intro, and "Take me Home Country Roads" by John Denver to end the show.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

Jason Wolfe of Prismatic - Podcast Episode 030

2013-05-07T14:30:00Z

One of the few unfortunate aspects to working at Relevance is that we sometimes know about people using Clojure in cool ways that we can't talk about. So when Ben Moss suggested that we talk to someone at Prismatic, I jumped at the chance. They have definitely put Clojure into production in a big and awesome way, and I was really glad to get the chance to talk to Jason Wolfe about their experience building Prismatic, including some of the cool stuff they've open sourced out of it. A big thanks to Jason, and to you, our listeners.

Download the episode.

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Jason Wolfe

Music

Jason chose "Fight Test" by The Flaming Lips as the intro, and "Get Free" by Major Lazer to end the show.

Action Items

Clojure Style Guide

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

Where to Find Relevancers: May Edition

2013-05-01T14:00:00Z

Want to meet a Relevancer in person? Here's where you can find us during the month of May:

Little Rock, AR 5/4
Made by Few
Attending: Kevin Altman

Norfolk, VA 5/6 @ 6pm
757.rb Ruby Meetup Group
Speaking: Russ Olsen: Looking Inside Your Ruby Implementation

Sydney, Australia 5/9
YOW! Night
Speaking: Stuart Sierra: Clojure: Lisp for the Real World

Cambridge, MA 5/9 @ 6:30pm
Clojure Meetup
Attending: Gabriel Horner

Sydney, Australia 5/10-5/11
Intro to Clojure Training
Trainer: Stuart Sierra

Brisbane, Australia 5/14-5/15
Intro to Clojure Training
Trainer: Stuart Sierra

Portland, OR 5/15
Refresh PDX
Speaker: Michael Parenteau: The Myth of the Half-Brained Designer… And Their Magic Vacuum

Brisbane, Australia 5/16-5/17
YOW! Lambda Jam
Speaking: Stuart Sierra: Keynote, Data, Visibility & Abstraction; Datomic Jam

Melbourne, Australia 5/20-5/21
Intro to Clojure Training
Trainer: Stuart Sierra

Melbourne, Australia 5/21
YOW! Night
Speaking: Stuart Sierra: Clojure: Lisp for the Real World

Lake Denman - Podcast Episode 029

2013-04-23T14:00:00Z

Michael Parenteau suggested to me that we ought to have Lake Denman on the show. "He's done some cool stuff, man." I remember hearing about a crazy and awesome Arduino project that Lake had worked on, so I thought, "Yeah, sure, we should totally have Lake on." And I'm very glad we did: Lake has indeed worked on some amazingly creative projects, and he has even more ambitious and interesting ones in mind. I hope you enjoy listening to him talk about them on this episode as much as I enjoyed hearing them.

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Lake Denman

@L4k3 on Twitter
His Relevance profile

Music

Lake chose "Cherub Rock" by The Smashing Pumpkins as the intro, and "Folk You" by Lake Denman/Nailbiter Society to end the show.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

The Shoemakers

2013-04-22T14:30:00Z

Over an email thread a little while back, one of my colleagues suggested that programming should be a required subject in high school. To survive in today's world, everyone should have a basic understanding of programming because technology is so essential to everyone's lives. His friend retorted that, "Since shoes are also essential, high school students should take a mandatory year of cobbling."

I think the shoemaker analogy falls apart in a big way. Companies don't need to modify their shoes every time they want to execute a business strategy. But I can't think of any serious strategic change that doesn't require software changes. I think this fallacy is also behind the notion of "outsourcing IT because it's not 'core'." Technology is core to every company's business today.

For example, suppose a big box shoe retailer in the US wants to enter the Canadian market by opening just one store. Building the store and hiring the staff is the single smallest part of that challenge. The retailer will need to set up at least one new distribution center, so their supply chain systems will have to change. Canada has a completely different tax system from the US, so that means they either have to extend their current tax system to support VAT and Canadian product classifications or, more likely, create a parallel tax system for the new market. Two tax systems means a new feed into their general ledger system.

Products in some parts of Canada have to be labeled in both English and French so their merchandising system needs to keep track of multiple languages. The same goes for their online content management system.

Employment laws are different in Canada, so the HR systems have to be extended, updated, or run in parallel to keep track of the workforce. Payroll is probably outsourced to an in-market provider, but a new feed will have to be built there.

Data privacy laws are different, and they prohibit storing customer data in any US jurisdiction (thanks to the PATRIOT act). That means a new "single system of record" for the customer database. Any system dealing with loyalty programs also has to be updated to know about both customer databases.

Oh, and the website should support single sign-on so a customer can use their one account ID and password no matter where they happen to be. A Canadian customer must be authenticated against the Canadian customer database, so identity federation comes into play.

I'm already tired just writing this much up. There are probably a half-dozen other major enterprise systems that would all have to be modified, duplicated, integrated, or federated just to open this one store in Canada.

But I bet the employees can still wear the same shoes.

Ben Vandgrift - Podcast Episode 028

2013-04-09T14:00:00Z

Pretty much every conversation I've ever had with Ben Vandgrift has involved him saying something profound or insightful. I think this episode of the podcast was no exception, as we talked about Ben's experience as a relative newcomer to Clojure, and he had some constructive (and slightly painful) feedback for the Clojure community. We also got to talk about (inc Relevance), the quarterly gathering of Relevance employees, which Ben spearheads. I hope you enjoy the show!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Ben Vandgrift

Music

Ben chose "Beat on the Brat" by The Ramones as the intro, and "I Wanna be Sedated" by The Ramones to end the show.

Credits

Episode Cover Art:

Kevin Altman

Production Assistance:

Abstraction or Leverage?

2013-04-04T13:30:00Z

A colleague of mine recently accused me of speaking too precisely. He was giving me a bit of grief, but I took it as a compliment. In this industry, we torture words until they give up all meaning. Keep your eyes open for the next time you read the word "significant." Does it actually refer to something that signifies? Or is it just salt, sprinkled in for flavor?

One of the words I see overused is "abstraction." I hear developers use it when they talk about the DRY principle and reducing common code. Reducing common code to eliminate repetition improves your leverage, but not necessarily abstraction. Leverage allows you to make a change in one place that affects all uses.

As an example, think about one of my favorite code-eating patterns in OO: the Null Object pattern. It concentrates a lot of "if not null" code into one place, which means you can change the logic in that one place. That's leverage: you can make a small changes with large-scale effects.

"Extract method" and "extract superclass" transformations fall into this camp. They add leverage, which is useful.

By drawing this distinction, I'm not criticizing leverage. It's useful and important. But there's an additional thing to consider. When you extract common code to gain leverage, you have little ability to ignore details. Indeed, the specific case is often tightly coupled to the implementation of the general case.

(I will admit that when I distill code down this way, it sometimes clears away the cobwebs enough to see the abstraction that was lurking in the corner. That's always a lovely feeling. So leverage can lead to abstraction.)

The essence of abstraction is cleaving. Abstraction separates a manifold thing into independent concepts. By separating the concepts, it also allows us to see connections between things that share one of those concepts. So abstraction also illuminates connections between things.

For example, consider Clojure's seq (pronounced "seek") abstraction. It calls out the concept "Sequential Iteration," thus bringing together everything that acts like an ordered sequence that can be walked. At the same time, it separates that concept from others. For a vector, seq separates that concept of "Sequential Iteration" from the concepts of "Random Access," "Stacklike Object," and "Lazy Evaluation." For a string, seq separates "Sequential Iteration" from "Unicode Storage" and "Regex Matching." That's the part about separating a manifold thing. At the same time, "seq" shows us that there is an important characteristic shared across all those implementations.

In each case, the abstraction allows us to emphasize a particular aspect of the thing while disregarding the rest. This is the dual nature of abstraction: it obscures and reveals.

As it happens, good abstractions also provide leverage, because they separate concerns. I think this is why the words get conflated. All abstractions offer leverage, but not all leverage is an abstraction.

As you work on your designs, think about where you can split those conceptual atoms. What ideas can you separate? How can you then reconnect the shared concepts?

Where to Find Relevancers: April Edition

2013-04-01T14:00:00Z

Want to meet a Relevancer in person? Here's where you can find us during the month of April:

Philadelphia, PA 4/2-4/4
Emerging Technologies for the Enterprise
Speaking: Stuart Sierra: Clojure/ClojureScript: One Language to Rule the Web

St. Augustine, FL 4/4-4/5
Ancient City Ruby
Speaking: Russ Olsen: Insight, Intuition and Programming

Panama City Beach, FL 4/11-4/14
LessConf
Attending: Maggie Litton, Ben Vandgrift, Marc Phillips, Ryan Neufeld, Kevin Altman, Sam Umbach

Durham, NC 4/16
West End Ruby Hack Night @ Relevance HQ
Attending: Sam Umbach, Yoko Harada

Alexandria, VA 4/23 @ 6pm
ClojureDC Meetup Group
Speaking: Bobby Calderwood: Datomic

Durham, NC 4/25 @ 7pm
Triangle Clojure Meetup Group @ Relevance HQ
Pedestal Q&A Leader: Ryan Neufeld
Attending: Chris Redinger, Daemian Mack

Columbia, SC 4/25-4/26
ConvergeSE
Attending: Ben Vandgrift

Portland, OR 4/29-5/2
RailsConf
Speaking: Yoko Harada: Datomic, From Ruby, From Rails
Attending: Jamie Kite, Lake Denman

Quick Bits from Speakerconf

2013-03-22T18:30:00Z

You know how one of the best parts of any conference is the "hallway track?" Speakerconf is designed to maximize that experience: conversation shaped by talks, but with more opportunity for many-to-many communication.

The speakers are diverse and so are the topics. I thought I would share some of the more interesting references that came up during our discussions.

LEDBAT (RFC 6817). "Lower-than-best-effort" data transfer reduces congestion by deliberately adding latency.
libutp - The uTorrent transport protocol library, which implements LEDBAT.
Iranian weavers deshredding documents from the U.S. embassy in Tehran in 1979, and the DARPA challenge from 2011.
crazyegg - a service for heat mapping overlays on web pages.
Computational Tree Logic - an extension of first-order predicate logic to cover future universes. Quantifiers include "there exists at least one universe in which!"
As We May Think - An enormously influential essay by Vannevar Bush.
Generalizing Monads to Arrows - One of John Hughes' many excellent papers about abstraction.
Probabilistic Graphical Models - A method for integrating uncertain beliefs to estimate likelihood of outcomes, like Bayesian networks on steroids.
Composing Contracts - A paper by Simon Peyton-Jones on using combinators in a functional language to model financial instruments.
Five flavors of logic: miniKanran, cKanran, alphaKanran, and a couple more that I can't remember.

As you can imagine, it was a stimulating week. Thanks to Jay Fields and Josh Graham for organizing it, and a special thanks to the sponsors Forward, innoQ, and DRW Trading who make it possible.

Tim Ewald on Pedestal - Podcast Episode 027

2013-03-18T20:30:00Z

As this post goes up, Tim Ewald and Stu Halloway are on stage at Clojure/West unveiling Pedestal, a set of open source libraries for writing web applications and services in Clojure. We here at Relevance have been hard at work on it for months, and we're extremely excited to be able to finally share it with the world.

In this episode, I talk with my close friend Tim about what Pedestal is, why we wrote it, and what makes it so interesting. Enjoy!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest,

Tim Ewald

Music

Tim chose "Darkness on the Edge of Town" by Bruce Springsteen as the intro, and "Blinded by the Light" by Bruce Springsteen to end the show.

Credits

Marc Phillips (@turbomarc), production assistance
Michael Parenteau (@parenteau), episode cover art
Daemian Mack (@daemianmack), production assistance
Kim Foster, production assistance

Where to Find Relevancers: March Edition

2013-03-01T16:00:00Z

Want to meet a Relevancer in person? Here's where you can find us during the month of March:

Aruba 3/4-3/8
SpeakerConf
Speaking: Justin Gehtland, Michael Nygard

Durham, NC 3/12
Girl Develop It: Intro to Git & Github Class @ Relevance HQ
Attending: Lynn Grogan

Portland, OR 3/17
ClojureScript Training
Trainers: Stuart Sierra, Luke VanderHart

Portland, OR 3/18-3/20
Clojure/West
Speaking:
Stuart Sierra: Clojure in the Large
Brenton Ashworth & Tim Ewald: Introducing Pedestal
Timothy Baldridge: Clojure & LLVM
Ryan Neufeld: Editing Clojure Effectively with Emacs
Stuart Halloway: Simulation Testing with Simulant
Attending: Almost all of Relevance!

Durham, NC 3/19
West End Ruby Hack Night @ Relevance HQ
Attending: Sam Umbach, Yoko Harada

Durham, NC 3/20
Triangle DevOps Meetup @ Bronto Software
Speaking: Michael Nygard
Attending: Sam Umbach

Boston, MA 3/28-29
Monitorama
Attending: Sam Umbach

Alex Redington - Podcast Episode 026

2013-02-19T15:00:00Z

If you've listened to the show more than once or twice, you've probably heard us mention pair programming. One of the things that people often ask us is how we make that work when so many of us are remote. So I thought that it would be good to talk with someone with whom I have done a fair amount of very successful remote pair programming: Alex Redington. The result was this episode, in which we talk about the various tools and pairing behaviors we use. And about setting things on fire.

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest,

Alex Redington

Music

Alex chose "Lateralus" by Tool as the intro, and "The Regulator" by Clutch to end the show.

Credits

Lake Denman (@l4k3), Lynn Grogan (@lynngrogan), Sam Umbach (@samumbach), production assistance
Michael Parenteau (@parenteau), episode cover art

Hash Maps as Mini State Machines

2013-02-18T13:11:00Z

While working with Ryan Neufeld last Friday, we came across a problem that I had experienced more than once in my programming career: how to toggle a hash map value between two (or more) states.

As a starting example, let's assume that we have a hash map that contains a single key/value, and that we want to toggle the value between -1 and 1. We could write the function thusly:

(defn toggle-value [m]
  (update-in m [:value]
    (fn [v]
      (if (>= v 0)
        -1
        1))))

Now while this code will work, it's not exactly elegant, nor concise. And if we had more than two states (-1, and 1) then we would be forced to change the "if" into a "case" (or a "cond") and then add more and more state transitions to the "if" expression, and our already ugly code would get that much uglier.

It was at this part of my pairing session with Ryan that I remembered an old trick from when I first learned to program in QBasic (more than a few years ago). The trick was that multiplying either -1 or +1 by -1 will cause the resulting value to "toggle" between the two numbers. Actually this works for any states that need to toggle between -N and N, but we'll use -1 today to demonstrate:

(defn toggle-value [m]
  (update-in m [:value] (partial * -1)))

This code is much more concise, but perhaps not as clear. However, for the problem Ryan and I were dealing with on Friday, we couldn't use numbers: we needed to use keywords. We had two states: ":checked" and ":unchecked" and possibly nil. We needed a clear, concise way to toggle between these three states, without writing an ugly case/cond block. In my mind I thought, "we need a function that will return a new state, given an old state. A function that takes a single arg, and returns the new state value." And then I remembered: "Clojure hash maps are functions! And they take a single value (key) and return a single result (value)." Well then, we can write our state transitions with this little bit of code:

(def toggle-states
  {:checked :unchecked
   :unchecked :checked
   nil :checked})

(defn toggle-value [m]
  (update-in m [:value] toggle-states))

This is what I consider good Clojure code. It's simple, clean, concise, and easily extensible. And hopefully it comes in as handy for you as it did for me.

GOTO Night on 2/25 with Michael Nygard!

2013-02-14T22:07:00Z

We're pleased to announce that Michael Nygard will be the featured speaker at GOTO Nights in Chicago on Monday, February 25. Space is limited, so sign up today.

Chicago, IL 2/25/2013, 5-7:30pm
GOTO Night - Devops Night
Speaking: Michael Nygard

In the mean time, check out Michael's interview with Michael Floyd from QCon 2012 about Redefining CAP.

Two Recent Talks

2013-02-11T14:00:00Z

Video for two of my recent talks has gone live. Here they are!

Whence Complexity

As programmers we grapple with complexity. Whether we classify it as "essential" or "accidental" complexity, we all agree that complexity is poison. At the same time, as living beings, we are sustained by complexity. Complexity allows us to sustain our existence far from thermal equilibrium. In natural systems, complexity produces robustness. In human-created systems (whether information, social, political, or economic) complexity tends to create fragility and extreme non-linear responses to stimuli.

At Clojure/conj last November, I discussed the origin of complexity as the result of ongoing dynamic processes.

Thanks to some last-minute calculations about spherical chickens, there was also an excessively vivid mental image included, for which I apologize.

This one is on Youtube.

Loopholes in CAP

Last November, I also spoke at QCon San Francisco on the subject of the CAP Theorem (a.k.a. Brewer's Conjecture.) CAP says that, of the three qualities of "consistency", "availability", and "partition-tolerance", a distributed system can achieve at most two.

I have to confess that this was a particularly nerve-wracking talk, because I followed Eric Brewer on to the stage. Yes, that Eric Brewer, he of the Conjecture itself and one of the field's top experts.

CAP went from "conjecture" to "theorem" in 2002, when Seth Gilbert and Nancy Lynch proved it. And, thus, a thousand database products were launched. Like any theorem, though, there are subtleties to consider. In this playful talk, I look at all the assumptions underlying CAP and try to violate them all, with examples from real systems.

You can see this talk at InfoQ.com.

Where to Find Relevancers: February Edition

2013-02-01T18:32:00Z

Want to meet a Relevancer in person? Here's where you can find us during the month of February:

Durham, NC 2/9-2/10
Girl Develop IT - RDU Intro to JavaScript & jQuery @ Relevance HQ
Attending: Lynn Grogan

Raleigh, NC 2/12
Triangle.rb Meetup Group
Attending: Daemian Mack

Atlanta, GA 2/18-2/19
DevNexus
Speaking: Stu Halloway
Talk 1: Simulation Testing
Talk 2: Concurrent Programming with Clojure
Talk 3: Clojure in the Field

Durham, NC 2/19
West End Ruby Hack Night @ Relevance HQ
Attending: Sam Umbach, Yoko Harada, Daemian Mack

Charlotte, NC 2/20
Charlotte Ruby Meetup, Lightning Talks
Hosting: Ben Vandgrift

Durham, NC 2/28
Triangle Clojure Meetup Group @ Relevance HQ
Attending: Daemian Mack

Alex Warr - Podcast Episode 025

2013-01-29T15:00:00Z

When potential customers decide that they might want to work with us, their first point of contact is our sales department. It's there that they get their first impression of what sort of company we are. So it has been clear to me for quite some time that we needed to have Alex Warr, the head of our sales team, on the show to talk about what sales is like at Relevance. Along the way, we also managed to talk about babysitting, gaming, law school, and a few other things besides. Enjoy the episode!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Alex Warr

@jalexwarr on Twitter
His Relevance profile

Music

Alex chose "Staying Alive" by The Bee Gees as the intro, and "Ziggy Stardust" by David Bowie to end the show.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

What We Learned, and the End of the Magic

2013-01-28T20:00:00Z

When we started the Word Magic project, there were some things we wanted to learn, and some things we wanted to confirm. This post is about those things.

First, we confirmed that the Big Data Reference Model holds true. We tried to operationalize analysis before we had really finalized the kind of information we wanted to produce. (Most specifically, how the numbers behind the "Word Usage Through the Decades" module would work.) In terms of the Reference Model, we were skipping the first two learning loops. Why would we do that? The same reason everyone else does. We thought we knew what we wanted.

Our initial mockups looked great, but they were foiled by the real data. The algorithm was to divide a word's occurrence in a decade by the occurrence of all words in that decade. That way, we expected to normalize out the general increase in written language since the 1600s. But, since we put both a phrase and the individual words on the same graph, we normalized to the largest occurrence of them. Sounds good, but in practice, the bubbles were all but invisible except for the most common word in the phrase. The most common word dominated the graph and caused the others to be invisible.

We eventually tried two visualizations and three algorithms before we got one that was suitable for launch. There are some different calculations that might have worked out well, but we couldn't try them. Why not? Because we jumped straight to operationalizing. Every time we changed the calculation in map/reduce, we had to spend multiple days reprocessing 400 years of humanity's entire printed output. We should have followed our own advice and done smaller samples in a quicker toolset.

The second thing we confirmed is that JRuby on Hadoop is a viable way to write map/reduce jobs. It's a thin wrapper on top of the Java API, but still an improvement.

Stuart Sierra's clojure-hadoop library also worked very well. Because this was a "20% project" for most of its life, we ended up with pieces written in JRuby, Clojure, and Java. The polyglot nature didn't really present a problem, because each piece was its own data flow.

On the deployment front, we used Pallet. It worked very well for deploying whole topologies of machines. We built several Hadoop + HBase + MapReduce clusters with it.

Finally, we confirmed that the Hadoop and HBase APIs are cumbersome, poorly documented, and frequently misleading. The processing model is expensive, and the query model is user-hostile. (We aren't the only ones to notice this.) Moving between our daily world of sharp tools like Ruby on Rails, Clojure, ClojureScript and Datomic and into the Hadoop world definitely felt like a step back in time.

Now that we've learned what we needed to learn from Word Magic, we have shut it down. Hadoop is great for scaling up with lots of servers, but it doesn't scale down well at all.

i-Human - Podcast Episode 024

2013-01-15T15:00:00Z

At Relevance, our mission is to help people solve hard problems through software. Most of the time, we do that through consulting. So it seemed well past time that we had one of our consulting clients on the show. And in my opinion we could not have made a better choice than Anne Tweet and Jay Patel of i-Human. We got a chance to talk not only about their experience working with us, but also about the really interesting software that they're building. I had a great time talking with them - I hope you, too, will enjoy this episode!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Guests

Our guests on this episode were Jay Patel and Anne Tweet of i-Human.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

Taming side-effects with mutability

2013-01-14T12:30:00Z

Clojure prefers immutable values and referentially transparent functions. But it also works well with the Java ecosystem, where everything is an object... mostly mutable.

I was recently dealing with ASM, the bytecode manipulation library. Most of the API is based on the Visitor pattern. To read a class from bytecode, you construct a ClassReader object from an array of bytes or an input stream, then tell it to accept an object that implements the ClassVisitor interface:

public interface ClassVisitor{
  void visit(int version, int access, String name, String signature, 
      String superName, String[] interfaces);
  void visitSource(String source, String debug);
  void visitOuterClass(String owner, String name, String desc);
  AnnotationVisitor visitAnnotation(String desc, boolean visible);
  void visitAttribute(Attribute attr);
  void visitInnerClass(String name, String outerName, 
      String innerName, int access);
  FieldVisitor visitField(int access, String name, String desc, 
      String signature, Object value);
  MethodVisitor visitMethod(int access, String name, String desc, 
      String signature, String[] exceptions);
  void visitEnd();
}

Well, now we see a bit of a problem. The ClassReader gets to call these methods, which mostly return void. When a ClassVisitor does return an object, it returns yet another visitor that the ClassReader calls to descend into members of the class. The methods of the visitor get access to information about the class, but how do we get it out?

In other words, how do you use an API that is 100% about side-effects and mutability inside a language that favors immutable values and pure functions?

One solution is to use mutability to contain the side-effects within a function, such that the function is referentially transparent from the outside. In so doing, we tame the side-effects and put a boundary around them:

(defn analyze [c]
  (let [classinfo (atom {})
        v         (reify ClassVisitor
                    (visit [this version access name sig supername interfaces]
                      (swap! classinfo assoc 
                             :static?    (static? access)
                             :interface? (interface? access)
                             :final?     (final? access)
                             :classname  name 
                             :super      supername))
                    (visitSource [this source debug])
                    (visitOuterClass [this owner name desc])
                    (visitAnnotation [this desc visible])
                    (visitAttribute [this attr])
                    (visitInnerClass [this name outername innername access])
                    (visitField [this access name desc sig value]
                      (swap! classinfo update-in [:members] conj 
                             {:type       :field
                              :visibility (visibility access)
                              :static?    (static? access)
                              :final?     (final? access)
                              :name       name})
                      nil)
                    (visitMethod [this access name desc sig exceptions]
                      (swap! classinfo update-in [:members] conj 
                             {:type       :method
                              :visibility (visibility access)
                              :static?    (static? access)
                              :final?     (final? access)
                              :abstract?  (abstract? access)
                              :name       name})
                      nil)
                    (visitEnd [this]))]
    (.accept c v 0)
    @classinfo))

In the let form at the beginning, I create a new atom that holds an empty map. This will be my mutable state for this function. Then I create an instance of ClassVisitor, where most of the methods do nothing and return nil. Only visitClass, visitField, and visitMethod are interesting to me. In those, I mutate the atom to attach the info that ClassReader passed to the visitor.

After the setup in the let, all that is left is to call ClassReader.accept with the new visitor. At that point the ClassReader does its bytecode interpretation and makes all its calls into the visitor. Finally, I get the resulting value out of the classinfo atom, which now has a nice, immutable Clojure-friendly map of information collected from that class file.

This technique is similar to using transients to build collections: allow mutability within a function, but don't let it escape. Transients do it for speed. The ClassVisitor does it to bridge to an object-oriented API.

Justin Gehtland - Podcast Episode 023

2013-01-03T15:00:00Z

Way back at the end of 2011, I had this idea that maybe I could record some conversations with various Relevancers and throw together a podcast. Our very first guest was Justin Gehtland, so as the end of the year and the anniversary of the show approached, I thought it would be only too appropriate to have Justin on again. Besides, he's a great guest, and we were clearly overdue to have him on the show again.

As is traditional for this time of year, we spent a good chunk of our conversation talking about the year gone by (verdict: it was crazy) and the year to come (verdict: it's super exciting).

I hope you enjoy this episode!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Justin Gehtland

@jgehtland on Twitter
His Relevance profile

Music

Justin chose "Mountain Song" by Jane's Addiction as the intro, and "Clean Elvis" by Dan Reeder to end the show.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

Where to Find Relevancers: January Edition

2013-01-02T16:30:00Z

Want to meet a Relevancer in person? Here's where you can find us during the month of January:

Sandusky, OH 1/8-1/11
CodeMash
Speaking: Jen Myers Talk: Straight-Up Design: Simple Ways for Devs to Make Apps Look and Work Better

Durham, NC 1/9-1/30
Girl Develop IT - RDU Intro to HTML & CSS @ Relevance HQ
Attending: Lynn Grogan

Richmond, VA 1/15
Richmond Ruby User Group
Speaking: Russ Olsen

Denver, CO 1/17
Den of Clojure Meetup Group
Attending: Timothy Baldridge

Durham, NC 1/24
Triangle Clojure User Group Meetup @ Relevance HQ

The Real Secret Behind Project Failure

2012-12-24T17:56:00Z

To the owner of a failed software project, there is little consolation that roughly forty to seventy out of every hundred other projects have also failed to meet their goals.

This sad math reflects a simple truth - delivering software solutions to hard business problems is, well... hard. Teams lacking sufficient technical skills are obviously doomed from the start. But a respectable lineup of developers merely gets you in the game. Projects making this first cut now must endure an onslaught of other challenges, from ambiguous requirements and unclear priorities to unrealistic schedules, poor communication, insufficient planning, and unmanaged risks. In other words, as a product owner embarking on a new assignment, the biggest threats to your success are not technical problems at all. At Relevance, we understand that software development is mostly a people problem.

This is why aside from employing an amazing group of developers, designers, and architects, most Relevance projects receive the direct attention of a Relevance coach to minimize the non-technical risks to your project, and to help great people accomplish great things.

So what exactly does a coach at Relevance do? A partial answer can be found in an episode of the Relevance podcast, but the conversation is sporadically interrupted by loud monkey noises (oddly not present at the time of the original recording). In true agile fashion we shall adapt and repeat, and our answer will focus on outcomes rather than activities. When clients ask us "What is the role of a coach at Relevance," we've found what they're really asking is "How will my project benefit by having a Relevance coach on the team?"

Saving you time and effort through efficient project management

As someone looking for a team to help grow your business, you've probably got a big problem, limited budget, and a tight timeline for finding a solution. And most likely you've got a lot of other very important things to do each day.

Once we kick off Iteration Zero, I like to tell clients they are now free to leave the engine room and snuggle comfortably in the captain's chair. As a busy product owner, having a Relevance coach on the job allows you to focus on the critical role of setting the course of the ship without the additional burden of monitoring and maintaining the complex propulsion, communication, and navigation systems required to complete your journey.

As an agile project manager, the Relevance coach saves you time and effort while achieving the necessary goals of transparent metrics, alignment between daily tasks and high-level goals, and protecting the unimpeded progress of the development team.

"But what if I have the project management expertise and time to devote to my project? Would I still want a Relevance coach on the team?"

For some projects, busy clients set broad targets and then connect with the development team for just a few minutes each day, trusting the Relevance coach as their near full time project manager. For others it may be a sliding scale of delegated responsibilities. For the most experienced and available clients, a Relevance coach serves to facilitate critical decisions such as changes to team priorities or core functionality. At these times, a Relevance coach enables the client to participate fully as product owner rather than project manager, to become an engaged passenger with someone else at the wheel until the matter is resolved.

There are also times where a Relevance coach will serve as proxy for product owners at both office and partner presentations, outlining plans and progress and gathering feedback for the team. Increasingly this role is extending to front line investigator and liaison with customers and executives alike.

Providing an objective perspective and minimizing risk

"I've spoken with other software consultancies who don't have coaches. Are their developers better than yours?"

This isn't about development expertise; it's about better overall outcomes. A Relevance coach on your project gives you a team catalyst, enabling a whole much greater than the sum of its parts. It means someone on your side with a dedicated, independent, and people-focused perspective - looking at the team and release in a different way.

Gil Broza in The Human Side of Agile describes the benefits of an 'agile team leader', though we believe a more accurate title is 'agile team enabler'. Broza notes that "by constantly observing the team's progress and learning about their hurdles, the leader can distinguish patterns, identify root causes, and stop small problems from growing into big ones." To accomplish this requires taking a birds-eye view of the situation, to focus on the big picture at all times.

"Why shouldn't one of the developers on the team play this role?"

Relevance developers are certainly capable of filling this critical role. At issue is whether this is the most cost-effective and efficient method to achieve your goals. Every moment spent by a developer preparing to facilitate a retrospective, discussing a shift in priorities with you, or evaluating upcoming staffing needs is a moment they are not delivering awesome software.

A developer in the role of coach would be by definition a part time responsibility. The "coach hat" and "developer hat" cannot be worn at the same time. To put their coach's hat on, a developer must temporarily leave the development team. Only computers are designed for multitasking. Human beings are only capable of switch-tasking, and switching between tasks is expensive.

Time spent context switching will come directly out of your project budget. Such teams can still be successful, but will have to be less efficient than those with a cleaner separation of roles.

As noted in "Resilience" by Andrew Zolli and Ann Marie Healy, such "cognitive diversity" within teams has been shown to consistently achieve better results. Relevance coaches help developers and clients consider a wider range of options, raising previously unseen issues, and helping to guide the team out of jams when they occur. Software projects are inherently risky, and unexpected obstacles can rise up suddenly to block the team. Teams consisting of individuals with similar modes of thinking can excel when the path is smooth, but adding individuals providing alternative views has been shown again and again to catalyze better performing, more resilient teams that can adjust faster and more effectively to difficult terrain, dramatically lowering the risk to your project.

"Why shouldn't I play this role as the product owner?"

To answer this question, please note that one of the key players on the team is you. As the Relevance coach identifies strategies for better team performance, this includes an objective view of the actions and interactions between all team members, including the product owner and other client stakeholders at a level that would be extremely difficult through self-reflection alone. Similar to the developer's dilemma, to become a part time coach necessitates giving up your role as full time product owner. In which capacity would you provide the most value to the project and your business?

Stacking the deck in your favor

In its simplest form, having a Relevance coach on your project is a really smart and cost effective method of risk mitigation. On most projects I spend a lot of time both asking and answering questions, proposing plans and providing feedback, and uncovering inconsistencies in goals and execution. I facilitate resolution of misunderstandings and help frame difficult decisions. At times I play the role of counselor and trusted advisor to developers and clients alike, serving as both relief valve and sounding board. And I love it.

In the end, technical difficulties are rarely the cause of project failure. Project failure is really team failure. Teams are made of individuals, including developers, product owners, and other stakeholders. In theory, if each individual performs perfectly, the input (your problem) should produce the correct output (your solution). But these people are not just boxes on an org chart. Inside those hard candy exteriors each team member has a creamy human middle. Any friction in the system generates heat, and things get sticky.

Unless sufficient attention is paid to the machinery of the team, the smooth and continuous flow of information and intentions can become clouded and spotty. When this occurs, even highly skilled teams can become bogged down or burnt out and fail to reach their goals. This is the real secret of project failure, and how a Relevance coach can help your project be a success.

Timothy Baldridge - Podcast Episode 022

2012-12-12T15:00:00Z

It may sound trite to say so, but Relevance is the awesome place it is because of the people that work here. So I thought it would be interesting to talk to our newest hire, Timothy Baldridge, about what it's like to get hired. He had a lot to say on the subject, and I found it pretty interesting. Just as interesting was finding out more about Clojure-Py, Timothy's implementation of Clojure on the Python runtime.

So please download the episode and enjoy!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guest, Timothy Baldridge

@timbaldridge on Twitter
On GitHub
His Relevance profile

Music

Timothy chose "Viva La Vida" by Coldplay as the intro, and "Don't Be Afraid" by Nobuo Uematsu to end the show.

Credits

Marc Phillips (@turbomarc), production assistance
Michael Parenteau (@parenteau), episode cover art

Relevance Blog

Entropy and Evolution of a Codebase

Chas Emerick, Mostly Lazy - Podcast Episode 031

Our Guest, Chas Emerick

Music

Links

Credits

Jason Wolfe of Prismatic - Podcast Episode 030

Our Guest, Jason Wolfe

Music

Links

Action Items

Credits

Where to Find Relevancers: May Edition

Lake Denman - Podcast Episode 029

Our Guest, Lake Denman

Music

Credits

The Shoemakers

Ben Vandgrift - Podcast Episode 028

Our Guest, Ben Vandgrift

Music

Credits

Abstraction or Leverage?

Where to Find Relevancers: April Edition

Quick Bits from Speakerconf

Tim Ewald on Pedestal - Podcast Episode 027

Our Guest,

Music

Links

Credits

Where to Find Relevancers: March Edition

Alex Redington - Podcast Episode 026

Our Guest,

Music

Links

Credits

Hash Maps as Mini State Machines

GOTO Night on 2/25 with Michael Nygard!

Two Recent Talks

Whence Complexity

Loopholes in CAP

Where to Find Relevancers: February Edition

Alex Warr - Podcast Episode 025

Our Guest, Alex Warr

Music

Credits

What We Learned, and the End of the Magic

i-Human - Podcast Episode 024

Guests

Links

Credits

Taming side-effects with mutability

Justin Gehtland - Podcast Episode 023

Our Guest, Justin Gehtland

Music

Links

Credits

Where to Find Relevancers: January Edition

The Real Secret Behind Project Failure

Saving you time and effort through efficient project management

Providing an objective perspective and minimizing risk

Stacking the deck in your favor

Timothy Baldridge - Podcast Episode 022

Our Guest, Timothy Baldridge

Music

Links

Credits