For developers using centralized version-control tools like CVS and Subversion, their workstations are often a complete mess of source code and checkouts scattered all over the place like pigeon droppings. They’ve got multiple checkouts of different upstream branches, a bunch of separate checkouts for projects based on the same branch, and even multiple copies of files within each checkout (with nicely dated suffixes or .bak.1, .bak.2-style naming). Frankly, it’s a mess. And it’s got two huge problems:

1. This workflow sucks for developers. It’s easy to lose work and hard to track progress.
2. Central IT and project management have no idea what’s going on. They can’t track or control anything and they can’t promote better practices.

Although some leading-edge developers will be using a tool like git that interface with CVS and SVN so they can work in a distributed, offline fashion and commit locally, most won’t. And even those that do work like that still suffer from a number of downsides since the whole team and the main repo aren’t doing the same.

The problem is that management is simply scared by the word “forking.” They envision fragmentation, disappearing commits, and invisible work, when in fact all of those things are already happening. Using a tool like Stash or GitHub Enterprise internally will counterintuitively increase transparency and control into an organization’s development practices, because people will push on a regular basis and use trackable forks within the context of the tool. In addition, there’s the clear benefits of smaller commits (because they’re fast and easy), feature branching (because it’s fast and easy), and maintain things in a way that enables better investigation of bugs (smaller commits make bisection easier).

In other words, using DVCS internally for corporate development improves the experience for both enterprise IT and individual software developers. The problem is that nobody seems to be telling that story to corporate IT.

Disclosure: Atlassian is a client, and GitHub has been one.

Last week I was at Adobe’s MAX conference in Los Angeles, where I grabbed some time with two of the key people behind PhoneGap, the incredibly popular framework for developing hybrid HTML5/native mobile apps. Over beers at Yard House (which has an outstandingly enthusiastic manager who loves craft and good beer), Andre Charland, Brian LeRoux, and I discuss PhoneGap, Adobe, craft and beer, the connection between designers and developers, and more.

I only had my phone on me at the bar, so this whole thing was recorded on a Nexus 4.

Some language may be NSFW, so watch at your peril.

Disclosure: Adobe is a client and paid for my travel and hotel at Adobe Max.

I saw two starkly different worlds of IT almost side-by-side last week, thanks to the absurdities of airline pricing, and it illustrated very clearly the contrast between how we perceive the world in our Bay-Area–centric bubble and how the world really is.

First, I spent some time at Amazon’s AWS Summit in San Francisco, where Amazon was pushing best practices at the bleeding edge of tech to one of the most technically sophisticated communities on the planet. Following that, I spent a day at DevOpsDays in Austin, Texas, en route to my home in Minnesota. (For some reason this was hundreds of dollars cheaper than a direct flight.)

In the Bay Area, I saw the same thing that’s endemic of the area. There’s a clear best way to do things, pretty much everyone is aware of it, and that’s what everyone does. Thanks to the heavy startup presence, there’s much less inertia in terms of existing cultures or infrastructure, so changes are easier. When you’ve got a next-door neighbor doing something amazing, it’s very hard to resist the peer pressure and the local culture, so everyone’s doing The Right Thing™. Very similar things hold true in the open-source world, where neighbors may be virtual but they’re still highly visible.

In Austin, it was an entirely different story. I saw yet another example of how the rest of the IT world, at least in this country, lives. I’ve seen it in places like Minnesota, Maine, and Oregon. It’s a world where trendy software vendors and startups don’t represent any meaningful part of the tech community, where businesses mostly don’t yet realize that software is eating the world. It’s a world where inertia rules the day, where business is king and sysadmins have little to no say in major changes. And it’s a world where even experimentation is difficult and must be done on the smallest of scales.

What happens in places like this? Let’s call it Everytown, USA. In Everytown, IT departments can’t afford to build a new infrastructure from scratch using Puppet or Chef in the cloud. They don’t have the freedom to do it externally or the resources to implement a private cloud internally.

Even at a conference like DevOpsDays Austin, if you ask people what they’re actually doing today, most of the time it has little to no resemblance to how a new Bay Area startup would set up its infrastructure. Don’t ask them about their plans, that’s often so ambitious as to be unusable. Maybe they’re maintaining cloud instances by hand in AWS, or maybe they’re slowly migrating a large datacenter full of pets to configuration management, which they’ve been working on for the past five years. If they’re open-source fans, chances are they’re running Nagios and have a huge collection of Nagios-related infrastructure that would need serious, dedicated effort to shift to anything different.

More modern shops could have migrated most or all of their servers to tools like Puppet or Chef, so everything’s at least under configuration management and thus documented and reproducible. But in many cases, this is for datacenter use only, either true on-prem or in a colo. Gaining the capacity, budget, and permission to even migrate to private cloud is impossible for many companies, and it could be that way for a while.

You can see the same thing at conferences for larger enterprise vendors like IBM — talking to attendees at IBM’s Pulse conference this spring, most of them are in exactly the situation I’ve described. IBM’s jump into both cloud and DevOps will make a significant difference to their adoptability in many places; it’s like a stamp of approval that these things are really ready for the enterprise.

“Shadow IT” developers outside the purview of IT-controlled infrastructure, on the other hand, often don’t have or don’t want to develop the expertise to learn DevOps philosophies and approaches. Developers may well be working in the cloud, but chances are they aren’t running tools like Puppet or Chef, and they don’t have any monitoring set up. They may hack things around by hand and hope everything doesn’t break too often, or they may outsource the infrastructure to somewhere external and run in a PaaS.

IT shops like this may be aware that better ways exists and they may have ambitions of going there, someday. The Bay Area view of the right infrastructure is always going to be years away for the rest of us — we even put William Gibson’s quote regarding this on our website:

The future is already here, it’s just unevenly distributed.

Update (5/5/13): Of course this is a generalization of reality, which is always more complex than a single answer at either end of the spectrum. I’ve just simplified it to communicate the overall points, which remain true regardless of the details. Reality looks like a distribution on both ends — but the distribution is shifted. I’m just talking about the most common cases within those distributions. There are clearly going to be some Bay Area companies with plenty of inertia, and some Everytown companies overflowing with cloud- and DevOps-based approaches. Even within a single company, there’s a distribution of approaches, with some areas more modern and others more legacy (heard of systems of engagement and systems of record?).

Disclosure: Amazon (AWS) and IBM are clients. Puppet Labs has been a client. Opscode and Nagios are not.

A couple of pieces of changes in direction regarding open source came out in the past few days, and they’ve gotten little coverage thus far, despite their fairly significant implications.

Gambling on traction with open source

NewSQL database provider Tokutek just went open source with its TokuDB v7 release yesterday. TokuDB is a MySQL/MariaDB storage engine based around an algorithm called fractal trees. What makes this move interesting?

For one, open-source NewSQL options are hard to come by. This is one market where open source isn’t yet table stakes, unlike NoSQL, so it does make companies stand out. VoltDB is one of very few OSS options, falling under the strongly copyleft AGPLv3. Tokutek went with GPLv2 for its engine (the same as MySQL), a slightly more permissive license in that you don’t need to provide source if it’s only available within a hosted service. Usefully, they also provided a patent license since that isn’t GPLv2′s strong point. This makes TokuDB newly interesting to service providers who want to incorporate an open-source NewSQL option into their products.

Secondly, it’s always interesting to look at the particular approach companies take to an OSS-centric model. In this case, it’s a combination of the classic models of support and proprietary add-ons (in this case tools for backup and recovery), according to SiliconAngle. As going open source with your core product isn’t a transition that’s easy to step away from, it can be useful to take a piecemeal approach, as you determine where your customers find the real value.

Maximizing the innovation window

Opscode, on the other hand, is moving in a more proprietary direction. As Adam Jacob, Chief Customer Officer, wrote in a post on the past five years:

One shift here is in the order of operations: before we wrote Chef, there was no Chef. We shipped the primitive first, then we built value (Hosted Chef and Private Chef) on top. As we move forward, we’ll shift to open sourcing new primitives after we build something cool on top of them that shows their power. [emphasis mine]

This shift to an “open-source the infrastructure” approach after you’ve already built a beautiful facade on top is a significant change to a model that’s entirely about differentiating on top (a la GitHub, Facebook, Twitter, LinkedIn) rather than being what I would call a true open-source company. It gives Opscode a new monopoly on the time window between when they create a new piece of infrastructure and when they release the proprietary frosting on top. It also has a detrimental effect on a leading subset of users who prefer a more composeable infrastructure, as we’re seeing now in the #monitoringsucks/#monitoringlove movement, and who will now be forced to wait for the core components until Opscode finishes building something on top of them. That said, much like the value of example code in SDKs, Adam is entirely right that building useful products on top of a core component will very clearly illustrate its values and some of its use cases.

So, two transitions: one shifting toward open, another shifting more closed. I’m looking forward to seeing what comes of both.

Disclosure: VoltDB is a client. GitHub has been a client. Opscode, Tokutek, Oracle (MySQL), MariaDB, Twitter, Facebook, and LinkedIn are not clients.

Here, I’m going to dig into data from Ohloh, which tracks a vast set of open-source software projects, to investigate some of the effects related to community size. I’ll look at a number of potentially connected variables centered around development activity, licensing choices, and hosting providers (GitHub, etc.).

As always, the caveats:

• This is only useful for active projects with active communities, because it contains only projects with commits during a 1-year period and members of the community must opt in to subscribe the project to Ohloh. This equates to 50,000+ projects, so it’s still a good-sized set.
• It is subject to any imperfections in Ohloh’s measurements, which is particularly relevant for license detection where it simply looks for strings in source files. It will miss any indirect references to licenses by name or URL. It also seems to miss some more obvious ones, which will set a lower bound on license discovery (but it should be independent of community size).
• In most cases, I’m ignoring the largest ~100 projects on Ohloh, which have ~150 or more committers per year, so these conclusions may not be generalizable to them. These tend to include well-known names like GNOME, Chrome, KDE, the Linux kernel, Mozilla, etc. There simply aren’t enough samples of things at similar size to aggregate data for general, non-project-specific conclusions.

To make these posts more easily readable, I’m going to try something new. All the methodology is now in the figure captions, so skip captions if you just want to read the what and ignore the how.

The size distribution of open-source communities

Before looking at the impact of size, I first wanted to gain an understanding of how big free/open-source software (FOSS) communities were, and how many project communities there were at each size. Plotting the community size against the number of FOSS projects at each size produced the plot shown below:

Ohloh data for projects active in the past year as of July 2012. Monthly data from the 30 days immediately prior to the Ohloh dump. The LOWESS fit shows a locally smoothed line in the noisier regions, using 1/8 of observations for smoothing each point. Not shown are the 92 projects with >150 committers per year.

Global features

What I find interesting about the shape of the above graph is two things: the helpfully linear behavior on this type of plot, and the gap between monthly and annual contributors. When this type of plot appears linear, it indicates behavior supporting a set of statistical distributions including the power law (the famous 80-20 principle stating that 80% of the effect comes from 20% of the causes). In the below section on specific effects, I’ll show some numbers indicating the kind of behavior we see as a result of this.

One interesting question I wanted to answer here was the relationship between monthly committers and “expected” monthly committers based on the year-long figures — we can get this by dividing annual committers by 12. However, appearances can be deceiving. This graph actually can’t tell us anything about that because there’s zero connection between projects with 80 annual contributors and 80 monthly contributors. Instead, what we can do to get at this information is directly correlate monthly and annual contributors at the level of individual projects, which I’ll show later.

What can that gap tell us, then? As it turns out, it’s not equally sized throughout. There’s a very real and linear increase in that difference as community size increases from 1 to 35 committers (it’s too noisy above that), with the logarithmic difference increasing from 0.57 to 1.29 (each unit of 1.0 indicates a 10x increase). The higher slope for the monthly committers indicates a set of values with a tighter overall distribution that are (unsurprisingly) biased away from high numbers of committers, which are much more easily attained for a project in a year than a month. Put simply, it’s an expected result — you get more unique committers in a year than a month.

Specific effects

The vast majority of projects are tiny, having just 1 or a few contributors. This is even more dramatic than it first appears if you look at the Y-axis, which is logarithmic rather than linear. From the full spreadsheet (embedded below),  we can draw some more quantitative conclusions. On an annual level, just over half of active projects (51%) have only 1 contributor, while 19% have 2, 9% have 3, 5% have 4, and 3% have 5 (see the PDF column below). Overall, 87% of projects have 5 or fewer committers per year (see the CDF column). Looking from the opposite perspective, merely 1% of projects have 50 or more committers per year, and a scant 0.1% have 200 or more (see the Rev. CDF column).

Contribution regularity is independent of community size

To directly compare monthly and annual committers, we need to pull the numbers at the level of individual projects and create a plot based on them, rather than looking at two independent histograms on the same graph as we did above. If we do that and aggregate it into 25-project bins to ease visualization, then fit lines to them, we can produce a plot much like the below:

Data points were created for each 25 projects, and percentiles were calculated for each data point. The lines indicate an observation-weighted cubic-spline fit with a smoothing factor of 1/1000.

This is a variation on a box plot, showing the median in thick black in addition to a number of percentiles to indicate the size of the distribution cores (25%–75%) and more extreme, non-outlier values (10%–90%). To interpret it, consider that medians represent the central values, while the thicker colored lines represent the “middle half” for each number of committers, and the thin colored lines represent nearly everything (the central 80%).

This plot shows a very clear typical range of annual committers, given a monthly number. Conversely, it could also be read the other way to suggest likely numbers of unique monthly committers, given an annual value.

I next wanted to look more specifically at the relationship between a simplistic prediction of monthly committers and the actual monthly values. It’s based purely on dividing the annual committers by 12 months, which means that a ratio of 1 would equate to each contributor making commits during only 1 month each year.

The ratio of expected monthly committers was generated from dividing 1/12 of annual committers by the monthly committers. Each semitransparent circle represents the median committer values of 25 data points, and darker colors indicate multiple overlapping circles.

Interestingly, while the data points distribute much more widely at lower committer counts (potentially due simply to larger populations), it remains near-linear and horizontal, going from a ratio of ~0.20 to ~0.25 as a function of community size. Values below 1 mean committers are making contributions during more than 1 month each year. In particular, if you multiple the ratio by 12 months, you get the average periodicity of someone’s contributions in months — so 0.25 * 12 = committing every 3 months, for a total of 4 months of contributions each year from each committer in large projects, on average, and 5 months from small projects using the same math. While many developers will contribute more, enough will also contribute less to make the final numbers come out around 4–5 months in a remarkably consistent fashion.

An important take-home from this result is that smaller projects are proportionately nearly as likely as larger ones to receive drive-by commits or have relatively inactive developers on a monthly basis.

Larger communities tend to get more engagement

To look for size effects at a finer-grained level than committers alone by looking at commits themselves, I took the ratio of monthly commits per committer and plotted it as a function of community size in the graph below. As the size of a project increases from 1 to ~10 developers, the median gradually doubles from ~5 to ~10 commits per committer, where it then holds steady as community size grows (beyond 20, the data become too noisy due to too few projects of that size).

Data points were created for each 25 projects, and percentiles were calculated for each data point. The lines indicate an observation-weighted cubic-spline fit with a smoothing factor of 1/1000.

A number of factors could explain this trend — for example:

• They receive or accept proportionately more drive-by patches that are credited to a committer rather than the patch contributor;
• Everyone is more active due to an effect of community interactions or peer pressure;
• They have a higher proportion of active committers, such as professional contributors who make more frequent commits;
• Most smaller projects will never gain the traction to grow larger, but the larger a project is, the more likely it is to have gained or be in the process of gaining developer traction.

However, the generally horizontal line at the 90th percentile (the peak around 7-8 appears to be an outlier due to some large projects with abnormally low committer levels that month) indicates that a subset of small communities do behave similarly to the larger ones. This suggests that it may be the last of these explanations.

“Post-OSS” licensing practices are a big issue in smaller communities

My eminent colleague James posted this succinct and bluntly honest tweet last fall:

younger devs today are about POSS – Post open source software. fuck the license and governance, just commit to github.

Luis Villa, open-source lawyer and Friend of RedMonk, wrote an excellent post following up on the topic, postulating that POSS behavior was an explicit rejection of permission-based culture. It’s easy to simply accept that this is happening, but as a scientist by training, I prefer to see whether there’s data to support or deny the assertion that licenses as a whole are growing less popular.

Ohloh is quite useful for licensing data because it goes beyond simply looking at COPYING, LICENSE or README files to directly examine the contents of each source file for strings found in licenses. While it’s undoubtedly imperfect because it looks directly for license strings so may miss poorly worded or obscure references to licenses, that will simply set a baseline for detection. Any changes relative to that baseline will still be valuable.

If we look at the percentage of active projects (1 commit in the past year) without licenses detected by Ohloh, it baselines around 20% for large projects, which one would hope embody best practices in open source. This is likely a combination of two factors, Ohloh’s detection ability and actual missing licenses (likely dominated by the former).

But once we start looking at the trends, that’s when things get interesting. Take a look at the graphs below:

Data points were created for each 25 projects, and percentiles were calculated for each data point. The lines indicate an observation-weighted cubic-spline fit with a smoothing factor of 1/1000.

I’ve classified project licensing into one of four categories: None, Copyleft, Permissive, or Limited (a.k.a. weak copyleft). Let’s walk through them in order.

First, unlicensed projects would qualify for the POSS designation and are shown in the top left. This is the largest trend among all of the license types in terms of absolute license share, indicating the importance of thinking about it. When looking at monthly contributors, this trend flattens out around 15 committers at 20% of all active projects and stays flat well beyond the right edge of this graph, to at least 70 commiters per month (after that point it’s too noisy). Regardless of whether this trend is due to a true rejection of the permissive culture, as Luis Villa suggests, or whether it’s a function of lack of licensing education, the shift from 50% unlicensed single-developer projects to below 25% unlicensed projects with 15 or more contributors cannot be ignored. My interpretation is that essentially no projects with ≥10 monthly contributors have licensing problems, while ~1/3 of one-developer projects do. The transition occurs in the middle. In other words, as projects grow, they tend to sort out any licensing issues, likely because they get corporate users, professional developers, etc.

Second, let’s look at copyleft licensing, the next-most-popular type. As a counterpart to the POSS trend, the use of copyleft licenses increases from ~20% to 35–40% around 15–20 monthly committers before the data get too noisy to draw any further conclusions. However, four of the five largest data points (25-project aggregates) hover around 45–50% copyleft, suggesting a potential upper limit that’s driven in part by the Linux kernel and Linux distributions, some of the largest collaborative projects around.

The lower two plots, permissive and limited (weak copyleft) licensing, show mild upward trends on an absolute scale. Permissive shows a small increase from ~20% to ~25%. Limited licenses, on the other hand show a small increase from ~7–8% to ~11–12%. While small on an absolute scale, this modest-seeming trend indicates that limited licenses are roughly 50% more popular in larger communities than small ones.

Hosting providers generally do not support large communities well

The other interesting data point I have is which code forge each active project (1 commit in the past year) is hosted at, so let’s examine the connection between code forges and community sizes. My expectations going in were that:

• Small communities would bias heavily toward GitHub, because it’s basically the center of open development today; and
• Larger communities would likely tend to host independently, because they have more complex needs in terms of service heterogeneity and scale.

Dots indicate the integer medians of each 25 data points in order of committer size, semitransparent so darker dots indicate multiple overlapping. points. Otherwise, data source and spline fit as described previously.

Small projects

On an overall level, we can see a strong bias for small projects toward GitHub (~50%), while just under 20% opt for both of SourceForge and Google Code. The remaining ~10% largely choose to self-host, with the last few percent going to Launchpad and Bitbucket.

Two points worthy of note are that GitHub and Launchpad both show global peaks at a committer count higher than 1, indicating a break with the global trend in the first graph that the most common situation is a single-developer project. This could support the importance of a low barrier to entry for collaboration. Getting those first few developers beyond the founder tends to be incredibly difficult, and anything that makes that easier is a huge deal.

Large projects

The downhill trends are clear for SourceForge and Google Code, while Launchpad and Bitbucket appear to remain roughly flat. GitHub seems to have a slight downhill trend. Interestingly, scaling to the needs of larger projects turns out to be a major issue for the older forges (SourceForge and Google Code), but GitHub seems to have largely defeated it.

Conclusions

Once a project reaches 15–20 monthly contributors, it seems to behave much differently, on average, than smaller projects in a number of ways. In larger projects, committers tend to be more active as a whole, licensing tends to be better-determined, and they’re much more likely to be self-hosted. Very small communities make up the vast majority of the open-source world, however, so we need to pay close attention to what’s happening even on solo projects.

Disclosure: Black Duck Software (which runs Ohloh) and Atlassian (which runs Bitbucket) are clients. GitHub and Canonical (which runs Launchpad) have been clients. Dice (which runs SourceForge) and Google are not clients.

My colleague Steve has posted about permissive vs copyleft licensing a number of times, but we’ve never done quantitative research into licensing choice to prove the extent to which any shifts are happening, the time frames involved, and the potential variations within different programming-language communities.

Approach: Classification, history, and languages

Using the Ohloh data for 57,930 active projects as of July 2012, I classified the top 30 open-source licenses into one of three categories: permissive (e.g. BSD, Apache), limited (e.g. LGPL, MPL, EPL), or copyleft (e.g. GPL, AGPL). This three-category classification accounts for 90+% of all projects with specified licenses, which means it should be representative. The total number of classified projects was 17,549, because a vast number of projects either have no license or Ohloh was unable to detect it. Limited licensing is quite rare, hovering around 2%–3% of projects with licenses, so for the purposes of this post, we will focus on permissive and copyleft licensing.

To attempt to identify historical shifts, I separated projects into buckets based on the date of their first commit. Since license changes between permissive and copyleft are quite rare, this should be a reasonable approach to examining trends over time.

Since I hypothesized that programming language might also play a role, I further split each year’s bucket by language. Here, I’m going to focus on the 11 most popular languages according to our rankings, as well as the total across all languages regardless of popularity. Any data points with 5 or fewer projects between permissive and copyleft are not shown, to remove noise.

Results: A clear trend toward permissiveness

I’m showing the data as a ratio between permissive and copyleft licensing to account for changes in absolute numbers of projects over time. Any number above 1 indicates a bias toward permissive licensing, while any number below one indicates a bias toward copyleft.

Remarkably, every single language shows an upward trend, starting either in favor of copyleft or near equilibrium and shifting upward in a more permissive direction. The overall total, shown as a thick black line, further supports and clarifies this trend since the individual languages can be rather noisy.

Two languages of particular note are the two extremes: Ruby on the permissive side and Perl on the copyleft side. While most languages cluster relatively tightly, Ruby rises far above them with a very clear and strengthening shift toward permissive licensing — 2x in favor of permissive in 2010, 6x in 2011, and 11x in 2012. At the other extreme, Perl shows a roughly 2x–3x bias in favor of copyleft, which is distinctly below the nearest neighbor, C++, but not nearly as large of a divergence from the primary cluster as Ruby shows.

Other than that, at the level of individual languages, it’s difficult to draw any strong conclusions based on their relative positions because they are much less distinct. More recent web-development languages (Ruby, JavaScript, Python) may bias toward permissiveness, as do languages that tend to be used on closed platforms (Obj-C, C#). The difference between Java, C, and C++ is likely cultural as well, with C and C++ being common in the copyleft community while Java is less so due to inertia from its OSS-unfriendly past.

Conclusions

The shift toward permissive open-source licensing is dramatic over the past decade. Since 2010, this trend has reached a point where permissive is more likely than copyleft for a new open-source project. Although there are language-specific effects, especially in the case of Ruby, the overall movement is clear. Outside the extremes, new projects in even the most copyleft-biased language (C++) in 2012 were given copyleft licenses less than 60% of the time.

Disclosure: Black Duck Software (which owns Ohloh) is a client.

If you look at a longer-term view of the full history of Google Trends from 2004 to present, other nearby countries show up as well, although lower-ranked: Uganda, Ethiopia, Zimbabwe, and Ghana. It’s likely no surprise to anyone that, outside of East and West Africa, South Africa (the country) also makes a strong showing, and Egypt appears to a lesser extent, visible on some of the maps but not on the top 10 of the lists. Here are the results for the search terms I used, “software engineering,” “programming languages,” “computer programming,” and “software development”:

It’s further supported by Web-traffic data from Alexa showing programming popularity in parts of Africa, with GitHub being a popular site in both South Africa and Nigeria (the same goes for Stack Overflow).

As another proxy for interest in software development and its future directions, we can look at website traffic to RedMonk.com. It shows the top 10 highest-traffic countries in Africa since 2009 as:

1. South Africa
2. Egypt
3. Kenya
4. Nigeria
5. Morocco
6. Tunisia
7. Ghana
8. Algeria
9. Mauritius
10. Uganda

South Africa and Egypt alone account for more than half of the traffic, however, so the rest appear behind in this respect. Comparing 2012 with 2010, we’ve seen a 13% increase in the proportion of our traffic coming from Africa, and none of that increase comes from Northern Africa (i.e. Egypt, Tunisia, Algeria) — it’s spread across the remaining regions. However, African traffic remains a quite small proportion of our overall traffic, hovering around 1% compared with our top 3 continents since 2009 at 50%, 32%, and 13%, so they won’t be taking over anytime soon. It’s roughly equivalent to the traffic from the #10-ranked US state.

If we look at another metric, that of LinkedIn members in any of the above African countries who match the job titles “software developer” or “software engineer,” we see very similar results (showing all countries with ≥50 results):

1. Egypt: 4080
2. South Africa: 2714
3. Kenya: 688
4. Nigeria: 597
5. Tunisia: 370
6. Mauritius: 231
7. Morocco: 167
8. Ghana: 176
9. Uganda: 153
10. Ethiopia: 144
11. Tanzania: 81
12. Zimbabwe: 80
13. Sudan: 70

While those numbers will be smaller than the true developer population, an estimate in IEEE Spectrum suggested roughly 200 full-time programmers in Ghana in 2005 compared to 176 on LinkedIn today, which suggests it’s not a completely unreasonable number. The correlation with hits to RedMonk.com suggests that these numbers, while perhaps not correct on an absolute scale, do reflect relative differences across Africa.

From these two lists, we can see that African software development extends somewhat more broadly than merely eastern and western Africa to include a broader group of generally stable, coastal African countries, be it north, south, east, or west.

What are they writing?

Language-specific searches of all RedMonk’s tier 1 and tier 2 languages on Google Trends with the pattern “$LANGUAGE programming” showed that Java and C/C++ were the primary languages in use. In fact, they were the only ones to show any meaningful population on searches. C/C++ shows up in Kenya and South Africa, while Java shows up strongly in Kenya and Nigeria, more weakly in South Africa, and finally weakest in Egypt.

The country-level popularity above shows an interesting correlation with entries to a World Bank software contest on global development, a domain in which many Africans have a keen interest in because it’s directly relevant to their lives (unlike many apps popular in San Francisco). The top submissions, in order, came from Uganda, Nigeria, Kenya, Ghana, South Africa, Niger, and Rwanda. Most interestingly, Africa had more submissions than any other continent.

I would also expect that as the living standards and cost of living in places like China and India continue to increase, we may see more outsourcing move to Africa.

Conclusion

Minnesota, Colorado, and Virginia are peers to Africa on the basis of RedMonk.com traffic, and most software companies don’t ignore them. If you aren’t thinking about Africa, it’s time to start. It’s already as significant as a top-10 US state, and it’s just going to get bigger from here.

Update (4/1/13): Joel Martin pointed out that this data also shows a reasonable correlation with Internet users in Africa.

Disclosure: World Bank is not a client.

The information? The input from ~2,500 developers over on Hammer Principle on the statement, “This language is expressive.” I mapped the top 10 and bottom 10 languages to my own median-based ranking, showing only the top two popularity tiers for simplicity, and got this:

Interestingly, it’s a very clear correlation — all expressive at one end, all poorly expressive at the other end, and a mix in the middle (indicating a bit of noise).

What do you think?

 What are we really measuring here?

As I mentioned as a major caveat in the initial post, lines of code (LOC) per commit is an imperfect metric as a window into expressiveness. It’s measuring something, but what does it mean? My take on these results is that it’s a useful metric when painting with broad strokes, and the results seem to generally bear that out. It’s more helpful in comparing large-scale trends than arguing over whether Ruby should be #27 or #22, which is likely below the noise level. I think the reason some placements seem so weird is that it’s measuring expressiveness in practice rather than in theory. That brings in factors like:

• The standard library and library ecosystem. Is there a weak standard library? Is there a small or nonexistent community of add-on library developers? In both cases, constructing a commit-worthy chunk of code could require additional lines.
• The development culture and its norms. Is copy-and-pasting common for this language? Are imported libraries often committed to the project repository (JavaScript is a prime candidate here)? Are autogenerated files committed (e.g., minified JavaScript, autotools configure scripts)?
• The developer population using it. Especially for third-tier languages, the number of developers is small enough that these results could reflect those developers more than the properties of the language itself. Some of the least-popular third-tier languages have fewer than 10 developers committing during a given month. I would generally disregard anything but the largest differences between third-tier languages, and treat even those with skepticism. Some languages are also more popular for beginning programmers, which could influence the results if the beginners make up a significant chunk of the language’s total userbase.
• The time frame of its initial popularity.  This can result in time-based influences upon tools and methodologies in use. For example, newer languages popularized in the agile and GitHub eras may tend to bias toward smaller, more frequent commits. Languages that grew up alongside waterfall development and slower, centralized version control may be biased more toward larger, monolithic commits. It even carries as far as things like line length — today, wide-screen monitors are common, and many developers no longer restrict their column width to 80 or less. This could have a language-specific impact, where older languages with a great deal of inertia change more slowly to a new “standard” of development. For example, perhaps fixed-format Fortran wasn’t typically maintained in version control at all, and full files were just committed wholesale? That could explain its similarity to JavaScript.
• Differences in project types by language. If a language is more likely to be used in larger, enterprise projects, this could influence the types of commits it receives. For example, it could get more small bugfixes than new features because it’s a long-lived codebase and requires additional stability. It could also see a different level of refactoring.

So … what should you get out of the results, then?

Frankly, given all the possible variables involved, the biggest surprise here is that the results look as reasonable as they do, at the level of broad, multi-language or cross-tier trends. Here’s what I would tend to believe, and what I would be skeptical about.

• Believe: multi-language trends
• Believe: cross-tier trends
• Believe: large differences between individual languages, but investigate why
• Believe: highly-ranked languages
• Be skeptical: anything involving third-tier languages
• Be skeptical: small differences between individual languages
• Be skeptical: individual languages that don’t fit into a group of similar ones
• Be skeptical: low-ranked languages, until investigated

Why do I suggest believing high ranks but not low ones? It’s the Anna Karenina principle, as Tolstoy wrote:

Happy families are all alike; every unhappy family is unhappy in its own way.

While there are a large number of ways to have a high median or high IQR, it seems to me that low values of both would indicate a number of good development practices in addition to a good language.

To wrap things up, I think this is measuring, with a fair amount of noise, a form of expressiveness in practice rather than in theory — a form that includes all the ways code is incorporated into a repository. That makes it an interesting window into a number of potential problems with how specific languages as well as language classes are typically used.

I recently got a hold of a great set of data from Ohloh, which tracks open-source code repositories, on the use of programming languages over time across all of the codebases they track. After validating the data against Ohloh’s own graphs, one of the first things I did was try out my idea on expressiveness of programming languages. Sure enough, it gave me results that made sense and were surprisingly reasonable.

Some caveats to this approach :

• This assumes that commits are generally used to add a single conceptual piece regardless of which language it’s programmed in.
• It won’t tell you how readable the resulting code is (Hello, lambda functions) or how long it takes to write it (APL anyone?), so it’s not a measure of maintainability or productivity.
• Ohloh relies on opt-in subscription from open-source projects rather than crawling forges itself. That said, it’s a vast data set covering some 7.5 million project-months.

Time to let the results speak for themselves. Enough words, here’s the data (enlarge by clicking):

It’s visualized in the form of box-and-whisker plots, which are effective for showing a distribution of numbers relatively simply. What numbers are we showing? It’s a distribution of lines of code per commit every month for around 20 years, weighted by the number of commits in any given month. The black line in the middle of each box is the median (the 50th percentile) for that language, and languages are ranked by median. The bottom and top of the box are the 25th and 75th percentiles, while the “whiskers” extend to the 10th and 90th percentiles. The “Total” box indicates the median of each value across all languages (median of all 25th percentiles, median of all 75th percentiles, etc.) to show a “typical” language.

I’ve also colored them according to our most recent RedMonk programming language rankings (red is the most popular cluster, and blue is the second-tier cluster, while black is everything else), and restricted languages here to the ones popular enough to be included in that set of rankings.

What conclusions can we draw from this?

Global effects

The trends generally make sense. If we focus purely on the tier-one languages shown in red, high-level languages (Python [#27], Ruby [#34]) lean toward better expressiveness while lower-level languages (C [#50], C++ [#45], Java [#44]) tend toward wordiness. Similarly in tier two, Fortran [#39/#52] and assembly [#49] are wordy, and “middle-aged” functional languages are intermediate while newer functional languages are best.

Expressiveness ranges broadly across languages. The medians go from lows of 48 for Augeas (#1) and 52 for Puppet (#2) to a high of 1629 for fixed-format Fortran (#52), which is a surprisingly large 31x variation.

Less expressive languages tend to show a much wider variability. There’s a clear, but not strong, correlation between the medians (black lines) and the IQRs (box heights). Languages with the largest IQRs also tend to have greater medians, and consistently expressive languages tend to also be more expressive.

First-tier languages are a mix of poor and moderate expressiveness. Of the 11 tier-one languages, 5 are moderately expressive and the remaining 6 are poor. The tier-one languages range from LOC/Commit ratios of 309–1485, which equates to 6x–30x lower expressiveness than the top languages. Perl (#26), the best tier-one language, is 5x more expressive than the worst, JavaScript (#51), and 3.5x more expressive than the classic C. That’s certainly respectable but falls well short of the 20x or greater improvement one could gain with one of the top languages.

Second-tier languages are well-distributed and reach into highly expressive languages. With 52 total languages on this list, the top ~17 constitute the highly expressive languages. Although none of those are first-tier languages, 9 of those 17 are second-tier — mostly functional with the exceptions of Groovy (#16), Prolog (#13), Puppet (#2), and CoffeeScript (#6).

Third-tier languages are heavily biased toward high expressiveness. Of the 15 third-tier languages on this list, 8 are in the top 1/3 of languages, leaving only 7 are in the remaining 2/3. Although these data do not directly show any correlation between age and expressiveness, it seems reasonable that newer, more expressive languages would begin less popular and may grow later.

Effects of language class/type

Functional languages tend to be highly expressive. On this list are Haskell (#10), Erlang (#22), F# (#21), Lisp variants (including Clojure [#7], Emacs Lisp [#14], Dylan [#12], Common Lisp [#23], Scheme [#31], and Racket [#11]), OCaml (#20), R (#17), and Scala (#18). Of those, only two fall below #30 out of the 52 languages included here.

Domain-specific languages are biased toward high expressiveness. Augeas (#1), Puppet (#2), R (#17), and Scilab (#19) are good examples of this, while VHDL (#38) serves as an outlier on the low end.

Compilation does not imply lower expressiveness. I was halfway expecting highly expressive languages to exclude all compiled languages but was proven wrong. Compiled languages in the top 17 include CoffeeScript (#6), Vala (#9), Haskell (#10), and Dylan (#12).

Interactive modes correlate with intermediate expressiveness. Languages with an interactive shell tend to be mid-range in expressiveness, with a few outliers on either side. For example: Lisp (#23), Erlang (#22), F# (#21), OCaml (#20), Perl (#26), Python (#27), R (#17), Ruby (#34), Scala (#18), Scheme (#31).

Specific language effects

CoffeeScript (#6) appears dramatically more expressive than JavaScript (#51), in fact among the best of all languages. Although the general trend is not particularly surprising because that’s the whole point of CoffeeScript, the magnitude of the difference seems unusual. I suspect JavaScript’s low placement could be at least partially due to wholesale copying of template JavaScript files rather than reflecting development in JavaScript itself.

Clojure (#7) is the most expressive of Lisp variants. There are a large number of Lisp variants that generally ranked quite well, described in more detail above in the functional-language section. In this context, it’s worth noting that the top one was the fairly popular Clojure, with a median LOC/commit value of 101, followed by Racket (#11) at 136 and Dylan (#12) at 143.

Among data-analysis languages, R (#17) and Scilab (#19) are most expressive.  With a median of 193 LOC/commit for R, it’s a clear top performer. R is followed by Scilab and Matlab (#35) with medians of 225 and 445, respectively.

Although Go (#24) is getting increasingly hot, it’s not outstandingly expressive. We keep hearing about new use of Go across a variety of startups, but it’s little better than Perl (#26) or Python (#27) by this measure. Despite that, it does trump all the tier-one languages, so someone who only had experience with them could certainly see an improvement when trying Go.

What if we sort by consistency of expressiveness, instead of the median?

Ideally a language should be:

• Easy enough to learn that the vast majority of developers using it can be highly productive; and
• Equally expressive across nearly its entire domain of usefulness.

To measure that, let’s take a look at the interquartile range (IQR; the distance between the 25th and 75th percentiles) as a proxy for these two criteria, and rank languages by that instead (enlarge by clicking):

What you’re looking for here is the height of the boxes. It starts small on the left side, with CoffeeScript doing best at 23 lines and increases to the right side, ending with fixed-format Fortran at 1854 lines.

A few new insights specific to this plot before we move on to considering them both together:

• As alluded to earlier but illustrated differently here, inconsistency and wordiness are correlated, as are consistency and expressiveness.
• Tier-one languages put in a much stronger showing here, with four in the top 1/3 of languages (Python at #11, Objective-C at #13, Perl at #15, and C# at #17). Shell nearly makes the cut at #19. Those IQRs vary from 90–167 LOC/commit, a fairly large difference even among the best performers.
• Consequently, tier-three languages make a poorer showing here, although they performed unusually well at levels of expressiveness. They are nearly proportionate with their population with 5 of 15 showing up in the top third, and the remainder are evenly distributed across the moderate and low consistency groups as well.
• Java turns in the strongest performance of “enterprisey” languages (C, C++, Java) when considering both metrics. Java comes in with nearly identical expressiveness as C++ (both at 823 LOC/commit) but a vastly greater consistency (IQR of 277 vs 476).
• CoffeeScript is #1 for consistency, with an IQR spread of only 23 LOC/commit compared to even #4 Clojure at 51 LOC/commit. By the time we’ve gotten to #8 Groovy, we’ve dropped to an IQR of 68 LOC/commit. In other words, CoffeeScript is incredibly consistent across domains and developers in its expressiveness.
• The outliers are particularly interesting — the ones with unusually high or low medians compared to nearby languages. If the median is higher than neighbors, than it’s an unusually consistent yet less expressive language. Conversely if the median is lower than neighbors, then the language is unusually inconsistent (a.k.a. shifted to the right on this graph from the rough correlation between consistency and median expressiveness).
  • Tier-one languages tend to be remarkably consistent, regardless of their expressiveness. In nearly all cases, their medians are higher than their neighbors, showing a general shift to the left from the expected placement. This suggests that a primary characteristic of a tier-one language is its predictability, even more so than its productivity.
  • Conversely, in most cases where languages appear shifted to the right, they’re third-tier languages. The lack of predictability has often held them back from even reaching the second tier.

So, what are the best languages by these metrics?

If you pick the top 10 based on ranking by median and by IQR, then take the intersection of them, here’s what’s left. The median and IQR are listed immediately after the names:

• Augeas (48, 28): A domain-specific languages for configuration files
• Puppet (52, 65): Another DSL for configuration
• REBOL (57, 47): A language designed for distributed computing
• eC (75, 75): Ecere C, a C derivative with object orientation
• CoffeeScript (100, 23): A higher-level language that transcompiles to JavaScript
• Clojure (101, 51): A Lisp dialect for functional, concurrent programming
• Vala (123, 61): An object-oriented language used by GNOME
• Haskell (127, 71): A purely functional, compiled language with strong static typing

Looking at the box plots again, I would tend to rule out eC based on the poor performance of the upward-reaching whiskers at the 90th percentiles, indicating a real lack of consistency as often as a quarter of the time (since the 75th percentile is quite good). I would also rule out Puppet and Augeas because they are DSLs.

Combining those with our RedMonk programming language rankings on popularity, the only highly expressive, general-purpose languages within the top two popularity tiers are:

• Clojure
• CoffeeScript
• Haskell

If you’re considering learning a new language, it would make a lot of sense to put Clojure, CoffeeScript, and Haskell on your list, based on expressiveness and current use in communities we’ve found to be predictive.

No tier-one languages fall in the top 25 on both metrics, although 5 make the cut on consistency alone.  Of the tier-one languages, lower-level ones tend to be both inconsistent and overly wordy, while higher-level ones have intermediate wordiness and very strong consistency. The most consistent languages are Python, Objective-C, Perl, C#, and shell, with the presence of Perl and shell supporting the initial assertion that expressiveness has little to do with readability or maintainability. Ruby is an interesting language, in that it violates the “rules” of expressiveness and consistency seen in the other higher-level languages. This could be an instance of a framework (Rails) truly popularizing a language that otherwise would’ve never taken off.

For projects that require an expressive language where it’s relatively easy to hire developers, Python is worth serious consideration. Of tier-one languages, Python, Perl, Shell, and Objective-C are the best overall performers, and I consider Python the strongest of those for general-purpose applications. In my opinion, it makes a lot of sense to take a polyglot approach to projects, writing as high-level as performance requirements allow. Fortunately many high-level languages like Python allow for modules based in more performant languages such as C. That means it’s easily possible to write the vast majority of a project in a more productive, more expressive language while falling back to high-performance languages where needed.

Update (3/26/12): I somehow missed Haskell on the final recommendations for second-tier languages, although it was on the initial list. Thanks to Chad Scherrer for pointing it out in the comments.

Update (3/26/12): I just wrote a post on the last day’s discussion and commentary about what this kind of metric means and what you can get out of it.

Update (3/26/12): I wrote a new post showing correlation of my data with external survey data on what languages developers think are expressive.

Disclosure: Black Duck Software (which runs Ohloh) is a client.