X⊕R | Chris Lonnen

My Community Boxen Setup

2013-11-12T00:00:00+00:00

Back in April I got a shiny new MBP and decided I wouldn’t transfer anything over by hand. I had spent too long bootstrapping development environments, for myself and others. “Restoring” the fresh machine from an old backup was an option, but it lacked nuance. My team had used Puppet to bring the setup time of Mozilla’s crash reporter down from one week to an hour, and I was looking for similar gains for dev machine setup. After a survey of the field I chose GitHub’s Boxen.

Boxen is a framework for creating automated, repeatable laptop setups. I created a Boxen for Mozilla WebDev, and have been slowly building our install base since April. After installing xcode and enabling full disk encryption, a single command will install Boxen and set up:

git + hub
homebrew
iterm2
nginx
node
python
ruby

All of these and a few more useful utilities are installed into /opt/boxen and set up for the machine – no more fussing with the difference between Apple supplied libs and the ones you Homebrew’d. Because I defined my own manifest, which was entirely optional, Boxen will install (just for me) my preferred chat clients, web browsers, editor, music player, and some other utilities. I’ve even set it up to install and update my public dotfiles. Once that is finished, BitTorrent Sync copies over my private files (unique to me) and in a few minutes my machine is ready to go.

I’ve been slowly evangelizing this way and growing the install base inside Mozilla. Some have used it as a foundation, customizing the rest of their machine manually. A few have contributed back their own manifests and modules. Matt Basta went so far as to set up project modules to make it easy to hack on Firefox Marketplace and Addons related projects.

Anyone meeting Boxen’s minimum requirements can run our flavor of Boxen, and I invite any curious parties to try. At the time of publishing, this means running OSX 10.8 or later – though support for other operating systems is slowly spreading through Boxen’s core modules. It works great for bootstrapping a fresh machine, and we’ve had no problems reported from installing it on already configured hardware either. You can read more about customizing it in the README, or jump right in via our new install page.

Boxen has already saved us a bunch of time bootstrapping new hires and recovering from hardware upgrades, component failures, and laptop theft. Many thanks are due to the community, at GitHub and elsewhere, that maintain the various modules that make our custom Boxen easy to maintain. We just pushed GitHub two-factor auth and improved OSX Mavericks support. For a sneak peak of what will be included in the future, check out Will Farrington’s PuppetConf 2013 talk about where the framework is heading or come hack on either Boxen modules or our boxen setup and build the features you want to have.

2013 Web Dev Intern Recap

2013-09-17T00:00:00+00:00

With the 2013 intern season winding down, we’d like to highlight some of the intern contributions to Mozilla Web Dev projects. This year five interns from four countries spent a few months at our Mountain View office building amazing things:

Ashish Dubey created a communication dashboard in the Firefox Markeplace for archiving communications between app developers and reviewers. The dashboard uses email notifications and syncs email responses, providing some flexibility for users. Ashish also created MarketPlay, an interactive sandbox for exploring the Firefox Marketplace API. * “Better Communication Flow in Firefox Marketplace” * “Communication Dashboard on Github” * MarketPlay Staging Server

Derek Ries created Kanbanzilla, a stand-alone app that uses the Bugzilla API to provide a Kanban interface to Mozilla’s bug tracker. This allows Mozilla teams to use Kanban without having to keep an external system in sync with Bugzilla. * “Cards and Columns, a Kanban process for Bugzilla” * Kanbanzilla Demo Site

Shu Hao Wu created a webapp for Mozilla’s support website (SUMO). The app provides offline access to the SUMO troubleshooting wiki for FFOS with fast, full text search that works well across SUMO’s localizations. Shu Hao also contributed to Mozilla’s crash reporter, where he added automated identification of rapidly growing (“explosive”) crash signatures. * “Building apps and doing math” * Offline Sumo Staging Server * Explosive Crash Report

Leon Zhang built the first iteration of the Brick web components library. Brick is a javascript library that provides common UI components for web development. Brick builds on the x-tags polyfill to offer the benefits of the emerging Web Components standard today across browser platforms. * “Building Bricks: An Introduction to the Brick Web Components Library” * Brick Website * Hacks Blog Post

Tony Young was a spring intern who refactored several modules of the distributed system powering Mozilla’s crash reporter. He wrote a faster filesystem module, brought consistency to the hbase module, and made storage modules composable to make Socorro easier to install and enable quicker development of new features and components in the future. * “Refactoring Socorro”

Many thanks and well wishes to our interns as they return to school! Also, a great big thank you to Mozilla University Recruiting, for furnishing us with such an excellent group this year!

All of this year’s intern presentations are available on Air Mozilla. If you’re interested in a Mozilla internship, you can get more information from university recruiting, see open positions on our careers page or meet with some Mozillians at a campus near you.

This post previously appeared on the Mozilla Webdev Blog.

Applied Matching Algorithms

2013-05-04T00:00:00+00:00

Matching problems concern themselves with pairing things up. There are many variations across graph theory, economics, and operations research where the goal is to seek some set of matches with a particular set of properties.

One of the best known variants is the stable marriage problem, explored in 1962 by D. Gale and L. S. Shapley in College Admissions and the Stability of Marriage. They describe an algorithm (now named after them) to solve the stability of marriage problem and the related college admissions problem.

Of Stable Marriages and College Admissions

In the stable marriage problem there are two groups: suitors and suitees. Each person ranks every member of the other group according to individual preference. Gale and Shapley demonstrate that for any set of preferences there is a stable arrangement, and as long as no ties are allowed in personal rankings the Gale-Shapley algorithm yields the optimal matching for the suitors.

By stable, we mean no pair would be willing to abandon their current partners for each other. Any individual may prefer someone else to their current betrothed, but the object of their affections will not reciprocate by virtue of already having a partner preferable to the aspiring adulterer. By optimal, we mean that every individual is at least as well off under the current assignment as they would be under any other stable assignment. In this algorithm, whoever is proposing has the upper hand and will receive a stable assignment. If only a single stable matching is possible, the solution is globally stable.

It works like this – unmatched suitors form a queue. One at a time they propose to the most desirable person who has yet to reject them. If the suitor has been rejected by everyone interesting the suitor simply leaves, unmatched. When a suitee is asked, the suitor is compared to any current beau. If the suitor is preferable a new engagement is made. Any scorned former partner is now unmatched and heads to the back of the queue. When one pool or the other is empty no more matches can be made, and all engagements are finalized into marriages.

The algorithm is robust to unequal populations. It is even robust to individuals having limited preferences that don’t rank all potential partners. Of course, this means that some people may end up unmatched even in an optimal matching.

The resulting arrangement must be stable because every suitor asks in order of their personal preference. Rejections only occur when the suitee already has a preferable partner, and engagements only break so that the suitee can trade up to someone strictly preferred to all previous partners. This way, no prior objections could possibly overturn under a new engagement.

The proof of optimality is left as an exercise to the reader [1].

Open Enrollment, Stability, and Suboptimal Results

Variants of this problem have practical applications. Combining the suitors and suitees into a single group yields the stable roommates problem. Allowing one group to accept multiple partners becomes analogous to the problem of college admissions, where each school has a fixed number of slots and a preference over the applicants. As with the stable marriage problem, there is an asymmetric advantage to the initiator. This particular problem has provided practical insight for the National Resident Matching Program, which seeks to optimally match medical students with residency programs. The NRMP has a slightly harder task, as since 1998 it has been accommodating joint submissions for couples seeking to be placed in the same geographic region. This particular variant has been shown to be NP-Complete.

So let’s examine a novel variant for a moment, which is set up like the college admissions problem except that the college has equal (no) preferences over the applicants. This resembles community college admissions with open enrollment, and it is isomorphic to dividing up a mixed bag of candy or a list of chores. The variations on the problem necessitate small changes to the algorithm and result in a change in the properties of the outcome. Here I present the Greendale algorithm for finding a stable matching in the community college problem:

People act as the suitors, as in Gale-Shapley, asking for their top choice and receiving it or moving to the end of the line. People who have exhausted their list will exit the process as unmatchable. Since the resources being matched have no preference there will be no trading up – the first person to ask for the resource will receive it and keep it. The process repeats until the line of people or the supply is empty.

The resulting matching is stable. Every item a person prefers to what he or she received must have been asked for first, and everyone asks in waves. This means anyone who gets an item must rank it at least as highly as anyone who asks later and finds it out-of-stock. Whatever those latecomers end up with will be considered strictly inferior by everyone who managed to get the out of stock item. If it was not, they would have asked for the inferior item earlier and received that instead. Thus, the result is stable and no one will be willing to trade.

Optimality usually cannot be achieved in this variation of the problem because the pool of resources effectively has an equal preference for any matching. Equal preferences are disallowed in Gale-Shapley’s construction of the problem, which is crucial to establishing optimality. In this construction, whenever a resource falls short of demand some people will end up with an inferior choice. Who ends up with what is determined by the initial order in which people interrogate the supply, since items have no way of re-entering the supply. Different orderings will all yield stable results, but individuals coveting the under-supplied resource will end up faring differently in different orderings. Thus optimality can not be achieved in any but the trivial scenario where resources outstrip individual preference. In this case the optimal matching is a global optimum because there is only one stable matching.

Inspiration and Application

I became interested in the community college problem because this year Mozilla Summit is being held simultaneously in three locations. Mozilla WebDev has a fixed number of spots at each location, and we need a means of splitting up the group. Studying these matching problems has lent some insight into how to achieve an equitable matching of Pro Mozillians [2] to locations.

I now know that having to keep small groups together greatly increases the difficulty of the problem. This isn’t a major problem, because we interpret one of the goals of Summit to be mixing up our usual groups and meeting Mozillians from other parts of the organization.

I also know that we cannot create a matching where everyone gets their top choice, but we can at least match people in a stable way. Of course, stability assumes that preferences are immutable, and many preferences may change when people find out where their friends will be going. This can be partly mitigated by putting everyone’s preferences in the open and being transparent about assignment. This can also be mitigated by reinforcing Summit as an opportunity to mix up our social structures.

The initial order of the queue is incredibly important, acting as an implicit ordering for the resource pool. Often it can be determined by the context of the problem – some problems may naturally order according to when preferences are submitted. Other situations may offer no useful clues, in which case you could certainly do worse than randomly ordering people in the queue.

Lastly, the algorithm may not acceptably accommodate extreme positions for real world applications. Extreme positions, in this case, means either not having any preferences, or abbreviating a preference list to reduce the possible matchings. The former is remedied by leaving them out of the matching and arbitrarily assigning them to whatever resources remain at the end. In case of the latter the individual risks not get matched at all. This risk is proportional to the number of people taking this extreme position. While stable, the global matching is presumably unacceptable if some people are unmatched. Possible mitigations are manual matchings, which will be easier before the algorithm is run, or preferential placement in the queue. It’s notable that all possible mitigations afford the extremophile advantages, and extreme positions should be examined or disallowed to prevent gaming the system.

Despite its imperfections, I believe the Greendale algorithm, above, could be useful for solving real matching problems. It is simple and robust, runs in polynomial time [3], and it has well understood weaknesses that can often be limited or mitigated. It is similar to algorithms that have solved other real matching problems for 45 years. For matching large groups it certainly provides an attractive alternative to hand calculations that will unavoidably be biased, time consuming, and may not reflect the actual market of preferences. The study of these algorithms also helps inform the trade offs made when considering different features, such as group matchings or allowing extreme positions.

I hope this work can be of use finding an agreeable MozSummit matching for my team.

Webapp Error Reporting

2013-04-11T00:00:00+00:00

I’ve been playing around with JavaScript error reporting, lately. It started last September, when my team started thinking about ways to offer Socorro as a Service. A month later I found myself at the Mozilla Festival, where Popcorn Maker 1.0 had just launched, complete with a client-side error reporter. I got to spend some time chatting with Dave Humphrey and David Bruant about the challenges and rewards of web app error reporting, and brought those discussions back to the WebTools team.

In February, WebTools got together in Mountain View to build some prototypes. I had spent some time investigating the modern ecosystem, and found heavy active development error reporters and aggregation software – including the open source Sentry + Raven.js combo – across platforms. I also noted some undeveloped areas for FFOS, and set about investigating. I created a simple client library, Ripcord.js, that provided an interface like Raven.js, but with drastically simplified internals. There was also Cypress, a simple web app that created JS errors in a variety of circumstances. On a Firefox OS phone I eventually supplemented my client library with a local web app for queuing and submitting errors. A custom web activity sent the error data to the app, which could then act on the errors intelligently.

Through these experiments we’ve learned a lot.

With FFOS it can be useful to have additional information about the state of the device. As you pile on information about the hardware, in addition to any information about the crash itself, reports can get large and quickly eat up a capped data plan. It can also impact network performance and extend the time a user spends dealing with each error.

Ripcord.js can use information about the device APIs with a local queue to bulk submit errors, conditionally submit based on Internet connectivity, present a unified user experience for handling errors, and locally aggregate common errors between submissions. However, any app wanting to use this library would have to go through extra certification steps and a potential reorganization to accommodate. By centralizing error submission in a separate app we can extend these features to the simplest web apps and provide a central location for historic information about errors on your device. Only the local submitter would have to go through extra certification steps. In order to ensure its ubiquity, it would have to come packaged with the OS, meaning it will have to undergo the extra certification anyway. Ripcord.js can then detect the presence of a submitter, and in its absence fall back to a simpler behavior based on the developers presence. This has the added benefit of reducing the size of the client library.

The experience of handling an error can be significantly improved by an error reporter. Uncaught errors can cause the UI to stop responding to the user with no warning or notification, which is frustrating. By capturing the error and triggering a MozActivity, we gain some control over the user experience when an app enters a bad state. I did not get to investigate this very thoroughly, but there’s opportunity here.

There’s lot’s of potential for improving error reporting in FFOS, but there are more fundamental challenges to error reporting from web apps in general. The top level window.onerror is supposed to catch unhandled errors, but it doesn’t fire when errors originate in certain event listeners, or in web workers. When it does catch the error, it provides only line number, file of origin, and the error message. There’s no access to the error object itself – for that you have to instrument try catch blocks throughout your code, and possibly add event listeners. Even when you manage to catch an error, you don’t get much more information. In the Firefox family of browsers the stack trace remains a mystery, though we’ve been able to revive a longstanding bug to improve this behavior that is now seeing a flurry of activity.

The few values you do get are nearly useless, because production JavaScript is compacted to minimize the number of lines and files. In practice, this means that most errors occur on the same line and file, making it difficult or impossible to differentiate between multiple errors of the same type, or isolate where a single error came from. In Webkit based browsers you can access a trace. In nightly Webkit and Chrome Canary you can even use source maps, which can map compressed JavaScript back to it’s source. The Firefox implementation will likely yield column number before a full browser source maps implementation, but column is enough for an external program to use, given the map and original files. When that is complete, error reporters will be able to usefully trace an error’s origin to the source – even if the source is a transpiled language like CoffeeScript or Fey. With full source maps support in the browser, error reporters will be able to skip the intermediate step and get the error against the original source.

A lot of the information that an error reporter could want is already fully or partially implemented in browser developer tools, which is encouraging. Not only does it make providing similar info in the browser window easier, but it hints at a possible future where a client-side error reporter could hook into the browsers developer tools directly.

The group within Webtools exploring this issue will be formalizing our efforts a bit more in the next few months, in a new project we’re calling Bixie. We will work towards a v1 with a minimal feature set. It will accept JavaScript errors, store them, and present them to users through a web app. To achieve this quickly we will build on other open source projects – leveraging what we can from the modular infrastructure of the Socorro crash reporter and relying on Raven.js as our client library. Subsequent iterations may introduce additional features like custom reports derived from raw errors, the client library improvements discussed above, and improved integration with other parts of the developer’s ecosystem.

Moving forward we will also work closely with other teams inside Mozilla to tackle the aforementioned bugs and more, in order to bring improvements to the web platform. The tools-bixie mailing list is pending, but until then we’ll be coordinating in irc://irc.mozilla.org#bixie. More details about the Bixie prototype and how you can get involved will follow as we ramp up.

UPDATE: the mailing list is available at https://mail.mozilla.org/listinfo/tools-bixie

Custom Error Responses in Flask: Revisited

2013-04-08T00:00:00+00:00

In an earlier post I wrote up a method of returning unimplemented status codes in Flask based on subclassing werkzeug.exceptions.HTTPException.

As it turns out, the output of Flask view functions are run through a liberal factory function that converts the return value into a response object. This makes it easy to output an otherwise unimplemented status code by returning a simple tuple from your view function.

This way there’s no need to subclass anything from Werkzeug.

Setting up OSX and iTerm2 for Emacs

2013-01-04T00:00:00+00:00

I’ve decided to give Emacs a spin as my primary editor in January. Learning a new editor has been refreshing, but OSX got in my way a few times when I was setting things up. Many thanks are owed to James Long for helping me with that. I wanted to share some of my experience for other curious parties encountering similar problems.

First: a note on my environment. I use iTerm2 instead of the native Terminal.app and I highly recommend you install it right now if you haven’t already. There is an older Emacs preinstalled on OSX, but since I’ve universally had trouble with other preinstalled dev tools I installed Emacs 24 via Homebrew. I used the optional --cocoa flag, though I run it exclusively in the terminal.

Emacs has two modifier keys used for issuing commands to the editor, referred to in the documentation as C- and M-, short for Command and Meta. By default these are control and option on an Apple keyboard, respectively. This presents two problems: first, control is awkwardly placed and uncomfortable to use frequently; second, the option key already has OSX-level functionality.

To help save my pinky, I opened up System Preferences -> Keyboard and hit the Modifier Keys button. From this preference pane I remapped caps lock to ^ Control, which changes the functionality throughout the OS. Now C- is comfortably within reach of my hands from the home row.

The M- char still presented a problem though, as option + [other key] is interpreted into special characters. To fix this, I used an iTerm2 feature. In preferences, under profiles and in the keys tab you can specify either the right or left option keys to send +Esc instead, which Emacs will treat as M-. I chose to alter only my right control so that I could still use the left if I needed default behavior.

At that point I had both C- and M- working, but I found option uncomfortable to reach for all the time. Another iTerm2 preference, this one under keys, allowed me to swap right command and right option keys. This is a little more comfortable to use frequently.

Custom Error Responses in Flask

2012-12-30T00:00:00+00:00

Update 2012-04-08: there’s a simpler way

Python’s Flask framework leverages Werkzeug’s HTTP exceptions module to make it easy to report error states in requests. The abort method works with most common HTTP status codes, but sometimes you may want to return a code that isn’t implemented by Werkzeug.

When I’m building services on services I like to use 424 - Failed Dependency to indicate that an unhandleable error occured in a dependent service. By subclassing Werkzeug’s HTTPException I can respond with that exception even though Werkzeug itself doesn’t implement it.

In this example it doesn’t matter if Github is down or if the application is running up against the rate limit – the app will indicate that through no fault of it’s own it cannot fulfill the request.

Through a similar subclass you’ll be able to implement custom, newly proposed, and otherwise unimplemented status codes.

Building a Shanty Town

2012-12-25T00:00:00+00:00

November and December have been a period of heavy investment in creating or enhancing the Vagrant infrastructure on Mozilla’s Socorro and DXR projects. Vagrant is a lightweight VM management tool for Virtual Box VMs. Coupled with Puppet or Chef it provides a reproducible sandbox for locally deploying projects in a production-like environment. You can store the scripts in your project’s repository and share the folder from your host machine into the VM. Develop your files locally, and changes are automatically mirrored into the VM without needing scp or external synchronization.

The Mozilla Ops group uses Puppet to automate large portions of our architecture, so it seemed the natural choice for our Vagrant VMs. Puppet is a declarative, object-oriented DSL with many built-in resource types. There are primitives for installing packages, running services, and copying whole files onto the VM. Declare the resources you need along with their dependencies and Puppet figures out the application order. Take a look at a basic manifest taken from the DXR project, containing a single class:

This manifest will install the apache2-dev package, then copy an apache conf file from the DXR repository and set some permissions on it. Then it will start apache running as a service. Iff mod_rewrite and mod_proxy are present they will be enabled, though when those commands execute is not known. It is, however, consistent – on multiple executions of Puppet the commands will execute in the same order.

Inheritance works classically, which can be used to great effect if you’re working with a distributed system. For example: the Socorro project has a number of small web components that talk to each other over HTTP. Socorro’s Puppet manifests declare a base class that installs common packages and starts process monitoring, logging. The base class is extended into a web service class, which starts an Apache service with associated configuration and mods. Individual components then extend the generic webservice and override just the resources that differ between them. The resulting code contains minimal boilerplate.

Puppet ensures that necessary environment changes are not passed around in IRC or just on one dev’s machine, but written down in a testable, executable fashion and shared throughout the team. It also minimizes the onboarding cost of new community and new fulltime developers.

Switching to a Vagrant+Puppet set up for local development brought the cost of setting up Socorro down from a day to an hour. DXR saw a proportional time savings. A scriptable set up lowers the risks and costs of major changes to the software stack, and bounds the worst case at destroying and recreating the VM. There’s plenty of room for improvement, though, especially when bootstrapping the initial Vagrant+Puppet set up. If you’re considering a similar set up for your own project, here are some of the pain points:

Changing the Puppet manifests usually involves destroying and recreating your VM to ensure some old state isn’t covering your mistakes. Even if you’ve cached the base VM locally this can mean between 10 and 45 minutes to test any change to Puppet code, which is painful to a web developer used to speedy interpreters and REPLs.

Furthermore, some of the design decisions in Puppet favor its primary use case – large, daemonizaed deployments on production hardware – to the detriment of single boxes. Errors tend to pass quietly. Puppet won’t abort or even log information beyond that something failed without an explicit directive. To help with this, we include the following in our initialization manifest:

The capital Exec will set the option for the exec statements that follow. You’ll still have to carefully read Vagrant’s debug output, though.

Lastly, but maybe most noticably, it can be difficult to translate shell scripts into Puppet code. A direct translation would have a series of exec statements requiring each other in a linear chain of spaghetti code. Puppet can be more expressive, but at times there isn’t a clear way to translate a bash script. When faced with this in a Vagrant+Puppet setup, it’s ok to just use the bash (or fab, Make, etc) script if that’s really the best way to describe what you’re after. A clean, consistent, reproducible environment is the goal, and fiddling with little changes can be extremely expensive on your time. You’ve got code to ship.

Despite these issues, Vagrant+Puppet have vastly improved our development processes wherever we’ve introduced them. If you’re thinking about tackling this for your own projects, there are some great resources to help. Richard Crowley’s excellent PyCon 2011 talk is a more thorough overview of why and how to use Puppet, targeted at beginners. Several Mozilla projects have Vagrant files and puppet folders to check out for real examples of how we use it for local development. Lastly, Puppet’s documentation is sometimes overwhelming, but cannonical.

Feedback and Interaction

2012-10-22T00:00:00+00:00

As an additional channel of communication I’ve cribbed Holman and opened up a feedback repo on GitHub. Questions, comments, requests, etc. can be filed there as issues; the README has details.

If it’s unused or unhelpful, I may close it in a few months. Other communication channels still work, but I check GitHub more often than email, or any other social network. Plus it’s public.

Pre-Integration Testing

2012-10-18T00:00:00+00:00

Reviewing a pull request to Socorro involves a series of filtering steps up front before the reviewer even considers the substance of the proposed change – does it violate the style guide? Does it pass our test suite? Most of these are checked by our CI server runs after an SCM change. It’s always embarrassing when you’ve thoroughly reviewed and merged a commit to find that it broke the build; and it effectively blocks the rest of the team from working while you sort out the failure.

But the preliminary steps get tedious, and that green merge button is so easy to click, and even when the reviewers do everything by the book differences in our developer environments and the servers we build and deploy from can lead to unexpected breakage.

If we run tests after code is integrated to master, why can’t we run them before?

Travis-CI does this, and it’s awesome. I’ve set up libraries to run there and pre-integration testing has been immediately useful. Especially when new contributors drop by with patches. Unfortunately, Socorro’s build process is a little more complicated and Jenkins is our CI of choice. Rather than maintain two separate build systems, I went looking for ways to get Travis-CI style testing from Jenkins. It needed to post the status of the build job back to the pull request, run automatically, and appropriately handle pull requests from other forks of the repo.

Introducing Leeroy

I found Leeroy, a small Flask app that listens to GitHub pull request events, parses them, and triggers a parameterized Jenkins build to test proposed changes before your merge them. As the job runs, Leeroy keeps the GitHub status API updated so the curious can follow along.

The project readme thoroughly covers installation. Leeroy requires accounts with access to your Jenkins server and your GitHub repo. You have to install a few plug-ins from the Jenkins UI, and then create a parameterized Jenkins job. I was able to clone our existing job and make a few tweaks to save space. Leeroy happily runs on a single Heroku dyno if you specify a port in your procfile (leeroy -p $PORT) and specify your own settings file with a heroku config variable. Once it’s running, newly opened or updated pull requests will have some additional UI to show their status.

No developer is prevented from merging, but the UI does encourage smart behaviors by default. While a job is running or after it fails, the merge button is still present but no longer an inviting green. Instead it comes with a cautionary message.

A status dot appears next to each commit that was run as a build. I’ve configured ours to build only the tip of the branch when multiple commits are pushed. The dot is a permalink to the Jenkins job testing that commit, and on hover it displays the job number and status. As a feature develops in Socorro it will have multiple builds in the history to look back on, which can be helpful for the reviewer.

On build success, there’s extra green on the screen confirming this change passed the CI tests. At this point, it’s ready for a person to come along and examine the substance of the change, knowing that Leeroy/Jenkins has caught any style guide violations and obviously broken tests.

How’s that new robot working out?

Our LeeroyBot is only set up for Socorro components, but it is nearing 150 build jobs for its first week. About one-third of those were failures. At 25 minutes of time to discover each failure manually, the back-of-the-envelope has LeeroyBot saving our team about 20 hours, or 2.5 work days this week.

Furthermore, it’s actually changed the way we review and develop code. Nobody is trying to be a wetware pep8, or checking every line for trailing spaces; the focus is shifted away from nits and onto the implementation. Quick fixes are much quicker, and feature branches have become more discussion like.

It’s not a perfect system. Builds break on external dependencies sometimes. Network traffic occasionally times out. Updates to master don’t cause existing pull requests to be re-run, so two commits that green light independently may have a poor interaction. All of these could be helped by manually triggering a build, but for the time being you need to modify the pull request commits to trigger a new build. I’ve filed the issue 9 to figure out a means of resolving this.

On balance, LeeroyBot has had an overwhelmingly positive response from the other devs, and, having survived his first week, will see only expanded use in the future. Many thanks are due to Joe Shaw and Charles-Axel Dein for their excellent work on Leeroy. I recommend it to anyone using Jenkins and GitHub in their day-to-day development.

Socorro Project History

2012-10-16T00:00:00+00:00

At Mozilla, I spend most of my time contributing to the Socorro Project. For the curious, I ran the repository history through Andrew Caudwell’s Gource. The result is a nearly two hour silent visualization of the project’s five year history.

My favorite moment is at ~1:18:00, where we landed a 74,000 line commit.

DNS Record Bankruptcy

2012-10-15T00:00:00+00:00

XOR has finished its move to a new domain: xor.lonnen.com

I’ve established a permanent redirect from its old home at xor.chrislonnen.com/(\S*) -> xor.lonnen.com. Ideally, I would like to map xor.chrislonnen.com/(\S*) -> xor.lonnen.com/(\S*), but my DNS host does not give me that level of control over the redirect. Shy of renting a server and using nginx or mod_rewrite, I’m not sure I can provide a better solution. My apologies for the inconvenience.

Other URLs related to the primary domain may have stopped working. Cool URIs don’t change, but everything 400’s sooner or later. Get in touch if you’re after something specific; I may have back ups.

Some things... are talking monkey work.

2012-10-10T00:00:00+00:00

“It’s a big universe Jerry, and some things in it are talking monkey work”

Gabriel, The Prophecy

During the 2009 H1N1 flu pandemic I was working as a research assistant in a computational epidemiology lab. Early on in the epidemic data sets were scarce, but the CDC was publishing what they had online in HTML tables. The dozen or so assistants in my lab were tasked with hand-scraping all of the CDC data and compiling it into CSV files for analysis. Our task was tedious, but required focus and attention, and so it kept me from even listening to a lecture while I copied and pasted and removed markup.

I put up with it the first time we had to collect a small sample. When the professors decided we’d be aggregating the entire data set, and potentially others, I wrote a script to do my part. After finding out what I was up to, they pitted me against my cohort in a race to complete our current data set. It was a modern twist on the steel drivers’ ballad, only I was running the steam engine, and racing a whole team of John Henrys.

Within 15 minutes I had a python script and my data set. In 20 minutes, the whole lab’s data scraping for the week was done. Totaling it all up, what would have been 100 man-hours or more was finished in half an hour. The machine had won.

Unlike John Henry’s story, though, no one died trying to beat my machine. No one even lost their job. Instead, everyone got to spend the reclaimed time on the important stuff – constructing and validating models of the disease.

That was a powerful experience, and it drove home a lesson that I’ve read about a dozen times since in one form or another: There’s so much interesting stuff in the world that sits firmly in the domain of talking monkeys. There’s so much time lost to work that could be better accomplished by transistors.

Embrace your talking monkey work. Leave the transistor work to the transistors.

Automate all the things.

Static methods and inheritance

2012-07-02T00:00:00+00:00

My colleague Peter Bengtsson recently published a blog post about the @staticmethod decorator introduced in Python 2.2. Specifically, he argues that @staticmethod is useful for functions called exclusively from object methods, where a developer may want to change the object’s behavior by overriding that static method in a subclass. Take a minute to skim his short post and familiarize yourself with the context.

If, like me, you’re are not a fan of @staticmethod you’d be tempted to define newlines as a function outside the class as Peter did in the second example. Naively then, to change the behavior of the subclass you might try:

Which results in an extra line of code and an extra method over the @staticmethod solution. This extra method can be an annoying indirection, and if you have many of these subclasses you might resent the growing number of *_newlines variants cluttering up your source tree.

As you write all this clutter, though, you might pick up on the similarity of all that copy-pasta. The differences between these functions boil down to the strings provided as arguments to the s.replace method. On examination, the strings start to stand out as strongly tied to the objects using the function and suddenly it becomes apparent: they are actually implicit properties of the object.

It might seem unfair; the example is intentionally simple, and there could be a dozen lines of business logic there instead of a quick one-line replacement. However, I conjecture that any time you see a subclass overriding a static method you are dealing with implicit state that could use some refactoring into the object proper. For our example it would look like this:

Which represents a three line improvement over the @staticmethod version given in the original post. The variable names could be improved, but the structure is explicit and clean. Depending on how often these subclasses are instantiated you could go further and pass in the line endings as arguments to the constructor, or create a class factory and use partial application to create the equivalent of the subclasses – but I’ll leave these ideas as an exercise for the reader.

Something still nags me about the example, though – with the static method gone it’s clear that we’re dealing classes having two methods, one of which is init. This is strong evidence that we should rewrite these as functions:

This cuts down the examples to 4 lines of code and increases readability by removing the object abstraction. It still suffers from the copy-pasta problem we had above, which can be overcome with a short factory method:

The more printers you require, the more this method pays off. The factory method is one-fourth the size of the above base class, and each new printer function is defined in one-third the size of a subclass, with equivalent functionality; which answers Peter’s challenge toward the end of the article: “How would you do this neatly without OO?”

Using GPG + Git

2012-06-01T00:00:00+00:00

Several weeks ago at the Mozilla Webdev offsite I gave a short presentation on using GPG with git. It aims to cover the basics without requiring a deep knowledge of either subject. Included are hints from my experience using it on the Socorro team, and command-line level examples to get you started.

Quarterly Stats

2011-12-28T00:00:00+00:00

The quarter (and year) are winding down, and on the Socorro team we’re looking back at what we’ve accomplished. In addition to stats from our bugtracker and quarterly goals lists, we use the following scripts to derive additional data points from our source control. These give us an idea of what we’ve done well, where we can improve, and what kind of performance we can expect for the upcoming quarter.

These tools rely on git log to do a bit of heavy lifting. You can manipulate the flags to drill deeper into the data set. Try filtering commits by author, date range, or use the built in --grep= for really powerful filtering. Read more with man git-log.

Building Vim

2011-07-17T00:00:00+00:00

OSX’s included version of Vim has done well by me for a while, but I’ve hit the point where I desire some features from newer versions. MacVim is popular among my colleagues, but I like staying in my terminal. It helps to keep my development environment consistent when I’m SSH’d into a dev box, or a virtual machine, or working on a non-Mac.

Updating Vim requires compiling from source, which goes beyond sharing dotfiles. I can’t just slap the binary in my dotfile repo and pull it down, because my environments are not homogeneous enough. Instead, I’ve put together a little compile script that will let me build a better Vim without requiring elevated privileges.

Introducing Introspecting

2011-02-28T00:00:00+00:00

Coincident with the release of Processing.js 1.1, I’d like to announce Introspecting – an add-on for Firefox 4+ to assist in Processing.js development. At this time, it provides a command-line-like-interface for interacting with Processing.js sketches embedded in webpages.

To install, visit the project’s github page, hit the ‘Downloads’ button, select ‘introspecting.xpi’ and drag the file onto your Firefox window. No restart is required.

When installed and active the add-on checks each page loaded into the browser for the Processing object. When a page has Processing, it will load a pseudo-command line at the top of the page that allows you to interact with various Processing instances on the page. If multiple sketches are listed in Processing.instances, it will also present a selector box that allows you to choose the sketch in which your commands are executed.

Most commands work as they would in Processing. For example, noLoop() and loop() will disable and re-enable looping. To be clear, though, the code is interpreted as JavaScript. In order to overwrite void draw(), then, you would write something like draw = function(){background(0);ellipse(mouseX,mouseY,pmouseX,pmouseY);} Undo and Redo will move you through the command history.

This project was conceived of as an intermediary step between println() and FireBug for new Processing.js developers during a discussion with Humph in the Processing.js irc channel. It was based on some earlier ideas for Processing REPL environments. Problems can be filed on Github and non-problem comments or suggestions can be e-mailed to me.

Homebrewing autoconf213

2011-01-04T00:00:00+00:00

Homebrew is an excellent package manager for OS X. When I went to build Firefox earlier today it had most of the required packages available for me, but was missing autoconf213.

A little digging revealed that there used to be an autoconf213 formula, but it was folded into the spidermonkey formula. In the ticket for Issue 3523 it was decided that this decision should be reversed, but the formula hasn’t reappeared yet.

I dug up the old formula from a prior version of the repo (version control seems magical, sometimes) and dropped it in my /Library/Formula folder, after which I ran brew install autoconf. The formula is ‘keg-only’, meaning it doesn’t get linked for general use. Running brew link autoconf created the appropriate symlinks and the Firefox build ran smoothly.

NOTE: instead of storing it locally, you can install using brew install http://bit.ly/autoconf213. Afterwards you will still need to link it.

Until the formula comes back, feel free to use the attached copy of the old one. After it reappears, I would use the more current version in the main repository.

Processing.py

2010-12-12T00:00:00+00:00

While Processing is a Java-based language, there have been implementations of it (with varying degrees of fidelity) in a few other languages. The best known of these is probably Processing.js, a brilliant bit of JavaScript that enables many Processing sketches to run unmodified in web browsers. There are other implementations that hybridize Processing with Ruby, Scala, CPython, and more. J. Feinberg’s Processing.py project is relatively new to this group, and I really like what he’s done with it so far.

Processing.py is built on Jython, a Python implementation running on the JVM, and it enables Processing functionality with Python’s sugary syntax. It supports both “static” and “active” sketches, sketch-specific global variables and methods, and all included Processing renderers. It is distributed as a .jar and to play with it you do not need Jython. Below is a toy syntax comparison:

Processing

Processing.py

These sketches produce the same result when run, but the Processing.py version doesn’t have brackets, semicolons, or return types. This is exciting in itself, but it does more. I’ve rewritten Daniel Shiffman’s Wolfram CA example to show off some of the differences you’ll encounter between Processing.py and pure Processing. Some of the examples are contrived, some are more practical.

More information

Full instructions and a quick start guide are available in the project’s readme. You can run my gist, try the included examples, and write your own. Additional information about the details of the implementation itself are available in a Google Docs presentation here.

Algorithm P(ython)

2010-12-05T00:00:00+00:00

A while ago a friend sent me a simple puzzle: write a program that enumerates all sets of nested parenthesis. I started with an intractable, naive solution and after some discussion and a dead end or two I wound up with a decent dynamic programming solution.

In Knuth’s The Art of Computer Programming: volume 4 section 7.2.1.6 he describes Algorithm P, which is much more clever than the one I came up with. It prints the nested parentheses in lexicographic order. I implemented it in Python and figured I would post it for folks put off by Knuth’s notation.

Undirected Graphs in Python

2010-11-13T00:00:00+00:00

A straightforward way to implement undirected graphs in python.

For more on implementing graphs in Python, see: Python Patterns - Implementing Graphs

Processing && Eclipse

2010-10-19T00:00:00+00:00

For the last several months I’ve been able to work full time on Processing thanks to generous support from the good folks at Fathom. Most of my efforts in that time have been devoted to developing a Processing Plug-in for Eclipse. Unlike p5exporter or some of the tutorials for using Processing Java libraries in Eclipse Java projects, this plug-in aspires to provide support for the language proper.

Right now the plug-in has many, but not all, of the features offered by the Processing Development Environment (PDE) that ships with the language. Syntax highlighting, creating, importing, and running sketches are all there. Additionally, it provides some error checking as you work. In the file system it stores some extra files, but efforts have been made to ensure that it works well with the PDE. Sketches created in one can be worked on in the other. It still has some rough edges and construction warnings, but it is fairly stable and functional.

As this is part of an open-source project, I am looking for community feedback and involvement. If you’re interested in playing with it, installation and getting started instructions are posted in the Processing Wiki and the Issue Tracker for Processing has been modified to include the “Component-Eclipse” label. Bugs and feature requests should be directed there.

I have tested and verified it as working on OSX 10.6.4 and Windows XP (SP3) in fresh installations of the Eclipse IDE for Java Developers. I’ve also verified it works on Ubuntu 10.04 LTS, using apt-get to install Eclipse 3.5 with JDT.

We can't keep it a secret any longer

2010-09-12T00:00:00+00:00

Donal Knuth:

[XOR] in fact, at one point, was a classified thing. I mean it was used so often in code breaking. I went to a conference in Los Alamos in 1976 where somebody wrote the truth table for XOR on the blackboard and the security man erased it immediately.

from his 2006 lecture titled “Platologic Computation”

X⊕R | Chris Lonnen

My Community Boxen Setup

2013 Web Dev Intern Recap

Applied Matching Algorithms

Of Stable Marriages and College Admissions

Open Enrollment, Stability, and Suboptimal Results

Inspiration and Application

Further Reading

Webapp Error Reporting

Custom Error Responses in Flask: Revisited

Setting up OSX and iTerm2 for Emacs

Custom Error Responses in Flask

Building a Shanty Town

Feedback and Interaction

Pre-Integration Testing

Introducing Leeroy

How’s that new robot working out?

Socorro Project History

DNS Record Bankruptcy

Some things... are talking monkey work.

Static methods and inheritance

Using GPG + Git

Quarterly Stats

Building Vim

Introducing Introspecting

Homebrewing autoconf213

Processing.py

Processing

Processing.py

More information

Algorithm P(ython)

Undirected Graphs in Python

Processing && Eclipse

We can't keep it a secret any longer