Developer Blog Posts

Look down. Now back up again. The Defects are now Issues.

mikeconley — Sun, 11 Jul 2010 02:44:46 -0700

From my software development experience, there are a few different words for the generic notion of a “problem”.

A bug or a defect, for example, is defined as the following from Wikipedia:

A software bug is the common term used to describe an error flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, and a few are caused by compilers producing incorrect code.

An issue goes a level higher – a bug is an issue, but an issue might not be a bug. Wikipedia says:

In computing, the term issue is a unit of work to accomplish an improvement in a system. An issue could be a bug, a requested feature, task, missing documentation, and so forth. The word “issue” is popularly misused in lieu of “problem.” This usage is probably related.

Where am I going with all of this?

Well, remember when I said I was going to add defect reporting/tracking capabilities to Review Board? I asked for some feedback on my UI mockups on the developer mailing list, and an interesting conversation on terminology erupted.

Anyhow, the long and the short of it is – we’re going to be calling “problems still existing within a review request revision” issues. And this is distinct from the sort of thing that might show up in an issue tracker.

Maybe down the line, we’ll have a way for administrators to set their own word for it. From the thread, it sounds like everybody and their brother has their own favourite terminology. ”Issue” will have to do for now.

Thanks again to everyone on the list who contributed to the conversation.

Filing Defects in Review Board

mikeconley — Sat, 10 Jul 2010 20:08:50 -0700

In my last post, I talked about an extension for Review Board that would allow users to register “defects”, “TODOs” or “problems” with code that’s up for review.

After chatting with the lead RB devs for a bit, we’ve decided to scrap the extension.

[audible gasp, booing, hissing]

Instead, we’re just going to put it in the core of Review Board.

[thundering applause]

Defects

Why is this useful? I’ve got a few reasons for you:

It’ll be easier for reviewees to keep track of things left to fix, and similarly, it’ll be harder for reviewees to accidentally skip over fixing a defect that a reviewer has found
My statistics extension will be able to calculate useful things like defect detection rate, and defect density
Maybe it’s just me, but checking things off as “fixed” or “completed” is really satisfying
Who knows, down the line, I might code up an extension that lets you turn finding/closing defects into a game

However, since we’re adding this to the core of Review Board, we have to keep it simple. One of Review Board’s biggest strengths is in its total lack of clutter. No bells. No whistles. Just the things you need to get the job done. Let the extensions bring the bells and whistles.

So that means creating a bare-bones defect-tracking mechanism and UI, and leaving it open for extension. Because who knows, maybe there are some people out there who want to customize what kind of defects they’re filing.

I’ve come up with a design that I think is pretty simple and clean. And it doesn’t rock the boat – if you’re not interested in filing defects, your Review Board experience stays the same.

Filing a Defect

I propose adding a simple checkbox to the comment dialog to indicate that this comment files a defect, like so:

No bells. No whistles. Just a simple little checkbox.

While I’m in there, I’ll try to toss in some hooks so that extension developers can add more fields – for example, the classification or the priority of the defect. By default, however, it’s just a bare-bones little checkbox.

So far, so good. You’ve filed a defect. Maybe this is how it’ll look like in the in-line comment viewer:

A defect has been reported!

Two Choices

A reviewer can file defects reports, and the reviewee is able to act on them.

Lets say I’m the reviewee. I’ve just gotten a review, and I’ve got my editor / IDE with my patch waiting in the background. I see a few defect reports have been filed. For the ones I completely agree with, I fix them in my editor, and then go back to Review Board and mark them as Fixed.

All fixed!

It’s also possible that I might not agree with one or more of the defect reports. In this case, I’ll reply to the comment to argue my case. I might also mark the defect report as Pass, which means, “I’ve seen it, but I think I’ll pass on that”.

I think I'll pass on that, thanks.

These comments and defect reports are also visible in the review request details page:

A defect has been filed.

All fixed up.

It's all good - just pass this defect report.

Thoughts?

What do you think? Am I on the right track? Am I missing a case? Does “pass” make sense? Will this be useful? I’d love to hear your thoughts.

Review Board Statistics Extensions: Karma, Stopwatch, and FixIt

mikeconley — Mon, 05 Jul 2010 19:22:13 -0700

I just spent the long weekend in Ottawa and Québec City with my parents and my girlfriend Em.

During the long drive back to Toronto from Québec City, I had plenty of time to think about my GSoC project, and where I want to go with it once GSoC is done.

Here’s what I came up with.

Detach Reviewing Time from Statistics

I think it’s a safe assumption that my reviewing-time extension isn’t going to be the only one to generate useful statistical data.

So why not give extension developers an easy mechanism to display statistical data for their extension?

First, I’m going to extract the reviewing-time recording portion of the extension. Then, RB-Stats (or whatever I end up calling it), will introduce it’s own set of hooks for other extensions to register with. This way, if users want some stats, there will be one place to go to get them. And if an extension developer wants to make some statistics available, a lot of the hard work will already be done for them.

And if an extension has the capability of combining its data with another extensions data to create a new statistic, we’ll let RB-Stats manage all of that business.

Stopwatch

The reviewing-time feature of RB-Stats will become an extension on its own, and register its data with RB-Stats. Once RB-Stats and Stopwatch are done, we should be feature equivalent with my demo.

Review Karma

I kind of breezed past this in my demo, but I’m interested in displaying “review karma”. Review karma is the reviews/review-requests ratio.

But I’m not sure karma is the right word. It suggests that a low ratio (many review requests, few reviews) is a bad thing. I’m not so sure that’s true.

Still, I wonder what the impact will be to display review karma? Not just in the RB-Stats statistics view, but next to user names? Will there be an impact on review activity when we display this “reputation” value?

FixIt

This is a big one.

Most code review tools allow reviewers to register “defects”, “todos” or “problems” with the code up for review. This makes it easier for reviewees to keep track of things to fix, and things that have already been taken care of. It’s also useful in that it helps generate interesting statistics like defect density and defect detection rate (assuming Stopwatch is installed and enabled).

I’m going to tackle this extension as soon as RB-Stats, Stopwatch and Karma are done. At this point, I’m quite confident that the current extension framework can more or less handle this.

Got any more ideas for me? Or maybe an extension wish-list? Let me know.

Review Board Statistics Extension – Demo Time

mikeconley — Thu, 24 Jun 2010 20:22:54 -0700

If I’ve learned anything from my supervisor, it’s to demo. Demo often. Step out of the lab and introduce what you’ve been working on to the world. Hit the pavement and show, rather than tell.

So here’s a video of me demoing my statistics extension for Review Board. It’s still in the early phases, but a lot of the groundwork has been taken care of.

And sorry for the video quality. Desktop capture on Ubuntu turned out to be surprisingly difficult for my laptop, and that’s the best I could do.

So, without further ado, here’s my demo (click here if you can’t see it):

Not bad! And I haven’t even reached the midterm of GSoC yet. Still plenty of time to enhance, document, test, and polish.

If you have any questions or comments, I’d love to hear them.

GSoC Update: My Review Board Statistics Extension

mikeconley — Mon, 21 Jun 2010 18:19:22 -0700

The Primary Goal

From the very beginning, my GSoC project has been mainly focused towards one primary goal: I want to build an extension for Review Board that will allow me to collect information about how long reviewers actually spend reviewing code.

That’s easier said than done. When I started, the Review Board extension framework wasn’t really in a state to allow such an extension to exist.

So I’ve been tooling around in the Review Board code for the past 2 months, preparing the framework, and getting it ready to handle my extension.

And last night, it started to work. I can now give rough estimates on how long a reviewer has spent reviewing code.

How It Works

My extension adds a new table to the database which stores “reviewing sessions”. Each reviewing session is associated to a particular review request and user, and also has a field to store the number of seconds that a user has spent in review.

I’ve created a TemplateHook that allows me to inject Javascript into key areas of Review Board (in particular, the diff viewer, and the screenshot viewer). The Javascript does the following: every 10 seconds, we check to see if the mouse has moved on the body of the HTML document. If it has, we send an “activity” notification to the server.

The server receives this activity notification through the Web API, and checks to see if the time lapsed since the last session update was greater than 10 seconds. If it is, we increment the working session by 10 seconds and return a 200 HTTP code. If it isn’t, we don’t change anything and return a 304 HTTP code.

Next, my extension waits for a user to publish a review. When it notices that a review is being published, it finds the working session for that user and review request, and then attaches it to the published review. If the user then starts looking at the diff or screenshots again, a new working session is created.

The result? A pretty decent estimate of how long a user has spent reviewing the code. No time gets recorded if the user gets up and has a sandwich. No time gets recorded if the user is on another tab reading Reddit.

Not bad. For a first draft, anyhow.

I think I’m going to try to chart the data somehow, so that users can track their inspection rates. I’ll let you know how that goes.

Review Board Tests: SCMTool Segmentation Fault

mikeconley — Wed, 02 Jun 2010 20:24:28 -0700

I can’t honestly say I’ve been doing much test-driven development on Review Board. Django is a really cool web-framework, but I miss all of the nice testing tools from the Rails ecosystem.

Anyhow, today, I decided to run the tests, and BAM – Segmentation Fault:

Testing Perforce binary diff parsing … SKIP: perforce/p4python is not installed
Testing PerforceTool.get_changeset … SKIP: perforce/p4python is not installed
Testing Perforce empty and normal diff parsing … SKIP: perforce/p4python is not installed
Testing Perforce empty diff parsing … SKIP: perforce/p4python is not installed
Testing PerforceTool.get_file … SKIP: perforce/p4python is not installed
Testing parsing SVN diff with binary file … ok
Testing SVNTool.get_file … Segmentation Fault

Yikes. Getting a segfault in Python is unusual, and a little jarring. However, it did let me narrow down the problem a bit: it must have to do with the pysvn bindings that Review Board uses to talk to Subversion, because those are written in C++ (which can certainly segfault).

The machine I was using had Ubuntu Hardy on it, and the Hardy packages have pysvn 1.5.2-1. Taking a look at the pysvn homepage, the most recent version is actually 1.7.2.

So, I uninstalled pysvn, downloaded the source for 1.7.2, compiled it, and installed it manually.

Segfault gone. Tests pass. Awesome sauce.

Python Eggs: Sunny Side Up, and Other Goodies (or How I Learned to Stop Worrying and Start Coding)

mikeconley — Wed, 12 May 2010 20:20:28 -0700

Cooking with Eggs

Every now and then, the computer gods smile and give me a freebie.

I’ve been worrying my mind out over a few problems / obstacles for my Review Board extensions GSoC project. In particular, I’ve been worrying about dealing with extension dependencies, conflicts, and installation.

I racked my brain. I came up with scenarios. I drew lots of big scary diagrams on a wipe board.

And then light dawned.

Batteries Come Included

Enter Setuptools and Python Eggs.

All of those things I was worried about having to build and account for? When using Python Eggs, It’s all built in. Dependencies? Taken care of. Conflicts? Don’t worry about it. Installation? That’s what Setuptools and Python Eggs were built for!

In fact, it even looks like Setuptools was designed with extensible applications in mind.

Wait, really? How?

Here’s the setup.py file for the rb-reports extension in the rb-extensions-pack on Github:

from setuptools import setup, find_packages

PACKAGE="RB-Reports"
VERSION="0.1"

setup(
    name=PACKAGE,
    version=VERSION,
    description="""Reports extension for Review Board""",
    author="Christian Hammond",
    packages=["rbreports"],
    entry_points={
        'reviewboard.extensions':
        '%s = rbreports.extension:ReportsExtension' % PACKAGE,
    },
    package_data={
        'rbreports': [
            'htdocs/css/*.css',
            'htdocs/js/*.js',
            'templates/rbreports/*.html',
            'templates/rbreports/*.txt',
        ],
    }
)

Pay particular attention to the “entry_points” parameter. What this is doing, is registering rbreports.extension:ReportsExtension to the entry point “reviewboard.extensions”.

“Hold up!”, I hear you asking. “What’s an entry point?”

Entry Points

An entry point is a unique identifier associated with an application that can accept extensions.

The unique identifier for Review Board extensions is “reviewboard.extensions”.

This is the first handshake, more or less, between Review Board and any extensions: in order for Review Board to “see” the extension, the extension must register an entry point at “reviewboard.extensions”.

This blog post shows how extensions can be found and loaded up.

Other Goodies

INSTALLED_APPS and Django

I remember also being worried about how to create tables in Django for extension models. I thought “holy smokes, I’m going to have to either shoehorn some raw SQL into the extension manager, or maybe even trust the extension developers to write the CREATE TABLE queries themselves!”.

Luckily, there’s a better alternative.

Django knows about its applications through a dictionary called INSTALLED_APPS. When you add a new model to a Django project, you simply add the model app to the INSTALLED_APPS dictionary, and run “manage.py syncdb”. Django does the magic, bingo-bango, and boom – tables created.

So if a new extension has some tables it needs created, I simply insert the app name of the extension into INSTALLED_APPS when the extension is installed, and call syncdb programmatically. Tables created: no sweat.

django-evolution

Creating tables is easy. But what if an extension gets updated, and the table needs to be modified? Sounds like we’ve got a mess on our hands.

And don’t expect Django to save you. When you modify a model in Django, they expect you to into that DB and alter that table by hand:

[syncdb] creates the tables if they don’t yet exist. Note that syncdb does not sync changes in models or deletions of models; if you make a change to a model or delete a model, and you want to update the database, syncdb will not handle that.
From The Django Book – Chapter 5: Models

Thankfully, there’s a mechanism that’s already built into Review Board that makes this trouble go away: django-evolution. Django-evolution, when used properly, will automatically detect changes in application models, and alter the database tables accordingly. This is how Review Board does upgrades.

And to top that off, RB co-founder Christian Hammond just became the django-evolution maintainer.

Wow. Everything is falling neatly into place.

Python Metaclasses in Review Board

mikeconley — Wed, 05 May 2010 01:31:43 -0700

So, after diving into the Review Board extension code, I hit a little snag.

It turns out I never learned about Python metaclasses. And the extension code in Djblets / Review Board uses them. In the map I made of the Review Board extension code, I represented my confusion with an image of some kind of quantum-divide-by-zero implosion.

I’ll get to the why for metaclasses in a second. First, I’ll demonstrate how it’s currently being used in the code.

The Way Things Are

This snippit is from the Djblets library, in the extensions folder, in base.py:

...

class ExtensionHook(object):
    def __init__(self, extension):
        self.extension = extension
        self.extension.hooks.add(self)
        self.__class__.add_hook(self)

    def shutdown(self):
        self.__class__.remove_hook(self)

class ExtensionHookPoint(type):
    def __init__(cls, name, bases, attrs):
        super(ExtensionHookPoint, cls).__init__(name, bases, attrs)

        if not hasattr(cls, "hooks"):
            cls.hooks = []

    def add_hook(cls, hook):
        cls.hooks.append(hook)

    def remove_hook(cls, hook):
        cls.hooks.remove(hook)

...

And here are those classes being used in the Review Board extension directory, where it defines its hooks:

...

class DashboardHook(ExtensionHook):
    __metaclass__ = ExtensionHookPoint

    def get_entries(self):
        raise NotImplemented

class NavigationBarHook(ExtensionHook):
    """
    A hook for adding entries to the main navigation bar.
    """
    __metaclass__ = ExtensionHookPoint

    def get_entry(self):
        """
        Returns the entry to add to the navigation bar.

        This should be a dict with the following keys:

            * `label`: The label to display
            * `url`:   The URL to point to.
        """
        raise NotImplemented
...

The idea here is that, if someone wanted to write an extension, they could add a hook to the navigation bar by subclassing the hooks, like so:

(from the rbreports prototype extension)

...

class ReportsDashboardHook(DashboardHook):
    def get_entries(self):
        return [{
            'label': 'Reports',
            'url': settings.SITE_ROOT + 'reports/',
        }]

class ReportsExtension(Extension):
    is_configurable = True

    def __init__(self):
        Extension.__init__(self)

        self.url_hook = URLHook(self, patterns('',
            (r'^reports/', include('rbreports.urls'))))

        self.dashboard_hook = ReportsDashboardHook(self)
...

So you can see the __metaclass__ thing up there in the DashboardHook. DashboardHook subclasses ExtensionHook, and has ExtensionHookPoint as a metaclass.

Why? And what is a metaclass anyways? Why is this useful at all?

Well, luckily, I think I’ve figured it out.

Behold – Metaclasses

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).
– Python Guru Tim Peters

There are plenty of resources available regarding Python’s metaclasses. And I ended up reading a ton of them trying to figure out just what the hell metaclasses actually do.

As it turns out, the best explanation I got was from Wikipedia:

In object-oriented programming, a metaclass is a class whose instances are classes. Just as an ordinary class defines the behavior of certain objects, a metaclass defines the behavior of certain classes and their instances.

In particular, check out their example on Cars. That, for me, more or less set the record straight on how metaclasses are used.

But why does Review Board use them when defining those hooks, like DashboardHook?

Why?

Forget the metaclasses for just a second.

Something is missing from that code that I posted up.

I’ll pose it as a question: How exactly is Review Board supposed to know about ReportsDashboardHook? Like…if we’re rendering the Dashboard, and want to fire off calls to all of our DashboardHooks…how do we do it? How can we find all of the active extension subclasses for DashboardHook?

One way you could do it, is to dive into the extensions directories, and use regular expressions to find defined classes. That sounds like a lot of work, brittle, and prone to error. What else?

Next, you could go back to those extension files, “eval” them (shudder), and extract the globals(), looking for ExtensionHook subclasses. That also sounds like lots of work, brittle, and prone to error.

You could go for class.__subclasses__…but this gives you every subclass, and we just want the hooks that are for active extensions. We could filter out all of the ones that aren’t activated, but we’d have to do that for every hook, and that blows.

Next, we could have developers add their hooks to a registry after defining them. So, we could have:

class ReportsDashboardHook(ExtensionHook):
  # ...
  # code goes here
  # ...

dashboard_hook_list.append(ReportsDashboardHook)

And then when an extension is shut down, we just remove it from the global hook list. That doesn’t look so bad, does it? The only problem, is now we have to trust that extension developers will know to add those hooks to the global_hook_list? It’s another step, another thing to forget, and another point of failure.

And it seems silly – I mean, why go through all this effort? Review Board has already seen these hooks…why should it be so hard to just access the list activated hooks easily?

And there it is. That’s why I think we’re using metaclasses.

If you go back to ExtensionHook and ExtensionHookPoint, you’ll notice that ExtensionHookPoint has a list as an instance variable called hooks. That’s the global hook list for that class of hook (DashboardHook). And then ExtensionHook, in its constructor, automatically adds itself to the Extension’s internal list of hooks, as well as the global hook list, with the add_hook method. Shutting down the hook removes that hook from the global hook list.

So now, all we need to do to add our implemented hook to the list of hooks, is to simply subclass DashboardHook, or NavigationBarHook, or any of those Review Board hooks. No need to add the hook to the global list manually – Djblets / Review Board will take care of it. So now, we can get all of the DashboardHook subclasses for activated hooks, easily and consistently, with no extra work for the extension developer. Same with the NavigationBarHook subclasses.

Smooth as glass. Nice.

Anyhow, that’s my understanding of it. If I’m totally off, I’ll let you know when I find out.

Looking Back on Review Board

chipx86@chipx86.com (chipx86) — Tue, 04 May 2010 09:53:00 -0700

Just over 3.5 years ago, David Trowbridge and I spent some time discussing the annoyances of the typical patch submission and code review processes in the open source projects we participated in and at companies, and decided to play with some ideas for improving this. At the time, we knew very little about what we intended to do. We had a name for it pretty early on, but that was about all we had. We didn’t even know whether we’d get past an early prototyping stage. But here it is, over 3 years later, and we have the leading open source code review tool with an active support and development community, hundreds of companies using it, and exciting new innovations for aiding in the code review process.

I was thinking a few days ago about how far we’ve come and some of the decisions we made along the way. I went digging through our commit history in order to relive some of the past of our little project. Since so few people were even aware of Review Board’s existence at the time, I thought I’d share some of our history with you. Particularly the interesting and funny bits.

“Add the reviewboard.”

Commit #1. The very first thing we put in our Subversion tree on September 27, 2006. I don’t even remember what was in this change now. We transitioned to Git last year and this commit is now just plain empty. Maybe it was jut the directory structure? Who can say.

Early on, we didn’t refer to “Review Board” as a proper name. It was generally “the reviewboard” or something similar. The codebase was young. We didn’t actually do code review on the project at this point (and it shows!). The first few months are littered with odd or nonsensical commit messages, small breakages, and bad decisions.

A few of my favorite commit messages are:

“I suck. Make submitting of reviews.”
“Don’t stuff the list of files in the bug list. It’s impolite.”
“Avoid failing out with Christian’s wacko form”
“Gum.”
“Holy apple pancakes. It worked!”
“I suck… The array was empty… The tests never had a chance to fail. :(”
“‘This is a summary’ sucks. Now we use fortune for the summary, description, and testing done. ‘You’re ugly and your mother dresses you funny.’”
“Unbreak things before ChipX86 notices”
“I’m just… garhgh”

Nowadays, our commit messages look nothing like that, but that’s the fun of a new project. You get to go commit-crazy while you try to figure out what you’re building.

Dashboard, quips and fortunes

The UI of old looked quite different than the UI of today.

We had a dashboard from the very beginning (before the review request pages, even) but it wasn’t anything like the dashboard we had today. It was a simple page with a table containing all outgoing review requests and a table containing incoming review requests. But it also had one more thing: quips.

The beginning of quips functionality was being built. Quips are just little random quotes that are inserted in the UI. I think the plan was to put quips on certain pages, making Review Board a little more fun. We were using them in the dashboard for empty lists, with variations all saying something about the dashboard being empty. Quips are a neat feature that just never survived the early days of development.

Fortunes are similar. On Linux/Unix systems, there’s a little program called “fortune” that just displays a random quote. Since we at first had to test review request functionality without actually having a repository backend of any sort, and we didn’t want to input all the information each time, we just used fortune to generate the summary, description and testing done text. This made for some really funny review requests early on, but this is of course something that had no reason to survive initial development.

Sometimes we would create a bunch of review requests just to see what kind of quotes we’d get. :)

Multiple repositories? Almost didn’t happen.

One of the really critical parts of Review Board today is the ability to talk to a variety of different types of repositories in one instance. But, it turns out, this almost didn’t happen.

The initial goals were not that ambitious. Review Board talked to one repository per instance. Everything was basically hard-coded with one repository in mind. That type of repository, as well as its information, was customizable. You just couldn’t have more than one. At the time, this wasn’t a problem, but it didn’t take long until we had a need to talk to two repositories.

We discussed this and at first decided that if we needed to talk to two repositories, we could just set up two instances. It would have been a lot of work to update it for multiple repositories, after all. And really, this was a small project. Who would really need more than a couple repositories? This started to nag at me, though, and so I spent a couple nights rewriting all of the code as an experiment. It ended up working pretty nicely, and we were able to ditch the multi-instance model.

The importance of rewards

It’s always nice to have a little reward for milestones. Developers sometimes compete over cool bug numbers, revisions, etc. Initially, we were going to use quips to add some fun to the site, but we ended up settling on our current trophy system.

One of our first Review Board instances started to approach review request #1000, which was a huge milestone for us. I decided to commemorate the event by staying up and quickly hacking in a hidden feature for showing a trophy for review request #1000. The way we implemented it, you’d see the first ever trophy at 1,000, and from there you’d see it at every milestone number (1,000, 2,000, 3,000, 10,000, etc.). I didn’t want to stop there, though, so I added support for a second type of trophy, one that has confused people with its appearance to this day. Mission complete.

Of course, when we updated the server and someone finally hit 1000, it triggered a bug in the new trophy code and broke his review request. Oh well, I tried.

Diff viewers are hard

If I could pick one point during the whole history of Review Board where I was ready to completely give up, it would be during the creation of our diff viewer. All three diff viewers.

See, the first diff viewer was a complete and total hack. We generated a side-by-side diff using the diff tools and just parsed the output, basically generating a table of that. It was ugly, though, and limiting. It also caused problems where text on a row would either be truncated or would break the parser. I spent a long time working on this before I totally gave up and went on to try a new approach.

My second approach was closer to what we have today, but also limiting and very, very buggy. We were using Python’s built-in diff generation module, which implements a basic diff algorithm. It gave us insert and delete information, but not replace information. We had to hack that in ourselves, and it was really a hack. Try taking a bunch of inserts and deletes and find out which of those are really changed lines. No, really, try it. It’s harder than you think, and it’ll often be wrong.

Still, we stuck with this for a long time. It was slow, buggy, and didn’t generate the sort of output people expected from diff tools. Most people see diffs from GNU Diff, which implements the Meyers Diff Algorithm (with a few additions and tweaks). These Meyers diffs are much nicer to view than what Python gave us. Another problem we hit was that we didn’t have real line number information, so we had to output fake line numbers. They weren’t really line numbers so much as row numbers in the table. Ugh. Even getting this far was really hard and frustrating, and the result still wasn’t good.

Attempt #3. I decided to build our own diff parser and generator from scratch. What a project. I knew nothing about diff generation and hardly knew where to start. I spent probably a good month or so just trying to work on this new diff code, and was so close to giving up so many times. It ended up being completely worth it, though, as we ended up with a very nice, extensible diff parser.

Without that third attempt, we’d be in the stone age. Review Board would not be as nice to use. We wouldn’t have inter-line diffs (where we highlight what changed in a replace line), syntax highlighting, move detection (coming in 1.5), or function/class headers (where we show which function/class the part of the diff is in — also coming in 1.5).

What else…

Well, there’s a lot more I could talk about. Our initial attempts at JavaScript code for the UI, our trials and challenges with database migration, or our early problems storing diffs with different encodings in databases. This is getting long, though, so I’ll cover these in another post on lessons learned.

The Review Board Extension Life-Cycle

mikeconley — Mon, 03 May 2010 02:55:07 -0700

According to the timeline, I’m still in the community-bonding period for GSoC. Coding for my project is supposed to start sometime towards the end of May.

So I’m using the time to do the following:

Close as many small, easy tickets as I can for the upcoming Review Board release. I’ve already posted some patches for review. More forthcoming.
Get to know the tools I’ll be using. Review Board is hosted on Github, and I’m relatively new to the whole DVCS thing. I’ve been figuring out how to use Git, how to post patches, merging, branching, etc.
Get to know the area I’ll be working in. I’ve been figuring out how Django apps organize themselves. I’ve also drawn up a map of the current state of the extension framework to help me visualize it.
Get to know the other developers working on Review Board. I’ve been hanging out in the #reviewboard-soc FreeNode IRC channel. Very nice, and helpful people to work with.
Develop a plan of attack for my project

And this last point is the one I want to talk about.

The Review Board Extension Life-Cycle

An extension isn’t just some isolated piece of code that gets crammed into an application. When you’ve got multiple extensions already installed and running, installing and activating a new extension is like introducing a new animal into an ecosystem. You have to make sure that your new animal plays nice with the others, and that, in the morning, there won’t be a pile of rotting corpses where your application used to be.

I may have gotten carried away with my metaphor.

So let’s look at what I’m envisioning as the life-cycle for a Review Board extension. I’ll start right from the top.

Getting the Extension

Ideally, this will work as nicely as WordPress’s implementation: a Review Board administrator is given a catalog of extensions to choose from within the Administrator interface, and one click later, the desired extensions are downloaded and ready to be installed.

For all you system administrators out there, that last idea might make your toes curl. A Review Board administrator is not necessarily a system administrator, and the system administrator knows what he/she likes on their machine. An application that can go and download other applications can be dangerous. We have to ensure that the application that we’re downloading is the one we’re trying to download. The last thing we need is some man-in-the-middle to do something cute and bork the code review machine. And we want to ensure that the extension functions as advertised. No hidden features. No self-destruct mechanisms. No back doors.

Do I have a plan for this part? Well….no, not really. I don’t imagine I’ll get that far – I consider it a little out of my scope. So, for my project, I think it’ll satisfy if the system admin (or Review Board admin) can manually download the extension, decompress it, and place it where Review Board can work with it. That other stuff can come later (and I’ll try to design so that it can come later easily).

Installing the Extension

Ok, so at this point, we’ve got our extension downloaded in a place where Review Board can see it.

So now what?

Now we need to install the extension. To me, that means letting the extension put its roots into the RB install by creating database tables, preparing initial data, and generally doing everything to make conditions suitable for the extension to function.

When an extension install begins, I imagine it will consider the following questions (in no particular order):

Do I (the extension) depend on other extensions to function? If so, are those extensions present? If not, let the user know so that they can go get them.
Are there some extensions already installed that will conflict with me, or make me behave badly? If so, let the user know so that they can either remove that conflict, or find an alternative extension.
Is the user entirely aware of what I can do? Make sure that my capabilities, limitations, and behavioural quirks are known to the user.

Once those 3 questions are answered, and everything is looking good for the install, the extension will create the database tables it needs (if any) in order to function. If anything goes wrong during this process, the database changes will be rolled back, and the user will get a full read out about what went wrong.

If nothing goes wrong, the extension will be installed. The user might then be asked to set some initial operating parameters for the extension.

Ok great – the extension is installed. Now what?

Activating the Extension

For something that sounds so dramatic, the explanation about what happens is pretty short: Review Board simply becomes aware that the extension is activated, and passes data through the necessary hooks in order for the extension to function properly. Upon activation, the extension should do a quick double-check to ensure that all prerequisites for the extension have been met. This is because we can’t trust users to activate an extension immediately after downloading them.

The Extension Runs

While it’s activated, the extension will probably react to various events that happen on Review Board. Tables will be updated. View methods will be run. Templates will be rendered.

If anything, ever, goes horribly wrong with an extension, the following will happen:

A log entry will be written, dumping the error, and information about what the user was trying to do
An error message will be displayed to the user, trying to tell them what exactly happened
The extension (and its dependents) will be deactivated. They will no longer react to events on Review Board. Their tables will still be there, the settings will still, but the extension will be, in essence, asleep. Dormant. Non-reactive.

The administrator could try to reactivate the extension at this point. They might try to contact the extension developer for support.

The Extension is Un-installed

If the user wants to rid themselves of an extension, they must first deactivate it. This will put it (and its dependents) in the dormant state. It just switches them off, nothing else.

Deactivated extensions can then be un-installed. If the user chooses to un-install the extension, the extension database tables and settings will be wiped out.

The extension itself won’t be deleted though – at least, not within the scope of my project. The extension files will need to be removed from Review Board manually.

At this point, any dependents that this uninstalled extension had will no longer be able to be activated.

Anyhow, that’s how I envision the life-cycle. It’s my first go at it, so I’d love to hear some feedback if you have any.

Code Spelunking: Review Board Extensions

mikeconley — Sat, 01 May 2010 00:25:04 -0700

So this summer, I’m working on Review Board for the Google Summer of Code.

Until my GSoC acceptance, my romps into the code had been relatively shallow. But with my proposal being given the green light, I’ve started doing more extensive explorations.

Review Board is built using the Django web framework. I haven’t worked with Django before, but I have quite a bit of experience with Rails, so that should be an asset. Using a web framework means having (relatively) predictable source code layout, and Review Board is no exception.

Djblets

At one point or another, the Review Board developers realized that a lot of their code wasn’t Review Board specific, and could be abstracted out into an external library.

That library is called Djblets.

Among other things, Djblets adds a DataGrid component for easy record sorting and pagination. There are improvements to Django’s Authentication system. Functions for easily displaying a user’s Gravatar.

And, low and behold, there is a branch of Djblets that provides classes and functions for giving a Django application an extension framework. The classes are abstract enough so that, in your Django application, you can specify different types and behaviours for your Hooks.

Djblets -> Review Board

The Review Board extension branch takes these Djblets extension classes, and extends them into DashboardHooks, NavigationBarHooks, ReviewRequestDetailHooks…lots of different hooks.

So, Djblets creates the foundation abstractions. Review Board makes these abstractions a little more specific. And then an extension writer needs to instantiate and use these classes to design their extensions. It sounds complicated, I know.

So Let’s Map It Out

When I start learning a new code base, I do a lot of drawing. To me, getting to now a code base is like getting to know a city, and that means walking around it, and mapping it out.

So I’ve taken the liberty of mapping out the extension classes that I’ve found, and how they relate to one another. Note that at the bottom of my map, a simple extension (RB Reports) is using some of those classes to hook itself into Review Board. You can find this, and other extensions,here.

Click here to check out my map of the current state of the extension framework

Now, before someone in the department starts complaining about my misuse of UML: I’m not a UML guy. I just wanted an easy piece of diagramming software, and the one that I found (Dia), did UML. I just wanted something to draw boxes and lines. So please don’t freak out if you think I’m using the wrong symbols.

One symbol you might be wondering about is the blue quantum-flux-capacitor-implosion.

I’ll save that for a future post.

My GSoC Project: Review Board Extensions

mikeconley — Tue, 27 Apr 2010 19:54:24 -0700

If you didn’t already know, Review Board is an open-source web-based code review tool. The MarkUs Team has been using Review Board for pre-commit code review for about a year now. This has given the team a number of advantages:

For a team that usually has a 4 month turnover, this allows us to quickly get new team members up to speed with how to contribute to MarkUs. We review every change that they propose, and give them tips/guidance on how to make it fit in well with the application. They learn, and the applications code stays healthy.
We catch defects before they enter the code base. Simple as that.
We get a good sense of what other people are working on, and what is going on in the code. Review Board has become a central conversation and learning hub for the developers on the MarkUs team.

So, the long and the short of it: I like Review Board. Review Board helps us write better code. I want to make Review Board better.

So what am I proposing?

How to Avoid A Bloated Software Monster

You can never make some people happy.

No matter how decent your software is, someone will eventually come up to you and say:

Wow! Your software would be perfect if only it had feature XYZ! Sadly, because you don’t have feature XYZ, I can’t use it. Please implement, k thx!

And so you either have to politely say “no”, and lose that user, or say “yes”, and add feature XYZ to the application. And for users out there who don’t need, or don’t care about feature XYZ, that new feature just becomes a distraction and adds no value. Make this happen a bunch of times, and you’ve got yourself a bloated mutha for a piece of software.

And we don’t want a bloated piece of software. But we do want to make our users happy, and provide feature XYZ for them if they want it.

So what’s the solution? We provide an extension framework (which is also sometimes called a plug-in architecture).

An extension framework allows developers to easily expand a piece of software to do new things. So, if a user wants feature XYZ, we (or someone else) just creates and make available an extension that implements the feature. The user installs the extension, activates it, and bam – our user is happy as a clam with their new feature.

And if we make it super-easy to develop them, third-party developers can write new, wonderful, interesting extensions to do things that…well, we wouldn’t have considered in the first place. It’s a new place for innovation. What’s that old cliché?

If you build it [the plug-in framework], they will come [the third-party developers who write awesome things]

And the developers do come. Just look at Firefox add-ons or WordPress plugins. Entire ecosystems of extensions, doing things that the original developers would probably have never dreamed of doing on their own. Hell, I’ve even written a Firefox add-on. And users love customizing their Firefox / WordPress with those extensions. It adds value.

So we get wins all over the place:

Our user gets their feature
The software gets more attractive because it’s flexible and customizable
The original software developers get to focus on the core piece of software, and let the third-party developers focus on the fringe features

And this is where I think I can help Review Board.

(Before I go on, if you’re interested, here’s another article on the how and the why of plug-in architectures)

Review Board Extensions

So if you look at the Review Board Wiki, or glance at the mailing lists you see numerous requests from users for new features, for example:

It would be nice if the review board had a “next comment” button that is always available to click, or had a collapse/expand button. This would make it easier to see other people’s comments in cases like this.

…

It will be nice to have post-commit support. Instead of every post-commit review being a separate URL, if we could setup default rules for post-commit reviews to update an existing review providing the diff-between-diff features, it would be very useful.

The Review Board developers could smell the threat of bloated feature-creep from a mile away. So, in a separate branch, they began working on integrating an extension framework into Review Board.

The extension branch, however, has been gathering dust, while the developers focus on more critical patches and releases.

My GSoC proposal is to finish off a draft of the extension framework, document it, and build a very simple extension for it. My simple extension will allow me to record basic statistics about Review Board reviewers – for example, how long they spend on a particular review, their inspection rate, etc.

Having been a project lead MarkUs for so long, it’s going to be a good experience to be back on “the bottom” – to be the new developer who doesn’t entirely have a sense of the application code yet. It’s going to be good to go code spelunking again. I’ve done some preliminary explorations, and it’s reminding me of my first experiences with MarkUs. Like a submarine using its sonar, I’m slowly getting a sense of the code terrain.

I’ll let you know what my first few sweeps find.

Ping!

mikeconley — Tue, 27 Apr 2010 18:03:05 -0700

I’ve done it again: I’ve let dust gather on my blog.

Quick update:

I’ve finished my courses for this semester, and have gone into full-blown research mode.
My research proposal is going through ethics review, in order to make sure that I’m not going to blow things up (or hurt anybody if I do)
While my paperwork is reviewed, I’m refining my procedure and apparatus. Better and better.
I’ve been accepted into Google Summer of Code this year – I’ll be working on Review Board. Details about my project will be the subject of an upcoming post, which I will toss up shortly.
I may or may not be co-directing a radio play. I’ll let you know.
The MarkUs team is about to release version 0.7, and a fresh batch of Summer students will soon be here at UofT to work on it!
I have not forgotten about the UCDP trip to Poland. I still have to tell you what we saw and did at Auschwitz. Cripes – it’s almost a year since I returned, and I’m only half-way through the whole story. And there’s a ton more to tell. Coming soon.

Stay tuned.

Review Board 1.0 Released!

chipx86@chipx86.com (chipx86) — Sun, 21 Jun 2009 07:47:36 -0700

Tonight, we hit a milestone in the Review Board project that we’ve been working toward for over two years. We finally pushed out our 1.0 release. A lot of blood, sweat and tears went into this release (okay, so not literally, but it was A LOT OF WORK!). The last few months in particular have been challenging, as we’ve had to solve some tricky bugs and scalability problems, but the end result is pretty great.

Just a short while ago, we announced the release and put up an overview of the entire release and product. We’ve already had some nice congratulatory e-mails and tweets, which is really nice :)

Some stats for this release:

2 years, 9 months, 25 days have passed since our first commit.
120 contributors have contributed to Review Board so far (in terms of code contributions).
2,019 commits were made.
899 review requests have been posted to our project’s actual Review Board server. 1,650 users are registered on there.
Our demo server, in comparison, has 2,082 review requests filed and 10,154 users.
938 bugs were filed. 812 were fixed.
232 feature requests were filed. 101 were implemented. Most remaining ones are scheduled for releases.
An estimated 200+ companies are now using Review Board. 26 have let us list them publicly.
The largest known Review Board install has over 83,000 filed review requests and over 2,000 users, doing upwards of 10GB of traffic per day.
5 presentations on Review Board are known to have been given, 3 by us, 2 by others.
552 users have joined our main mailing list, and 3,674 e-mails have been sent.

Now that Review Board 1.0 is out, we can get started on some awesome new features we’ve had planned. I have a little notebook full of ideas for our 1.1 and 1.5 releases (which may become 1.5 and 2.0, respectively, as this list grows). Some of the new features are actually ready to be committed within the next couple of days, so those of you using nightlies will start to see them soon.

We were accepted into this year’s Summer of Code, and have three students working on exciting projects for us, so hopefully we’ll start to see these trickle into the upcoming nightlies as well. Among these projects include diff viewer improvements (moved region detection, better whitespace-only change detection), IDE integration with Eclipse, and improved notification hooks and e-mail support.

We’re also working on providing support for third-party extensions, which will allow developers to extend Review Board in new, exciting ways without having to modify Review Board itself. This is especially handy for companies who wish to integrate better with their sandboxes, bug trackers or unit testing services. This will likely land in 1.5 (2.0?) at the earliest, as it’s a large change, but the code for this mostly works today. It’s just a matter of getting the codebase ready and figuring out what APIs we want to stabilize and expose.

As I mentioned in the release announcement, we’re planning a release party, tentatively on July 11th, 2009, in the Bay Area (somewhere around Palo Alto, CA). If any Review Board users want to join us, please RSVP!

Review Board: Summer of Code, Roadmap and Future Plans

chipx86@chipx86.com (chipx86) — Thu, 02 Apr 2009 21:53:39 -0700

Summer of Code

This year, we (the Review Board project) was given the opportunity to participate in Google’s Summer of Code. We’ve received some great student proposals so far, and I think we’ll see exciting work done on Review Board this summer.

The deadline for Summer of Code is coming up fast (April 3rd, 19:00 UTC). If you’re interested in working on Review Board and haven’t yet applied, it’s not too late, but you’ll want to hurry. Skim through our ideas page and, if you find something interesting or have a great idea not listed here, then apply and tell us your plans. I can say we’ve received several proposals so far for the installer and admin UI, so unless you feel strongly about either of those, you’ll increase your chances with other proposals.

We’re also offering free Review Board hosting for open source projects participating in Summer of Code. If you’re a mentoring organization and would like to give Review Board a try for reviewing and managing student code, go ahead and contact us and we’ll get you set up.

Roadmap

We’re finally nearing 1.0. We recently put out our 1.0 beta 2 release and are now in a feature freeze. We’re working to get some bug, performance and usability fixes in for beta 3, which I’m shooting for in a few weeks. Then we’ll branch for 1.0, put out a Release Candidate or two, and then finally release 1.0!

There’s a lot of really cool features planned after 1.0, namely extensions and policy customization.

Extensions

Our bug tracker is filled with feature requests for all kinds of things, ranging from bug tracker integration, instant messaging, a method for offering bribes for code reviews, and so on. We clearly can’t put all the requested features in the codebase, so we’ve decided instead to add support for third-party extensions. Coming soon, developers will be able to write extensions to Review Board in the form of Python modules to extend or alter the functionality of Review Board. The extension framework will allow them to do the following:

Access the database using the existing Review Board database models.
Add new database models for storing data.
Listen for signals (new review request published, review request submitted, etc.) and act on them.
Add custom URLs.
Replace existing URLs, for advanced capabilities such as replacing the diff viewer.
Add new API handlers.
Add “action” links to existing review requests and reviews.
Add columns and sidebar entries to the dashboard.
Add pages to the administration UI.
Communicate with other extensions.
Provide a settings page, which stores data in Review Board-provided models (we even auto-generate the settings page for the extension by default).
And more!

A lot of this already exists in a private development branch, and it will be one of our primary focuses as soon as 1.0 goes out.

In time, we’ll add a new section to the Review Board website where developers can list their extensions for download and for sale. Administrators will be able to browse and search for extensions directly from the administration UI and install them without having to even open a terminal (in most cases).

We’re hoping this will solve a lot of in-house integration issues. For example, many companies have custom sandbox architectures, bug trackers, and statistics software which they’ll now be able to tie in with Review Board.

Policy Customization

We’ve found that a lot of companies have very specific ways they want to handle policy and access restrictions. For example, many companies want to limit who can see certain parts of a repository (and therefore certain diffs), or want to allow anybody to create review groups, or want to disallow people from joining review groups. Some also want to dictate what constitutes approval for submitting a change.

We’re looking into the various requests and attempting to come up with a policy model that is flexible enough to handle these needs. One of the ideas is to provide some basic level of access control on a per-repository, per-path, and per-group basis. We’d then piggy-back on the extension framework to allow for more specific policy control. The advantage is that developers could write their own policy rules that interface with some part of their company’s infrastructure.

If people have any input on this, we’d love to hear it.

Improving browser performance in Review Board

chipx86@chipx86.com (chipx86) — Tue, 27 Jan 2009 12:14:29 -0700

This past Sunday, I landed a set of changes into Review Board that provide improved performance, such as aggressive browser-side caching of media and pages. It’s just a start, but has already significantly reduced page load times in all of my tests, in some case by several seconds. We implemented these methods for Review Board, but they’re methods that can be applied to any Django project out there.

There are several key things that Review Board now does to improve performance:

Tells browsers to cache all media for one year.
Only sends page data if new data is available.
Compresses all media files to reduce transfer time.
Parallelizes media downloads.
Loads CSS files as early as possible.
Loads JavaScript files as late as possible.
Progressively loads expensive data.

A lot of the performance improvements come straight from Yahoo!’s Best Practices for Speeding Up Your Site. We’re not doing everything there yet, but we’re working toward it. That’s a great resource, by the way, and I recommend that everyone who has even made a website before go and read it.

So what do the above techniques buy us, and how are we doing them? Let me go into more details…

Caching all media for a year

The average site has one or more CSS files, JavaScript files, and several images. This translates to a lot of requests to the server, which may leave the site feeling slow. On top of this, a browser only makes a few requests to a server at a time, in order to avoid swamping the server, which will further hinder load times. This happens every time a user visits a page on your site.

Aggressive caching makes a huge difference and can greatly reduce load times for users. Review Board now tells the browser to cache media files for a year. Once a user downloads a JavaScript or CSS file, they won’t have to download it again, meaning that in general the only requests the browser needs to make is for the page requests and AJAX requests.

The big pitfall with long-term caching is that the cached resources can go stale. For example, if a new version of an image was uploaded, the browser wouldn’t even know about it, since it was told it should keep its old version for a year before checking again.

We solve this by introducing “media serials,” timestamps that are appended to all media paths. Instead of caching /js/myscript.js, the browser would cache /js/myscript.js?1273618736.

These media serials are computed on the first page request by our djblets.util.context_processors.ajaxSerial context processor. This quickly scans all media files known to the program, finding out the latest modification timestamp. It then provides a {{MEDIA_SERIAL}} variable for templates to append to media URLs as part of the query string.

The benefit to this method is that we can cache media files for a year and not worry about users having stale cached resources the next time we upgrade a copy of Review Board. The filenames requested will be different, browsers will see that the new file is not in the cache, and make a request, caching the new file for a year.

Only send page data if new data is available

Aggressive caching of media files is great and saves a lot of time, but it doesn’t help for dynamically generated content. For this, we need a new strategy.

When a browser makes a request, it can send a If-Modified-Since header to the server containing the Last-Modified value it received the last time it downloaded that page. This is a very valuable header, and there’s some things we can do with it to save both the server and the browser a lot of trouble.

If the browser sends If-Modified-Since, and we know that no new data has been generated since the timestamp provided, we can send an HttpResponseNotModified (HTTP response code 304). This will tell the browser it already has the newest version of the page. The sooner we do this, the better, as it means we don’t have to waste time building templates or doing any expensive database queries.

Djblets, once again, provides some functions to help out here: djblets.util.http.set_last_modified and djblets.util.http.get_modified_since.

The general usage pattern is that we first build a timestamp representing the latest version of the page. This could be the timestamp for a particular object that the page represents. We then check if we can bail early by calling:

if get_modified_since(request, timestamp):
    return HttpResponseNotModified()

Further down, after building the page, we must set the Last-Modified timestamp, using the same timestamp as above, like so:

set_last_modified(response, timestamp)

We’re using this in only a few places right now, such as the review request details page, but it drastically improves load times. If the review request hasn’t changed and nobody’s commented on it since the browser last loaded the page, a reload of the page will be almost instant.

Compress all media files

Our Apache and lighttpd config files now enable compression by default. By compressing these files, we can turn a relatively large JavaScript file (such as the jquery and jquery-ui files) into a very small file before sending it over to the browser. This reduces transfer times at the expense of compression/decompression time (which is small enough to not worry for deployments of this size, and can be offset by caching of compressed files server-side).

Parallelize media downloads

It’s important to not mix loads of media files of different types. The browser parallelizes media downloads of the same type, in page load order, but if you load one CSS file, one JavaScript file, another CSS file, and then another JavaScript file, the browser will only attempt one load at a time. If you load all the CSS files before all JavaScript files, it will parallelize the CSS file download and then the JavaScript downloads. By enforcing the separation of loads, we can achieve faster page download/render times.

Load CSS files as soon as possible

Loading CSS files before the browser starts to display the page can make the page appear to load smoother. The browser will already know how things should look and will lay the page out accordingly, instead of laying the page out once and then updating that once the CSS files have loaded.

Load JavaScript files as late as possible

JavaScript loads block the browser, as the browser must parse and interpet the JavaScript before it can continue. Sometimes it’s necessary to load a JavaScript file early, but in many cases the files can be loaded late. When possible, we load JavaScript files at the very end of the document body so that they won’t even begin downloading until the page has rendered. This provides noticeable performance for script-heavy pages.

Progressively load expensive data

There are types of data that are just too expensive to load along with the rest of the page. For a long time, Review Board would parse and render fragments of a diff for display in the review request page, but that meant that before the page could load, Review Board would need to do the following:

Query the list of all comments.
Fetch every file commented on.
Apply the stored patch to each file.
Diff between the original and patched files.
Render the portion of the diff commented on into the page.

This became very time-consuming, and if a server was down, the page wasn’t available until everything timed out. The solution to this was to lazily load each of these diff fragments in order.

We now display a placeholder table for each diff fragment in roughly the same size of the rendered fragment (to avoid excessive page scrolling on loads). The table contains a spinner showing that something is happening, and, one-by-one (to avoid dogpiling) we load each diff fragment.

The code to render the diff fragment, by the way, takes advantage of the If-Modified-Since header and is also cached for a year. We use an AJAX_SERIAL (same principal as the MEDIA_SERIAL above) to allow for changes in new deployments.

With these caching mechanisms in place, the review request page now loads in roughly a second in many cases (or less once cached), with diff fragments coming in lazily (and then almost immediately on future loads).

More to come…

This was a great first step, but there’s more we can do. Before we hit our 1.0 release, we’re going to batch together all our CSS files and JavaScript files into a couple of combined files and then “minify” them (basically compressing them in such a way to allow for smaller files and faster load times of the interpreted data).

Again, these are techniques we’re now making use of in Review Board, but they’re not in any way specific to Review Board. Anyone out there developing websites or web applications should seriously look into ways to improve performance. I hope this was a good starting point, but seriously, do read Yahoo!’s article as well.

Infect your application with Parasite!

chipx86@chipx86.com (chipx86) — Tue, 20 Jan 2009 02:52:47 -0700

Ever find yourself stuck debugging an application because the UI is just doing something weird that you can’t track down? Maybe a widget isn’t appearing correctly, or you just need more information about the overall structure and logging statements aren’t doing you much good. Debugging complex UIs can be a pain.

We’ve had some real challenges at VMware, due to the complexity of our applications. It was enough to drive me mad one day, so rather than write more logging statements, I wrote Parasite.

Parasite is a debugging tool that David Trowbridge and I have been working on to give developers an interactive view of their entire application’s UI. It provides a number of really useful features, including:

See the entire widget hierarchy of your UI.
Watch properties update live.
Modify existing properties on a widget.
View all registered GtkActions.
Toggle GTK+’s debugging of graphic updates.
Inject custom code while the application is running.

Yes, you can inject new code into an application. With Python. Parasite runs in-process as a GTK+ module, so it has access to some internals of your application. We provide a Python shell equipped with PyGTK support for creating and modifying your UI on-the-fly, regardless of the language it was written in. Handy when you want to test out new concepts for a UI without writing new C code.

David has a nice screencast available showing some of what Parasite can do.

For more information on Parasite, including screenshots, a mailing list, and where to get the source code, see the Parasite homepage.

Review Board 1.0 alpha 1 released

chipx86@chipx86.com (chipx86) — Mon, 19 Jan 2009 23:20:13 -0700

Roughly two years ago, David Trowbridge and I began development of Review Board for use in our open source projects and our team at VMware. During that time, we’ve turned Review Board into a powerful code review tool that works with a variety of version control systems. Most of VMware has moved over to it, as have an estimated 50-100 companies world-wide. We’ve had over 100 contributors to the project, people providing volunteer support on the mailing list, and people have developed third party tools for integrating with Review Board.

After all this time in development, with this many people contributing, we decided it’s probably time to get a release out there. Sure, we could have done this a long time ago, but there’s a number of large things we were hoping to get in (a recently-committed UI rewrite, for instance). Now that we have most of the major features we want for our 1.0 release, we decided it was time for an alpha.

Over the coming months, we’ll be working on stabilizing the codebase, fixing a few large remaining usability quirks, enhancing performance, and writing some proper documentation (which is coming along nicely).

We’re eager to get a quality product out there and to begin development on the next release. There’s a lot of neat things planned:

Support for writing extensions to Review Board.
A fully-featured API covering every operation you’ll need to perform.
Some degree of policy support (specifying which users/groups can see which parts of a repository, for instance).
Reviews with statuses other than “Ship It”. This will probably be customizable to some degree.
Possibly some theme customization to allow Review Board to blend in better with corporate sites, Trac installs, etc.

Along with this, I plan to roll out a new website for the project that will have a browseable list of third party extensions, apps, Greasemonkey scripts, and more.

We have more information on our release on our release announcement.

Armed robbery in progress

chipx86@chipx86.com (chipx86) — Fri, 26 Dec 2008 06:04:46 -0700

Our Christmas this year was very nice, with lots of good presents, time spent with family, delicious food, and an opportunity to foil an armed robbery. Yes, an armed robbery. How do I continually get myself into these situations?

While heading back home from my stepdad’s parents’ house, I spotted a guy standing in a small field at a street corner near my parents’ house, holding what looked to be a gun. The guy was probably late teens/early adult, Caucasian, and dressed in all black. His actions looked instantly suspicious. He was pointing the gun to the ground and made some motion as if he was checking the ammo or something. He seemed pretty lost in his own world, apparently not even realizing he seemed very suspicious.

My Mom was driving, and decided to slowly drive away from home instead of toward it, and then turn around, giving us time to watch and see what was happening. We knew there was a gas station across the street and wanted to see if he was going to head in there. If so, there was a good chance we’d be witnessing a robbery.

Sure enough, he started walking across the street to the TowerMart, a gas station/convenience store. We parked the car and my Mom grabbed my phone and called 911 while my brothers and I carefully watched from behind a building, making sure we weren’t noticed. We saw the guy walking back and forth at the side of the building, looking very nervous. He was wearing long sleeves with his left hand exposed and his right hand (which was holding the gun) completely covered. As my Mom talked to the police, we continued to watch, and the guy eventually psyched himself into going into the store. He put the gun in his pocket and went in.

The store was pretty crowded, and we weren’t sure what to expect. Would people be running out screaming? Would people be locked in? Would we hear gun fire? Or would he just leave?

We never saw him actually leave, but we could only see the one side of the building. A couple minutes later, six police cars drove up, two right beside us. Two cop cars pulled up alongside us. One of them hopped out, grabbed a big ol’ semi-automatic, and approached the building, pointing the gun, ready to fire. Three other cops did the same, surrounding the building. A couple other cops went in and got people out of there, with another couple cops asking those people if they had seen anyone matching the description we gave, or saw other suspicious activity.

They spent some time going through the TowerMart and eventually came back out once they were sure he wasn’t hiding in there. It seems at some point, he had left the building. We weren’t able to see when. However, he was definitely in there. Practically everyone they talked to noticed him, as he was nervously walking in circles around the building, completely covered. He had either grown nervous with the number of people in there, or heard the sirens come. Either way, he got out of there before doing anything.

As most of the cops started to exit the building, the one who had pulled up closest to us walked up and asked what we saw. Then he said, “Oh, I talked to you before.” We all thought that was strange, since he hadn’t actually talked to us, but then he pointed out that he remembered us from the assault back in May. Good memory.

We walked with him to the field and pointed out roughly where we saw him standing. He found the footprints and was calling out the detectives to take pictures or whatever. We chit-chatted briefly before returning home. He told us we very well may have saved the store and a lot of people from experiencing a robbery on Christmas Day. I just hope the guy didn’t make another attempt elsewhere.

We found out that an hour or two later, the police were still all over that place. About an hour ago, they apparently had arrested someone not too far from here. Whether related or not, I don’t know. Given that they had video footage of the guy, testimonies and descriptions from a bunch of people in the store, and his footprints, it’s probably only a matter of time.

The rest of Christmas proceeded without police intervention. Merry Christmas!

Ubuntu and Desktop Notifications

chipx86@chipx86.com (chipx86) — Thu, 25 Dec 2008 07:09:12 -0700

This past Monday, Mark Shuttleworth wrote about their plans for an overhaul of desktop notifications (the little popup bubbles telling you someone IM’d you or there’s updates available). Many people have asked me about this, some concerned, and wanted to know what I thought. Being the maintainer of libnotify, notification-daemon and the Desktop Notifications specification, some people were concerned that this work would supersede my own.

The reality is, this isn’t a replacement of my project. This is a new joint effort between Ubuntu, KDE, and myself. What’s been written in Mark’s post may not end up being the end result. We’re still deciding how this will progress, and Ubuntu wants to experiment a bit. Aaron Seigo’s post on the subject sums up a lot of my thoughts on this, and I recommend reading it, though I’ll go further and discuss where this all started and how it’ll affect the project.

First of all, libnotify/notification-daemon isn’t going anywhere. It’s not being replaced, nor is an alternate spec being drafted. Ubuntu’s User Experience team has some new things they want to try, and that’s fine. The plan is to see how this stuff goes in their codebase, with the intention of migrating code back upstream.

A couple of weeks ago, during the Ubuntu Summit, I was invited to speak with some of the developers and User Experience guys from Ubuntu about their plans. They showed me the mockup that’s on Mark’s blog and told me about their plans. They have things they want to do that could definitely improve the experience. Some things are pretty controversial, such as the removal of actions on notifications. We sat down, discussed and debated various aspects of the proposals for a while, and I believe reached a general course of action for the project. The highlights include:

Actions will be removed for applications packaged in Ubuntu. The developers will try to replace them with something that they feel makes more sense, with the hope of pushing these changes upstream.
The released notification-daemon will, I believe, support actions, since many non-packaged applications do use them. They will, however, appear as old-style notifications and not appear in the new window stack demoed in Mark’s blog post.
We’ll be drafting new additions to the notification spec in order to address the needs of KDE and Mozilla.
The work will be done by their developers in either a fork or a whole new notification-daemon implementation, allowing them a greater ability to experiment. These changes (or parts of them) will make their way upstream. It will be spec-compatible so users won’t have to worry too much about losing features (aside from actions, perhaps).
In the end, it’ll likely be that the Ubuntu theme specifically works this new way, and that other themes will work differently (they may support actions more directly, for example).

Now I should point out that I don’t believe actions are a bad thing. There’s many use cases where actions are very much warranted, and it seems Aaron agrees. While the Ubuntu team has discussed the possibility of deprecating this in the spec, I believe the end result will be that actions will live on. I’m also pretty adamant that upstream notification-daemon will still support actions and some other features. Ubuntu can choose what experience to give their packaged applications, but that may differ a bit from what we decide upstream.

Time will tell how this all turns out. I personally think it’s great that there’s some momentum on this project. I know I haven’t had much time to work on it as of late, between VMware and Review Board, which is why getting some new people on board with fresh thoughts will probably be a good thing.

On a related note, I’d like to welcome Andrew Walton to the project. He’s going to be working as a co-maintainer and helping out when I’m busy (which will be a lot of the time for a while).

Over the next month or two, the project should start to pick up. We’re beginning to look at ways to improve the spec and at what work needs to be done in the near future for the project. These discussions will take place on xdg-list.

And with that, I wish everyone a Merry Christmas (or just a very good December 25th for those who don’t celebrate Christmas)!