First, I feel sad that designers fear being ignored, feeling that their work or expertise is not appreciated. Second, I think designers need to do the work and research to understand agile better, so that they better understand its perspective and the benefits it can provide. Conversely, I think developers need to learn about UX design and user research, for the same reason. Otherwise it seems to me that even if you get both groups into the same room (or on the same team) to learn how to work together and to love each other better, they won’t have enough shared understanding to make real progress.

When we started writing these tests back in 2007, we used the best technology we knew of that was then available, a Rails plugin called funkytown, since-abandoned brainchild of Pivotal Labs. Funkytown’s code is a little weird, and we’ve had to fix bugs and inflexibilities from time to time. We monkeypatched its rake tasks. Over the years we updated the bundled version of the selenium RC jar file several times.

We wrote another library (selenium-spec, which we open sourced once a while back) on top of funkytown that integrated selenium tests into rspec, provided the ability to chain examples (see this blog post), as well as some webrat-like helper methods, except better, enabling code like element('results').should be_visible.

So that’s what we’ve been using for a few years now. It’s not pretty, but it works. When cucumber and webrat came out, all the buzz made me think that we’d inevitably switch over some day in the beautiful future when everything will be perfect. However, I recently had the opportunity to try them out on a another pet project of mine, and I was surprised to learn that even though they have a much bigger install base than our install base of one, our crazy stack turns out to be better.

There are three major problems with cucumber/webrat/selenium RC, as far as I can tell:

1. You can’t really use fixtures.
2. Debugger breakpoints in your application code don’t work.
3. I swear there was a number three.

All of those (well, at least the first two) aren’t true of our crazy stack, and I wouldn’t want to give them up.

There are some nice things about cucumber/webrat. The annotations are neat, and it’s kind of cool that you can run your tests as both Rails integration and selenium RC tests (although I imagine once your tests get realistically complex at all that is no longer so assured).

Perhaps a good opportunity, one day in that beautiful future, for us to clean up our stack, rebase it on some more modern infrastructure (or rewrite our own), and open source what for us has been a really useful tool.

No agile methodology I know explicitly discusses how user experience design activities fit into the workflow. In the agile context, design usually means the design of code, not the design of the user experience. The standard response of agile to this accusation is “of course each project involves other roles, and they will just fit in wherever makes sense that fits with the principles and practices of the team”. While this may be true, by leaving design out of the explicit picture, many agile software development teams who are doing lots of other things right continue to ignore the importance of incorporating iterative UX design into their process, and therefore are delivering software efficiently that is not as effective as it could be.

I assume something similar is wrong with standard UX practice. For one, it seems that perhaps the design community still tends to think of design as something only done Big and Upfront at the beginning of a project, missing an opportunity to learn from their mistakes and iteratively improve their design as implemented in the hands of real users.

Based on my personal experience, this seemed like a great idea, so I was really looking forward to attending the retreat. At CDD we do work with designers, and although (if I may be so bold) we’re doing better than most informatics companies in this regard, it’s still clear to me that we could do even better. Thus far we’ve have a hard time testing/validating and iterating on user designs and working these activities into our software development workflow.

The retreat was inspiring and motivation building, having so many people in one room who care passionately about making great software from both design and development. One enduring theme was the abolition of “Us and Them” thinking. Two points are important here. One, we should think of designers and developers primarily as people, rather than as roles, people who work together and bring their competencies to bear on a common goal. Two, a given set of competencies that normally would be thought of as belonging to two separate roles can be found in the same person, so we should never be limited by thinking that a given person is either a designer or a developer, isolating them accordingly from activities that involve the other discipline. Design and development must work together, even within the same brain.

I would have liked more detailed discussion at the retreat, however this was difficult to accomplish in two short days. I also would have liked more representatives of the “business” or “money” role, because I think the same principle applies: abolish us vs. them thinking, and find ways for enlightened practitioners from a mix of all of these roles to combine their expertise effectively.

Further activity is planned, targeting something concrete to be presented at the Agile 2010 conference. Stay tuned.

I recently had a weird problem applying updates to Ubuntu Hardy, due to a failure while dpkg was configuring python2.5, giving me error messages like:

dpkg: error processing python2.5 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 python2.5-minimal
 python2.5

I fixed this by allowing the pycentral rtinstall command to overwrite “local” files (which weren’t actually local, I think they were left over from previous versions of Ubuntu–I have upgraded the OS several times). In /etc/python/debian_config, I added:

overwrite-local = True

“aptitude safe-upgrade” then ran without a hitch.

The PI noted that he has seen it to be helpful when more than one lab conducts an experiment, so that each can help the other avoid finicky and/or tacit experimental conditions that would prevent others from reproducing their results. I have wondered for some time (and for the case of microarrays in particular) whether the practice of “pair programming” that we use in software development would be more helpful than minimum information standards to increase the reproducibility of complex experiments. The problem with this, as the PI pointed out, is that duplicating every experiment can get expensive, and in the world of soft money (especially today’s world), people are always looking for ways to make the research dollar go farther. The possible long term efficiency of duplicating some efforts to increase data value and reduce a tendency to go down blind alleys might not be easy to quantify, and thus not easy to weigh quantitatively against the immediate penalty of “getting half as much work done”. (That’s certainly true in software.)

The PI pointed out that even if direct duplication was too expensive, he still advocated some kind of collaboration on experiments. In particular he advocated getting people together in the same room to look at the experiment together as it was being performed, so that the collaborator might catch important things that weren’t immediately apparent to the person performing the experiment. This, at most, only costs a small amount of travel funds.

I asked the PI if others shared his views, and he said that most of the larger microarray efforts had some sort of distributed work going on, but he wasn’t sure that this idea had been formalized anywhere.

I’m interested in this not only because of its parallel with software work, but also because I work for a company focused on facilitating collaborative science. I’m very interested in the different forms that scientific collaboration can take, and how best to help them along.

hotspot$target:::method-entry
/(self->class = copyinstr(arg1)) != NULL && self->class >= "org/apache/solr/" && self->class < "org/apache/sols"/
{
  /* action goes here */
}

A little ugly, but it worked. The choice of “org/apache/sols” as the upper bound was somewhat arbitrary.

Both approaches have trade-offs, and their biggest downsides both have to do with wasting programmer time. If you test with real objects, then your tests run slowly (especially if you use an ORM that binds your domain objects tightly to the database like ActiveRecord). Your tests hit the database, and this is slow. There are other sources of slowness, but nothing has anywhere near as great an effect as hitting the DB.

If you test with mock objects, once your app has any kind of complexity, your refactoring and test writing processes become slow. This is not immediately apparent when you start using mock objects. But as you start writing more and more code, eventually you start having to come up with a crazy number of mock expectations just to test some of your methods. It is true that this is good feedback that the class you’re testing presents too complex an interface to other collaborating objects, or that it collaborates with too many objects, etc. What starts simple will eventually become too complex, and at some point you’re going to need to refactor.

More insidious than this, however, is the effect this web of mocks you’ve wrapped yourself in has on refactoring. You don’t notice how thoroughly you’ve painted yourself into a corner until you want to refactor some ugly aspect of a core class that collaborates with many objects in your system. Suddenly all of those collaborating objects’ tests break because they expect certain method calls from this core object. These tests break because you’ve changed a method signature, a method name, or even worse just an implementation, because no matter what those BDDers tell you, if you test with mocks, to too great a degree that means you’re testing implementation, not behavior (or is that behaviour).

So, now you have to go through all of those mock-based tests and “correct” them, i.e. change their expectations so that they fit the new method name/signature/implementation. This is horrible. The whole point of tests during refactoring is to verify that your refactoring hasn’t changed the behavior of the system (that being half of the definition of refactoring). The tests should pass before you refactor, and they should pass after you refactor. Not only does this break the fundamental refactoring process, it also can take a lot of time, because you have to remember the context of each of those test cases that you have to fix.

You can do something about slow running tests that hit a database (in extreme cases you can use a faster in-memory database, or even parallelize your tests). Of course they won’t run as fast as they would if they didn’t hit a database, but in my opinion it’s something you can live with. Dav Yaginuma has a good suggestion for what to do with this time: Go write that email you need to write, go take the bathroom break, go walk around the office and stretch your legs. It’s not like that’s really wasted time. It is wasted time, however, if you’re sitting there squinting at the screen fixing all of your mock expectations. You can’t do anything else with that time.

I’m kind of half-convinced now that people who advocate the heavy use of mocking either have really nice IDEs that make fixing the expectations a breeze, or they don’t refactor. And if they’re using Ruby that means they don’t refactor. Ok, tongue out of cheek. Seriously, I’d love to hear from folks who have used and continue to use mocks heavily on long running projects, to hear how they handle the refactoring issue. I have pretty much sworn off mocks except in old-school traditional cases (“mocking out an external dependency too expensive to call directly”) because of it.

Fortunately, there was a third option. I spent some time massaging Joyent’s Ruby DTrace patch so that it would compile with Apple’s version of DTrace (subtly different from Solaris’s), and so it would play nice with the other patches in the official Ruby MacPort distribution. Anyway, long story short: you can get it via my newest RubyForge project, rubyport-dtrace. You can install either from the tarfile or by checking out from Subversion, see the instructions in the distribution.

Why I like MacPorts: I like being able to cleanly remove software I install. I also like that I can compile Ruby with DTrace and other patches that I might want (such as the Railsbench GC patch, which I’m also working in to the rubyport-dtrace (dare I call it) code, it might already work but I haven’t tested it).

Rereading a paper months later often makes you wonder whether you read the paper at all the first time. This reading really clarified for me what the Z factor is, that it is not just for high-throughput screening, and raised a number of questions (especially after discussion with colleagues) not addressed in the paper.

The Z factor is the ratio of the “separation band” of the data to the assay dynamic range. A picture helps:

where μ₊ is the mean of the positive controls (in this case the controls with desired effect), μ_s is the mean of the data, σ₊ is the standard deviation of the positive controls, etc. The assay dynamic range in this diagram is μ₊ – μ_s. The screening window is then (μ₊ – μ_s) – (3σ₊ + 3σ_s), and the ratio of this to the dynamic range is the Z factor = 1 – (3σ₊ + 3σ_s)/(μ₊ – μ_s).

(If you’re reading this in an RSS reader, the story continues on my website.)

Zhang et al go on to describe desirable values of Z: Z = 1 is an “ideal assay” (the standard deviations are negligible compared to the difference between the means), 1 > Z ≥ 0.5 is an “excellent assay”, 0.5 > Z > 0 is a “double assay”, Z = 0 is a “yes/no assay” (no separation band, the two 3σ regions touch), and for Z < 0 “screening is essentially impossible”. Note that when the two distributions completely overlap (μ_s = μ₊), then Z is -∞.

As I mentioned above, the Z factor’s usefulness is not restricted to high-throughput screening assays. Indeed, it can be applied to any assay that measures a number of experimental subjects in an identical way and measures control values. However, when discussing application to assay optimization, the paper does point out that use of the Z factor requires “relatively large data sets”.

Three major questions arise:

1. What do these significant values of Z mean, especially 0.5? What is a “double” or “yes/no” assay?
2. How large is a “relatively large data set”, i.e. how large does your data set need to be to use the Z factor?
3. Why is the range of of Z -∞ to 1? Is there another parameter with a more intuitive range?

A nice companion reading for deeper understanding is the paper published in March 2007 by Sui and Wu. They take on question #1, by examining the statistical power of an assay at different Z factor values. The statistical power, in the case of a drug screening assay, is the probability that an active compound is scored as a hit (i.e. the probability of “true positives”). Z factor is calculated without referring to the hit threshold, the value beyond which a compound is scored as a hit. Often people score all outliers three standard deviations outside the mean as “hits” (in our diagram above, this would be any measurements falling above μ_s + 3σ_s, i.e. within the separation band or above). Sui and Wu show that if the standard deviations of sample data and controls are equal, then a Z factor of 0.5 corresponds to a statistical power of 0.999, i.e. there is only 0.1% chance that an active compound is not scored as a hit.

However, they also show that if the standard deviations are not equal, then interpreting the Z factor becomes considerably trickier. They also show that although the Z factor calculation does not necessarily rely on the error distributions being normal, for non-normal error distributions (where the sample (or the control data) is not well described by the normal distribution N(μ_s, σ_s²)) the Z factor does a poor job of describing the reliability of the assay, demonstrated by the fact that the Z factor is different if non-normally distributed data is transformed to be closer to normal.

Sui and Wu suggest caution when interpreting Z (and Z’) factor values, and recommend that analysts confirm normality of the data (transforming if necessary) and calculate the statistical power corresponding to the distributions of the sample/control data and the hit threshold to get a more reliable measure of assay reliability.

Humorous side note: Sui and Wu interpret “double assay” to mean “doable assay”. Who knows.

As for question #2 (“how large does a data set need to be to get a reliable estimate of Z factor?”), this can be calculated from the standard error of the estimators used to calculate the means and standard deviations. That is, when calculating Z factor, one can’t actually use the real means and standard deviations of the underlying distributions of the samples and controls, one can only estimate these quantities by making a number of measurements of the samples and controls. I plan to calculate these and publish them here at some point, but any statistician can do the same (probably more efficiently and correctly than I).

Finally, question #3: Zhang et al‘s objective was to develop a dimensionless constant that took three distributional parameters into account: the difference of the means of the samples and controls, the variability of the controls and the variability of the sample data. There are other ways to combine these parameters into dimensionless constants that are different from the Z factor. For instance, one could calculate the ratio of the separation band to the sum of (3σ₊ + 3σ_s), call this parameter C. C varies between -∞ and +∞, and if σ is the same for controls and samples, then C = -1 when Z = -∞, C = 0 when Z = 0, C = 1 when Z = 0.5, and C = +∞ when Z = 1.

This gets rid of the weird “0.5″ and the upper limit of 1, but I think that actually the Z factor is a more accurate reflection of reliability. The reason is that the reliability of an assay with Z of 0.9 vs. 1.0 is really quite small, even if the dynamic range in these two cases is very different, because 0.9 is already good enough. By contrast, the difference between the reliability of an assay with Z = 0 and Z = -∞ is huge, because we go from having the data variability ranges ([μ - 3σ, μ + 3σ] with appropriate subscripts) touch to having the two distributions completely overlap. By contrast C would only vary from 0 and -1 between these two cases.

I may retouch the explanation above for clarity at some point, especially if people ask questions.