Apple keyboard and Incase origami case

Apple keyboard and Incase origami case

The first setup I looked at was to just use the Apple bluetooth keyboard together with the innovative Incase origami case. There’s not much point in me writing a review of the keyboard as there are already a tonne out there. Suffice it to say that it’s built well, is full sized (minus the number keypad) and feels great. The case is the interesting part of the kit. Based on the japanese art of origami, the case folds into a stand for the iPad when you want to type. When you first put it together before you put the iPad in place it seems a bit flimsy, but as soon as the iPad is in place, its super stable and looks pretty neat too. There are velcro straps to hold it all together; I saw some reports that these might come unstuck after a lot of use, I’ll have to give an update at the end of the trip.

Zagg Zaggfolio case

Zagg Zaggfolio case & keyboard

The other setup I looked at was the Zaggfolio case from Zagg. The nice thing about this setup is that it combines the case and keyboard in one package. There are also function keys specially programmed for the iPad to search or mimic the home key (something Apple’s offering is lacking). The case replaces the Apple Smart cover if you have one, and the iPad sits face down in the case when not in use. Although this seems like a neat solution, I ultimately decided against the Zagg for a couple of reasons. First up, the keyboard isn’t full sized, but a little shorter as it needs to fit within the dimensions of the iPad. I’m a pretty fat-fingered typist at the best of times, so I was worried that the keyboard would end up just being too annoying. Secondly, because the case and keyboard are all in one, you can’t really go traveling without the keyboard and just the tablet. I think I will only be using the tablet for writing 5-10% of the time, so I don’t want to have to bring the keyboard along for the ride all the time.

First impressions

So far, I’m really happy with the keyboard setup. I really like the key-feel of the Appple keyboard, and writing an article like this is almost as painless on the iPad as it is doing it on my laptop. I’ve found a couple of gripes with using the keyboard so far; one general, and one Apple related. The first is that keyboard arrow key navigation doesn’t seem to be built in to Safari. If I’m typing a url or search term, auto-complete suggests some options and my muscle memory keeps trying to use the down arrow key to select an option. I assume this would be the same no matter what keyboard I chose and it’s just something I’ll have to get used to. The second gripe—and it’s minor—is that when I’m done with the keyboard, I have to remember to turn it off, else I don’t get the on-screen keyboard when I’m using the iPad. It’s easily fixed by switching bluetooth off, but nevertheless has been a little bit of a gripe from time to time. That said, I think an external keyboard is an essential accessory for the tablet if you are going to be writing more than the occasional email; so far my money is on the Apple / Incase combo.

On the road again, and learning to be more like a car than a truck

For most of us, being connected all the time is as natural as breathing the air around us. A typical day for me involves checking emails on my ‘phone in the morning, working on my laptop during the day, and watching tv on the ipad at night. What happens if you take the laptop away? Do we need to have a laptop to be a functioning (and contributing) member of digital society? When Steve Jobs was interviewed about this a couple of years ago, he likened the relationship of tablets to PC’s as that between trucks and cars:

“When we were an agrarian nation, all vehicles were trucks…. but as we moved to a more urban society, now most people use cars, so that now something like one in every twenty five vehicles is a truck. PC’s are like trucks.” —Steve Jobs

I couldn’t agree with this more. I’ve seen how people have moved from having a desktop PC at home, to having a laptop—before long I should imagine its rare to see anything other than a tablet in the house. I can imagine a similar shift for many people at their workplace over time. Right now, the tablet is still in it’s nascent form in terms of both software and hardware. We are starting to see reasonable content creation apps becoming available (I’m writing this post on one now). As I’ve been planning my trip, I’ve tried to predict what sort of scenarios I might find myself in, and setting up my ipad in preparation for that. I’ll share these over the next couple of weeks as I get my stuff together for the trip.
I couldn’t agree with this more. I’ve seen how people have moved from having a desktop PC at home, to having a laptop—before long I should imagine its rare to see anything other than a tablet in the house. I can imagine a similar shift for many people at their workplace over time. Right now, the tablet is still in it’s nascent form in terms of both software and hardware. We are starting to see reasonable content creation apps becoming available (I’m writing this post on one now). As I’ve been planning my trip, I’ve tried to predict what sort of scenarios I might find myself in, and setting up my ipad in preparation for that. I’ll share these over the next couple of weeks as I get my stuff together for the trip.

“The greatest reward and luxury of travel is to be able to experience everyday things as if for the first time, to be in a position in which almost nothing is so familiar it is taken for granted.” —Bill Bryson

I’m excited to travel. Not just to parts of the world I haven’t been before, but also to find out what makes for a good tablet lifestyle. As Bill Bryson said in the quote above, one of the great rewards of travel is to experience everyday things anew. I think this is just as true for my digital journey as for the physical trip I’m going on. What I’m looking forward to are the unexpected changes in online behavior—both positive and negative—that will suggest interesting opportunities to differentiate and enhance tablet-living away from it’s PC roots.

Don't overpromise and foreclose on your project

Performance as money

I first was introduced to the idea of performance as money through watching an MIT lecture series on algorithms. In the opening lecture, Charles Leiserson talks about performance as currency:

A good way I think about performance, and the reason it’s on the bottom of the heap, is sort of like performance is like money, it’s like currency. You say what good does a stack of hundred dollar bills do for you? Would you rather have food or water or shelter or whatever? And you’re willing to pay those hundred dollar bills, if you have hundred dollar bills, for that commodity.

when he talks about performance being on the bottom of the heap, what he’s talking about is how other things are more important than performance; features, security, ease of use and so on.

Performance constraints as the debt limit

I’ve posted before on non functional requirements every application should have, and first and foremost amongst those constraints is performance. The deal you need to make with yourself and your users is how much of that performance stack of bills are you willing or able to spend to get there? all of the decisions you make, from language choice to functionality to visual design are informed by this decision. Setting your performance constraint is just like setting your household budget (or the government’s debt ceiling).

And the credit rating?

To extend the metaphor a little bit further than is natural, the credit rating is akin to the likelihood that your project will fail overall. We don’t have a good measure for that in software development, but I do think drawing the parallel is interesting. If you read the S&P downgrade overview, you will see that the decision was in part due to the rising debt burden (technical debt?), but more significantly it is due of the infighting and lack of a clear plan forward:

More broadly, the downgrade reflects our view that the effectiveness, stability, and predictability of American policymaking and political institutions have weakened…

This sounds so much like entrenched behavior of stakeholders on a poorly managed software projects (and its effect) that I couldn’t pass on the opportunity to use it. Simply put, if you have a lack of alignment and clear plan about what is acceptable, what is important and how you are going to spend your resources (or generate income) things aren’t going to be plain sailing.

Performance—Avoid foreclosure

O.K. I really am stretching it here, but I wanted to finish up with how I think you best avoid getting mired in software rot or simply just building something that no-one wants to use. And it’s really similar to being financially prudent:

• Understand that almost everything you do bears some cost to performance
• Decide what is an acceptable level of performance…
• …and spend within your means
• Don’t go into debt(let performance drop below acceptable level)
• But sometimes it’s going to happen. If it does, get a plan in place quickly to address it.

We think we are getting this...

...but often end up stuck with this

As Mateo mentioned, we are pretty much trained to reuse code wherever we can (and that includes our own—more on that later). The assumption is that just because it’s on google code or GitHub it’s good to go. If you’re not careful you will spend more time working through someone else’s library than you would coding from scratch. In practice the decision to reuse or not is a very delicate balancing act that you can easily get wrong. Here are some things to consider when you are faced with the choice:

Good times to consider reuse

It’s generally accepted to be best in class

No-one is going to regret choosing to use JQuery as a base framework for their javascript app. If you know the library is a de-facto standard for the problem you’re trying to solve you should be in good shape.

You don’t know much about the problem / algorithm

I’m not a statistics guy, and when I decided to code up an application that uses some simple estimates to get a more realistic prediction of development time I needed to calculate the shape of a beta distribution chart. Rather than do it myself, I found a library that saved me having to learn how to code a bunch of statistical formula.

There is an a active community around the code

If there are O’reilly books on the subject, if there are questions (and answers) about the code on stack overflow, then you can probably feel a little more comfortable that when you get stuck, your questions will get answered.

It’s not the main problem you’re trying to solve

Again, in the example of the software estimation app I built, the main problem I was after solving was showing people how you can quickly get a sense of the length of your project, not hardcore statistical analysis on data. I knew I didn’t need to do too much with the statistical analysis, and if it all went wrong, I could back out and use simpler, back of the napkin estimates

When to be wary

The code isn’t being maintained

If the code is marked at version 0.1 and was last updated three years ago, just be aware that the owner has probably moved on to other things and if you find problems, you might end up having to fix someone else’s issues.

You know a lot about the problem domain and could put something together quickly

If I know a problem domain really really well, then I can probably write my own stuff in the time it takes me to learn how someone else has approached the problem. Essentially I can do it quicker than ‘most any learning curve.

The code doesn’t have supporting tests

At the risk of sounding like an agile hitler, if code doesn’t have a decent, usable test harness written against it, I have very little faith in it’s utility. It will mean that I have to proceed more slowly and cautiously and likely not understand half the intent of the code.

It’s your own stuff

So you wrote some stuff on your last project that applies to what you’re doing now? Good on you; should save a tonne of time, right? Think again—just because you wrote it doesn’t mean it’s any better than someone else’s work. This one deserves a little more explanation below

It’s going to be really difficult to back out of later

If the library you’re going to use is going to form the backbone of your application or the way its structured, then you need to be careful about the decision you’re making. Things like an architectural framework deserve careful though because the cost of changing your mind can get prohibitively high.

On re-using your own code

There are a couple of reasons why we need to be wary of our own work. First off, we tend to take a revisionist view on things; we assume that the work we did in the past is completely applicable to the problem we have at hand today. And I don’t know about you but I’ve become a better coder over time, so my old code is not as good or clean as the code I write today. Any time you reuse work, you need to weigh your own code with the same rigour as you would someone else’s. Don’t get me wrong, I’m really not advocating re-writing your own code for every project; what I’m suggesting is that if you are going to reuse your own code, make sure it’s fit for purpose, well-tested and as good as you can make it.

So, what can De Niro teach me about reuse?

The decision about whether to reuse or not is a tough one. You need to invest some time evaluating the library, and at some point make a judgement call on how steep the learning curve is versus just going it alone. The smart thing to do when you decide to reuse code is to make it easy to back out of the decision. As De Niro says in Heat:

Have no attachments; allow nothing to be in your life that you cannot walk out on in thirty seconds flat if you feel the heat around the corner

What I mean by this is that if you are going to use a library or someone else’s code, make sure you have it well isolated in your application. Try to only interact with it in one place, and make sure you have solidly tested that integration point. That way, if you later decide to back out of your decision, you can cleanly, surgically remove the library and replace it with your own work (or vice versa). Lowering the cost of changing your mind is the thing that can make the difference between a happy marriage and a bitter, bitter divorce that bankrupts you in the process.

The estimation app in action

The application

The application is really simple. You just break down your project into a few tasks and enter three estimates for each—Optimistic, Nominal and Pessimistic (To get what I mean about these terms, check out this post). What it then does is calculate the expected duration of the tasks as well as a high and low end of a range that you can have 95% confidence in. On top of that, it consolidates all of your estimates into a rolled up view that tells you how long you can expect your full project to take. See, there can be some science behind software estimation after all (if you want the details behind how the estimates are worked out, they’re covered in a previous post).

The sparklines

One of the things I wanted to do over and above a spreadsheet was to make something that was clean and simple to read yet packed a lot of information into each row. What originally was six data points has been reduced to two thanks to using sparklines. I could have possibly also done away with the low and high end estimates and had them as rollovers (like in the main result chart) but I just found it to be too overloaded and hard to read. I’ve found that there has been some really nice outcomes from using the sparklines:

• I can see which tasks have the most uncertainty in them (those with the broadest range)
• I can see how different tasks stack up against one another by comparing the sparklines vertically (they’re all on the same x axis scale). I can do this much quicker than if it were a numeric table
• I can present a hierarchy of information. The ranges are most important, then the expected time (which you can see on rollover), followed by the variance and the shape of the estimate which you can infer from the shape of the chart

The results script

One of my big frustrations is that when people come up with software estimates, they give you values that imply a far greater precision than is reasonable. I’ve seen software estimates for a project in terms like “3042.5 hours” What this does is imply that the estimate is between 3042.45 and 3042.55 hours. Far more likely is that you have added up a number of estimates and at best you might be precise to within ten, twenty or even fifty hours (tough to say without knowing more about the estimate, but you get the drift). With the results script, I’ve tried to represent this by using natural language to lessen the implied precision. A few examples:

• Anything less than 1 is represented as “A few hours” “less than a day”
• Up to a week you have “A couple of days” or “A few days”
• Up to a month you have “About a week, a couple of weeks, three weeks
• More than that, time is measured in months

While looking into this, I also found this article talking about the psychology of the zero. In essence it says that people are more likely to ‘accept’ a number with implied precision than one without. If you want to try this out and use the calculated numbers, rolling over the dots on the results plot will give you them. I also left in the calculated numbers for the individual tasks so you can see what’s going in to the final estimate.

What’s next?

Well, I’m hoping you’ll tell me. Right now, all estimates are in days and part days, but I thought I could add an option so you could put estimates in hours for smaller tasks. Maybe some export options or something. If you use this app and get some value from it, let me know if there’s anything else you’d like to see it do. So go ahead and try it out and let me know what you think!

How long would it take you to build "the free world's most formidable weapon"?

The real reason why your project will take longer than you think

What the Navy realized is that when you give a baseline estimate, the number you tend to give is how long you think it will take in ordinary circumstances; the outcome that will happen most of the time. This isn’t however the average amount of time the task will take. How so? Consider this extract from Bob Martin’s excellent book The Clean Coder where he describes a typical software estimation conversation:

Mike: “How likely is it that you’ll be done in three days?

Peter: “Pretty likely.”

Mike: “Can you put a number on it?”

Peter: “Fifty or sixty percent.”

Mike: “So there’s a good chance that it’ll take you four days.”

Peter: “Yes, in fact it might even take me five or six, though I doubt it.”

Mike: “How much do you doubt it?”

Peter: “Oh, I don’t know … I’m ninety-five percent certain I’ll be done before six days have passed.”

Mike: “You mean it might be seven days?”

Peter: “Well, only if everything goes wrong. Heck, if everything goes wrong, it could take me ten or even eleven days. But it’s not very likely that so much will go wrong.”

As Uncle Bob goes on to explain, what’s going on here is that Mike is describing a probability distribution, something like this:

What is happening is that when you describe how long you think it will take, you are describing the mode – what will happen most often. However because there is more chance of running over than under, you need to take that into account. What the Navy did was find a really quick simple way of finding the mean by looking at three simple estimates for the duration of a task…

Three little estimates

They figured out that if you can get people to estimate these three possible outcomes, you can calculate the expected duration for a task:

Nominal (ψ)

This is the standard issue estimate, what people are used to giving—the amount of time you would expect a task to take under normal circumstances. In the example above it would probably be 3.

Nukem (ω)

This is the longest it could take if pretty much everything went wrong. This is the one in a thousand worst case scenario, the Nukem (for more on this read my last post). Mike’s Nukem is 11.

Mu-nicorn (μ)

I took extreme license with naming this one, but bear with me. This is the your estimate of how long this would take if the stars aligned, unicorns pair programmed with you and generally everything went so well you have to check yourself.

Super Simple Math

Now you have these estimates, you can calculate the expected duration of the task using this formula:

T_e = (μ + 4ψ + ω)/6

You can also get the standard deviation of the estimate, which gives a measure of how uncertain the task is:

σ = (ω – μ )/6

Now, you can tell people that you are pretty confident that your task will take between T_e-2σ and T_e+2σ. This statement is not strictly correct, and I will explain at the end why this is so, but to keep it simple, I would use this.

Putting it to practice

In order to put this into on your project I would recommend breaking your app up into somewhat smaller, more manageable pieces for estimating. In theory, this helps because the law of large numbers implies that your errors cancel out. In practice, it’s also just plain easier to understand your app in small pieces. When you have your estimates for a number of tasks, you can simply add the expected times together to get the expected duration for the project. To get the standard deviation, you have to calculate the square root of the sum of the squares:
σ _project = √ ∑ σ ²

Conclusion

There is nothing like real development data and experience to understand how long things are going to take. Unfortunately you will often have a situation when you don’t have that information available and you are still called upon to give a estimate. This technique can help you do that and have solid reasoning to back you up. I hope you find this useful, and if you want to look into estimation more, I highly recommend both Bob Martins book The Clean Coder mentioned above and Mike Cohn’s book on Agile Estimating and Planning

Post-script – there (probably) be dragons down here

This gets a little involved down here, so if you want to keep it simple, don’t worry about this stuff. As you are usually at the beginning of a project when you do this and are wildly imprecise anyway it shouldn’t make much difference.

The more astute amongst you will have seen an issue with how we worked out our range by simply adding / subtracting two standard deviations from the estimated duration of the project. That assumes that its equally likely we run over as it is that we run under—that is, it’s a normal distribution like this:

Trouble is, as we saw in the example, there is a tail off to the right. This is what’s known as a beta distribution (or at least an example of one), something more like this:

If you look at these overlaid onto one another, you see that the range estimate when using normal distribution actually underestimates the duration a bit. If you want to keep things simple you are probably ok, but it’s worth knowing:

The shape of the distribution is called a beta distribution, and it makes the math for working out the range quite a bit more complicated. You need to calculate a couple of variables that describe the shape of the curve: α and β. These are a bit complicated to work out, but for the sake of completeness, I hunted them down for you here. They are:

α = ((mean-min)/(max-min)) * ((((mean-min)*(max-min))/σ²)-1)

β = (max-mean)/(mean-min)*α

Once you have these, you can use the Excel BETAINV function to get your minimum and maximum ranges. For example, if you want to calculate an duration as a range you can have 95% confidence in:

low end of range = BETAINV(0.025,α,β,min,max)

high end of range = BETAINV(0.975,α,β,min,max)

Duke Nukem Forever Launched after 14 years development; what can this big chested antihero teach you about your project?

Why it overran

There are a number of reasons why the Duke took so long to get his politically incorrect ass back out in the wild. As an adoring public were salivating at the thought of a new Nukem, the development team and its management were stuck in a pattern of events that pretty much ensured a death march of a project:

Scope Creep

Anyone who has ever been on a development project will be familiar with this problem. After the success of Duke Nukem 3D, George Broussard—the game’s creator—kept seeing opportunities for ever more awesome features to put in the experience:

Broussard simply couldn’t tolerate the idea of Duke Nukem Forever coming out with anything other than the latest and greatest technology and awe-inspiring gameplay. He didn’t just want it to be good. It had to surpass every other game that had ever existed, the same way the original Duke Nukem 3D had.

Snow scenes? You got it. Online Multiplayer? Of course! These sorts of things may seem price-of-admission now, but bear in mind that these features were being added over ten years ago. And – I’m not saying that these were bad ideas or unnecessary ideas, just that the goalposts kept moving, which makes it kind of tough to finish.

Technology Death Spiral

When Duke Nukem 3D came out, it was built on a game engine named “build” by a child prodigy called Ken Silverman. In order to keep up with ever increasing needs of the project (see above), Broussard swapped out for a new game engine not once, but twice. This, again, may have been necessary, but it meant that the development team were always playing catch up.

Too much cash, not enough business

Here’s the thing: Duke Nukem 3D was a runaway success. So much so, that unlike before, Broussard could basically fund development themselves. Creative freedom is great, but without the business reminding you that what your building is for people to use and enjoy you run the risk of getting stuck into some sort of artwank vortex.

Fear of Success—gilding the lily

I’m not a console game developer, but when I heard it takes up to a year from “locking a game down” to launch I really felt sorry for those guys. The projects I work on have some level of locking down code to close out the last remaining bugs, but if it took a year, I can see how it would be an almost paralysing decision to make.

You’re no different from Nukem

To some extent or another, your project will have some of these issues. Or the server will blow up. If you don’t have complete freedom and never know when to finish the business will be on your ass. Or the IT department. Or someone else’s sucky APIs will kill you slowly as you realise you’re the first person to use them. Or you will get a fundamental set of assumptions so wrong that you need to reset. Or you hire someone that turns out to be worse than useless (and yeah – I know you will fire them quickly, but it will still suck energy and time from the team).

Once you realize that some or all of these things will happen to your project, you can start to use this information to your advantage…

Duke Nukem Forever is at the limit of how long things will take

See here’s the thing: Duke Nukem is pretty much at the outer limit of how long you can get away with a project overrunning. Any longer and it would have been be laid to rest. If it weren’t for the cult-like following DNF’s only achievement would have been 3 Vaporware of the year awards (and a lifetime award to boot). Gearbox Software, the company that picked up development after the original developer went bankrupt probably realized that they were better off launching a mediocre game and starting from scratch than suffering yet another rewrite. This gives them the opportunity to reset the ‘development time’ clock.

Enter the Nukem as a unit of development time

With all this in mind, I propose that we consider using the lesson of Duke Nukem Forever to give us a new unit of software estimation:

A Nukem: The amount of time a project or part thereof will take when pretty much anything that can go wrong does.

So – how long will your next project take?

I can’t be precise, but I can be accurate. This is how long your next project will take longer than you expect it to to and less than a Nukem. Or, in mathematical terms:

ψ < x < ω

Where:

ψ = the amount of time you expect it to take

x = the amount of time it will actually take

ω = a Nukem

What’s interesting here is that many of the issues that Nukem had weren’t really down to code or developers, yet developers are the ones who get asked to estimate project duration and get held to it. Oh well, tough shit, that’s what you get for being a creator. Broussard’s stock response of “screw you publisher, it’ll be ready when it’s ready” actually ended up killing his baby. Of course if you go to someone who is paying the bills with this range, you’ll probably not get vey far. But there is hope. In my next post I’ll show you how you can use this information to arrive at a range that is more palatable to them, has some science behind it, and allows the business to make some decisions based on your assumptions.

The trouble with typical non functional requirements

“it’s not fast enough…” That was the response I always had with my client / UX lead when I was developing when I asked them what they thought of the application. The trouble was, the client was used to seeing a vision demo which had canned data and used smoke and mirrors to mimic the application without having to worry about anything like network latency or the size of data. During development, the software you are building will go through various degrees of speed and performance as you strike a balance between functionality, flexibility and speed. In order for you not to get the shaft as a much harried developer, you need to arrive at an understanding with the client about what performance is acceptable and in what conditions.

Make your requirements SMART

First off, the creation and agreement of how the application needs to behave has to be a conversation between UX, the business and the technical team; you all fail if they are crappy, and you all need to sign up to them. What you need to do is come away with a clearly defined set of goals for ‘quality’ of the application. Like any other requirement, they should be clear, simple and understandable. A great way to check this is to use the SMART principle first written up by Mike Mannion and Barry Keepence in a 1995 paper on this. Basically they suggest that requirements should be:

Specific - Clear, consistent and simple
Measurable – If you can’t measure it, you won’t know when you’ve acheived it
Attainable – The requirement must be something that the team thinks is acheivable given what they know
Realizable – Can it actually be done given the constraints you know about the way the project is being executed (is there enough money / time to acheive this goal for example)
Traceable – It is important to understand why a requirement exists so that it can be justified (and to question if it is still needed if the original driver or assumption changes)

A common structure for defining the requirements

I did quite a bit of hunting looking for a way that I could describe requirements as diverse as conformance to coding standards and ability to cope with year-on-year growth needs. Fortunately I found this post by Ryan Shriver that talks about how he uses a method by Tom Gilb

[requirements] can 1) be specified numerically and 2) systems can be engineered to meet specific levels of performance. Qualities can be specified using a minimum of six attributes:

Name: A unique name for the quality
Scale: “What” you’ll measure (aka the units of measure, such as seconds)
Meter: “How” you’ll measure (aka the device you’ll use to obtain measurements)
Target: The level of performance you’re aiming to achieve (how good it can be)
Constraint: The level of performance you’re trying to avoid (how bad it can be)
Benchmark: Your current level of performance.

Some sample requirements and reports

I think it’s best to take a look at a few examples from my last post on essential non functional requirements for building quality software. What I’m showing here is how I would specify test and report the constraint:

Processing times

First off a relatively simple one. Measuring processing time is a matter of allowing the system to log out how long it took to do stuff.

Name: Parsing of a large search result
Scale: Elapsed milliseconds from receiving an XML response in the client containing 1000 results
Meter: Log messages with a defined signature that output the parsing time in milliseconds
Target: <1200ms (i.e. 1200 ms or less is a target time)
Constraint: >5000 ms (i.e. more than 5000 ms is an epic fail)
Benchmark: Build 155; test environment 3/21/11: 4500ms

Reporting this over time could give a chart something like this:

Architectural standards

This one seems a bit more tricky at first glance. How can you measure how compliant a codebase is against architectural standards? Fortunately there are some great static analysis tools out there that allow you to inspect your code for potential problems. We have used PMD and FlexPMD in our projects. You can write your own rules or draw from common rulesets. Using this we are able to reduce things like architectural conformance to something quantifiable and thus testable:

Name: Conformance to standard Flex architectural best practices
Scale: # of PMD high warnings from the ‘cairngorm’ and ‘architecture’ PMD rulesets
Meter: Flex PMD warnings reported by the nightly Hudson Jenkins build divided by the number of classes
Target: <1200ms (i.e. 1200 ms or less is a target time)
Constraint: >5000 ms (i.e. more than 5000 ms is an epic fail)
Benchmark: Build 155; test environment 3/21/11: 4500ms

You can see here how architectural standards started to lapse in sprint 3. The good news was that it was caught and addressed. This sort of conversation is far easier to have with your team when you have data to support the conversation.

Year on year growth requirements

Last one to demonstrate some breadth. If you want to test growth requirements, you are basically going to have to think about what your likely point of failure is going to be in your other requirements as your user base grows. It could be storage needs or response times, or more likely it’s peak throughput. In the example below, we increase the standard load on the system until the throughput test fails. We use that to predict how much runway the current system has before breaking down.

Name: 60% year-on-year growth
Scale: Number of months for system to hit peak throughput failure
Meter: Standard Load Scenario increased by 5% per month and measured in each scenario until failure
Target: 8 months
Constraint: 3 months
Benchmark: Current application; 3/21/11; Production environment; 2 months

Simplicity and Consistency

This pretty much wraps it up for non functional requirements (and FWIW, I agree with Mike Cohn that we should be calling these things constraints). Non functional requirements can be a gnarly area as they attempt to describe areas of the software most prone to gradual decay and entropy, but I think these are the areas that are important to test if you are serious about building decent quality software. What I like about the approach is that the output reports are nice, clean and simple. They allow you to report on really varied requirements in a consistent manner and allow you to have a data driven conversation if things start to slide.

How I arrived at this list

I started to look at this a little while back when I was putting together our playbook for how we create software for our customers by bringing together agile and user centred design together. We articulate our requirements as User Stories, but outside of the functional aspects of the system, I realized that there are a common set of non-functional requirements that we saw the need for over and over again. Mike Griffiths has put together a great checklist of non-functional requirements, but I felt that a number of them could be covered off and scheduled in standard stories. What follows are the ones that to me are about how a development team needs to think and behave over time:

Performance

This one shouldn’t come as a surprise. Quality software has to be fast. Or at least feel fast. As a front-end guy, this is the one I always feel first. “It’s not fast enough” is a battle I never want to get into. I had this reported as a bug against one of my projects, but the client wouldn’t specify what “fast enough” was; talk about moving goalposts. When you think about an app being performant, thing about specifying the following:

1. Response times

How long should your app take to load? What about screen refresh times or choreography?

2. Processing times

Can I get a spinning beachball please? How long is acceptable to perform key functions or export / import data?

3. Query and Reporting times

This could be covered off with general reporting times, but if you’re providing an API you should probably consider acceptable query times too.

Capacity and Scalability

How much do you need to cope with now, and how much do you think you’ll need to cope with in the future. Unless you are truly blessed with an incredibly forgiving audience (or a great illustrator), you won’t be able to get away with your own fail whale.

Your audience had better be pretty charmed by you if you can get away with regularly futzing on capacity

4. Throughput

Think about how many transactions your system needs to handle. A thousand a day? A million? When Amazon solved this for their needs, they decoupled systems and created a queue service that became the foundation of AWS.

5. Storage

How much data are you going to need to store to do the awesomeness you need it to do?

6. Growth Requirements

This is a tough one, because truly you don’t know how popular your app is going to be until it’s out there. But you can bet (or hope) that someone has made predictions about how wildly successful your app is going to be. Be wary of over engineering here, but at least make sure you aren’t constantly laying down track in front of a moving train.

Availability

Does your system need to be available all the time, and to everyone? Most of the time you can get away with the occasional vacation, but knowing what’s important helps you make good decisions. Quality software ain’t quality if its down when your precious users need it

Can you get away with a product that is rarely available?

7. Hours of operation

When does your app need to be available? If you need to do a database upgrade or a system backup, can you take the system offline while you do it?

8. Locations of operation

A few things to think about here: Geographic location, connection requirements and the restrictions of a local network prevail. If you are building a killer app for use behind the corporate firewall, you’d better make damn sure you aren’t using any exotic ports.

Maintainability

When you’re running a software project you are at constant war against software entropy. This is your chance to save yourself from having to live with shitty code, or leave shitty code for others to pile more shit onto. I’m passionate about this; can you tell?

Don’t live with broken windows

9. Architectural Standards

What’s important here is that the code is going to feel like a consistent body of work, not some ad-hoc glom of a bunch of developers’ whims. Define what the standards need to be else you’ll always be discussing the merits of different styles.

10. Coding Standards

Again – decide on the coding standards for the application and stick to them!

Recovery

You are prepared for a disaster, right? Right? And if a disaster happens, how long’s it going to take to get back up and running?

Backing up your stuff somewhere is smart, but how long will it take to get it out of storage?

11. Restore time

I know you don’t want to think about the worst happening, but if it does, ask yourself how long should it take to get back up and running?

12. Backup time

Speaking of which, how long does it take to back your data?

It doesn’t need to be all doom and gloom

I know when I first put together this list, I felt a bit deflated at all the extra specification I was laying onto my magic development world. In truth though it’s not all that bad. In fact I’ve done the hard legwork for you in editing down the list. If you just think through these things early on in development, you can be pretty sure you are going to save yourself either under- or over-engineering your app, and be much more likely to produce decent quality software that you can be proud of.

In order to try and do this I decided to use as much of the code as possible that the FlexCover guys had written, and only change what I needed to get it working.Once I had it up and running I could worry about making it faster / leaner / more neatly coded…

The approach

So here’s the basic plan:

1. Swap out the default localConnection reporting mechanism for one that will pick and choose depending on environment the app is running in
2. Create a headless agent that will pull the code coverage data collection and reporting side of the coverageViewer into the app
3. Use commandline options to swap between the headless and localConnnection agent
4. Default to using the localConnnection

How it works

When your app compiles using the instrumented SDK created for using FlexCover, your application makes use of a class called CoverageManager. This manager is a patch that allows you to plug in a custom code coverage agent for use in your app. By default it uses a class called LocalConnectionCoverageAgent which broadcasts code coverage metadata to the coverage viewer application. What my patch does is get in and allow you to use a different, headless agent. To do this simply call during the preinitialize event of your main application:

private function injectAgent():void{
CoverageManager.agent = new CoverageAgentSwitch();
}

As a quick test, run your app as you had previously and confirm it still works by sending data over a localConnection to the CoverageViewer. This is expected as by default, the switch will create a LocalConnectionCoverageAgent. In order to use the headless agent you need to set a few commandline properties. First you need to tell the switch that you want to go headless. Then the headless agent needs to know where it’s getting its metadata from, where it should output the code coverage report. The commandline options are:

-coverage-agent-type=headless <em>Headless is only option, anything else will default to localConnectionAgent</em>
-coverage-metadata='/full/path/to/coverage/metadata.cvm'
-coverage-output='/full/path/to/coverage/reportInOriginalFormat.cvr'
-emma-report='/full/path/to/coverage/reportInEmmaFormat.xml'

Note that if you don’t specify the metadata path and at least one of the report formats, the headless agent will log errors but otherwise fail silently.
In terms of the edits I made to get it working, that is pretty much it. The headlessCoverageAgent swc is probably larger than it needs to be and is definitely grossly inefficient. I will update this soon to improve this, but right now I only have time to get this post up.
Obviously you will have to change your build script to pass these new commandline parameters in to the test harness when you run it. If there is any interest, I’ll post my updated build script and test harness modifications that dispense with the log parser altogether and make for a more repeatable build script.

The Result

My build process is now much quicker because I don’t have to wait to be sure that the coverage viewer has initialized. It’s also completely stable and the code coverage trend has been a great motivator for the team.

Code Coverage trend without crazy variance found using the external viewer
Download headless library

Update

A few of you have asked for a copy of the source code so you can play around with it and undoubtedly make it all work better for your needs. Attached here—with no warranty or support—is the headless agent project. In order to get it to work, you will need to have the CoverageAgent and CoverageUtilities projects in your workspace, as well as the modified CoverageViewer project from previous posts. Good luck with it – It’s code I haven’t looked at for a while so don’t hate on me for it :/