Figment Engine

Installing weighttp on AWS/EC2: ev library not found

2011-12-23T09:35:00.001Z

when installing weighttp on AWS EC2 using the Amazon Linux AMI you might get errors such as:

"ev library not found." when running ./waf configure or
libev.so.4 not being found

This series of steps fixed it for me:

Sutton with analytics

2011-11-19T21:23:00.001Z

Sutton: a website to test performance & seo has not reach #1 yet :-)

The next stage is to get more information about visits, before adding google analytics the page performed something like this:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.355s	0.321s	0.383s	31	0.355s	1	3 KB	0.597s	3	8 KB
Repeat View	0.279s	0.240s	0.259s	31	0.279s	1	3 KB	0.290s	2	3 KB

After adding it looked performed like this:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.364s	0.331s	0.393s	36	0.364s	2	16 KB	0.687s	5	22 KB
Repeat View	0.338s	0.225s	0.369s	36	0.338s	2	4 KB	0.498s	3	4 KB

Overall its looks ok (a few extra KB - hopefully that users have already loaded from visiting other sites), however the repeat view time has crept up by nearly 0.200s, something that is not too surprising with the need to force the client to fetch a non-cacheable tracking image (GET /__utm.gif?...).

Comparing S3 with EC2 micro instance with go

2011-09-01T10:26:00.001+01:00

Created the same page on both S3 and on an EC2 micro instance running linux & go.

Using Google's web master tools as a bench-marker: the S3 instance is for Croydon:

And on EC2 micro instance for New Malden:

Time spent downloading a page (in milliseconds)

Looks pretty good - this is the instance that bench-marked well.

All assets etc are the same, so should be a fair comparison, I'll give it a few days to get a good average.

Perfomance of Go as a web server on Amazon AWS

2011-08-30T18:14:00.000+01:00

Continuing the series on performance and exploring the Go language I wanted to see how many requests/second I could get out of an Amazon micro instance.

I used Siege for the testing of the "Sutton" server, and using the same code that gives us this sort of response:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.355s	0.321s	0.383s	31	0.355s	1	3 KB	0.597s	3	8 KB
Repeat View	0.279s	0.240s	0.259s	31	0.279s	1	3 KB	0.290s	2	3 KB

I also added some caching on the server side, to stop it regenerating the content every time.

Using Siege from a large instance (a small instance could not generate enough requests) I got the following results:

$ siege -c 40 -t 20s -d0 http://wimbledon.chart.is

Lifting the server siege... done.
Transactions: 28178 hits
Availability: 100.00 %
Elapsed time: 19.78 secs
Data transferred: 68.07 MB
Response time: 0.03 secs
Transaction rate: 1424.57 trans/sec
Throughput: 3.44 MB/sec
Concurrency: 37.69
Successful transactions: 28178
Failed transactions: 0
Longest transaction: 1.51
Shortest transaction: 0.00

So over 1,000 requests/second on the smallest AWS instance! the server is running at 100% CPU so some simple performance analysis could yield some significant improvements. Also some more effort around making the benchmark more trustworthy, especially with higher numbers of concurrent requests.

Hosting favicon.ico on a different server (Amazon S3)

2011-08-17T12:36:00.003+01:00

To increase the performance of our very simple Where is Croydon test site static assets are hosted on S3.

One of the assets that is requested for a site is favicon.ico the icon is used against the site name in browsers (normally to make spotting it among many tabs easier). By default this is requested from the root of the site, however this can be changed by adding:

to each page (head element) on the site.

To host it on S3 requires a bucket matching the domain name, and mapping a CNAME to point to the amazon end-point. In this case using a "static" sub-domain to host these kinds of static assets. Note: if you get "Access Denied" from these its probably because of permissions for "Everyone" to open/download are not set (per asset).

Doing this means the browser can use a separate connection to retrieve the icon, and therefore process the download in parallel with other content. Its worth considering multiple sub-domains if you have many static assets. This also has the benefit of reducing requests to your site's servers, and improving server-side cache hits as well.

During testing of this a little "feature" of chrome was discovered, it requests "/favicon.ico" from you site if you do a "view page source" - even if you specify a different location.

Google Ads Async (asynchronous)

2011-08-12T15:59:00.005+01:00

Making Google Ads asynchronous seems to be critical to achieving better performance, in this post in the continuing series on optimization of sites (Performance: the effect of google ads and Go Faster Stripes for site performance and google ads), a simple idea has some interesting implications.

The current ad code for Where is Croydon looks like this:

<script type="text/javascript">
</script>
<script type="text/javascript"
src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script>

And performs like this:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	1.939s	0.289s	0.426s	187	1.939s	12	52 KB	2.438s	14	53 KB
Repeat View	1.054s	0.228s	0.574s	182	1.054s	2	6 KB	1.054s	2	6 KB

Document complete in about 2 seconds is not good enough, (given our page has nothing on it), and the advert script is going to slow down any other asset downloads (by putting the browser into serial mode).

So how can we fix this? a bit of re-plumbing should fix this, our ad script becomes:

<script type="text/javascript">
</script>
<script type="text/javascript">
</script>

How does this work? Google's Ad script is quite "nasty" as it uses document.write, generally not a good idea as it can only be used during page creation. So this script does two things:

it dynamically inserts the Google ad script, allowing this to be separate from the main page render, hence the rest of the page can load without waiting for the adverts
it re-plumbs the document.write function to instead add the ads to a placeholder element. If this is not done then the Google code will try to call document.write and it won't output anything as the document is already complete. For completeness the code re-plumbs document.write after the first call, but really there should be no other code calling it.

Now this code is far from bullet proof, or tested (WOMM), but it shows the basic principle - it works only because its specific to the Google code, a more generic solution might need to follow this approach.

What does all this effort yield us:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.484s	0.332s	0.584s	191	0.484s	1	3 KB	2.353s	13	55 KB
Repeat View	0.347s	0.211s	0.611s	186	0.347s	1	0 KB	1.185s	2	6 KB

Fully loaded has not changed (as expected, we are still loading the same amount), but document complete is massively improved - from 2 seconds to less than half a second. Whilst this might seem to be moving numbers around, it means that the browser can render the rest of the page before adverts, and since adverts can be very very slow, this can make the site appear much faster.

And for those of you who do want to cheat performance numbers, then changing the "}), 1);" line to "}), 4000);" gives you this. A page that is "fully loaded" in 0.39 seconds - this is of course cheating, the adverts appear 4 seconds later, but https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/quick-start-quide#TOC-Fully-Loaded: looks for 2 seconds of no activity to judged fully loaded time.

Go Faster Stripes for site performance and google ads

2011-08-09T16:17:00.002+01:00

Carrying on our performance experiments with Amazon AWS and Go..

Repeating the same experiments, but comparing EC2 with S3.

S3 looked like this last time:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.518s	0.391s	0.000s	14	0.518s	1	1 KB	0.518s	1	1 KB
Repeat View	0.363s	0.237s	0.000s	14	0.363s	1	0 KB	0.363s	1	0 KB

Document complete in under half a second, with TTFB of 0.391s and document complete 0.518s

These look pretty respectable, but now compare with an EC2 instance (t1.large linux) using Google Go language as a HTTP server

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.342s	0.212s	0.000s	14	0.342s	1	1 KB	0.342s	1	1 KB
Repeat View	0.389s	0.224s	0.000s	14	0.389s	1	1 KB	0.389s	1	1 KB

Faster across the board - TTFB has halved to 0.212s and this is time saved off document complete and fully loaded. I was expecting that S3 would be very fast at serving static content, and its looks reasonable, but compared to this single server (large admittedly) its a fair % slower.

Now what happens with Google adverts? we get a similar google ads penalty to last time, 1.5s penalty on document complete and fully loaded:

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	1.582s	0.283s	0.411s	175	1.582s	10	51 KB	1.582s	10	51 KB
Repeat View	1.184s	0.210s	0.523s	175	1.184s	2	6 KB	1.184s	2	6 KB

Whats hidden in these numbers is a little gem - the ~0.200s saved using the EC2 instance is still there, i.e. this performs ~0.200s faster than S3 even when using Google Ads.

So even if you have a very heavy page, and are using CDN or other techniques to speed things up, you really need to see if you can get your TTFB down - and that may mean looking at async HTTP servers such as Go or Node.js

Here is the Go code for this example (the HTML is literal using back-quotes `), based on one of the Go examples:

package main

import (
"fmt"
"http"
)

func Hello(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w,
`
<!doctype html>
<html lang="en-GB">
<head>
<meta charset="utf-8">
<title>Croydon</title>
<meta name="description" content="Croydon">
</head>
<body>
<article>
<p>Croydon</p>
<p>Information on Croydon: shops, restaurants, property, schools, council &amp; government</p>
</article>
<script type="text/javascript">
</script>
<script type="text/javascript"
src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script>
</body>
</html>`)
}

func main() {
http.HandleFunc("/", Hello)
http.ListenAndServe(":8080", nil)
}

Note that the previous tests were rerun to ensure no significant variance in timings. The next set of test will be to see if we can keep sub-0.300s for template based content (as opposed to hard-coded HTML)

Don't use Google Mail to test email aliases: sent mails vanish

2011-08-01T12:18:00.003+01:00

If you setup a new email alias and want to test it then don't use Google Mail.

In order to save space gmail does not store duplicate messages, so messages sent to an alias that are redirected to your gmail account (via pop etc) simply vanish.

If you check the gmail logs (settings, accounts and import, view history) you can see it fetched the email, but then it simply discards it.

So its best to send the email using a separate client (such as swaks), this problem also appears with sub-domains

Performance: the effect of google ads

2011-07-29T16:09:00.002+01:00

I've created a test page (on the subject of croydon) that is hosted on Amazon's S3 as a static site. This is part of a series of tests I'm doing to see how best to use AWS and other technologies (mongoDB & Go), and how performance effects ranking (more on this in a later blog).

I'm expecting some delays due to the DNS setup, but the first test looks pretty good

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	0.518s	0.391s	0.000s	14	0.518s	1	1 KB	0.518s	1	1 KB
Repeat View	0.363s	0.237s	0.000s	14	0.363s	1	0 KB	0.363s	1	0 KB

Document complete in under half a second, and fully loaded about the same! - this shows how fast Amazon's AWS is for this kind of simple site (and it took minutes to set up as well)

Now looks what happens when we add some google JavaScript for the second test

					Document Complete			Fully Loaded
	Load Time	First Byte	Start Render	DOM Elements	Time	Requests	Bytes In	Time	Requests	Bytes In
First View	2.251s	0.382s	0.703s	173	2.251s	11	51 KB	2.251s	11	51 KB
Repeat View	1.046s	0.264s	0.552s	173	1.046s	2	6 KB	1.046s	2	6 KB

Document complete is now over 2 1/4 seconds, that seems a heavy price in performance to pay - and as a website owner there is little you can do about this, an irony as Google ranks faster performing sites higher.

XML for UML State diagrams

2011-04-28T09:35:00.001+01:00

W3C again...

"..Harel State Tables are a state machine notation that was developed by the mathematician David Harel [Harel and Politi] and is included in UML [UML 2.3]. They offer a clean and well-thought out semantics for sophisticated constructs such as a parallel states. They have been defined as a graphical specification language, however, and hence do not have an XML representation. The goal of this document is to combine Harel semantics with an XML syntax that is a logical extension of CCXML's state and event notation.."

State Chart XML (SCXML): State Machine Notation for Control Abstraction

I like the microwave example:

<scxml datamodel="ecmascript" initial="off" version="1.0" xmlns="http://www.w3.org/2005/07/scxml">

  
  <datamodel>
    <data expr="5" id="cook_time">
    <data expr="true" id="door_closed">
    <data expr="0" id="timer">
  </data></data></data></datamodel>

  <state id="off">
    
    <transition event="turn.on" target="on">
  </transition></state>

  <state id="on">
    <initial>
        <transition target="idle">
    </transition></initial>
    

    <transition event="turn.off" target="off">
    <transition cond="timer >= cook_time" target="off">

    <state id="idle">
      
      <transition cond="door_closed" target="cooking">
      <transition event="door.close" target="cooking">
        <assign expr="true" location="door_closed">
        
      </assign></transition>
    </transition></state>

    <state id="cooking">
      <transition event="door.open" target="idle">
        <assign expr="false" location="door_closed">
      </assign></transition>

      
      <transition event="time">
        <assign expr="timer + 1" location="timer">
      </assign></transition>
    </state>

  </transition></transition></state>

</scxml>

The Ubiquitous W3C: Battery standards

2011-04-28T09:16:00.001+01:00

Batteries? wow!

I saw this release from the W3C and it made me sit up:

"The Device APIs and Policy Working Group has published the First Public Working Draft of Battery Status Event Specification. This specification provides tools for developers to access device battery status (charge level, time remaining, etc.) within Web Applications. This specification defines a new DOM event type that provides information about the battery status of the hosting device and associated auxiliary devices. Learn more about theUbiquitous Web Applications Activity."

why? because I thought - why is the W3C publishing standards about batteries at all? W3C stands for "World Wide Web Consortium", yet here we are looking at how to monitor battery status within a web page.

it just show you how Ubiquitous they are trying to be!

OCD

2011-04-15T16:55:00.000+01:00

love these

http://www.adaptcreative.co.uk/2010/08/mental-disorder-posters/

Print before reading

2011-04-15T16:51:00.000+01:00

especially when you print an email out, and you get an extra page with "please think about the environment before you print this..."

I wonder how much energy is waster by sending these extra bytes around the internet...

Spare us the e-mail yada-yada

http://www.economist.com/node/18529895
“IF THIS e-mail is received in error, notify the sender immediately.” “This e-mail does not create an attorney-client relationship.” “Any tax advice in this e-mail is not intended to be used for the purpose of avoiding penalt...

If it's complex it can't work until it's empty

2011-04-15T14:11:00.000+01:00

Amazing quote

An aside, what made all these things work so well is that they were empty inside. Almost skeletal. Hard to believe there isn't more to it. I asked one of my mentors how this could be and he said it has to be that way. If it's complex it can't work until it's empty. These days we have another way to describe this, my friend and former colleague David Weinberger called it Small Pieces, Loosely Joined. I've never heard a better description of the architecture of the Internet.

http://scripting.com/stories/2011/04/15/letsBuildANewInternetInAca.html#p6247

must be a line from blade runner...

Cloud Providers the new war: It Depends

2011-04-15T09:29:00.000+01:00

Some interesting reading about rackspace vs AWS in terms of price/performance

http://gasoftwareblog.com/home/2011/3/31/amazon-ec2-vs-rackspace-are-all-cloud-hosting-services-the-s.html

http://scripting.com/stories/2011/04/14/rackspaceBeatsEc2ByALot.html

http://news.ycombinator.com/item?id=2448812

reading the hacker news thread I can see that beginning of a new religion - which cloud provider do you use? its going to be one of those windows/linux, firefox/IE wars... and ultimately there no real way to say which one is best, as it depends on what you need.

I moved my hosting from a dedicated server (windows) to a AWS micro instance (linux fedora). Its only been running for two weeks, but its only cost $10 compared to $50 - of course I had a physically dedicated machine before, but the micro instance seems happy enough running email (postfix). I'll write a post about setting it up later. So the cloud will save me $1000/year - not bad!

Connected world irony

2011-04-14T17:45:00.002+01:00

Before the Internet individual computers were used to accomplish general tasks: such as word processing; spreadsheets etc.

The Internet allowed computers to be massively interconnected, allowing a user to access many other computer's resources.

Now the Cloud is moving these basic tasks to work on the Web, allowing users to work on their documents etc from any computer.

Image: jscreationzs / FreeDigitalPhotos.net

The irony of the Cloud is that an unconnected computer becomes useless, as all documents etc are in the cloud. The advent of the cloud has turned a useful general purpose tool into a type-writer with no ribbon.

This train of thought was caused my me being in Ireland, in the country side with no Internet access or mobile signal. Happily I was writing code (64bit assembly on Fedora), and I suddenly thought:

"are programmers the only people have a use for a disconnected computer any-more?"

I'm backing this watch

2011-02-05T14:55:00.001Z

this seems so smart...
http://kck.st/hyrZpz

New blogger app for android

2011-02-05T14:49:00.001Z

my first post using my phone. The main limiting factor seems to be that insert picture does not allow a web address, which is a bit daft.

stating the (un)obvious

2011-01-09T17:30:00.001Z

Red Gate

Learning some more languages...

2010-12-06T12:55:00.000Z

going to try this:

Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages (Pragmatic Programmers)

Bog Standard

2010-12-06T12:45:00.002Z

I've been having a rethink about my core skills recently - looking at my resume/CV I see that I have spent a long time in the windows camp (10 years+).

So I've decided to jump to the Unix (Linux) camp again for a while.

Why?

My main reason is that I don't buy the proprietary platform story anymore - I don't see enough value add. I'm not happy with the constraints, and the monolithic nature of the solution space.

But more than anything, I don't like being on one side of the fence - I don't want to be limited by a particular view point, or a fan boy for linux or windows.

So I'm moving from:

Windows 7 to Fedora 14
C# to C++, especially C++0x

or Node.js
or anything that looks interesting
and maybe something like ruby

Silverlight to HTML 5

I want to be more standards based, and less proprietary, hence bog standard..

Its been a while since I did anything on Unix, so I expect it will be painful, but as I type this in firefox running under fedora I feel I'm investing in my knowledge portfolio...

Surely You Must Be Listening to Mr. Feynman!

2010-09-16T20:41:00.000+01:00

OMG - Can't wait....

The Feynman Variations

if you can't then watch:

BBC Archive > Richard Feynman

The Pragmatic Capability Model Of Software Delivery (PCM-OSD)

2010-08-15T17:01:00.004+01:00

The “Agile” vs “Formal” delivery discussion is well underway – with many interesting posts on the subject (such as The Agile Flywheel).

I was interested in this discussion because I needed to answer the question:

“How good are we at delivering software products?”

The reason for asking the question was a “platform” that we had created, a platform in the sense that other teams were expected to build applications on top of it. If we were expecting other teams to build on top of it we needed to ensure that it was “good quality”, that it would provide a firm foundation for their work. Also as it was a platform with many teams using a shared infrastructure, applications built on top of the platform had to be of “good quality” as well, otherwise they would degrade the platform for others.

Of course it’s hard to quantitatively define “good quality”, we can all recognised a good product like, but it’s harder to quantify all the attributes and processes that go into making a good product. It’s even harder to definitively show that a particular product is good enough against the nebulous label of “good quality”.

Unfortunately that’s exactly what we needed to do.

Creating a model

The approach taken was to create a model that captured the definition of “good quality”, which turns out to be a measure of how well a particular team delivers software products.

There are many metrics to measure individual activities, but what was needed was a holistic cross-functional measure – something that could cover the following areas:

Development: how we work with code
Building: how we turn code into deployable artefacts
Deploying: how we deploy into environments
Testing: how do we ensure that the deployment is good
Reporting: how we drive transparency (for quality)
Support: how we manage incidents and requests
Product Management: how we understand & communicate the product needs
Architecture: how we manage our technical direction
Operations: how we manage the run-time environments

Ultimately the model could have been created using different functional areas (for example we did not include Operations in this version), and the expectations are just opinion. The model presented here is easy to change and should be adaptable by others, for example tweaking to match local concerns.

The model is a mash-up of Agile & Formal product delivery views:

Agile: how to do software development, Scrum focuses within that on how to project manage and handle requirements, DevOps looks at the deployment and infrastructure aspects.
Formal: CMMI (Capability Maturity Model Integration) and ITIL (Information Technology Infrastructure Library) look at process improvement and IT Management.

Our model takes the “Maturity” concept from the formal stream, and a “Pragmatic Capability” from the agile stream. We did not want a model that was about a team’s ability to improve (e.g. continuous improvement), we wanted a snapshot of the capability of the team in a functional area. It was also heavily informed by the work we were doing with Thought Works on testing and CI, and by the highly recommended urban{code}’s anthillpro whitepaper:

Enterprise Continuous Integration Maturity Model

Combining these concepts and streams together gives us the “Pragmatic Capability Model Of Software Delivery” (PCM-OSD).

The implementation

The model was implemented in Excel and consists of the a spreadsheet with three tabs:

Model Definition & Product Preview

This first worksheet “Model” defines the model (in terms of functional areas and attributes) and allows you to select a particular project and see what the maturity results are for each attribute in the functional areas. The Project Rating ranges from “F” to “A++”, this stops us trying to present precise % which would probably cause more arguments (an idea taken from thought works)

Assement data collection

The second worksheet “Projects” is used to collect the data from the maturity assessment. Each Project has a column that the data can be collected against, the list of projects at the top drives the drop-down on the “Model” worksheet.

The calculation is very simple, each attribute is given a score of 0-5, and the % maturity for the functional area is calculated by adding all the attribute scores up and taking that as a % of the maximum score possible for that area. Note this calculation is therefore very dependent on how many attributes are in each area, and how you group into functional areas. Finally these % are mapped to a rating (F to A++)

Reporting – benchmark projects

The final worksheet “Reports” is used to visualize the results. We used the Radar chart (a type of polar chart) to visualize the data.

Applying the model

In order to use it for our projects we created two benchmarks:

A “baseline” project, that assumes the lowest level of maturity that is acceptable
A “Agile Platform” project, that assumes good practice, not not to the “insane level” (see anthillpro’s whitepaper)

By creating these two benchmarks we can show our projects relatively by overlaying them:

Visualization of Maturity: Our platform before any improvement work

This first shows how our platform looked before we tried to improve it – the red peeking though shows us areas we have concerns that need addressing.

Visualization of Maturity: the platform after targeted improvement

And this diagram shows how our project looked after we improved our practices, no more red shows so have reached a base level of maturity, and we have made substantial steps towards become an “agile platform”.

Assessment process

We carried out this assessment on a 10 projects, and the diagrams produced coincided with our intuition of the teams maturity. We carried out the assessments by interviewing the technical leads on the projects (about one hour), and the information can be used to allow us to identify future training needs and capital expenditure.

During the assessment process the focus was on ensuring the teams understood the purpose of the process, and our preference to give a lower score for an attribute rather than over estimate the maturity of a team. For attributes that were not applicable to a particular team (for example internal facing only teams) the approach was to give a 5 for the maturity rather than penalize due to non-applicability. For the teams the opportunity to identify training opportunities, mentoring or improved working was the main benefit. In the future a quarterly review might allow teams to see where they should target their continuous improvement efforts.

Our opinionated model

The model is based on our views of what’s important, and the maturity levels defined by our views. Applying this model unchanged is arguable, so its worth understanding our reasoning behind the attributes, so they can be changed to match your views.

The model is documented in the same structure of the Excel Workbook, in terms of Areas with Attributes, each which has a description/benefit statement (what benefit you see from increasing this attribute)

Development: how we work with code

Check in frequency: as you move to a CI process the frequency of check in drives how quickly problems are found.
Behaviour: how responsible do developers act about ensuring the code is good, follows the broken window theory
Int. Documentation: what kind of documentation is available internally – not quantity, more quality. Preferably automated and easy to maintain.
Ext. Documentation: what kind of documentation is available internally – not quantity, more quality
Feedback loop: how long before a developer gets feedback on the quality implementation for a user story. This reflects the efficiency of tools, process and project. For example having too many dependencies may make build time too long. This feedback must have value - so tests and environments like live.
Code Management: coping with the need for multiple developers to work on the same code at the same time, using source control effectively allows parallelism and root cause tracking.
Source control: how do we use source control, making source control the start of an quality process rather than just a repository.

Building: how we turn code into deployable artefacts

Build: how easy is it to create a new build, making this quick and painless reduces friction to introducing CI and other automated tools
Binaries: how to manage the output of a build, is it versioned and tracked – and do we test the same artefact through all environments up to production
Configuration: often ignored it can turn a well designed system into an un-deployable mess, ultimately should be automated and by convention over config as much as possible

Deploying: how we deploy into environments

Deployment: how we make a release live, making this painless and automated reduces friction to testing and rapid evolution
Resource: what skills are needed to deploy, having special skills constraints becomes a blocker, ultimately it should be the final output of CI
Database: repositories and databases need to be deployable as easily as the application
Environment: having well defined standard environments with automation allows for creation of test environments and additional live capacity automatically

Testing: how do we ensure that the deployment is good

Code Quality: how do we ensure that code is good, pair programming and automated tools help, and toxicity reports can drive technical debt reduction and help in adjusting estimates for changing code in toxic areas.
Manual test: how is manual testing carried out, having some targeted smoke/sanity tests can quickly discover defects, user confirmations for stories allows dev to tests themselves.
Unit test: how do we do unit testing, using TDD improves the design, and increased coverage improves the ability to detect regressions earlier –whilst there is no defined lower limited for coverage, 60% seems to be a reasonable target for most teams. This should be higher for critical systems (and platform).
Acceptance test: how are acceptance tests carried out, these normally start out manual, but moving to automation allows cheaper tests cycles, and frees up testers to do targeted manual tests
Performance: often left until after a release causes performance issues (loss in revenue or increased costs), doing it earlier is cheaper (than fixing after) and can be ROI based (http://www.phpied.com/the-performance-business-pitch/)
Security: testing the security of the application can be automated to detect the main problem areas (OWASP top 10), and ultimately ovoid creating weaknesses in the first place.
Automation: moving from highly skilled resources running low-value manual processes, to automating as much as possible and targeting manual work where is leverages skill sets most

Reporting: how we drive transparency (for quality)

Visibility: increasing transparency drives the team to higher quality, and allows cross project comparison & learning
Traceability: being able to track why change occur, from end user feature to code, config and environment changes allows quicker fixing and trend detection
Integration: how well integrated is information sharing within the project team, is there one information repository or unconnected datasets (bugs not related to builds for example)
Defects: how defects are tracked and related to changes in the system enables better investigation and reporting

Support: how we manage incidents and requests

Support process definition: do we have a defined process for managing a incident, is it reviewed for effectiveness?
Support structure: what resource is allocated to providing support, having designated support reduces interrupting people unnecessarily & increases velocity.
Triage: how do we decide the priority to assign problems, and the resolution time. Ensuring that there is agreement to priorities across the whole team (especially with the business) ensure focus on what matters.
Self-service support: how do external teams/users get support for themselves (forums/wiki), can the community improve the knowledge base
Support SLA: do we have publicised goals, do we have processes for managing exceptions and ensuring we monitor compliance

Product Management: understanding & communicate the product needs

Vision: understanding the direction of the product and aligning with technology changes
Requirements: gathering new features and change requests, communicating stories in an actionable and testable way, prioritising by value. Better quality stories reduce confusion and increase velocity.
Documentation: gather and report in a consistent way, understand all the documents other roles create (dev & arch esp.)
Communications: communicating to internal & extern teams and customers. Is everyone aware of the product vision and changes etc
Co-Ordination: how does product management integrate with dev, architecture and project management office – is it aligned, reducing confusion and arguments between the different functions.
Delivery: moving from ad hoc delivery to repeatable delivery with selection of stories into releases rather than holding up releases for stories

Architecture: how we manage our technical direction

Documentation (A): ensuring we have light-weight documentation and diagrams that aid continual product evolution
Technical Debt management: keeping track of decisions that reduce velocity or options for future delivery, proactive planning to remove over time
Technical Risk management: keeping track of technical risks to the project in the existing solution, understand and communicate the impact of risks and costs to fix them.
Vision (A): understanding to technology vision for the product, aligning it with the product road map

Operations: how we manage the run-time environments

this area was exclude from our model as we are still working out the details of the attributes that would best capture capability levels

Next steps

The next phase is to complete the assessments for all the teams, and see what proactive action we can take to improve each team. Reports that are worth considering:

tracking the change in a team over time,
understanding the gaps in external supplier teams,
and looking at a set of teams in aggregate to find repeating patterns..

One of the unsurprising findings from the process is that is a lot easier to introduce best practice at the start of a project, and this model allows the make and processes of a team to be tuned before starting. It would be interesting to see what others think of this approach, where they think there are weaknesses, and if they try to apply it any issues or changes they made.

The latest version of the Pragmatic Capability Model Of Software Delivery spreadsheet uses generated data (RANDBETWEEN function), just replace this with your values.

Web Performance & Dev Ops

2010-07-15T22:14:00.000+01:00

A few links that are relevant to web performance, how to improve it and its effect on the bottom line:

And a book that has some good hints on rethinking the approach to building scalable architectecure for the web:
Scalable Internet Architectures (Developer's Library)

Well worth looking into Dev Ops as well

Growing Object-Oriented Software Guided by Tests

2010-07-07T13:23:00.001+01:00

I've just finished reading Growing Object-Oriented Software, Guided by Tests (Beck Signature) it is without doubt a very good book (I'd go so far to say the best book I have read this year).

Understanding how to do agile development well is not something you learn and then are finished with, it is something that you can always do better...

The first part of the book is about the fundamentals, and the rest is then applying to an example project. This involves creating user stories, the high level architecture and writing the acceptance, unit test and finally the functioning code - all at one vertical slice at a time. By this I mean that they show how you can evolve the architecture and code during the development process - something that is hard to understand without a worked example.

Going back over the fundamentals, with the clarity of these authors really taught me a lot. And the example allowed you to see what thay mean. The book is short and the diagrams nicely done - well recommend this one.