Planet TW

Ross Pettit: Introducing alphaITjournal.com

noreply@blogger.com (Ross Pettit) — Thu, 24 Jul 2008 04:03:22 +0000

I'm pleased to announce the launch of alphaITjournal.com, an online magazine focused on the execution, management and governance of IT investments that can produce outsized (or "alpha") returns. The mission and purpose are summarised in the welcome message on the site and in the press release that was issued in early July.

There are a few things that I hope stand out about the site.

The first is the site layout. It's designed to give attention to writers and their articles, and make the content easy for readers to navigate without being overwhelmed by a polyglot of messages.

Another is the absence of advertising. Aside from the "Presented by ThoughtWorks" message on the left navigation and bottom menu, there is no advertising on alphaITjournal.com. Being practitioners, this affords us flexibility in dealing with changing project demands and work priorities that will affect content production and editing.

Still another is the continuous release of content. Rather than having monthly editions, there will be one or two articles released each week. This will make it easier for the reader to stay current, and it will also make it easier to sustain fresh content on the site.

Last but certainly not least is the diverse community of writers. While ThoughtWorks is sponsoring the site, the community of writers are from all corners the IT universe. They, in turn, are producing content on a diverse collection of topics, all with a common theme: how to maximise returns on IT investments.

I hope that alphaITjournal.com consistently provides compelling content so that you'll be a regular reader, even a promoter: add it to your RSS news reader, share it with your peers and customers, add a link to it from your blog. I also hope that you'll consider being a contributor. We have a number of writers, but we are always looking for more. If you have ideas for individual articles, a series or a column, drop me an email.

We've just gone live, so there are a few additions we'll make once we establish our rhythm (such as reader comments). Meanwhile, if you haven't done so already please give alphaITjournal.com a visit.

Neal Ford: Rich Web Experience 2008 x 2

noreply@blogger.com (Neal Ford) — Wed, 23 Jul 2008 16:22:40 +0000

Last year, one of the most refreshing conferences where I spoke was the Rich Web Experience. Too often, conferences are mono-cultures, and mono-cultures are generally boring. RWE is fundamentally different because it's what I call a condiment conference. The focus is on how to make applications better, not how to make the plumbing better. Nothing wrong with plumbing, mind you, it's just refreshing to look at faucets for a change.

This conference has an eclectic mix of developers. Hallway conversations lack the implicit assumptions you can generally make at main course conferences. For example, all Java developers have an implicit context. At The Rich Web Experience, you have to throw away your base assumptions, both in sessions and conversations. Just like travel broadens you because you meet people with different contexts and experiences, attending the Rich Web Experience does the same for technologists. Instead of the usual low-level animosity that each technology tribe exhibits for the non-tribe members, everyone focuses on common ground. This year, there are 2: one near the east coast and another on the west.

Agile Hong Kong: Build Jamming

Wed, 23 Jul 2008 15:59:54 +0000

Builds are a fundamental part of any project. They are the bedrock of a project. Done well they make development and maintenance of a project easier. Done poorly and they can increase the time it takes to develop and maintain a project and even make development difficult. A well structured build will even help reduce defects by making continuous integration easier and by ensuring tests are run during each integration cycle. A good build will also aid in the deployment of a system.

In our next Jam we’ll look at build systems. Identify what makes a good build system, looking at how they might be structured for best practice. We’ll also look at various tools that support builds. Specifically we’ll look at tools that support continuous integration, prevent code duplication, bug detection and style checking amongst other tools. This Jam will include both a theoretical and a practical part. Some of the practices introduced in this session will form the basis to some future code jams.

So, dust your laptops off and come along. If you don’t have a laptop (or it doesn’t work) then come along anyway, we’ll work in pairs so there should be every opportunity for you to team up with someone. Don’t forget your laptop charger!

Drinks will be provided.

When: 7:30pm, Tuesday 29th of July 2008

Where: ThoughtWorks Hong Kong Office

Address: Room 1304, 13/F, Tai Tung Building, 8 Fleming Road, Wanchai

Map: ThoughtWorks Hong Kong

Jay Fields: Move Mail.app RSS Feeds to NetNewsWire

noreply@blogger.com (Jay Fields) — Wed, 23 Jul 2008 11:30:00 +0000

Like Dave Thomas, I recently decided to give Mail.app a try as an RSS reader. Overall the experience hasn't been great, but it wasn't poor enough that I thought the effort to switch was justified.

And then I decided to quit my job*. Next week will by my last week with ThoughtWorks and I'll need to turn in my laptop. I tend to store as much as possible online so this type of situation isn't a big deal, but going with Mail.app seems to have been a decision in the wrong direction. Maybe Mail.app stores my RSS feeds online somewhere, but it wasn't immediately obvious to me.

A quick google search revealed a Mac OSX Hint on how to dump your RSS feeds to the command line. This was quite helpful, but what I really needed was an OPML file that I could send to NetNewsWire.

I wrote the following script for anyone that finds themselves in the same situation. This script will grab all the feeds and write them to an OPML file. I'm sure it can be cleaned up or done more easily, but this works.

feeds = %x[IFS=$'\n';for i in $(find ~/Library/Mail/RSS/ -name "Info.plist");do grep "<string>http://" $i | sed "s/.*\(http[^<]*\).*/\1/";done].split("\n")

feeds = feeds.collect { |element| element.gsub(/<string>/, "<outline xmlUrl=\"") }

feeds = feeds.collect { |element| element.gsub(/<\/string>/, "\"/>") }

xml = <<-eos

<?xml version="1.0" encoding="UTF-8"?>

<opml version="1.1">

  <head>

    <title>mySubscriptions</title>

    </head>

  <body>

#{feeds.join("\n")}

  </body>

</opml>

eos

File.open("mail_app_export.ompl", "w") { |file| file << xml }

note: This file works in NetNewsWire, but I had better success by logging on to newsgator.com and importing on their website. When you import on their website it pulls the feed names and other information. The NetNewsWire didn't seem to pull the additional information.

*For those interested, I'm joining DRW Trading.

Chris Leishman: Defining Throughput

noreply@blogger.com (Chris Leishman) — Wed, 23 Jul 2008 11:22:08 +0000

In this second part to my series on Throughput Accounting, I'm going to explore the concept of Throughput, starting with a definition.

Every system has a goal it is trying to achieve. For most business processes this goal is revenue - money given to the organisation by customers due to sales of product or services. Given this, 'Throughput' is simply defined as a measure of the rate at which a system is achieving its goal. It is not a measure of profit or any indicator of efficiency, it is just a measure of flow.

Throughput is often expressed in terms of 'goal units'. In for-profit organisations, these units may be dollars of revenue. For non-profit organisations it may be expressed by another measure, such as patients successfully treated in a hospital emergency ward.

When dealing with monetary goal units, Throughput (T) is often measured as gross revenue from sale of a product (S) minus totally variable costs (TVC), which are only the costs that vary directly with the system output. In a manufacturing setting, TVC is often just the cost of raw materials. Thus T = S - TVC.

In the example from my last post, a company was producing 800 widgets per month. Their TVC, for materials, was $40 per widget. If those widgets were being sold at $50, the throughput per widget would be $10 and the system throughput is $8000 per month, or around $0.18 per minute.

In calculation, this is identical to the cost accounting metric 'contribution margin'. The contribution margin is used to determine the fraction of sales that contributes to offsetting fixed costs and can be used to disambiguate some decisions, such as the purchase price example from my previous post. As Throughput Accounting manages fixed costs separately, it is not required for this purpose - but has a much broader role.

It is important to note that Throughput is measured only as the ultimate achievement of a system, for example it may be the final sale of a product or service. It is not measured in terms of production of assets that will then be processed further to achieve the ultimate goal (eg. assets sold to achieve revenue). This is inventory - which I will discuss in a later post.

A good question is how to define Throughput in the context of a software delivery engagement. Ultimately the software system supports a business process which is in turn producing revenue - so it could be defined in terms of dollars made by the business. Pragmatically, however, it is more useful to define Throughput as the rate at which software additions and/or modifications are delivered into an environment where they can be utilised by the business process.

It is tempting to measure this rate in terms of 'business benefit' (possibly also in dollars), which is the ultimate goal unit of the entire business process. This is an important thing to focus on as often the best improvements can involve organisational changes as well as software changes. However, when the software production process is the organisation's constraint (as it often is) then we are focussed on just this process alone. In which case we need to use a 'goal unit' that is reflective of the amount of the work done to produce some change in software.

In Agile software delivery methodologies there is the concept of 'story point', which is simply a relative measure of the effort required to deliver a particular requirement. Given its relative nature, we can measure it over an arbitrary time period to determine the rate of delivery of story points, which we can use as Throughput. This is effectively the same as the Agile term 'velocity' and this comparison is useful for applying some TA concepts to software delivery. I'll return to this point in future posts...

Ram - Sriram Narayanan: The Reply-To Header

Tue, 22 Jul 2008 19:41:27 +0000

If you receive a mail on the Belenix-discuss list and you hit reply, your reply goes to the author of that mail. I never did like this because I thought it didn't help build up content and knowledge on the list, and that this didn't show traffic on the list either.

Then I was pointed to the following URL, which reasons why the default behaviour should be "Sent to author", and why Reply-TO mungling actually gets in the way.

http://www.unicom.com/pw/reply-to-harmful.html

I quite agree with most of the reasoning - enough that I am convinced and am starting to train myself to hit Reply to All.

Carlos Villela: Git Iterator

Tue, 22 Jul 2008 18:11:09 +0000

I wanted to generate some visualizations of our project’s growth, so I decided to put together a little shell script that looked at the output from git log to spit out some metrics.

So git-iterate was born: run anything through your entire project’s history, and get the results in something easily converted into a beautiful chart!

It does that by running git-reset --hard $COMMIT for every commit in the repository, and then calling the script given to it as the first argument. It passes the commit ID to the script too, so this:


git-iterate echo

…will generate a list of all your commit IDs, most recent last.

The code is on GitHub, as usual. I’m running a few stats on some projects I have access to, and will upload a few charts as soon as they’re ready. Meanwhile, feel free to send me the output of this:


git-iterate 'echo `flog app` `flog spec`' # if needed, replace "spec" for "test"

…and I’ll chart those for comparison as well. Also, it shouldn’t be difficult to port git-iterate to other source control systems (all of them have a checkout command, right?) and, if you do that, make sure to plug it in the comments.

Have fun!

Update: fresh off the oven, here’s Rails’ total lines of code. Neat, huh?

Chris Leishman: Throughput Accounting

noreply@blogger.com (Chris Leishman) — Tue, 22 Jul 2008 16:48:26 +0000

Let me declare today that I have a new enemy - Cost Accounting.

In 1981, at a conference of Management Accountants, Eliyahu Goldratt declared Cost Accounting to be "Enemy #1 of Productivity". In my work as a consultant for ThoughtWorks, I value productivity highly. Hence the enemy of my friend is also my enemy.

Goldratt, author of the popular business novel "The Goal", showed how traditional cost accounting easily and trivially leads intelligent managers and other decision makers to make bad decisions - decisions that have massive negative effects on performance and profitability. This was actually not surprising news, as the problems with cost accounting had been recognised. But Goldratt also proposed an alternative.

His alternative was titled "Throughput Accounting", and was described in the 1998 book by Thomas Corbett. It is an accounting model which does not focus primarily on local cost optimisation and instead focuses primarily on overall system throughput. Importantly, it does not attempt to allocate overhead costs across products and services.

Why is this important? Cost accounting attempts to allocate a companies fixed costs over a given time period to the products or services produced in that period. It enables managers to ignore the fixed costs and focus instead on the results of each period in relation to the products produced. This made sense in the 1890s, when cost accounting was developed, as most companies had extremely low fixed costs. The majority of costs were labour, which at the time was fully variable, as workers were only paid for the hours they laboured at the demand of the employer. Although there was distortion due to the overhead allocation, it was extremely small.

Modern businesses, however, have a much lower proportion of fully variable costs. Most costs are fixed, especially labour as now most employees work fixed hours regardless of demand. Due to this environmental change, the distortion caused by cost allocation is much larger and can cause management decisions, based on cost accounting, that directly result in large loses of productivity and profit.

As an example, imagine a company has fixed overheads of $10,000 (plant and labour) per month and currently produces 800 widgets with a materials cost of $40. The unit cost per widget would be $52.50 ($40 + $10,000/800). If an order now came for an additional 500 widgets at a price of $50, then management based on cost accounting should reject the order as it would result in a loss of $2.50 per widget. However, as the majority of costs are fixed, the increase in production would actually result in a new unit cost of $47.69 ($40 + $10,000/1300), so profits would actually increase by $1,153.85. This might seem clear, but imagine this example with many different products and prices in the mix and quickly it becomes hard to see the mistake. This is only one example of a poor decision made due to a cost accounting metric.

So why am I so concerned? In the software industry, especially, there is very little variable costs. The overwhelming expense is for developers, which are typically in short supply, so they are not a variable cost. Basing our decisions around software projects and the ROI from software delivery on cost accounting can lead to incredibly poor outcomes.

In future blog posts, I'll also explore what I believe is another critically important aspect of Throughput Accounting - the focus on throughput and inventory reduction over costs. I believe this has the potential for a massive effect on the software delivery process, especially when combined with the Lean practices that ThoughtWorks is pursuing in the industry.

Believe me when I say this - Throughput Accounting will become of major importance in our industry. It will make a dramatic change to the way we finance software projects and it has the power to put real numbers around why we use Agile and Lean principles and practices (as shown in David Andersons book "Agile Management for Software Engineering").

If your interested in another good introductory text, in addition to Corbetts book, try the book by Steven Bratt, also titled "Throughput Accounting".

Marc McNeill: The scourge of Document Driven Design

Tue, 22 Jul 2008 15:18:52 +0000

Documents, or rather words are the scourge of product design. Because words can never convey the true meaning or emotion of what is really required. All to often, software development projects are driven by the documentation - agile projects can be equally guilty of this- driven by words on paper (or card) that convey what the requirement is. Issue #1. Developers don’t read!

Except for a few exceptions, software is all about a user interacting with a system in order to accomplish a goal. Trying to describe that premise in terms of features and functionality is fundamentally flawed as it will be nested in the language of technical implementation, not in the user interaction. “Find” becomes “search”, “buy” becomes “shopping cart”, “check” becomes “validation” and so on. Issue #2. The desired outcomes become lost in a smog of technical jargon.

Solution: Picture Driven Design. Up front. Yeah! I’m all for up front design! A picture tells a thousand words. It has the power to remove ambiguities, clarify the vision, showing what the journey to realise outcomes look like. Start with a day in the life of… scribble out the flow, the user journey. Nothing complicated, boxes and arrows. Then scribble out what the user interface might look like. Done. That’s your up front design. That’s your documentation. That’s your scope. What you do next is up to you, write loads of documents that describe it and produce the software in a waterfall way if you want. I’d prefer you were more lean and adopted agile practices, but whatever you do, start with the picture. I’m convinced it will save much pain later on.

Jay Fields: ActionScript: Literal XML

noreply@blogger.com (Jay Fields) — Tue, 22 Jul 2008 12:00:00 +0000

One feature of ActionScript I really like is the ability to use XML within the language. In ActionScript there's no need to use strings to represent XML, you can use XML inline just as you would any other literal (numbers, strings, etc).

Recently I was working on some dynamic code that needed to update based on an XML response. We were just getting started on the card, so instead of worrying about the service, how it was triggered, etc, we added a button to the interface that called the response parsing method and passed in an XML literal as an argument. The code is similar to what's shown below.

parseResponse(<user><firstName>Jay</firstName></user>);

Sure, we had to change the code later and make a real service call, but this quick solution let us keep focusing on the task at hand instead of how to get the XML. In a language like Ruby we could have used a builder or just a string, and given such a small amount of XML it would have been fine. However, the actual XML we were working with was significantly larger than the example and would have been a decent mess of a multiline string or several builder calls. Being able to write XML natively was significantly easier.

The beauty of literal XML is the simplicity. I don't have to represent it in any way other than what it actually is -- XML.

If you aren't a fan of all the angle brackets, that's okay, ActionScript has you covered there also. You can add elements and attributes as you would expect to be able to if you prefer method calls.

var request:XML = <smart_list/>;

request.sort.order = "highest";

request.sort.field = "Average Position";

request.max_results = 10;

request.toXMLString(); // "<smart_list><sort><order>highest</order><field>Average Position</field></sort><max_results>10</max_results></smart_list>"

If you only ever use XML similar to the above syntax, then there may be little value in literal XML, but I don't think you're usage would be limited to the above syntax.

I write tests, a lot of them. I prefer to see the actual XML in my tests, instead of the builder versions of XML. I don't like angle brackets any more than the next programmer, but I strongly prefer testing with expected literals. I find my resulting tests to be more readable and reliable. If you've ever had a test fail because of whitespace issues in your XML, you should know what I mean by reliable.

Literal XML also ensures XML compliance at compile time and encourages IDEs to provide syntax highlighting. Two things that aren't essential, but are definitely nice to have.

ActionScript is the first language I've used with literal XML support, and I'm very happy with the experience. As programmers we spend a lot of time hiding other languages with object relational mappers (orm) and builders, but literal XML is a refreshing step in the other direction. The creators of ActionScript have embraced the fact that XML isn't going anywhere. They built first class support for XML in the language itself instead of hiding the problem with a framework, and that helps me significantly more than any orm or builder ever has.

Ian Cartwright: Marketing Driven Development

noreply@blogger.com (Ian Cartwright) — Tue, 22 Jul 2008 11:25:47 +0000

Marketing Driven Development is a process where the features and priorities for a new release of software are driven primarily by a marketing department. So a feature is valued more for the ability to create a marketing campaign around it over any usefulness or value that it may have for an end user.

Marketing Driven Development is primarily a reaction to the fear of commoditization.

It's consequences often include

Well known bugs (or to use the correct term: sub-optimal feature) remain for many versions as fixing them is never a priority
Usability does not improve as this is rarely seen as valuable by marketing and anyway it was sold as easy to use the first time around
It tends to be expensive to buy, maintain and customize i.e. marketing driven development does seem to allow the avoidance of commoditization.

Jon Pither: Flow, Pair Programming and Emotional Intelligence

Tue, 22 Jul 2008 08:03:32 +0000

I was involved in an interesting chat with some colleagues about the apparent incompatibilities between pair programming and being in the ‘zone’; a Zen like state where everything just clicks and one is able to perform seamlessly to their optimum ability given the present moment. From a developer viewpoint our egos become almost on autopilot as our hands reach out and tap away at the keyboard, the mind harmoniously serving up a variety of designs and strategies that may help us solve the problem at hand. We effortlessly approach excellence.

Psychologists refer to zone as ‘flow’. Daniel Coleman - author of Emotional Intelligence - describes flow as being “emotional intelligence at its best… the ultimate in harnessing the emotions in the service of performance and learning”. Emotions generate a plethora of thoughts for us, including distracting worries and concerns. Therefore having our emotions working for us enables us to be more focussed and to think more clearly and objectively about a specific problem.

Pair programming is a not stress free activity. Though it has been proven to raise productivity it does require more focus on the part of the developer - asking much more of them than were they coding solo. Being in a “pair” you are constantly required to explain your intricate thought processes to another person. You are challenged, and you have a responsibility to ensure that the person you are pairing with gets up to speed with the material at hand and that you are providing them with a progressive learning experience.

We need to appreciate that developers are humans, and sadly are not perfect coding punching-out machines. Pair-programming introduces an emotional burden on the developer. Thoughts such as how the other person in the pair is coping and performing are one of many, along with speculation as to what the other person is thinking… Are they frustrated that I can’t keep up? Are they worried I’m going too fast? Or worse still do they feel guilty for slowing me down? - a common concern as developers strive to be productive. In order to deal with these worries a whole new - totally justifiable - plateau of thinking opens up. The brain switches over from analysing the current software problem to instead work on strategies for making the pair more effective: ‘they have a need to see this area of the code-base as it will help them to contextualise the problem better… I want to get some “quick wins” to build our momentum as a pair, therefore it makes sense to pick off this little bit of low hanging fruit…’

I believe that whilst pair-programming has the capacity to inhibit flow, the emotional intelligence within us is the tool to help minimize the impact distractions have allowing us to closer reach a level of excellence on an individual level. EQ is essentially about being aware of our emotions and the impact they have on us - to realise when we’re worried, or when our frustrations are mounting. A higher EQ will not only help us to better meet our own needs, but it will give us more empathy and understanding to better meet the needs of the person we’re working with - to help us create a more effective “pair”. EQ is essentially about being aware, and the best thing about it is that unlike IQ it can be learned.

Lastly, I want to refer to the premise that “stress makes you stupid“. We underperform when our emotions run unchecked. I’ve seen first hand developers resigning from jobs they sought hard for because pair programming has been too much of a difficult experience. If we can improve our awareness of the distractions and difficulties pairing introduces - and in doing so our EQ - then we’ll likely make it easier for ourselves and the person we’re pairing with.

ThoughtWorks Studios: CruiseControl Enterprise: 10 Best Practices

Tue, 22 Jul 2008 05:54:20 +0000

I'm an Infrastructure Specialist at ThoughtWorks. In my role I make sure that we are building our software so it can successfully be deployed to production. In this series of blog posts I hope to pass on my top ten tips for using CruiseControl Enterprise effectively. I'm writing these with the developers or systems administrators in mind: the people who most often manage CruiseControl. However, I hope that anybody who is interested in Continuous Integration will get something from these articles.

# 1: Publish with a Publisher

Many projects use the concept of publishing: placing built artifacts into a repository, or making test results available to end users. The ArtifactsPublisher is a very popular way to publish to CruiseControl's own repository of timestamped directories, making logs and archives available via the CruiseControl dashboard or legacy reporting application. But ArtifactsPublisher is just one of a family of Publishers, my favourite being the AntPublisher. If your Ant build finishes by publishing artifacts to a repository or tagging your version control system, there may be another way to do things. Consider the Ant example below:

<project>

 <target name="dist" description="build everything">

 </target>

 <target name="clean" description="delete all build artifacts">

 </target>

 <target name="test" description="run unit tests">

 </target>

 <target name="publish" description="publish to repository">

 </target>

 <target name="cruise" depends="clean,dist,test,publish"/>

</project>

The end state of the build is to publish unit-tested artifacts to an unspecified repository of some kind via the publish target. This is well and good, but the concept of publishing doesn't really fit the developer build; if we can disconnect the publishing activity from the rest of the build, we can run the same build in CruiseControl as the developers. To do this, have CruiseControl invoke your developer build, leaving publishing as a separate Ant target:

<project>

 <target name="dist" description="build everything">

 </target>

 <target name="clean" description="delete all build artifacts">

 </target>

 <target name="test" description="run unit tests">

 </target>

 <target name="publish" description="publish to repository" if="logfile">

 </target>

 <target name="dev" depends="clean,dist,test"/>



</project>

The 'if' attribute on the 'publish' target ensures that the target can only ever be run by a publisher or a determined person. Next, you need to make some changes to your CruiseControl configuration:



 <schedule interval="60">

 <ant antWorkingDir="${checkout.dir}" buildfile="build.xml" target="dev"/>

 </schedule>

 <publishers>

 <onsuccess>

 <antpublisher antWorkingDir="${checkout.dir}" buildfile="build.xml" target="publish"/>

 <onsuccess>

 </publishers>

And you're done. Once you apply the new configuration to CruiseControl, it will start to run the same Ant build as the developers, meaning that there's no mystery when the build breaks. It may also shorten the build feedback loop by reporting success faster. While the developers celebrate or lament the state of the build, the AntPublisher can get on with the work of pushing artifacts around your network. I also use it to tag the codebase on a successful build - another thing that is specific to Continuous Integration. Publishers can be configured unadorned inside the <publishers> element in the config file to run every time a build completes regardless of status, or they can be wrapped in <onsuccess> or <onfailure> to run conditionally.

Publishers have a cousin element called Bootstrappers, but I'll cover that another time.

©Copyright 2007 Julian Simpson. All rights reserved.

Simon Brunning: Links for 2008-07-21 [del.icio.us]

Tue, 22 Jul 2008 05:00:00 +0000

Simple Top-Down Parsing in Python

Ram - Sriram Narayanan: Post release bugs can be expensive to fix

Tue, 22 Jul 2008 04:32:04 +0000

Just read at the register that Apple's having problems with helping MobileMe users sync information with Outlook. And to apologize (atone was the word used, actually), Apple has given the MobileMe users 120 days of free credit.

Awesome move by a company where they felt bad for what they did to the end users.

And I sympathize with the developers at Apple who would have surely tested their work, but are now having to fix issues reported by end user (who may have lost their data).

Hopefully, the devs at Apple would be feeling bad about end user losing data, because not all software devs actually feel the pain when customers lose something - money, sales accounts, jobs.

Brandon Hastings Byars: Orthogonality

Tue, 22 Jul 2008 02:36:00 +0000

Orthogonality means that features can be used in any combination, that the combinations all make sense, and that the meaning of a given feature is consistent, regardless of the other features with which it is combined. The name is meant to draw an explicit analogy to orthogonal vectors in linear algebra: none of the vectors in an orthogonal set depends on (or can be expressed in terms of) the others, and all are needed in order to describe the vector space as a whole.

– Michael Scott (Programming Language Pragmatics)

I’ve used Delphi and Visual Basic at previous jobs. I disliked both of them. VB has this annoying distinction between objects and primitives, so I’d consistently type

Dim server
server = "localhost"

Dim catalog
catalog = CreateObject("MyCompany.MyObject")

...only to be greeted by an ugly Windows popup box that Object doesn't support this property or method. The problem, as all you grizzled VB programmers no doubt spotted immediately, is the last line should start with the keyword Set. VB requires you to prefix assignments on “objects” with Set. But if you try to put a Set in front of assignments on what VB considers primitives (like the first assignment above), you get an error that reads Object required: '[string: "localhost"]'.

Delphi likewise frustrated me with mundane annoyances:

DoOneThing();

if value = goal then
    DoSomething();
else
    DoSomethingElse();

The code above doesn’t compile, and it won’t compile until you remove the semicolon from the call to DoSomething(). Semicolons complete a statement, and the statement is the entire if-else clause.

These problems in VB and Delphi are related to the concept of orthogonality mentioned in the opening quote. VB doesn’t let you compose assignment to objects the same way it lets you compose assignment to primitives. Delphi doesn’t let you end an if clause the same way it lets you end an else clause. These inconsistencies encourage even experienced programmers to make silly syntax mistakes and make the language harder to use.

What is orthogonality?

The key principles I extracted from Michael Scott’s quote listed above are consistency and composability. Composability means that features can be combined, and consistency stands in for the Principle of Least Surprise—features act how you expect they would, regardless of how they’re being combined. VB’s assignments lack consistency. Delphi’s semicolon parsing doesn’t act consistently when composed within a surrounding statement.

Scott claims that a highly orthogonal language is easier to understand and easier to use. Nowhere does he mention that it’s easier to implement. I’m sure the Delphi grammar was simplified considerably by refusing to allow the DoSomething statement to end in a semicolon when contained within an outer if-else clause. It’s also likely that the implementation of Visual Basic was simplified by having the programmer tell the compiler whether an assignment referred to an object or a primitive.

I suspect many non-orthogonal aspects of languages are there to make them easier to implement. However, some languages that are trivially easy to implement can also be amazingly orthogonal. Lisp is a prime example; it is a highly orthogonal language, and yet an entire Lisp interpreter written in Lisp fits on just one page of the Lisp 1.5 Programmer’s Manual.

I found it instructive to list out syntactic constructs that make languages less orthogonal. It’s amazing how mainstream most of them are:

Statements

Statements aren’t necessary, and a number of languages avoid them altogether. Having only expressions makes the language easier to work in. Compare the inconsistency of Delphi’s statement parsing to the composability of Ruby’s expressions:

puts if x == 0
  "Zero"
else
  "Not Zero"
end

The if clause is an expression; it returns the last value evaluated (e.g., either “Zero” or “Not Zero”). And that value can be composed within another expression.

So what happens when you don’t have a reasonable return value? Smalltalk always returns self, which is convenient because it allows method chaining. In the Ruby example above, the entire expression returns the result of the call to puts, which happens to be nil.

The beauty of expressions is that the composability doesn’t just go one level deep:

x = if a == 1; 1; else; 2 end

y = x = if a == 1; 1; else; 2; end

puts y = x = if a == 1; 1; else; 2; end

puts "returns nil" if (puts y = x = if a == 1; 1; else; 2; end).nil?

As the example above shows, orthogonality doesn’t guarantee good code. However, it allows the language to be used in unanticipated ways, which is A Good Thing. Moreover, since everything is an expression, you can put expressions where you wouldn’t normally expect them. For example, in Ruby, the < operator represents inheritance when the receiver is a class, but the superclass could be an expression:

class Test < if x == 1; OneClass; else; TwoClass
end

Pushing a language like this, only to find that it’s turtles all the way down, is a signpost for orthogonality.

Primitives

We already saw the clumsiness of VB primitives, but most mainstream languages share a similar problem. Java, for example, has a confusing dichotomy of longs and Longs, the first a primitive and the second a full-fledged object. C has stack-allocated primitives, which are freed automatically when they fall out of scope, and heap-allocated variables, which you have to free yourself. C# has value types, which force another abstraction – boxing – into the programmer’s lap.

private static object DBNullIf(int value, int nullValue)

    return (value != nullValue) ? (object)value : DBNull.Value;

The code above should be a head-scratcher. Isn’t object the superclass of all other types? And if so, why do we have to explicitly cast our int variable to an object to make this code compile? Why don’t we have to do the same for the DBNull?

I mentioned above that ease of implementation is a common source of non-orthogonality. With primitives, we can see another, more legitimate reason: performance. There is a cost to keeping primitives out of the language. Below, we’ll see several more non-orthogonal language features that make performance easier to optimize.

Nulls

Nulls have been problematic enough that an entire design pattern has been created to avoid them. Access violations and object reference exceptions are some of the most common programmer errors. Most languages divide object references into two types: those that point to an object, and those that point to nothing. There’s nothing intrinsically non-orthogonal about that division, except that in most languages the references that point to nothing work differently. Instead of returning a value, they throw an exception when dereferenced.

In Ruby, the division is still more or less there -- some references point to objects, and some point to nothing -- but both types of references still return a value. In effect, Ruby has built the Null Object pattern into the language, as the references that point to nothing return a nil value. But, like everything else in Ruby, nil is an object (of type NilClass), and can be used in expressions:

1.nil?        # returns false
nil.nil?    # returns true

You never get an NullReferenceException in Ruby. Instead, you get a NoMethodError when you try to call a method on nil that doesn’t exist, which is exactly the same error you’d get if you called a method on any object that didn’t exist.

Magic Functions

Most object-oriented languages have certain functions that aren’t really methods. Instead, they’re special extensions to the language that make the class-based approach work.

public class Magic

    public static void ClassMethod()

    public Magic()

    public void NormalMethod()

Magic.ClassMethod();

Magic magic = new Magic(5);

magic.NormalMethod();

Notice the context shift we make when constructing new objects. Instead of making method calls via the normal syntax (receiver.method()), we use the keyword new and give the class name. But what is the class but a factory for instances, and what is the constructor but a creation method? In Ruby, the constructor is just a normal class-level method:

regex = Regexp.new(phone_pattern)

Typically, the new class-level method allocates an instance and delegates to the newly created instance’s initialize method (which is what most Ruby programmers call the “constructor”). But, since new is just a normal method, if you really wanted to, you could override it and do something different:

class Test
  def self.new
    2 + 2
  end
end

Test.new    # returns 4!!

Operators also tend to be magic methods in many languages. Ruby more or less treats them just like other methods, except the interpreter steps in to break the elegant consistency of everything-as-a-method to provide operator precedence. Smalltalk and Lisp, two other highly orthogonal languages, do not provide operator precedence. Operators are just normal methods (functions in Lisp), and work with the same precedence rules as any other method.

So here we have yet another reason, on top of ease of language implementation and performance, to add non-orthogonality to a language. Ruby adds operator precedence, even though it adds an element of inconsistency, presumably because it makes the language more intuitive. Since intuitiveness is one of the purported benefits of orthogonality, there is a legitimate conflict of interest here. I think I would prefer not having operator precedence, and leaving the language consistent, but it seems more of a question of style than anything else.

Static and sealed Methods

Class-based object-oriented languages impose a dichotomy between classes and instances of those classes. Most of them still allow behavior to exist on classes, but that behavior is treated differently than instance behavior. By declaring class methods as static, you’re telling the compiler that it’s free to compute the address of this function at compile-time, instead of allowing the dynamic binding that gives you polymorphism.

Ruby gives you some degree of polymorphism at the class level (although you can’t call superclass methods, for obvious reasons):

class Animal
  def self.description
    "kingdom Animalia"
  end
end

class Person < Animal
  def self.description
    "semi-evolved simians"
  end
end

In most mainstream languages, not even all instance methods are polymorphic. They are by default in Java, although you can make them statically-bound by declaring them final. C# and C++ take a more extreme approach, forcing you to declare them virtual if you want to use them polymorphically.

Behavior cannot be combined consistently between virtual and sealed (or final) methods. It’s a common complaint when developers try to extend a framework only to find out that the relevant classes are sealed.

Instance Variables

Only the pure functionalists can get by without state; the rest of us need to remember things. But there’s no reason why the mechanism of retrieving stored values has to be treated differently from the mechanism of calling a parameter-less function that computes a value, nor is there a reason that the mechanism for storing a value has to be different from the mechanism of calling a setter function to store the value for you.

It is common OO dogma that state should be private, and if you need to expose it, you should do so through getters and setters. The evolution of the popular Rails framework recently reaffirmed this dogma. In prior versions, sessions were exposed to the controllers via the @session instance variable. When they needed to add some logic to storing and retrieving, they could no longer expose the simple variable, and refactored to a getter/setter attribute access. They were able to do so in a backward-compatible way, by making the @session variable a proxy to an object that managed the logic, but it was still a process that a more orthogonal language wouldn’t have required. The language should not force you to distinguish between field access and method access.

Both Lisp and Eiffel treat instance variables equivalently to function calls (at least when it comes to rvalues). Lisp simply looks up atoms in the environment, and if that atom is a function (lambda), then it can be called to retrieve the value no differently than if the atom is a variable containing the value. Eiffel, an object-oriented language, declares variables and methods using the same keyword (feature), and exposes them - both to the outside world and to the class itself - the same way (Bertrand Meyer called this the Uniform Access principle):

class POINT 
feature
    x, y: REAL
            -- Abscissa and ordinate

    rho: REAL is
            -- Distance to origin (0,0)
        do
            Result := sqrt(x^2 + y^2)
        end

Once you get passed Eiffel’s tradition of YELLING TYPES at you and hiding the actual code, Uniform Access makes a lot of sense. Instance variables are just like features without bodies. C# has properties, which provide similar benefits, but force you to explicitly declare them. Instead of a getRho and setRho method, you can have a property that allows clients to use the same syntax regardless of whether they’re using a property or a field. Because Ruby allows the = symbol as part of a method name, it allows a similar syntax.

However, the separation between variables and properties is superfluous. For example, there’s no need for them to have separate access levels. If other classes need the state exposed, then declare it public. If the language doesn’t offer instance variables, then you’re simply exposing a property method. If you run into the same problem that Rails ran into, and suddenly need to add behavior around exposed state, refactoring should be easy. Just add a private property method that is now the state, and leave the public property method.

So, in my hypothetical language, we might have the following:

class Controller
  public feature session
end

And when you feel you need to add behavior around the exposed session dictionary, it should be easy:

class Controller
  public feature session
    # added behavior goes here
    private_session
  end

  private feature private_session
end

One thing that this hypothetical syntax doesn’t allow is separate access levels for getting and setting, but it shows the basic idea.

Inconsistent Access Levels

Since we’re on the subject of access levels, Ruby’s private access level is not very orthogonal at all. Unlike C++ derived languages, Ruby’s private is object-level, not class-level, which means that even other instances of the same class can’t directly access the private method. That’s a reasonable constraint.

However, instead of making object-level private access level orthogonal, the implementors simply disallowed developers to specify the receiver for private methods. This undoubtedly made implementing object-level private access much easier. Unfortunately, it means that you can’t even use self as the receiver within the object itself, which makes moving a method from public or protected to private non-transparent, even if all references to the method are within the class itself:

class TestAccessLevels
  def say_my_name
    puts self.name
  end

  private
  def name
    "Snoopy"
  end
end

# The following line throws an exception
TestAccessLevels.new.say_my_name

Second class types

Having special types, like null, is a special case of having primitives, and it’s a common performance optimization. Being a first-class type has a well-defined meaning. Specifically, its instances can:

be passed to functions as parameters
be returned from functions
be created and stored in variables
be created anonymously
be created dynamically

It’s becoming increasingly common for modern languages to move towards first-class functions. In the Execution in the Land of the Nouns, Steve Yegge parodied the typical transmogrifications Java programmers had accustomed themselves to in order to sidestep the language’s lack of first-class functions. Java’s cousin (descendant?), C#, has more or less had them since .NET 2.0 in the form of anonymous delegates.

What neither Java nor C# have are first-class classes. Both have reflection, and even allow you to create new types at runtime (painfully…). But, because both languages are statically typed, you can’t access these runtime-created types the same way you can access normal types. Assuming Foo is a runtime-created type, the following code won’t compile:

Foo foo = new Foo();

The only way to make it work is by heavy use of reflection. Ruby, on the other hand, makes it trivially easy to add types at runtime. In fact, all types are added at runtime, since it’s an interpreted language.

Single Return Values

Since you can pass more than one argument to functions, why should you only be allowed to return one value from functions? Languages like ML add orthogonality by returning tuples. Ruby works similarly, unpacking arrays automatically for you if you use multiple lvalues in one expression.

def duplicate(value)
  [value, value]
end

first, second = duplicate("hi there")

This feature allows you to combine lvalues and rvalues more consistently.

Inconsistent Nesting

Nesting is the standard programming trick of limiting scope, and most languages provide blocks that you can nest indefinitely if you want to limit scope. For example, in C#:

public void TestNest()

    string outer = "outer";

        string middle = "middle";

            string inner = "inner";

In addition to blocks, Ruby allows you to nest functions, but the nesting doesn’t work in an orthogonal way:

def outer
  inner_var = "inner variable"

  def inner
    "inner method"
  end
end

outer
puts inner
puts inner_var

In this example, the last line will throw a NameError, complaining that inner_var is undefined. This is as we should expect – since it’s defined inside an inner scope from where we’re calling it, we should not be able to access it. However, the same is not true for the inner method defined in the call to outer. Despite the fact that it’s defined within a nested scope, it actually has the same scoping as outer.

Ruby’s scoping gets even weirder:

def outer
  begin
    inner_var = "inner"
  end

  inner_var
end

puts outer

This code works, printing “inner” to the console. It shouldn’t.

JavaScript similarly suffers from strange scoping rules. Because all variables have function scope, and not block scope, a variable is available everywhere within a function regardless of where it is defined:

function testScoping(predicate) {
  if (predicate) {
    var test = "in if block";    
  }
  alert(test);    // works, even though it's in an outer block
}

Fixing your mistakes

Speaking of JavaScript, few things bug me more about programming languages than those that try to fix your mistakes for you. JavaScript automatically appends semicolons for you at the end of a line if you forget to. Most of the time, that works fine, but every now and then it creates a ridiculously hard bug to diagnose:

return
{
  value: 0
}

The return statement above looks like it returns an object with a single property. What it really does, though, is simply return. The semicolon was appended on your behalf at the end of the return keyword, turning the following three lines into dead code. It is for this reason that Douglas Crockford recommends always using K&R style braces in JavaScript in JavaScript: The Good Parts.

Summary

Orthogonality makes the language easier to extend in ways that the language implementors didn’t anticipate. For example, Ruby’s first-class types, combined with its duck-typing, allows the popular mocking framework Mocha to have a very nice syntax:

logger = stub
logger.expects(:error).with("Execution error")

do_something(:blow_up, logger)

The fact that classes in Ruby are also objects means that the same syntax works for classes:

File.stubs(:exists?).returns(true)

I picked on JavaScript a couple times, but it did a better job of any other language I know of with literals. JavaScript has literal numbers and strings, like most languages. It has literal regular expressions and lists like Perl and Ruby. But where it really shines is in it’s literal object syntax. Objects are basically dictionaries, and Ruby and Perl have hash literals, but JavaScripts objects include function literals:

function ajaxSort(columnName, sortDirection) {
    setNotice('Sorting...');
    new Ajax.Request(
        "/reports/ajaxSort",
        {
            method: "get",
            parameters: { sort: columnName, sortDir: sortDirection },
            onSuccess: function(transport) { 
                $("dataTable").innerHTML = transport.responseText;
            },
            onComplete: function() { clearNotice(); }
        }
    );
}

In time, that syntax was leveraged to form the JSON format.

In a more dramatic example, when object-orientation became all the rage, Lisp was able to add object-orientation to the language without changing the language. The Common Lisp Object System (CLOS) is written entirely in Lisp. The reason Lisp was able absorb an entire new paradigm is largely due to the language’s orthogonality. Not all function-like calls work the same way; some are called “special forms” because they act differently (for example, by providing lazy evaluation of the arguments). However, Lisp allows the programmer to create their own special forms by writing macros, which are themselves written in Lisp.

It helps to have turtles all the way down.

(Update: 7/22/08 – fixed some code formatting issues when rendered through an RSS feed)

Kiran Bellubbi: NetFlix is so 1995 - it sux!

Mon, 21 Jul 2008 22:26:26 +0000

In a moment of weakness I gave in to advertising jingles and decided to try NetFlix out - its my second day using the service and one of the main reasons I decided to try it was their new WatchNow feature

The WatchNow feature only works on Internet Explorer, buffering takes way too long - it took 1:25mins for a movie to be ready to be played on my 10mbs line at home.

People at NetFlix: wake up! Go look at Hulu, watchTVSitcoms, joox, surfthechannel and compare your sucky, sub-par, paid service against what these guys provide for free!

Ram - Sriram Narayanan: Opportunity: Sun hardware and software in demanding environments.

Mon, 21 Jul 2008 20:37:50 +0000

As an erstwhile hardware technician, I have always loved reading about hardware. And as an erstwhile Industrial Automation technician, machine uptime is something that I enjoy learning about.

A long time discussion in the Industrial Automation Industry has been around how people would live to have the reliability of Sun hardware, the stability of Solaris, and the feature set of Windows based Automation tools.

Some years ago, I was fortunate to see a Bayer Diagnostics' Advia Centaur being commissioned at Asia's largest Thyroid testing center at Thane (pronounced: thaaney), Maharasthra. This machine (called an immunoassay system) did a whole bunch of tests at very high speeds. When you have to process something like 30,000 independent blood samples in 4 hours at night at very high speeds, then you need reliability and accuracy. The high speed system would process the test-tubes which had barcoded stickers on them.

The only thing that counted as a minus for me was on how the data was finally made available. All the data collation was made available via some custom software. There was some different mechanism of providing the data over serial port, and this was in terms of "ask me for a very specific and I'll provide you that". There was no notion of making the data available as a stream, or via some http mechanism, or even some TCP stream.

It was not as if the hardware platform was limited - the equipment was driven by a dual SPARC Solaris box running Solaris 8.

Well, this is not really a post discussing how data access could be provided better, so let met get back to the point.

I learned then from the senior technicians from Bayer, that they trusted Solaris and the Sun hardware a lot. They had a few war stories to tell me about how some Sun hardware that they had would just _refuse_ to fail ! We all wondered if Sun was actually competing with itself !

These war stories of Sun hardware just working on and on, never failing, never dying, is something that I have heard from other guys who worked on Bank ATM systems too. They'd have OS/2 Warp on the kiosks (Windows 95/98 on rare occassions), and Solaris 8 on Sun servers.

With the addition of awesome tools and technologies such as ZFS, DTrace, FMA, Trusted Security, multi core awareness, the plethora of storage options, and the soon to be integrated Crossbow - the Solaris OS and Sun hardware combination could become more attractive as a platform to delivery great solutions on.

What, then is preventing system integrators from selecting Sun for Automation Solutions ? The lack of a Automation IDE targetting Solaris would be one thing. The absence of device drivers would be another. Writing high quality device drivers is an important job, especially when writing them for devices such as high speed counters (where accuracy is important) or Analog to Digital convertors (where precision and timing are important). National Instruments' LabView did have some Solaris based releases, but that was never really marketed too well - at least in India, unfortunately.

There was an Eclipsecon presentation by Siemens' Brazil some years ago, but I haven't heard more since then.

Bottomline: Having a killer IDE, a healthy eco system of device drivers, and strong marketing could position Solaris well in those worlds where other OS can't cut it.

Manish Chakravarty: I am Immersed!

Mon, 21 Jul 2008 19:31:01 +0000

I am Immersed!

Originally uploaded by Manish Chakravarty

I completed my Immersion at ThoughtWorks.

It was a wonderful experience, both personally and professionally.

I will blog about my experience about it in a later blog post

More Immersion photographs at

http://flickr.com/photos/manish_chaks/sets/72157606218107832/

Ram - Sriram Narayanan: Belenix 0.7.1 - Amazing initial download reports

Mon, 21 Jul 2008 19:28:09 +0000

Here's some exciting news about Belenix, and OpenSolaris based distribution that you ought to give a try !

+ About 1.1 TB served this weekend :)

+ About 1700+ downloads in two days :)

+ Independent seeding has begun :)

+ More downloads in progress :)

Yup, readers, that's smiles all the way :) (Err.. a Hindi saying to ward off the evil eye comes to mind.. "Buri nazar waley, tera muh kaala !" ;)

The Google Analytics on the site tell us more things:

+ 3000 Page Visits (where people stay around for more than a few minutes and read various things on the site)

+ Hits: 10344 on Sat and 6550 on Sun (We'd announced Belenix on Saturday)

1. A few people have written in expressing their satisfaction with Belenix 0.7.1

2. Someone wanted help with testing Belenix with 802.11 WEP. I have Enterprise WEP at work, but forgot to test this ! Ugh !

So, folks, get started with OpenSolaris ! :) Download Belenix from here http://belenix.org/content/Download There's even a Metalink on that page ! :)

Here are the release notes

Contact #belenix @ freenode.net on IRC

Feel free to file bugs here : http://sourceforge.net/tracker/?group_id=216118

We take bug reports very seriously, and future development always includes time for bug reports.

Website: www.belenix.org

Discussion List : belenix-discuss@opensolaris.org

Developer List: belenix-dev@opensolaris.org

IRC : #belenix @ freenode.net (late night India Standard Time is when the core team is around - there are others who hang aroudn too ! :)

Feel free to get in touch with us. This is one group that's incredibly friendly, end user focussed as much as we can think of being, and driven by user inputs on what people want to see !

Carlos Villela: Networks Are Smart at the Edges

Mon, 21 Jul 2008 17:20:42 +0000

A toothpaste factory had a probem: they sometimes shipped empty boxes, without the tube inside. This was due to the way the production line was set up, and people with experience in designing production lines will tell you how difficult it is to have everything happen with timings so precise that every single unit coming out of it is perfect 100% of the time. Small variations in the environment (which can’t be controlled in a cost-effective fashion) mean you must have quality assurance checks smartly distributed across the line so that customers all the way down the supermarket don’t get pissed off and buy someone else’s product instead.

Understanding how important that was, the CEO of the toothpaste factory got the top people in the company together and they decided to start a new project, in which they would hire an external engineering company to solve their empty boxes problem, as their engineering department was already too stretched to take on any extra effort.

The project followed the usual process: budget and project sponsor allocated, RFP, third-parties selected, and six months (and $8 million) later they had a fantastic solution — on time, on budget, high quality and everyone in the project had a great time. They solved the problem by using some high-tech precision scales that would sound a bell and flash lights whenever a toothpaste box weighing less than it should. The line would stop, and someone had to walk over and yank the defective box out of it, pressing another button when done.

A while later, the CEO decides to have a look at the ROI of the project: amazing results! No empty boxes ever shipped out of the factory after the scales were put in place. Very few customer complaints, and they were gaining market share. “That’s some money well spent!” - he says, before looking closely at the other statistics in the report.

It turns out, the number of defects picked up by the scales was 0 after three weeks of production use. It should’ve been picking up at least a dozen a day, so maybe there was something wrong with the report. He filed a bug against it, and after some investigation, the engineers come back saying the report was actually correct. The scales really weren’t picking up any defects, because all boxes that got to that point in the conveyor belt were good.

Puzzled, the CEO travels down to the factory, and walks up to the part of the line where the precision scales were installed. A few feet before it, there was a $20 desk fan, blowing the empty boxes out of the belt and into a bin.

“Oh, that — one of the guys put it there ’cause he was tired of walking over every time the bell rang”, says one of the workers.

Simon Brunning: Happy Birthday me!

Mon, 21 Jul 2008 09:01:06 +0000

28 today!

Brian Guthrie: GitHub, briefly

Mon, 21 Jul 2008 06:03:45 +0000

I know that I’m late to the party here, that all of the cool kids have had accounts for what seems like Internet-decades, and that I can do nothing but add my voice belatedly to an already rapturous chorus; nevertheless, forgive my enthusiasm: GitHub rules, it is worth joining, and, once signed up, it took only a week for my workflow to be forever improved. I am, in short, a believer.

It’s never been easier or more convenient to contribute to other open source projects.
It’s never been easier to work remotely.
It’s never been easier to track the progress of projects and developers you’re interested in.
Unicorns! Rainbows! Puppies!

Over the next few weeks I’ll gradually be migrating all of my projects over to it, and I encourage you to do the same.