Inversionism | Paul Hammant's blog

Call to Arms: Average Story Sizes of One Day

Tue, 24 Apr 2012 00:00:00 PDT

Synopsis: Story sizes should average as close to one day as possible¹. If they don’t your Agile project is going to be harder for nearly everyone involved. If your average is significantly greater than that one day, then change something until you get there. Change technologies if you have to, or split stories, or eliminate NFRs (or delay them). Of course, I assume you know what ‘Agile’ and ‘Stories’ are. *So, I’m hereby calling on PMs and BAs to do their part to developers live in this zone (details below).

A week ago, I talked about the Glass Ceilings in Application Development and that’s generally applicable to this topic too. But this blog entry is concerned with the costs of having longer estimates. Developers cannot estimate longer stories accurately. It is just impossible to consistently go above that and be accurate.

So what was intended for Agile/XP when all this stuff was new (and projects were smaller)? I suggest (synopsis) that the XP pioneers said “we can’t tell you how many iterations it is going to take, or what the final cost will be, but we can commit to shipping software in iterations, concentrating on business value, and giving you moments where you can choose to complete, pause, or abandon development of the product/service”, and to that end, they envisaged smaller than 3-day stories.

What to do if you have longer averages?

What if you average is significantly longer than that one-day mark? What if it is, say, 3 days with the increased risk to measured progress and team morale? Work out what is wrong, and change it, is the answer.

Wrong People?

It could be that you could change the developers for the project. Developers are the biggest percentage of resource costs when considering the price of a story. Say you have a project that uses Hibernate, you may get a boost from hiring Gavin King (who is/was the Lead Developer for Hibernate). He may not thank you for plucking him from his yacht, but he really knows that stuff. How much can be gained from having experts on those technologies which are just part of a larger stack? How much faster are the experts versus good generalists within the team? It seems that perhaps only modest gains could be made, and as you implicitly lose the generalists, you lose the blended-team productivity benefits.

Wrong Technologies?

Maybe the technologies are wrong. Following on from that topic from a week ago, I suggesting that we can, unfortunately, choose technologies that really make it hard for us to complete what were trying to do. Here’s an example: Hardy Leung has made the excellent Tagxedo:http://www.tagxedo.com/ site. It is mostly HTML of course, but a critical design tool for word clouds is written using Microsoft’s Silverlight technology. Hardy might have started with a HTML site and quickly imagined an eternity of effort to get his vision complete. He settled on Silverlight for the best ‘bang for the buck’, and having used the app on my Mac, I can say that he has done a great job. Near the bottom of his FAQ he goes into his rationale for choosing Silverlight. That is one of a few ‘right’ technologies for this job, as of Q2 2012.

Changing technologies is not cheap of course. Sometimes you’ve gone too far to change, and it’s better to soldier on. Quickly identifying the need to change is best. Perhaps best of all during the ‘experimenting with technolgies’ phase before ramping up the project. Again, be careful not to choose one with glass ceilings.

Rule of thumb

For the technologists amongst us, critically estimate the number of lines executed (non-framework, non-library) from the ‘click’ the user did all the way down the stack and back up again for the response that changes the UI. That counts JavaScript, Java, Ruby or C#, and stops at SQL. If you’ve busted the facade pattern, then count all those tendrils down the stack. That is, unless your client caching strategy can massively mitigate² that. Aim at a max of 100 lines executed. If you’re not hitting that, ask for simpler technologies in order to be able to hit the one day story average.

Story Analysis wrong?

Perhaps the stories do not adhere to the Extreme Programming ideals of thin vertical slice or the INVEST principle. That spelled out is “Independent, Negotiable, Valuable, Estimable, Small and Testable”. Perhaps before estimation more representatives of the development team can advise on where to make splits in stories, and the Business Analysts still have time to rework them with the customer’s agreement before that estimation session. These should be outliers though. Story splitting I mean. A team that is zipping along will be feeding developers stories where the sizing is optimal, and if they are too large (from somebody’s perception) they can’t be split any further, and something else is ‘wrong’. I’d never like BAs to just accept the large-story scenario though; I’d implore them to respectfully push back at the project manager, stakeholders and architects in the team to try to change something so that the one-day average could become true. If that means they have to say “it looks like our technologies are too complicated” ad they see it, then so be it.

NFRs killing you?

Non-functional Requirements; Hmmm. The cost of implementing something in a technology is one thing, but Agile teams have an additional factor to worry about: They frequently revisit code that is complete to a certain level, and move it incrementally as new stories come in. That can include refactoring. So is the technology or the way you are using it friendly to refactoring? Quite often the NFRs for an Agile team mandate the least-refactorable technologies in to the mix.

CORBA

CORBA is traditionally a technology that’s not easy to refactor in the best of breed IDEs. It is bound to be an NFR, as its not often going to be chosen given free choice.

Say you have a three-tier architecture, and the architects stipulated CORBA as the RPC linkage between a service tier and its client in the same language. You might have a future requirement to bolt on a second client in a different language, making CORBA a common choice for later on in the project. At the beginning you had many technology choices. Maybe the language in question had something more canonical for it (like Ruby’s DRb). Using that might be a lot easier than crafting all the IDL that CORBA deems necessary. Perhaps you should leverage an open source library that makes for RESTful endpoints into the service tier, that is again easier to use, but with the added benefit of maybe being good enough for the future second client. Even if you ‘must have’ CORBA in that final multi-client technology version, you could have gained initial functional productivity using easier technologies and added a small CORBA layer later. I.e. delay is a good choice, if you plan carefully. There are other RPC choices that are multi-language that are also viable, and perhaps easier to deal with that CORBA: Thrift and FlowType Bridge are just two.

Persisting to a normalized database

Maybe its a requirement for a relation schema that’s been normalized to the nth degree and having referential integrity turned in with constraints and triggers. If you ‘only’ have a UI interpreting with that store, and that UI needs a JSON document to and from into today’s cheapest web technologies (AngularJS or Knockout) then why bother with the relational schema? Use a document store for the time being, and migrate to a relational schema later in the project. Read up on NoSQL people.

You might do a document-store solution for pure productivity reasons, but have a known relation schema need later in the project for operation or managerial reports. You still have choices. You could create a extract process that takes the written document data and writes it to relational tables to be used in reports. That could happen in real time, or in a daily batch extract.

Multi-tier per se.

Is it possible that you’ve got the tiers that the architecture team want for the final product, and there’s a lot of fairly predictable transformation logic going on up and down the stack to separate the models. It could be that you could gain productivity by delaying a tier separation. Perhaps Conway’s Law ³ workshops should be and yearly and mandatory for managers, project check-book holders, executives and architects with IT organisations :)

Technical-debt stories.

Obviously by delaying work we may end up with a number of technical debt stories. They are controversial, as the customer may not ultimately map them to features they have asked for. In later phases of the story, the customer may be used to the horse-trading stage of iteration planning and killing stories that have no business value. Finding the right way of describing what is being paid for later is key. I’ve seen such stories explain well in terms of business value. Your risk at this moment is that a huge piece of work has been delayed, and someone want to do this on a branch over a month, rather than Branch by Abstraction. Maybe as you decide to delay non-functional work, you

Lastly.

Colleague Derek Hammer reminds me of one the fundamental benefits of this proposal: Consistent throughput/liquidity because ‘small’ stories are the average. There are some stats involved, but the gist is that we have less waits on stories being completed. We also have less ‘hangover’ of nearly completed stories, the points/days for which ‘banked’ in the next iteration. Here that problem in a chart form:

How is the project doing? What is the trend for story points per iteration? I don’t think we can’t really tell from that chart.

Footnotes.

¹ - Stories have estimates and actual times. I’m assuming here that your team has these more or less the same as you get into your stride.

² - Please read Steve’s Souder’s Cache them if you can blog entry.

³ - Thanks to colleague James Lewis for the reminder.

Application Development Glass Ceilings

Sun, 15 Apr 2012 00:00:00 PDT

Sometimes in app-dev we’re using technologies that have a delayed cost. Simple to use now, but with shortcomings that mean that later features might be much harder to add. The term ‘Glass Ceilings’ (whilst classically meaning something else) seems to apply here.

An Example

I’ll Assume you are familiar with Agile. The customer/product-owner asks for a change to an existing dropdown in the UI. She wants a picture of the fruit next to the name of the fruit. He remembers Visual-Basic application development from the mid 90’s where this was easy. Also, She’s seen it in Java-Swing applications not so long ago. Now, that most enterprise apps are web-based, and fifteen years have passed since She first saw it, She thinks it should be easy!

Well, it is not. HTML does not support this be default. Most formal Web-Frameworks don’t do icon+text combo boxes. If they do it at all, it is because the authors have written something special in JavaScript. It’s a very custom control in HTML-land.

Lets say that the original feature (dropdown of fruit names load and saving to the database for the ‘thing’ in question) took a day to write. If we’re using a technology that does not have that feature we are going to have to go back to the product owner with some bad news or VERY bad news. It could take an eternity or be impossible to do it with the technology we are currently using.

What you really hope for is an ‘easy’ addition of something that’s correct for your technology choices. If you were using JQuery as your substantial in-browser technology, or were compatible with it, and you had this exact problem, then you’d be very happy that you found Fairway Technology’s FlexBox. Refer example 7 on their demo site like so:

Punching through the glass ceiling yourself

If we are using a commercial closed-source technology we are beholden to the vendor to even think our change request is important, or have a channel open for such communication. Moreover, we hope they are a competent outfit, and can ship a release quickly. If the technology is open source, then we’ve recourse to fork it and make a change ourselves. In that eventuality, we hope that the canonical version’s team will accept back the changes, and not make too many changes. Incidentally, this is one place that open source teams fails – not-facilitating someone else’s idea for that technology.

The worst-case scenario is that you have to change technologies. You might be halfway through the product buildout cycle, and heads will roll on this bad news!

The best-case scenario, is that the open or closed source team has ‘plugin’ mechanism for adding capability. Even better if there’s a way of leveraging/extending supplied components.

Even in the best-case scenario we’re going to have to suggest to the product owner that the feature She needs is going to take five days, but after that’s done, similar changes to other dropdowns will take half a day.

Thus, in summary, we hope that if we encounter glass ceilings during the building of an app, we can punch though them. Perhaps we are unsure about how much effort that will be in advance of encountering it. It is probably best that we are diligent in technology choices up front. I’m going to return to that particular thought in my next blog entry.

Not just UI controls

The problem of glass ceilings does not just apply to UI affordances and effects. You could encounter it in mismatches between a self-designed relation-schema for your app, and perhaps the Object-Relational Mapping (ORM) technology you chose. Perhaps your synchronous RPC technology should have been asynchronous.

Footnotes

I’ve talked about this before Ruby versus JavaScript for Web3.0 in 2006. That one was proposing changes to web-app coding that have not come into fruition. Perhaps they could with a future version of Dart, but I’m not chasing that agenda for the time being.

I’ll link to other proper HTML combobox solutions if people email me, or comment below.

Performance Testing Knockout, Angular and Backbone with Selenium2

Thu, 12 Apr 2012 00:00:00 PDT

Following on from Monday’s Selenium2 demo of ‘functional testing demo’ for Angular and Knockout, I thought I’d expand what I had into a performance test seeing as there are no server GETs or POSTs after loading each TODO-MVC app. Monday’s test scripts have been morphed into ones that are less a demonstration of fluent-selenium, and more a harness for the repeated addition of TODOs into the TODO-list. Specifically, rather than loop through items, I’m going straight to the last matching one for an XPath selector via ‘//whatever[xxxxx and position() = last()]’. I have a github project for the test scripts and the results (CSV). Selenium2 has a ‘feature’ whereby it needs to have the widget that it’s intersecting with visible on the page, and consequentially does a lot of scrolling for these apps. I want to add 1000 TODOs for this performance test, and fear the cost of the constant scrolling within the browser. I measured it, and the cost is real – like up to 10% of the over all time taken for the test, so I’ve forked Addy Osmani’s TODO-MVC demo apps to make the input field for “new TODOs” at the bottom of the list rather than the top. I’ve also deliberately detached the CSS, so that thus is a raw performance test of the Client-Side MVC framework. I could not measure much of a difference for that last though. The test comprises i) Adding a TODO, ii) Locating it in the list and verifying that it was added, and iii) clicking the “done” checkbox. That means that the substantial testing of the Client-Side MVC framework was an a) add to the model, b) a change to an item in the model, c) view updates to reflect that, and d) a consequential change to a ‘remaining todos’ counter. I’ve also tested that Selenium2 itself is not responsible for the slow-down by timing some of its DOM traversals that are not associated with a change to the model.

Results.

Can you see that? Backbone and Knockout are kinda constant at 132ms and 123ms respectively per iteration. Angular starts slight better than those two, but moves up to 400ms per iteration by #1000. Click here to see a large PDF of the whole chart (sideways scrolling required).

I’ve done multiple runs on this 8Gb SSD Mac to ensure consistency. The AngularJS chart, for one, was odd enough to warrant further inspection. Here are two runs overlaid on the same chart (and zoomed in):

Weird huh? Almost identical. I can only think that’s an interaction between AngularJS, the JavaScript environment of Firefox 10.0.2, and the vanilla JavaScript model. It could be that Selenium2 is erroneously waiting for interactions to finish slightly longer than necessary, meaning the slow-down is not real. Specifically, AngularJS polling for changes in either direction, might be enough to thwart Selenium’s “document ready” detection.

Conclusion

I still think that the productivity benefits of AngularJS makes it very much ‘worth it’ compared to Backbone. AngularJS at the 2001 model-bound nodes in the view (1000 output fields from a model + 1000 input/output rows from that model + 1 calculated field) is still working faster than humans can type. KnockoutJS takes slightly more coding that the equivalent AngularJS app, but has the best performance at that 2001 node limit. Backbone takes much more coding, making your productivity benefits dissapear. I’d still use Knockout and AngularJS instead of Backbone every time.

JQuery Foot-note.

I was going to similarly measure the performance of Addy’s JQuery+Moustache example, but Selenium could not locate the input-field when I moved it down the page. If I left it at the top, and ran the test anyway, it was taking 600ms per iteration at the end. Here’s a video that has NOT been slowed down:

PS – thanks to colleague Tarek Abdelmaguid for the backbone version of the groovy script.

Testing Knockout and Angular with Selenium2

Mon, 09 Apr 2012 00:00:00 PDT

Selenium2 (AKA —WebDriver) is the de-facto standard web testing technology. It is good that Selenium now garners more job postings than QTP, but that’s a different topic. the Client-Side MVC frameworks are a boon to web app development, but what about writing test scripts for the finished product? In this blog entry I will show how testable AngularJS and KnockoutJS applications are. In my opinion these two are the leading client-side MVC technologies, though there are alternatives like Backbone.js that have significant market share. I prefer the two I’m highlighting over Backbone for a range of reasons.

Testing a KnockoutJS app with Selenium2

There’s a TODO app on Addy Osmani’s technology comparison site. It uses Knockout 2.0 (Knockout 2.1 is in beta-testing presently). Here’s what it looks like:

In testing the app, I’m going to try to write for an idiomatic Knockout app. Here’s the DOM as Firebug sees it:

I’ve highlighted ‘1’ – the input line where new entries are posted, and ‘2’ an attribute where Knockout notes that it is looping through todo items. Prevalent througout the DOM are many more ‘data-bind’ attributes. These are evidence for Knockout managing the page. HTML does not have that attribute, only Knockout understands it. The browser just ignores the attributes it does not understand.

Here is the Selenium2 script (Groovy) that tests this app. It is going to add and item, then mark it as ‘done’, then confirm that the UI displays it as done with a strike-through:

@Grapes([
 @Grab("org.seleniumhq.selenium:selenium-java:2.20.0"),
 @Grab("org.seleniumhq.selenium.fluent:fluent-selenium:1.5.1"), 
 @GrabExclude('xml-apis:xml-apis')
])

import org.openqa.selenium.By
import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.Keys
import org.openqa.selenium.StaleElementReferenceException
import org.seleniumhq.selenium.fluent.FluentWebDriverImpl
import static org.hamcrest.CoreMatchers.equalTo;
import static org.junit.Assert.assertThat;
import static org.seleniumhq.selenium.fluent.FluentBy.attribute
import static org.seleniumhq.selenium.fluent.Period.secs

def driver = new FirefoxDriver()
def fluent = new FluentWebDriverImpl(driver)

driver.get "http://addyosmani.github.com/todomvc/architecture-examples/knockoutjs/index.html"

def todo = "abc 123 def 456"

def start = System.currentTimeMillis()

// there are many angular examples in one page

def form = fluent.within(secs(10)).div(By.id("todoapp"))

// add an entry
form.input(By.xpath("contains(@data-bind, 'enterKey: add')")).sendKeys todo + Keys.RETURN

assertThat("no items checked initially", form.lis(By.className("done")).size(), equalTo(0))

def lis = form.ul(attribute("data-bind", "foreach: todos")).lis()
// loop through the todo items ..
for (li in lis) {
  if (li.getText().contains(todo)) {
    // .. click the checkbox for the matching TODO item
    li.input().click()
	break
  }
}

assertThat("one item checked", form.lis(By.className("done")).size(), equalTo(1))

println "Test Duration: " + (System.currentTimeMillis() - start) + " milliseconds."

driver.close()

The best elapsed time for the test was 326 milliseconds. Certainly faster than a human could type.

Testing a AngularJS app with Selenium2

Angular is gearing up for a 1.0 release presently. That’s about time after 2.5 years! As such, I’ve copied their current TODO example to a stand-alone GitHub-Pages site. Here is what that app looks like:

Here’s the DOM as seen in Firebug. Again the input field and looping construct are shown:

Angular uses more fine-grained attributes than Knockout. Again, attributes that only it understands. ng-repeat, ng-model, ng-click, ng-show are just some of them. This might make it slightly easier to leverage them for Selenium2 locators.

@Grapes([
 @Grab("org.seleniumhq.selenium:selenium-java:2.20.0"),
 @Grab("org.seleniumhq.selenium.fluent:fluent-selenium:1.5.1"), 
 @GrabExclude('xml-apis:xml-apis')
])

import org.openqa.selenium.By
import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.Keys
import org.openqa.selenium.StaleElementReferenceException
import org.seleniumhq.selenium.fluent.FluentWebDriverImpl
import static org.hamcrest.CoreMatchers.equalTo;
import static org.junit.Assert.assertThat;
import static org.seleniumhq.selenium.fluent.FluentBy.attribute
import static org.seleniumhq.selenium.fluent.Period.secs

def driver = new FirefoxDriver()
def fluent = new FluentWebDriverImpl(driver)

driver.get "http://paul-hammant.github.com/angular_todo_app/"

def todo = "abc 123 def 456"

def start = System.currentTimeMillis()

def form = fluent.div(attribute("ng-controller", "TodoCtrl"))

// add an entry
form.input(attribute("ng-model", "todoText")).sendKeys todo + Keys.RETURN
//form.input(attribute("type", "submit")).click()

assertThat("one item checked", form.spans(By.className("done-true")).size(), equalTo(1))

def lis = form.lis(attribute("ng-repeat", "todo in todos"))
// loop through the todo items ..
for (li in lis) {
  if (li.getText().contains(todo)) {
    // .. click the checkbox for the matching TODO item
    li.input().click()
	break
  }
}

assertThat("two items checked", form.spans(By.className("done-true")).size(), equalTo(2))

println "Test Duration: " + (System.currentTimeMillis() - start) + " milliseconds."

driver.close()

The best time for the Angular/Groovy test was 330 milliseconds which is more or less the same as the Knockout one. Still faster than humans can type. It is also important to note that the Angular team have changed from ‘ng’ namespace found in the pre-1.0 releases that meant that colons were prevalent in the HTML soure, to a design where a dash is used instead. What was ng:show is now ng-show (etc). This is great news for Selenium2 testing as the colons were a bit of a problem. I blogged previously on that.

Conclusion

Angular and Knockout provide some handy hooks for Selenium2 to navigate by. Though IDs managed by the framework would be better, what’s there is good enough. Angular using more fine-grained attributes rather than the little language that Knockout has in a single attribute and that is going to be easier to navigate with. That said convenience locators could be made for either:

// Angular
form.input(ng.model("todoText")).sendKeys todo + Keys.RETURN
def lis = form.lis(ng.repeat("todo in todos"))

// Knockout
form.input(ko.enterKey("add')")).sendKeys todo + Keys.RETURN
def lis = form.ul(ko.foreach("todos")).lis()

AngularJS and Selenium (Again)

Thu, 15 Mar 2012 00:00:00 PDT

Update (Apr 9, 2012): A new version of Angular is much easier to test with Selenium2 now. Read about that in the follow up blog entry: Testing Knockout and Angular with Selenium2

Synopsis: Using Selenium with Angular’s not so easy right now.

Refer to my previous JBehave ‘Story Navigator’ app again. Here’s the URL to the site, if you wanted to play with it, or look at it with Firebug:

http://paul-hammant.github.com/StoryNavigator/navigator.html

What I want to be able to do is use the model references to locate elements within the page. The most ‘sophisticated’ example is the “Showing 3 Of 3 Stories”

That looks like this in the source:

<th colspan="2" class="section"> Stories<br/>
  <p id="stories-total">Showing {{data.xref.stories.$filter(search).$size()}} of {{data.xref.stories.$size()}} stories</p>
</th>

And in the rendered DOM, that looks like:

Getting to the first number should be as simple as:

//span[@ng:bind = 'data.xref.stories.$filter(search).$size()']

The problem is that $ is an illegal char in respect of a xpath lookup inside Firefox. I’ve tried a couple of escape sequences for it, but have not been able to overcome that. Simpler model references like “search.description” should work though.

The second, and perhaps more serious problem is that using xpath to find attributes “ng:bind” is impossible too. The colon is the issue, and despite there being advice on the StackOverflow as to how to search for namespace-centric things using xpath, none seem to work in Firefox. This one should work, but does not:

//span[local-name()='bind' and namespace-uri()='http://angularjs.org']

OK, I think the AngularJS team needs to get inventive here. Perhaps, for a certain mode of operation it should generate legal IDs for each element that are data-bound. The IDs could be as is, munged somewhat to replace ‘illegal’ chars, or a hash function of the string (or part of it). It’d only be used by Selenium, and the mode would not be turned on for production deployments of the app. This dogged us previously with GWT applications where the XHTML was very sopisticated and locators for widgets required a PhD to make elegant. For a client today, I’m writing Selnium2 scripts for an app that has duplicate IDs, so I don’t partocularly care whether, for testing, IDs are duplicated. Selenium’s Firefox or InternetExplorer driving can handle that.

Anyway, I can get Selenium working with the Story Navigator app. Here’s some groovy:

@Grapes([
 @Grab("org.seleniumhq.selenium:selenium-java:2.20.0"),
 @Grab("org.seleniumhq.selenium.fluent:fluent-selenium:1.0"),
 @Grab("org.hamcrest:hamcrest-all:1.1"),
 @GrabExclude('xml-apis:xml-apis'),
])

import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.StaleElementReferenceException
import static org.openqa.selenium.By.*
import org.seleniumhq.selenium.fluent.FluentWebDriverImpl
import static org.seleniumhq.selenium.fluent.Period.*

def driver = new FirefoxDriver()
def fluent = new FluentWebDriverImpl(driver);

driver.get "http://paul-hammant.github.com/StoryNavigator/navigator.html"

fluent.within(secs(1)).p(id("stories-total")).text.shouldBe "Showing 3 Of 3 Stories"

fluent.input(name("search.description")).sendKeys "Hat"

fluent.within(secs(1)).p(id("stories-total")).within(secs(1)).text.shouldBe "Showing 1 Of 3 Stories"

driver.quit()

Here’s the resulting video, which is too quick to see (less than half a second for the actual browser interaction):

You might have to advance the video frame by frame in order to see it working – sorry!

One more small fact – Angular has about a 50ms delay between events in the page (I type “hat”) and the model-view-controller results of those changes. Humans cannot see that, but Selenium can, hence the build in delays.

The Importance of Design-Mode for Client-Side MVC

Mon, 12 Mar 2012 00:00:00 PDT

Synopsis: Design mode (the same view but without Angular or Knockout loaded) is a great way to see the multiplicity of the view as you develop.

Remember that JBehave ‘Story Navigator’ blog entry I did a year ago? Well, give it a quick click to remind yourself. It has been tidied up some since – as I get better with Angular. I forked it on github, to a version that’s published in the same way (github pages) but without JavaScript running; No AngularJS specifically.

Here’s a screenshot:

And here’s the link, which is a better way of quickly seeing the shrunken picture above.

http://paul-hammant-fork.github.com/StoryNavigator/navigator.html

Moustache or handlebar style template fields everywhere.

You can see them everywhere – things that would be output text in the final version, but are now just model references in design mode. Some are simple field references, some are more compex expressions, or have formatters attached.

Only one row versus many in the running version.

There are three rows in the running version, and only one row in the design mode.

Multiple alternates concurrently shown in design-mode

The passed OR failed icon in in the running version, is now passed AND failed (side by side) in design mode:

For each column in the table, according the the design of the JBehave StoryNavigator, there are three possibilities in respect of sort order:

this column is the sorted one in descending order
this column is the sorted one in ascending order
this column is not the one being sorted

All three of those realities are shown for each column, and it is that which forces the columns far out to the right of the view. There’s an obvious improvement that someone could make to that – as there is no need for me to repeat the text of the column name as it does not change.

Summary

Hopefully you get to see the beauty of of Angular and Knockout: You get to see a design preview of the finished page, with all of the components visible, and with CSS hooked up as it would be in the finished product.

Replacing JQuery With Angular

Sat, 03 Mar 2012 00:00:00 PST

Synopsis: Angular webapps can have a lot less code that JQuery ones, and be easier to follow.

I took the source of an e-zine article from 2007 using an older version of JQuery, checked it into to Github and incrementally converted it to Angular. I’ve squashed that series of commits into on to make it easier on the eye (and hide all the mistakes I made en route), and you can see that here. It is worth perusing that commit for a few moments to see the comparison, and the shear weight of code deleted.

The original version (JQuery 1.2)

Digital Web’s article supporting the JQuery version

A live version of that JQuery demo

Please ignore the fact that nobody in real life would actually commission a seat plan app with this functionality.

After Refactoring to Angular 0.9.19

Play with a live version of that Angular_Air demo

What did I change?

I replaced JQuery with Angular of course. The HTML file was huge before, but is now 95% smaller because I’m using the ng:repeat to repeat the rows/seats through the airplane map, and also the passengers table below. The JQuery using JavaScript (global.js) has been entirely deleted (as has JQuery), I have new JavaScript in the HTML page for the model setup, and a couple of functions I’m calling to support the view. That’s about an 80% JavaScript reduction.

I changed the design of the passenger table. It used to be a tabbed dialog, with which row you were viewing changing in its entirety each time you clicked a row in the airplane. Now, it shows all the individual seats you have selected, in an expanding table. Perhaps that should be constrained in terms of height and have a scroll bar. In the older version, the passenger details for selected seats were highlighted in yellow, and editable, whereas the unselected passengers were not yellow, and not editable. In my version, the unselected passengers are not shown (see use of ng:show in the HTML source)

The Angular version is much easier to understand. Perhaps even easy enough for relative novices. That would not be true for a Backbone redux of the same app. The JavaScript models for backbone I’m beginning to think are MORE VERBOSE than the Java equivalent for GSON / Jackson serialization. A Knockout version of the same would be roughly the same implementation and lines of code to the Angular one.

I’ve also appended a debug panel, that shows the two pertinent model objects in the app. You can see that live updating as you click around in the airplane image, and the passenger details.

What’s less than perfect?

The model is less than perfect to me. What I’d like to have :

seats[
  row : 
  {
    num: 1,
    a : {
      passenger_name : "Fred"
    },
    b : {
      passenger_name : "Wilma"
    }
    // etc    
  }
]

seats[
  row : {
    num: 1,
    positions : [
      { 
        posn : "a",
        passenger_name : "Fred" 
      },
      { 
        posn : "b",
        passenger_name : "Wilma"
      }
      // etc    
    ]
  }
]

But what I have instead is something less hierarchical:

seats[
  {
    row: 1,
	posn: "a"
    passenger_name : "Fred" 
  },
  {
    row: 1,
	posn: "b"
    passenger_name : "Wilma" 
  }
  // etc
]

The reason I’ve had to compromise that the seats are rendered from a <li> tag within the HTML, and the are no grouping expressed within that list of 25 × 6 seat permutations. Un-orderd lists don’t have the ability to support groupings in a way that would be legal for HTML. The model in Angular has to support the view, and if it does not, then it has to change. Thus I flattened it. I would refactor my backend logic to be able to serve that up. If the app were to send the whole seat plan back to the server to persist it, I’d change that handling of that too, to support a receiving flatter seat list.

Final thoughs

JQuery has moved on quite a bit since 2007. The core library itself has improved, but there has also been a slew of controls in the “JQuery UI” space. Lastly both Handlebars and JavaScriptMVC are technologies that independently pushed the capabilities of JQuery in the directions I’m talking about here. With a mix of these, code reductions towards that I’ve achieved above, are possible.

Client-Side MVC: browser plugins

Tue, 14 Feb 2012 00:00:00 PST

I’ll do a quick bit of blue-sky thinking. The range of ultimate ‘winner(s)’ of the Client-Side MVC frameworks shakedown, could do worse than have sympathetic plugins for developers to use. I can think of two things that would help developers.

View Source (on Angular source):

View in Page (after some hot-key is pressed):

Client-Side MVC frameworks compared

Mon, 13 Feb 2012 00:00:00 PST

A month ago Gordon L. Hempton wrote about twelve JavaScript frameworks in the Client-Side MVC space. His rating criteria were different to mine. One that really sticks out is that I like the logic not forcing the template HTML to migrate to <script> tags. Depending on the sophistication of the app, I like to be able to see the app in a browser or DreamWeaver when the framework is not running. It gives me a way of gauging the composition of the app. It appeals to a WYSIWYG leaning that I have. I like my UI frameworks to be built for designability if you like.

Addy Osmani has a number of implementation of a TODO app on a github pages site. For composition purposes, this really is the definitive place presently. Using those, I’m going to scrutinize the HTML and how it the app looks without JavaScript. I checked out Addy’s repo, then recursively deleted all javascript files, before loading the main page for each into a browser.

Angular

In the browser:

In DreamWeaver:

What I like is that I can see the repeating item in it’s mustache templating style: {{todo.content}}. What I don’t like is that I can’t see the “X item(s) left” message, that live updates. Angular leaves your template HTML in-situ – where it would be if there were no angular logic happening to it. Here’s the todo list in code (ng:repeat is the looping construct):

<div id="todos">
    <ul id="todo-list">
      <li class="todo" ng:class="'editing-' + todo.editing + ' done-' + todo.done" ng:repeat="todo in todos">
        <div class="display">
          <input class="check" type="checkbox" name="todo.done" / >
          <div ng:click="editTodo(todo)" class="todo-content"> {{ todo.content }} </div>
          <span class="todo-destroy" ng:click="removeTodo(todo)"></span>
       </div>
      <div class="edit">
        <form ng:submit="finishEditing(todo)">
            <input class="todo-input" my:focus="todo.editing" my:blur="finishEditing(todo)" name="todo.content" type="text">
        </form>
      </div>
      </li>
    </ul>
</div>

Backbone

Missing is the repeating TODOs. Instead, in the page, there is a <ul id="todo-list"></ul> placeholder for Backbone to insert child elements to later. Here’s the template, in canonical Backbone (hidden inside a <script> tag):

	<!-- Templates -->

    <script type="text/template" id="item-template">
      <div class="todo <%= done ? 'done' : '' %>">
        <div class="display">
          <input class="check" type="checkbox" <%= done ? 'checked="checked"' : '' %> />
          <label class="todo-content"><%= content %></label>
          <span class="todo-destroy"></span>
        </div>
        <div class="edit">
          <input class="todo-input" type="text" value="<%= content %>" />
        </div>
      </div>
    </script>

There’s no ng:repeat equivalent code in the HTML, that’s done in JavaScript.

Knockout

I’m not sure if the way the TODO app was coded with knockout is the only way, but the line that would show an individual TODO item from the list of TODOs looks like <div class="todo-content" data-bind="text: content, event: { dblclick: edit }" style="cursor: pointer;"></div>. Without the frameworks attached, there’s nothing to see for that DIV. If I hard-code TODO-CONTENT in that DIV, and reload, then it looks more like Angular:

It is nice to see the inline templating rather than the template in a <script> tag like Angular:

<div id="todos">
    <div data-bind="visible: todos().length">
        <input id="check-all" class="check" type="checkbox" data-bind="checked: allCompleted" />
        <label for="check-all">Mark all as complete</label>
    </div>
    <ul id="todo-list" data-bind="foreach: todos">
        <li data-bind="css: { editing: editing }">
            <div class="todo" data-bind="css: { done : done }">
                <div class="display">
                    <input class="check" type="checkbox" data-bind="checked: done" />
                    <div class="todo-content" data-bind="text: content, event: { dblclick: edit }" style="cursor: pointer;"></div>
                    <span class="todo-destroy" data-bind="click: $root.remove"></span>
                </div>
                <div class="edit">
                    <input class="todo-input" data-bind="value: content, valueUpdate: 'afterkeydown', enterKey: stopEditing, event: { blur: stopEditing }"/>
                </div>
            </div>
        </li>
    </ul>
</div>

Knockback

Screenshot – as Backbone’s

Knockback is a very pragmatic merger of the best features of Backbone and Knockout. Unlike many of the websites for these technologies, Knockback’s site tries to instantly sell the technology to the developer. Like Backbone, Knockback keeps templates separate. Here’s their todo list: <ul class="todo-list" data-bind="template: {name: 'item-template', foreach: todo_list.todos}"></ul>. Here is the template for that:

<script type="text/x-jquery-tmpl" id="item-template">
  <li>
    <div class="todo" data-bind="css: {done: done, editing: edit_mode}">
      <div class="display">
        <input class="check" type="checkbox" data-bind="checked: done" />
        <div class="todo-text" data-bind="text: text, dblclick: toggleEditMode"></div>
        <div class="todo-destroy" data-bind="click: destroyTodo"></div>
      </div>
      <div class="edit">
        <input class="todo-input" type="text" data-bind="value: text, event: {keyup: onEnterEndEdit}" />
      </div>
    </div>
  </li>
</script>

Broke

Screenshot – as Backbone’s

The template logic is in JavaScript files, which I really don’t like:

todo.templates= {
    list: '<ul class="items">' +
        '{% for task in taskList %}' +
            genericTaskTemplate +
        '{% endfor %}' +
    '</ul>'
    ,view: genericTaskTemplate
    ,create: ''
    ,update: '<li class="item" data-app_label="{{ task.__class__._meta.appLabel }}" data-model="{{ task.__class__._meta.modelName }}" data-pk="{{ task.pk }}">'+
        '<form action="#/task/update/{{ task.pk }}/">' +
            '<input type="text" name="title" value="{{ task.title }}" />' +
        '</form>' +
    '</li>'
};

Sammy

Screenshot – as Backbone’s

The template logic is in separate .template file, which seems unnecessary given it’s just an extended HTML format.

<h2 data-type="list" data-id="<%= list.id %>"></h2>
<% $.each(todos, function(index, todo) { %>
  <li data-type="todo" data-id="<%= todo.id %>" class="<%= todo.done ? 'done' : '' %>">
    <div class="todo">
      <div class="display">
        <input class="check" type="checkbox" <%= todo.done ? 'checked' : '' %>/>
        <span class="trashcan" data-type="todo" data-id="<%= todo.id %>"></span>
        <span contenteditable="true" data-type="todo" data-id="<%= todo.id %>" class="todo-item"><%= todo.name %></span>
      </div>
    </div>
  </li>
<% }); %>

Ember

<script type="text/x-handlebars">
{{#view id="todos"}}
  {{#collection id="todo-list" contentBinding="Todos.todosController" tagName="ul" itemClassBinding="content.isDone"}}
    {{view Ember.Checkbox titleBinding="content.title" valueBinding="content.isDone"}}
  {{/collection}}
{{/view}}
</script>

You should know by now that I’m really going to like Ember’s templating. From my point of view this one is in last position!

ExtJs

Screenshot – as Backbone’s

The template logic is in JavaScript files. ExtJs is not so much HTML augmented with Client-Side MVC capability like the rest are. It’s more of a departure from HTML and a reality built on it’s own DSL which can be highly attractive in its own right. I’m only including it because Addy has it on his list. In this case, the ExtJS grammar is showing inlined HTML in the pertinent list of TODOs section:

Ext.define('Todo.view.TaskList' , {
    store: 'Tasks',
    loadMask: false,
    itemSelector: 'div.row',
    extend: 'Ext.view.View',
    alias : 'widget.taskList',
    tpl: Ext.create('Ext.XTemplate',
        '<tpl for=".">',
            '<div class="row">',
                '<input type="checkbox" {[values.checked ? "checked" : ""]} />',
                '<span class="{[values.checked ? "checked" : ""]}">{label}</span>',
            '</div>',
        '</tpl>',
        {compiled: true}
    )
});

Fidel

Screenshot – as Backbone’s

<script type="text/template" id="item-template">
  <li class="todo {!= todo.done ? 'done' : '' !}" data-todoid="{!= todo.guid !}">
    <div class="display">
      <input class="check" type="checkbox" {!= todo.done ? 'checked="checked"' : '' !} />
      <div class="todo-content">{!= todo.name !}</div>
      <span data-action="destroyTodo" class="todo-destroy"></span>
    </div>
    <div class="edit">
      <input class="todo-input" type="text" value="" />
    </div>
  </li>
</script>

Like Backbone, or this example, there’s no ng:repeat equivalent code in the HTML. It is done in JavaScript instead.

JavaScriptMVC

Screenshot – like Backbone’s. Template nesting (inside <script> tags) looks powerful:

<script type='text/ejs' id='todosEJS'>
	<% for(var i =0; i < this.length ; i++){ %>
		<li <%= this[i]%>>
		<%= $.View('todoEJS',this[i] ) %>
		</li>
	<% } %>
</script>
<script type='text/ejs' id='todoEJS'>
	<input type='checkbox' name='complete' 
		<%= this.complete ? "checked" : "" %>>
	<span class=''><%= (this.text || "empty todo ...")%></span>
	<span class='destroy'></span>
</script>

JQuery with Handlebars

Screenshot – like Backbone’s. Separate template again:

<script type="text/x-handlebars-template" id="todo-template">
{{#this}}
<li {{#if done}}class="done"{{/if}} data-id="{{id}}">
	<div class="view">
		<input class="toggle" type="checkbox" {{#if done}}checked{{/if}}>
		<label>{{title}}</label>
		<a class="destroy"></a>
	</div>
	<input class="edit" type="text" value="{{title}}">
</li>
{{/this}}
</script>

Spine

Screenshot – as Backbone’s. Template again leverages <script> again but is inline with the HTML that contains it. Looping is not in the grammar though, so is performed by JavaScript as the template is acted on.

<div class="items">
  <script type="text/html" id="task-template">
    <div class="item {{if done}}done{{/if}}">
      <div class="view" title="Double click to edit...">
        <input type="checkbox" {{if done}}checked="checked"{{/if}}>
        <span>${name}</span> <a class="destroy"></a>
      </div>

      <form class="edit">
        <input type="text" name="name" value="${name}">
      </form>
    </div>
  </script>
</div>

YUI Library

Screenshot as Backbone’s. As is the norm the template is in a <script> tag, with the looping in Javascript.

<script type="text/x-template" id="todo-item-template">
    <div class="todo-view">
        <input type="checkbox" class="todo-checkbox" {checked}>
        <span class="todo-content" tabindex="0">{text}</span>
    </div>

    <div class="todo-edit">
        <input type="text" class="todo-input" value="{text}">
    </div>

    <a href="#" class="todo-remove" title="Remove this task">
        <span class="todo-remove-icon"></span>
    </a>
</script>

Batman

(added 14 Feb)

Batman’s TODO is in it’s own Github Repo, and not Addy’s one (yet). Batman uses CoffeeScript instead of JavaScript, which is alluring in its own right. Here’s the screenshot:

Like Knockout, the bound field is not visible in design mode. Similarly, the number of items statistic is also missing. Lastly the <a> tag for ‘delete’ is missing a and thus does not appear as a link. The other frameworks were using fancy images for this, so that is probably the same for them too, but not apparent in their demos. I’ve hacked those things in (hard-coded) and taken another screenshot for your enjoyment:

The good news (to me) is that the Batman extends HTML like Angular and Knockout.

Conclusion.

Only Angular, Knockout and Batman allow me to see what’s what in design mode, and see the mechanism of looping in the extended HTML grammar. That’s the opposite of one of Gordon L. Hempton’s “good” criteria.

It is also interesting that one of these has Microsoft patronage, and another has Google support. I wonder if there’s a Dart port “Dangular” in the future from the Google fellows :-P

After thought.

Angular’s TODO list implementation could be even better in design mode, if it encoded stats like so:

<div id="todo-stats">
  <span class="todo-count" ng:show="hasTodos()">
	{{statsCount()}} item{{statsPlural()}} left.
  </span>
  <span class="todo-clear" ng:show="hasFinishedTodos()">
    <a ng:click="clearCompletedItems()">
      Clear {{finishedTodos()}} completed item{{finsihedPlural()}}
    </a>
  </span>
</div>

Instead of the way it is now:

<div id="todo-stats">
  <span class="todo-count" ng:show="hasTodos()">
    <ng:pluralize count="remainingTodos()" when="{'0' : 'No items left.', '1': '1 item left.', 'other' : '{} items left.' }">
    </ng:pluralize>
  </span>
  <span class="todo-clear" ng:show="hasFinishedTodos()">
    <a ng:click="clearCompletedItems()">
      Clear <ng:pluralize count="finishedTodos()" when="{'1': '1 completed item', 'other' : '{} completed items' }"></ng:pluralize>
    </a>
  </span>
</div>

We’d get a design mode view like so:

Client-Side-MVC applications : best practice (part 1)

Fri, 10 Feb 2012 00:00:00 PST

I’ve made a quick game – “Whack a Mole – Angular Edition”. Try clicking on the moles or holes:

Hmm, the mole image has some lasso artifacts after I reappropriated it from wikipedia. Damn you CorelPaint!

Now, look at the source (it’s in an iframe so you’ll see the minimum amount of HTML). Of if you trust me not to do some sleight of hand, you can look at the github source for that

Aside from the 2 × 2 JSON “moles” document with a not-so-random “up” versus “down” state, the angular magic is:

The thing I want folks to really grok right now, is that there are two images in the source apparently side by side within each of the TD elements. Specifically “mole up” and “mole down”. It is Angular that’s making one one of the two show at any one moment in time. Gecko and Webkit are fast enough to not make it apparent that there is momentarily two or none of the images visible for a cell. Perhaps Angular has no such tweening state, and is even in charge of the browser repaints. The ng:click stuff is where the model gets changed – or rather the “up” becomes “down” or vice versa. ng:click is attachable to any meaningful element of course.

Here is what the source looks like in DreamWeaver’s WYSIWYG mode for the single apparent cell before Angular re-does the DOM:

Yes that’s the mole both apparently alive and dead at the same time.

Here’s an alternate way of doing the same thing without the two image and ng:show & ng:hide trick, that I would count as a refactoring:

Whichever way you do it, the model drives the view (no JavaScript to shuttle back and forth values). And the view mutates the model – in this case via expressions co-located with HTML. The Angular guys suggest that these things are better done with real JavaScript functions that can touch the model, and they are right as far as testability is concerned. But there’s a preotyping community that’s really going to dig Angular. There’s also a the likes of User Experience (UX) and Business Analysts folks who may like to join in somewhat in the actual development of functionality.

It is interesting too, what Angular does to the DOM at runtime. Below is a screen-shot from Firebug. The ghosted lines are not visible in the browser frame. Thus it’s conformation (if you needed it) that only one image is visible at a time for one cell:

Business apps

You’re going to be able to compose larger applications fairly easily, and leverage plenty of model following UI changing activities. You’re principal duty is to keep the model supporting the view, and refactoring as appropriate to fit the growing vision. The hard stuff, you will still do in JavaScript, but I think that’s the exception rather than the rule, as the Angular makes it all pretty easy.

Consider a table of fruit in a page:

If clicked, a line could expand a little to accommodate more text:

HTML’s rowspan might be the Angular triggered UI trick to accommodate that, by the ng:click would trigger an 'expanded = true' model change for the fruit in question. As retrieved from the database the fruit never had a 'expanded = false' node, let alone 'expanded = true', but that does not matter as in the worse case scenario, your could ignore such properties when attempting to write the fruits back to the database (if they were editable in-situ). Your original JSON document to support the list of fruit might only have enough text to support the collapsed list of fruit. If that’s the case, you’d do an AJAX wire call to get the expanded fruit description, and populate an ‘expandedFruit’ object outside the ‘fruits’ model. Only one fruit is expanded at a time, so it could easily be a single field. Judicious use of ng:show again will make the right control of two show within the <td> .. </td> elements.

Or some form of overlay could happen:

The same backend model, AJAX fetch, and expandedText field, but a different view still driven by ng:show. You could refactor from one design to the other. Indeed the UX person could do so without developer help.

All of a sudden, web apps became simpler to make (again).

The document is the single source of truth

Wed, 08 Feb 2012 00:00:00 PST

I’m inspired by the title of a blog entry that mulls Angular. The title in question, for those that have not clicked is The model is the single source of truth, and it refers to Single Source of Truth concept, which is occasionally bandied around in enterprise computing. Anyway, it came up today at work, so I thought I’d blog.

Models in the normalized database era.

Codd and Date gave us relational database theorems/designs in general and SQL in particular. From the 80’s through to the turn of the millennia, database design was the foremost consideration in application development. The initial work was done the old fashioned way, but later tools like Erwin allowed us to super-achieve in the realm of database design, with “physical” and “logical” models mapped and workflowed to completion. Outputs were DDL that teams could use to deploy databases and fill them with starting records (most likely reference data). Applications could be built on top of that, we were assured.

The Agile software movement (2000 onwards), decided that source control was the reference place for SQL tables/indexes/views, and so was happier to not use such modeling tools upfront. If they wanted pretty-printed models, they were happier to generate them from actual deployed schemas. Having everything on one SCM tool for a single-click-build was the highly desirable thing, and table-shapes changed as needed according to the ‘stories’ being developed in an iteration. Later the tool-chains improved for each language, to allow table-shape delta scripts to be automatically generated for day to day use, or more importantly between production releases. This would be to ‘upgrade’ a database shape, or ‘downgrade’ if a production push were lamented. Really though, for Agileists, the conflict was in the between object modeling and relational-schema modeling. In the eyes of the Agileists the object model was much more important than the table design. But it was not the Agile community per se that drove the next change: to document stores.

Models in the Client-side MVC era.

So everyone gets by now that I love the client-side MVC frameworks, and think they are the future. When it comes to data storage the obvious conclusion is that the backend should save something pretty close to the document that the client presents, mutates, and sends back to the server for posterity. I don’t though, as yet, think in “models” for the data. Instead I think in JSON documents. Indeed, it is the representation on the wire for a GET to populate a page, and the PUT for a hypothetical return of a changed document that my minds-eye considers. Ideally, the document’s structure in those two scenarios should be pretty much the same. Like for a normalized database in a previous era, there are one-to-many relationships etc, but everything for an canonical key is represented in one textual document. Now a backend could split up such a JSON document in a PUT operation, and write/overwrite dozens of records in many tables (in one transaction), but why bother? Use a document store instead. When would you use a normalized DB design today? The answer to that is: only when you have other processes reading and writing to your database. For the most part these days, we’re interfacing to other systems via service calls (Web Services, REST etc), and using yet more documents on Enterprise Service Buses (Tibco, JMS, etc). There is also the possibility of writing a reportable database later, as a compromise.

The development-time economies for avoiding a formal normalized database are huge, and teams should be starting with document stores, and justifying “why not” if needs be. In short folks, get up to speed with the NoSQL movement.

Web-app model quandary

Mon, 06 Feb 2012 00:00:00 PST

Web applications take effort to develop for the most part. Serving static pages is much easier. Over the years the patterns and techniques to develop web-apps have changed, and so has the division between logic we write on the web-server tier versus logic in the browser. We now write much more logic in the browser, and indeed we’re on the cusp of an era where Model-View-Controller in side the browser is the dominant design. Micro-web-frameworks have made that possible, and there are new problems to solve.

In this blog entry, I outline the difficulties making solid web-apps where the model is being heavily mutated. Earlier today I mulled the technologies in play in 1993, 2000 and 2006 for enterprise web-applications, and this blog entry is intended to continue the cataloging of technologies to the current day.

2012: The Micro Web Framework era.

I’ll phrase this as a status quo for the technologies in question.

Micro Web Frameworks are largely divorced from the back-end server technology that hosts the larger web app. As such, we can consider them interoperble with many alternate backends. In that web-server tier, Rails is still an immensely popular technology, but there are new kids on the block: NodeJS, Grails, Scala’s Lift and Play. Rails has competition with Sinatra. It ate its previous competitor – Merb and Rails merged to form Ruby on Rails 3.0. SpringMVC and the one of .Net equivalents are one more the more ‘safe’ enterprise choices. WebSockets and HTML are the cutting edge. PHP is still an amazingly popular choice for startups of course.

The frameworks competing in this space include Backbone, Knockout, KnockBack, Spine, Ember, BatMan, Angular, Sammy, JavaScriptMVC, YUILibrary, SproutCore, Broke and Fidel which all ship as JavaScript scripts from 30K and upwards. The genus kicked off in 2009, but the competition did not heat up until the middle of last year. They are all variations on an MVC theme. HTML markup is kept (and the tool-chains and techniques around that). But the page is now able to have programmed series and other turing complete functionality with an absolute minimum of hand crafted JavaScript. The libraries perform something akin to magic to rewrite the page according to JSON data sources and a meld of more functional enhancements to HTML (that the library recognizes), and a spattering of JavaScript for the more advanced functions.

Back in the web-server tier, the chore is now serializing to and from JSON. Some languages have that built in, some require libraries. Java’s XStream falls by the wayside for Java as it is better transforming to JSON than from it, Luckily there is GSON and Jackson. Indeed having classes that support specific hierarchical structures of JSON is the challenge. The dynamic languages have the advantage of course.

But where should the dominant model be?

The shift in onus is from the model being (mostly) in the web-server tier, to there being no clarity about whether the model is there or in the browser tier. This is the inherent problem right now. A starting position for design (once you’ve decided to go with a micro web framework) is to have the browser tier do all the MVC for the app one ‘page’ at a time, and retrieve JSON from a vestige app-server as well as put it back when done. That makes the back-end more of a document store, and thinner by necessity. It is easy to see that the persistence technologies built to support such apps should naturally retrieve and save such documents rather than normalized sets of records. Old fashioned DB technologies (Oracle, SQLServer, MySql) still have a place, but the direct page-supporting tables should be more like document stores, and have CLOB fields to hold JSON, with simpler key fields. You should be more drawn to the likes of Cassandra, MongoDB, or CouchDB really though.

With these browser-based MVC designs, if the web-app tier is doing any business logic, it is validating data a second time, as one can’t really trust what goes on in the browser with the likes of GreaseMonkey and Firebug around. If it finds validation issues, it is more than likely to send that back to the browser for re-presentation. An easy way to do that would be to embellish the document received with errata, and have the MVC app just display them inline. More about that in a follow up article.

Things get more complicated when a larger document is served to a HTML page, but only smaller sections may be changed at a time. There’s a temptation to POST back the subsection, have the backend validate/persist that, and simultaneously rework the model in the browser to reflect what “would have been served” for the whole document. Why not PUT the whole document to the backend, and reload if needed? If the document is as big as a word document, and your app functions like GoogleDoc’s word processor, then it would die with the overhead of the whole document going up and down per keystroke. Google, has a very efficient wire language to describe the keystrokes being made in a POST operation labelled ‘mutate’. It hints that their model is mirrored to both sides of the wire. It has to be really, as other people can concurrently edit Google Docs.

The GoogleDocs design should make us hope for safety and transactionality with apps that use micro-web-frameworks. The Page could be alive a long time without pushing data back to the web-tier and hence the data storage. A beautiful aspect of MVC is the fact that you can in fact abandon (garbage collect if you like) the model, if the changes it holds is no longer required. The older data-binding style of models is much harder to coerce into abandoning DB writes if required. GoogleDocs allows the rolling back of a series of changes. It feels more like a source control system in that respect. Yet, as long as you are connected, everything is safe, and the chance of loss of data is down to a small number of keystrokes (and for when the browser/machine fails).

By contrast the traditional install of Microsoft Excel means that it’s going to be editing a file-system available XLS document. While you’re editing it, it keeps a backup version just in case it crashes for some reason. You could safely reply on that backup, until you’re ready to for a formal SAVE and close of the worksheet. Your alternate is to close without saving, which is the equivalent of abandoning the changes. With either action, the backup file is deleted. Excel’s metaphor allows for working detached from the network which is handy in for “on an airplane” scenarios.

An SCM-like Workflow?

Is there a model where the web-server initiates a workflow for ‘record’ of some sort, where work is ongoing on that record, and will be completed later? Maybe the record is locked for the duration. maybe not. That would be ‘work-in-progress’ and the representation of it should be mirrored like in GoogleDocs. All changes to it would be safe. You also get the late “submit” or “abandon” reality of a detached model. As it would be document-centric, there could well be a “diffs” that describe the modifications from a starting position, or from last advice. That loses some of the intent of the changes being made though. Martin has an article from seven years ago on Event Sourcing that is well worth a read. Event Narrative is a follow up. Martin is much clearer on the chronalog of changes made to a model, during an implicit save/publish phase. Would the micro-web-frameworks grow a synchronization side that can keep a backend informed with deltas every few seconds? Right now you’re on your own. You write code to POST fragments to keep your models in sync, or you push whole documents. There’s no mirror/sync capability built into the current micro-web-frameworks. The metaphor for the intended save or abandon, feels closer to the commit/merge concept from source control systems.

2/8/11 Update:

Alan Huffman, in email, says the name for what I’m talking about here is “Client-Side MVC”. Yep, that’s it – nice and short.

Previous Web Architectures

Mon, 06 Feb 2012 00:00:00 PST

For enterprise app dev, and to support a following blog entry, I’d like to recap technology choices previously used in the era of the web.

What’s the synopsis? Over the years, logic is progressively migrating from a web-server tier into the browser. Pattens were established in the web-server tier, and were starting to be defined in the web page.

Web application development in 1993, 2000 and 2006.

1993

Web applications were still tightly bound to the server hosting them. Java, Python, PHP and Ruby were a few years from becoming realities for web development. Perl perhaps was a possibility for web development, but you were most likely a C++ developer if you were making web-apps. If Awk were viable then, I can’t glean that from the wikipedia page for it. JavaScript was a year away, and wouldn’t be called JavaScript for another two. If you submitted a form and hit the back button, you had little hope of anything useful happening.

Beyond the declarative nature of HTML programming, and the understanding of the contracts behind URLs, there was no functionality in any turing-complete manner present in web pages.

2000

In the year 2000, Sun’s Java Servlets or Java Server Pages (JSP) apps were a possibility with Model-View-Controller being the recommended pattern. Microsoft had recovered from the mistake of ignoring the web in 1995, and had made Active Server Pages (ASP) the most popular enterprise solution. Indeed they had their fork of Java (google for “Project COOL”) .Net ready for enterprise app dev (see later). PHP had come from nowhere as a CGI solution, and Apache with that mechanism of linkage was the most popular web server. You might have even seen /cgi-bin/ in the URLs of web applications, though rewrites were possible. Allaire’s Cold Fusion (an independent IIS data-binding solution) had by then been snubbed when Microsoft didn’t buy the 90% of the company they didn’t already own and launched ASP instead. ASP apps followed the data-binding solution that fatter MS-Access solutions had pioneered for years. ASP.Net was still two years away though. Python and Ruby may have been used in CGI mode, but Rails (in 2003 I foolishly told DHH it wouldn’t scale in the enterprise) was still three years away, and Django two more. Microsoft had pioneered AJAX style functionality in ActiveX controls, and you might have encountered applications that used them. Similarly Sun could still make a claim that Java Applets were a reality on the web at large. Macromedia’s Shockwave/Flash was a viable platform that competed with, and would ultimately beat both ActiveX & Java Applets.

Cookies are in every app by now (thanks to Netscape and Microsoft’s subsequent adoption of them). JavaScript if used, was mostly for simpler validation concepts within pages, before allowing the POST to go through.

2006

AJAX was the big thing. Between them Microsoft, Netscape and Google had made it popular. Most enterprise teams were building apps that were somewhere between Web 1.0 and Web 2.0. Roy Fielding’s REST is big but it required some decoding for the layman. Ruby on Rails had been a significant productivity force for a year or two. The .Net offerings for the web were solid 2.0 a year before and before the end of the year 3.0 would be released. Model-View-Presenter was the paradigm, and Martin talked about it after some initial ThoughtWorks projects with .Net. Microsoft’s patterns and practices group would later talk about it too. That’s not to say that MVC is dead – most of the non-.Net frameworks are coded in accordance with it. All web-app best practitioners were now doing POST-redirect-GET and you finally hit the back button after posting a form – take that Tim Berners-Lee. ThoughtWorks had a year or two before pushed out Selenium for web-testing (and was one of the early pioneers of Comet as a result), then Sahi for another go at the same idea, with WebDriver (now Selenium2) to come a year later. Web applications were now beautifully decomposed solutions, illustrating Separation of Concerns, Inversion of Control, if not Dependency Injection in particular.

JavaScript usage is quite sophisticated now. Applications might issue AJAX operations to the back end to help with adaptive parts of the page, or individual controls. Libraries exist to allow developers to economize and standardize on idioms. These were YUI, Prototype, script.aculo.us, Dojo, and Mootools, but would see future “winner” of the genus JQuery arrive by the years end. Early adopters would try out (in order of revolutionary zeal) ExtJs, and Google Web Toolkit, but the industry remained divided as to whether devs should embrace JavaScript or avoid it completely in the browser. Cappuccino (even more revolutionary) would come a couple of years later, and suffer the same question.

For the purposes of the following article, the take away is that Model-View-Controller in the web-server tier (as delivered by web frameworks) is a big thing in 2006.

Angular and Selenium

Wed, 01 Feb 2012 00:00:00 PST

I’m the co-creator of the original Selenium 1.0 (or 0.something as it was then) but I mean Selenium2 (nee WebDriver) of course. As UI’s get more sophisticated, the ability to locate the right part of the page for purposes of a query or interaction, is increasingly hard. I have personal experience with GWT that, on occasion, has driven me nuts and a pathological fear of the the super-advanced frameworks like Cappuccino

Finding Elements for and Angular app is not especially hard. Indeed all of Backbone, Knockout, KnockBack, Spine, Ember, BatMan (and the rest) will paint relatively normal HTML and you could get busy with ID, CSS or XPATH based locators for widgets. This blog entry could end there, but I’d like to do a “what if”.

Model-Centric Angular Locators

Here is the model for from monday’s blog entry again:

scope.cart = {
    authenticity_token: "xotITlobeVi6BUnqBIqAd4eFy5+yJf0JlSyog8rK5hQ=",
    items: [ 
        { 
            imgSrc: "index_files/grace-wireless-player.jpg",
            link: "grace-wireless-internet-radio",
            name: "Grace Wireless Internet Radio",
            price: 179.99,
            qty: 1
        },
        { 
            imgSrc: "index_files/plantronics-headphone.jpg",
            link: "plantronics-m40-mobile-in-ear-headset",
            name: "Plantronics M40 Mobile in Ear Headset",
            price: 9.99,
            qty: 2
        }
    ]
};

Here is part of the Angular page:

<tbody id="line_items">
    <tr ng:repeat="item in cart.items" class="">
        <td><img alt="{{item.name}}" ng:src="{{item.imgSrc}}" /></td>
        <td class="description"><h4><a href="http://demo.spreecommerce.com/products/{{item.link}}">{{item.name}}</a></h4></td>
        <td class="unit-price"> {{item.price | currency}} </td>
        <td class="operator"> X </td>
        <td class="quantity"><input style="width: 32px;" name="item.qty" ng:validate="integer:1:5" /></td>
        <td class="operator"> = </td>
        <td class="total"> {{item.price*item.qty | currency}} </td>
        <td class="total"><a ng:click="cart.items.$remove(item)" class="delete button">Remove</a></td>
    </tr>
    <tr class="totals">
        <td colspan="6"> </td>
        <td colspan="2" class="totals"> Subtotal: {{cart.items.$sum('price*qty') | currency}} </td>
    </tr>
</tbody>

What I would like to be able to do is locate parts of the page via the model that’s presenting them. Please excuse the Java:

WebElement quantityInput = webDriver.findElement(id("cart")).findElement(angularModel("items.$filter('{ name: \"plantronics-m40-mobile-in-ear-headset\"}').qty"));
quantityInput.sendKeys("15");

The initial findElement() is for the DIV that contains the ‘MyController’ Angular reference, and is from the regular Selenium2 API. The ‘angularModel’ method is a Selenium2 locator yet to be written for Angular interop. The expression within is more or less handed over the wire to an Angular enabled Selenium2 controlled browser, and the lookup is against the model of course. The Angular/Selenium integration piece returns the widget that is bound to that model node. If there is no widget bound directly, then an exception is thrown all the way back to the test script. If there is more than one widget bound directly to the model node, then a different exception is thrown. Use of ‘Quantity’ in formulae would be considered secondary.

I wonder if $filter() as opposed to a more desirable $find(). The former will subset an array to all that match. What we want in this instance is a way to find an single matching element.

View-Centric Angular Locators

Let’s look for something less model-centric in the page, using a hypothetical Selenium/Angular locator that traverses view references.

WebElement cost = webDriver.findElement(id("cart")).findElement(angularView("item.price*item.qty"));
assertThat(cost.getText(), equalTo("$123.45"));

Neat huh? We could find anything Angular managed whether output or input. Of course, Angular’s recommended test idiom is via Misko’s other stroke of genius – JsTestDriver.

Command Line: A Freebie

What if we had a custom control for Angular that allows an effective command line interaction with the current model:

.. snip ..
      </table>
    </div>
    <p class="actions"> <a ng:click="todo()" class="continue">Continue shopping</a>  
      or   <a ng:click="todo()" class="button checkout">Checkout</a> </p>
    <div ng:commandline style="width: 500px; height: 100px;"/>
</div>
.. snip ..

We could interact with that using the same model traversal techniques as described above. ThoughtWorker colleagues would love for a command line interface to our “time and expenses” app. It is currently a Rails app, and was previously a pre-Django Python app when it was the test bed for the earliest version of Selenium 1.0. Before that it was a Lotus Notes app. Anyway, it could easily be an Angular app with a command line for expedited time and/or expense entering:

Conclusion

In the race for dominance between Backbone, Knockout, KnockBack, Spine, Ember, Batman, Sammy, YUILibrary, JavaScriptMVC, Broke, Fidel and my current favorite Angular, part of the criteria for winning is going to include:

Elegant left-to-right style model traversal expressions.
Selenium2 integration for low effort full stack testing.
A command line interface :)

Angular and DreamWeaver

Wed, 01 Feb 2012 00:00:00 PST

DreamWeaver feels like something from yesteryear, and maybe most web application development is done without it these days. Flipping between a text editor like TextMate and a running browser like Firefox with Firebug and other plugins running is normal.

That said, Dreamweaver has always claimed to be a WYSIWYG editor for HTML source, that is aware of a number of markup regimes, with every intention of leaving stuff it does not recognize alone. Stuff in this instance being elements and attributes principally, but also macros that it might see in between tags. There were many years thought where DreamWeaver would be an admirable tool for round trip editing for ASP, but would desecrate a JSP page if it had the chance. I’m not sure if that’s fixed in version 11.5, as who uses JSP anymore?

Here’s the the excellent SpreeCommerce demo from monday’s blog entry cart again, this time in Dreamweaver. Specifically the version of the original saved source, but after I used Dreamweaver to reformat the HTML source:

Actually, the version before the reformat looks identical in DreamWeaver.

Viewing Angular source with DreamWeaver.

Here is what the same thing looks after I did the initial conversion to Angular:

The Mustache style expressions into the JSON model are visible where they are inlined into text. They are somewhere between unobtrusive and useful in this form. Dreamweaver recognizes that I’ve made changes outside the editor, and reloads. If I’m using either of Intellij IDEA or WebStorm they save on loss of focus automatically, so I’m less likely to be in a state where I’ve got unsaved changes in both, and no ability to merge them.

Making a change in DreamWeaver

I changed some plain text and a mustache expression, and moved a column that I know to have an invisible Angular attribute in:

And here’s how the source looked as DreamWeaver saved it. Diff format for the file obviously, to show how utterly clean this was:

diff --git a/index.html b/index.html
index e239f09..5d33561 100644
--- a/index.html
+++ b/index.html
@@ -51,7 +51,7 @@
     <nav><a href="http://demo.spreecommerce.com/products">products</a> <a href="http://demo.spreecommerce.com/login" class="cart">Log In</a> </nav>
   </header>
   <div id="main" role="main">
-    <h1>Shopping Cart</h1>
+    <h1>CHANGED</h1>
       <h3>You have {{cart.items.length}} items in your cart</h3>
       <div>
         <table id="cart-detail">
@@ -60,25 +60,25 @@
               <th colspan="2">Item</th>
               <th>Price</th>
               <th> </th>
+              <th> </th>
               <th>Qty</th>
               <th> </th>
               <th>Total</th>
-              <th> </th>
             </tr>
           </thead>
           <tbody id="line_items">
             <tr ng:repeat="item in cart.items" class="">
               <td><img alt="{{item.name}}" ng:src="{{item.imgSrc}}" /></td>
-              <td class="description"><h4><a href="http://demo.spreecommerce.com/products/{{item.link}}">{{item.name}}</a></h4></td>
+              <td class="description"><h4><a href="http://demo.spreecommerce.com/products/{{item.link}}">{{item.CHANGED}}</a></h4></td>
               <td class="unit-price"> {{item.price | currency}} </td>
               <td class="operator"> X </td>
+              <td class="total"><a ng:click="cart.items.$remove(item)" class="delete button">Remove</a></td>
               <td class="quantity"><input style="width: 32px;" name="item.qty" ng:validate="integer:1:5" /></td>
               <td class="operator"> = </td>
               <td class="total"> {{item.price*item.qty | currency}} </td>
-              <td class="total"><a ng:click="cart.items.$remove(item)" class="delete button">Remove</a></td>
             </tr>
           <tr class="totals">
-            <td colspan="6"> </td>
+            <td colspan="7"> </td>
             <td colspan="2" class="totals"> Subtotal: {{cart.items.$sum('price*qty') | currency}} </td>
           </tr>
             </tbody>

There is one mistake concerning the column in moved leftwards: there is a total-row cell that spans 7 columns now instead of 6. It’s a shame that DreamWeaver does not have a “move column” refactoring. When I said “cut/paste”, the reality was “cut-column”, “insert blank column” & select it, paste the clipboard into the blank column, then remove the now-blank rightmost column. Removal was done in the source view as “Delete Column” does not actually delete the column, it just wipes out the content. That was the only operation not performed in WYSIWYG mode.

The hard part for DreamWeaver to do right here was the preservation of the ng:click=“cart.items.$remove(item)” attribute. It worked a treat, but this is not surprising as DreamWeaver has been preserving the extra attributes and elements for Coldfusion Markup syntax from the mid-90’s onwards.

It seems to me that best practice with Angular if you’re using DreamWeaver, is to re-format as needed in DreamWeaver, and not so in your preferred JetBrains IDE. Then ping between the two tools at your leisure and feel smug at how clean the diffs are.

(almost) No JavaScript prototyping with Angular in a single source file

Sun, 29 Jan 2012 00:00:00 PST

[ Following on from last week’s blog entry ] So what is this Angular stuff like in a real world app? We’ll I’ve gone to the demo site for SpreeCommerce’s open source shopping cart and saved a live snapshot of the page using the MAFF plugin for Firefox. SpreeCommerce is a good target for me presently, as the copyright/license for it does not preclude hacking to showcase Angular.

I checked that into Github as a new repository : https://github.com/paul-hammant/spreecommerce_angular. There are three commits, if you look here. The first is a checking as was saved. The second, was the result of loading the page into DreamWeaver, and reformatting. The third is the result of replacing the Web 1.0 functionality (Ruby on Rails) with Angular. If you click on the diff link in the github interface you can see what I did precisely. Generally though, I removed all other JavaScript, and changed the first row of the cart to a repeating row in the Angular style. There’s also a JSON model for the cart contents, which for the purposes of honoring this blog entry’s title, I’m going to consider that not JavaScript as such. The only real JavaScript is a wrapper for an an alert box, that’s hooked into “Continue Shopping”, and “Checkout” as they don’t actually work. The won’t work until the backend is reworked to handle a JSON version of the cart contents on a POST. Incidentally, if you’re new to this, ng:repeat=“item in cart.items” is where the magic is really hooked in.

You can play with the finished product here: http://paul-hammant.github.com/spreecommerce_angular/ (as well as look at the source in situ). The page looks like this:

The difference to the original is that the “update” button is missing as POST-redirect-GET to the server side is not longer needed to recalculate the cart size.

Angular’s advantage restated

The expressions that are possible on augmented arrays in particular, and angular ‘functions’ generally, are the stroke of genius power boost in my opinion. They are allowing me to prototype without writing JavaScript functions, yet build a highly functional client-side application. In the example above we have $sum(expression) used for a subtotal summing all the result of price times quantity for ALL the items in the model:

<td colspan="2" class="totals"> Subtotal: {{cart.items.$sum('price*qty') | currency}} </td>

For the individual item ‘row’ in the cart, $sum(..) is not used, but an expression is:

<td class="total"> {{item.price*item.qty | currency}} </td>

Misko has also added convenience functions to modify the model, for user interaction:

<a ng:click="cart.items.$remove(item)" class="delete button">Remove</a>

Not shown in this example, are $filter :

<tr ng:repeat="item in cart.items.$filter('price > 9.95')"/>
  <td>..</td>
</tr>

Nor is $orderBy :

<tr ng:repeat="item in cart.items.$orderBy('price', false)"/>
  <td>..</td>
</tr>

Expressions can be hard coded in single-quotes, or variables passed in. In the case of ordering tables, you’d assign the order column to a variable, and have some other UI control change that (as well as ascending or descending). I’ve an example of that, and filtering here from a year ago: http://paul-hammant.github.com/StoryNavigator

Powerful stuff huh?

A better Etsy.com example.

Etsy.com (a fantastic shopping site for hand-made designer craft items) have a items within stores within the cart. It’s more of a model. I’ve made an Angular version of their cart too, but presently don’t yet have permission to push the source to Github (it’s copyrighted to them, not open source). Here’s the screen-shot:

The JSON source for the ‘cart’ looks like so:

scope.cart = {
    stores:[
        {
            cartId: 138229652,
            name: "GoldenSection",
            items:[
                {
                    id: "89166902",
                    txt: "January - photography print calendar zodiac Aquarius Capricorn winter birthday owl romantic gift for her him unisex home decor wall art love",
                    short: "january-photography-print-calendar",
                    price:  26,
                    shipping: 5,
                    qty: 1,
                    note: "",
                    img: {
                        url: "index_files/il_75x75.297883532.jpg",
                        w: 75,
                        h: 75
                    }
                }
            ],
            payment: "Paypal"
        },
        {
            cartId: 138148701,
            name: "RetroModernArt",
            items:[
                {
                    id: "69841278",
                    txt: "Bike Art Black Tandum Bike 8x10 Linocut art print",
                    short: "bike-art-black-tandum-bike-8x10-linocut",
                    price:  24,
                    shipping: 4,
                    qty: 1,
                    note: "",
                    img: {
                        url: "index_files/il_75x75.249358938.jpg",
                        w: 75,
                        h: 75
                    }

                },
                {
                    id: "85933033",
                    txt: "Valentines Day LOVE Red Block Print - LOVE art in RED Typography Woodblock 5 x 7 on white paper",
                    short: "valentines-day-love-red-block-print-love",
                    price:  12,
                    shipping: 2,
                    qty: 1,
                    note: "",
                    img: {
                        url: "index_files/il_75x75.297883532.jpg",
                        w: 75,
                        h: 75
                    }

                }
            ],
            payment: "Paypal"
        }
    ]
};

There are some interesting things to note, that are not as prominent in the SpreeCommerce cart that I have released the source for, but still true:

I’m mixing reference information in the cart’s JSON, alongside mutable entities like ‘quantity’.
index_files the directory does not really hold any images, that’s just the way the MAFF plugin saved a local copy.

In the case of the former, I could do a refactoring to separate the ‘never changes’ reference data from the cart contents that does change (quantities change and lines disappear). In the end though it does not matter. The whole JSON document being pushed back to the server for a ‘checkout’ action is hardly going to hurt it. I’ve optimized the JSON document to support the user interface composition, which is what I think you should do for Angular apps.

Etsy’s Cart Size.

Remember that items are within stores that are within the cart. Here’s the use of $sum() to flatten that to a single number:

{{cart.stores.$sum('items.length')}}

Etsy’s orders have a shipping amount too.

The per store expression to calculate a total for the items obeys the standard order of operations of course:

{{store.items.$sum('price*qty + shipping') | currency}}

Etsy’s grand total:

Well this isn’t possible yet, in an expression. Just as well its not needed for the Etsy cart:

{{ cart.stores.$sum('store.$sum('price*qty + shipping')') | currency }}

Yeesh!

More Aggregate Functions

I’m really enamored by thinking of the JSON document as a live database within the web page. I was pretty good at SQL in the early 90’s and loved the composition of reports with judicious use of SQL’s GROUP BY, HAVING and UNION. If you ever meet me for an interview (ThoughtWorks or Client) and have put SQL on your resume, I’m going to see how far you can go with those.

Take a look at a tutorial for Group-By. We can used that to explore a few more array (aggregate) functions in Angular. Functions that don’t exist yet, but I hope will be implemented in due course.

scope.orders = [
    {
        OrderDate: "2008/11/12",
        OrderPrice: 1000,
        Customer: "Hansen"
    },
    .... etc
];

Group-by:

<tr><td>Customer</td><td>Total Order</td></tr>
<tr ng:repeat="custOrd in orders.$groupBy('Customer').$select('Customer, sum(OrderPrice) as OrderTotal')" />
  <td>{{custOrd.Customer}}</td><td>{{custOrd.OrderTotal}}</td>
</tr>

Having (equivalent):

<tr><td>Customer</td><td>Total Order</td></tr>
<tr ng:repeat="custOrd in orders.$groupBy('Customer').$select('Customer, sum(OrderPrice) as OrderTotal').$filter('OrderTotal > 1000')" />
  <td>{{custOrd.Customer}}</td><td>{{custOrd.OrderTotal}}</td>
</tr>

Min, Max, Average:

<tr><td>Customer</td><td>Minimum Order</td><td>Maximum Order</td><td>Average Order</td></tr>
<tr ng:repeat="custOrd in orders.$groupBy('Customer').$select('Customer, min(OrderPrice) as MinOrder, max(OrderPrice) as MaxOrder, mean(OrderPrice) as AverageOrder')" />
  <td>{{custOrd.Customer}}</td><td>{{custOrd.MinOrder}}</td><td>{{custOrd.MaxOrder}}</td><td>{{custOrd.AverageOrder}}</td>
</tr>

Simpler aggregate Functions.

It is possible to consider the following …

orders.$min(‘OrderPrice’)
orders.$max(‘OrderPrice’)
orders.$mean(‘OrderPrice’)

… on the whole array, or an array after being filtered:

<span>Hansen average: {{ orders.$filter("{ Customer: 'Hansen'}").$mean('OrderPrice') }} </span>

It is much less useful than the pivoted table version that Group-By gives us though.

UI Technology Paradigm Shift

Mon, 23 Jan 2012 00:00:00 PST

I’ve blogged about Angular before in 2009, and I’ve been aware of it since May 29th, 2009 when Ola Bini and had breakfast with Misko Hevery at Google before rolling into the first “Wave hackathon” the day after it was announced at Google I/O. We (I ?) chuckled at Misko then for his in-a-web-page idea, and since finding out more, I’m regretting that.

What has changed?

Well it has been over two and a half years since Misko and Adam Abrons started this thing. Initially they were thinking of a data-service priced by the mega-byte and you’d host your Angular pages anywhere. Now with Adam not so active for the time being, and Googlers Igor Minar, Vojta Jina as well as many contributors, Misko’s Angular is more than ready for large app development. Production ready, despite not being a 1.0 release yet (that’s coming soon I’m told).

There’s also new competition in the “declarative HTML extending tech” space – Backbone (October 2010), Knockout (July 2010), Batman (January 2011), Knockback (November 2011), YUI Library (urggh; when??).

Knockback is a serious attempt to merge the best of Backbone and Knockout by Kevin Malakoff, that deserves further attention I think. If nothing else, it’s comparison table is a powerful starting point to think about the pros and cons of these technologies.

What’s the criteria for selection?

Can bind to nodes in a JSON document, especially for repeating ones.
Feels like an extension to HTML (albeit not a W3C approved one)
HTML is round-trip editable in DreamWeaver. e.g. DreamWeaver does not take out HTML attributes it does not recognize on ‘Save’ so technologies that rely on attributes for implementation are safe.
Doesn’t require much Javascript/CoffeeScript in order to get into the rapid prototyping of the whole UI experience.

Not making the cut as far as I’m concerned.

JavaScriptMVC doesn’t make the list because of its verbose templating / binding
Ember (some time in early 2011?) sadly still has a verbose HTML syntax
Spine (when?) has some ugliness around its logic that makes it feel like it’s not an natural extension of HTML.
Sammy (March 2009) has separated templates that feel like a whole other world. This is only a little further than backbone which I like a little, so maybe I’m being too hard on Sammy.

Not sure about my selections? – take a look for yourself: Addy Osmani has a repo that compares implementations of a TODO list on Github That project is the definitive place for comparing these presently. That said the reader some take some caution, as some of the TODO list implementations are minimalist, and some have ‘bells and whistles’.

The Paradigm shift.

If you get into the way of working with these technologies, you start to think of the document on the wire as the contract. That document is JSON of course, and emerged through iterations of UI development. The development is so rapid that you could almost sit next to your end user and ever minute or so say “how about this?”

This is Model View Controller (MVC) in a web page at its best. Purists will say that it’s Model View View Model (MVVM) which is a derivation of Model View Presenter, which as far as I’m concerned was necessary because of the unit-testing complexity of earlier ASP.Net implementations. I’m really happy with thinking of this as MVC though.

It will force you to change your back-end from traditional designs. It is going to have to serve up that document for the page load, and accept it back (modified) and subject to verification when the end-user finishes with the page. Maybe incrementally as the page is used are a possibility. As such the storage for the data behind the document could potentially be a lead weight. If you choose a traditional design (highly normalized data), that’s going to definately be the case. Perhaps if you are hell bent on that reality, you should delay its creation as long as possible while developing the app, and use something lightweight for the time being. In terms of backend technologies you’re still wide open, though ones that implement JSON are a must. Node, Scala, Ruby, Groovy have JSON built in, though even Java can play well too with the likes of GSON.

The whole development experience is going to be faster. Perhaps 10x over what you’re used to. I’m reminded of Microsoft-Access applications from the mid 90’s where non-technical people could put together shared DB apps for small teams with forms/ reports etc quite quickly. Lotus Notes plays (played) to the same space. Cold Fusion and PHP, back then, were attempts to get the simplicity into web app development, and to lower the bar for development generally. Anyway, the data-bound yet MVC nature of Angular and its ilk really are an accelerator on rapid app dev. I’ve also been measuring page performance with the latest release across for all tier one browsers, and some four thousand data-bound fields on an local Angular page are now being rendered inside of a second. As a result, I’m happy this is quick enough for most apps.

This stuff is so fast for application development, that it should really be your default for new projects, and you then determine if you need to shift away from it for more sophisticated applications. To give you a feel to suitability – booking apps, shopping experiences, and anything that would have fitted a 100 table schema are all very much within reach of Angular, Knockback etc. Content Management systems, Games, GoogleDocs, apps that feel like desktop ones are perhaps not. For those think of GWT, ExtJS and Cappuccino if they must be web-apps.

If these technologies take off, Agile practitioners should be pleased because it shifts the emphasis away from data driven / long-cycle design towards an naturally iterative and emergent one. UX driven development/design, in order words.

Thoughts on Angular per se.

It’s the leader as far as I’m concerned because of the ability to drive a single page from model mutations and Angular attributes sprinkled throughout the page. Even occasional or modal parts of the page. The breakthrough thinking for me was a realization that you could have more than one model in a page. One for the data that came from the server, and one for the flow-control for the page that isn’t going to be saved anywhere. One more for the reference data that came from the server (or was inlined in the page, if it doesn’t change too much). More than once I’ve saved a HTML page using Firefox’s MAFF plugin, then operated on the source to a) remove all JavaScript that normally makes the page, replace the JSON and models that the original web-technology placed in the page with an Angular loaded JSON document. You could do so quite easily for a Amazon’s cart page, or any Airline’s “available flights” page. With Angular in particular, I’m not having to push HTML into ‘script’ tags to make them into templates.

Angular sells itself via the tag line “What HTML would have been, had it been designed for web apps” which is better than their page title “Superheroic JavaScript MVC Framework”. Decipher Inc suggest The model is the single source of truth. I think there’s some general messaging that could be improved, especially when compared to the competing frameworks. It is also lamentable that the AngularJS.org site is (a) not presently indexed by Google, and (b) a place to get stuck on, as the back-button doesn’t allow you to leave it.

Lastly, I think they’re headed in the wrong direction on one specific aspect. Array functions (as they call them), that would have lead into more sophisticated aggregate functions in the style of SQL’s Group-By and Having. In their live example on the lower right of their front page, the remaining number of TODOs is presently:

<span>{{remaining()}} remaining</span>

Mapping to a JavaScript function:

scope.remaining = function(){
  return angular.Array.count(scope.todos, function(todo){
    return !todo.done;
  });
};

This used to be the following expression, without an accompanying JavaScript function:

<span>{{todos.$count("!done")}} remaining</span>

I hope they’ll revert the commit that took that out.

Next up : “(almost) No JavaScript prototyping with Angular in a single source file”

PS – I’m still dreaming of a nested UI markup that’s not an derivative of XML that has inline turing completeness and closures.

Jan 28 – update

Kevin has a thorough response on his blog. Well worth a read!

Sauce Labs: Tunnels or not?

Sun, 11 Dec 2011 00:00:00 PST

Below are three architecture diagrams pertaining to Sauce Labs OnDemand usage for parallel testing of webapps with Selenium 1 or 2. You’re using Sauce Labs, of course, if you need 50 to 500 parallel browsers testing a web-app constructed from one or more developer commits – right ?

Using a Sauce Connect Tunnel for content

An illustration of Sauce Labs’ Sauce Connect technology being used to poke through your corporate firewall to the web-server that otherwise is not accepting connections from ‘outside’

Using a Sauce Connect Tunnel for content and commands

As above but the commands from Selenium go the browser also go through the tunnel. While the normal way sends commands over port 80 and/or 443, and your firewall will see them as normal traffic, there could be benefits from marshaling all traffic through on connection (as shown). Sauce’s tunnel doesn’t use ‘Beep’ but if you don’t understand the concepts, a quick read will be helpful.

Publishing your test app directly on the web.

If you deploy the web on a public sub-domain, OnDemand can test it without a tunnel. This is preferable of course, as it will be faster. Most likely you are choosing temporary domain names like 904f628c-23ad-40e6-83f8-48600d242e36.example.com and discarding that as soon as the test suite is complete. There’s some tooling complexity there as you’re doing to have make your DNS dynamic, but nobody said you had to use your main corporate domain.

Cookie Cutter Scaling

Tue, 29 Nov 2011 00:00:00 PST

It is true that the industry generally no longer does vertical scaling – buying bigger and bigger boxes to host single deployments of nodes in a stack. For many years, horizontal scaling has been the normal scaling technique. There is some variation in the implementations of that idea though. One particular style, where all horizontally scaled nodes are identical (which I’ll call cookie cutter for the sake of this posting) pays dividends in terms of easy deployment and the ability to simply add boxes to add capacity to the stack.

Deliberate Multi-Tier Designs

One common one for enterprises is to have a multi-tier architecture with TCP/IP separating each tier, and each horizontally scaled according to CPU load:

Explanation: In the above, the observed (or estimated) CPU loads of the production environment drive the numbers of each node at each tier. Assuming identical hardware for each process, it is easy to predict how many and where new hardware has to be added for a horizontal increase in capacity. In the diagram above, to denote an intended capacity increase, I have two ghosted nodes at the W (Web) tier, and one ghosted node for each of the P and Q nodes (whatever they are). In our diagram we model two ultimate backends. They could be an big-bucks Oracle DB, ERP or legacy IBM mainframe. They can add their own constraints like a max number of concurrent connections, that would affect everything I’m trying to sell here.

Scaling problem

When the production load on the stack is getting close to the maximums or starting to degrade capacity, then CPU percentage loads can be used as a way to work out which tier needs a new node to alleviate load (the shaded boxes in the diagram). That, or some more complex analysis according to Amdahl’s law. With each deployment in one tier as opposed to another, there are configuration considerations. Lastly, your architecture (and its configuration) might be able to suffer nodes being provisioned while the stack is up, or it might require being taken down totally (like the ‘Apple store’ today, though that might be because of mystique).

Cookie Cut Designs Recommended

The premise is that you compress up as many tiers into the top tier as possible. Optimally into the same process, so you’re avoiding between-machine TCP/IP if you can. This assumes that you are using the same base technology in a relatively homogenous solution. Corporate Java or .Net teams are used to that reality. Quite often startups are not, as they’ll use any technology that’s reasonable to get functionality live quickly. If you cannot get everything inside one process (one Java or .Net VM), then a localhost TCP/IP interface will suffice.

Anyway, here’s a diagrammatic representation of the cookie cutter version of the previous architecture.

Here we have W, P and Q in in one process. The designers should have drunk the Inversion of Control and Dependency Injection kool-aid to allow a sane structuring of the components of this application, and to minimize the side effects of running multiple things concurrently in one virtual machine. We’ve eliminated some but not all of the downstream calls. With this design we still have to worry about timeouts and caching to a different degree. Other things will become problems that were not before. Specifically we have more classes in one VM (JVM or CLR for .Net) or now, and should worry about getting the memory allocations correctly. Then there is the sidestepping all the TCP/IP formerly involved, as well as the implicit error handling or failover for outage. We should also think about whether there is garbage collection that might have been workable before, but has tipped over a threshold since putting more in one JVM/CLR. Perhaps also, there is dynamic class loading and implicit unloading that could cause the issues like they have for others in the past. The downside could also include reaching the constrains of the big or legacy back-end systems. The mainframe may only be able to support a fixed number of concurrent connections.

In some senses, this is moving ‘functionality via remote services’, to ‘functionality in an embeddable library’. Whereas services can have their own release schedule, and the larger stack need not be deployed in lock-step, embeddable libraries are different. Specifically, if there is a defect in the embeddable library version of the same thing, how quickly can all systems that use it be redeployed. Is your deployment organization adept enough at rolling out upgrades, or are you 18 months at a time with the same release?

Examples

We could consider that verifying Zipcodes or calculating Geo-ip data are candidates for external services, with elegant RESTful interfaces. There would be a team around that of course, and it would get its own release schedule. That would be a traditional enterprise solution right? A sizing exercise would work out how many nodes were required in production, and the release engineers would get busy adding domain names and port numbers to configuration. With the cookie cutter alternative, those two would be deployed in the same process as the things that would use them. You would do some sums to determine whether the added data for each were was worth the cost.

Actually Zipcodes and Geo-IP data are edge-case use in a typical stack. I blogged before about Google Accounts, and maybe Google would do Geo-ip via a redirect on a missing or out of date cookie. At least they are tangible examples though.

Deploying A Cookie Cutter Solution

From the deployer’s point of view, there are far fewer nodes types to think about. If the stack has load-balancing (say F5) in front of the web tier, then registering new nodes with that is the only configuration ‘chore’. In terms of deployment at a pertinent go-live moment, and assuming humans are still involved (versus Continuous Delivery), a slider metaphor may be appropriate:

Edging the slider to the right is a safer than an all or nothing flipping of a switch for a release. Using the measured incidence of exceptions from the logs is a good way of gauging the success of the upgrade while 10% of the way in. Of course whether your app is stateful or stateless has a bearing on whether this will work or not. If it is stateful, F5 is going to try to keep people attached to instances hosting the old version. You’re going to have to log session out periodically if their session is inactive, in order to free up nodes. If you use Gmail, you notice that Google logs you our periodically. Sometimes sooner than you’d expect, or two separate gmail accounts get logged out at the same time, and that could be correlated with a upgrade progression like this. If the stack is stateless, then moving folks from one node to another mid-session is far easier. Although, that would only work for releases where there a few discernible differences between the old and the new.

If you ignore CDNs and similar in-house solutions for static content, in terms of hardware you only have one set of machines. Each of those machines is either on one version or another. With the other style of “all upgrade at once” solutions, you will need two sets of machines, if you want to minimize downtime to seconds.

The Developers View

From the developers point of view, there is a chance that they can bring up a larger chunk of the system on their own box. Not only does this help a desk-check of the functionality that is about to be committed, but support folks are going to find it easier to attach an IDE for interactive debugging of issues they are trying to reproduce. Sometimes dev boxes given out in enterprises are particularly powerful. Anything to relieve the amount of resources needed for the processes needed to stand up the stack, would be a good thing. Also, having the production configuration closer to the development config, is a good thing. In a way, this is faithfully scaling the production architecture downwards, to an individual developer.

Caching and Latency Considerations

While this blog entry promotes the simplicity aspects of deploying in a cookie cutter style, there are a couple of other things to think about.

Caching Issues

The designers of the reference multi-tier stack at the top, may have caching in place at any of the tiers to reduce the number of downstream calls. That could either be a separate cache per node because there is only a fixed set of relatively static data (refer to the traditional use of EhCache), or something in a Memcached style design that could support all of the nodes in the same tier. This all amounts to a performance boost to the stack. The suggested cookie-cutter solution can similarly benefit from caching. Depending on the variable range of items being cached, and the numbers of keys, the same caching solutions can apply.

Latency Issues

It is true to say that each physically separated tier in a production stack can add latency to the end-user experience. Caching, as mentioned, can reduce that, but only for transactions that are meaningfully cacheable. What sort of figures can this amount to in a tuned production stack? It can be as low as single-digit milliseconds per tier, and many would argue that this verging on insignificant. What makes the TCP/IP multi-tier physically-separated machine style more complex in terms of design and deployment, is that care has to be taken with timeouts. If the bottom most tier can handle timeouts appropriately, then a tier above that calls it should have a longer time out. Or put another way, the closer to the client (browser) it gets, the longer the timeout should be. Perhaps they should not be inordinately long. Instead a sensible design should occur, with shorter and shorter timeouts towards the bottom of the stack.

Conclusion

Once you have matched your development and deployment practices to the cookie cutter style, you are able to feel the benefits of provisioning interchangeable machines. Not only that, but the flexible resource allocation, automation of server management tasks, far fewer involved deployment engineers should provide additional (smug) comforts :)

PS – Related Blog Entries

I’ve blogged in and around this space for a few months:

The Architectural meaning behind Google’s Accounts system

- Google are likely not calling side systems with every request. There’s a notable exception with Search’s OneBox which is essentially a server-side include. It has to be said that there are less ‘OneBox inserts’ into pages today than in 2006 when the linked blog entry was written. Aggressive caching of the JavaScript fragments no doubt alleviate a lot of the latency that would be implicit otherwise – Google really care about every millisecond in their stack, especially for search.

A forgotten aspect of the Facade Pattern

- Try not to make your stack function like a sewing machine up and down the stack, for a single user transaction.

The Five port rule for stack configuration

- Merge service endpoints and avoid port-centric configuration bloat.

Thanks to..

Charles Haynes and Cosmin Stejerean for comments and fixes.

Google's accounts system - architectural meaning

Tue, 22 Nov 2011 00:00:00 PST

Google provides hundreds of web applications. Many are merged into its google.com domain, but most exist on their own domain like blogger.com or analytics.google.com. They want one mechanism to login across all web-apps for multiple reasons. The reason I would like to concentrate on, for this blog article, is the desire to be able to perform a lock-step upgrade of “accounts” for all web-apps. In order to achieve this they have not embedded core account logic in each separately released web-application. Instead they redirect from the web-app in question to accounts.google.com and back again after successful authentication, if the user is not logged in. If the user is logged into ‘accounts’ but not the web-app in question, the redirect to accounts and back again can be invisible.

Visiting a Google site with no pre-existing cookies.

Google’s Blogger site is a good place to examine here as it is fairly basic (sorry Blogger team!) in terms of technology. I’m using a Plugin for Firefox called HttpFox to show the sequence of GETs and POSTs from the browser’s point of view, but there are other ways of getting the same thing:

	Arriving At	Cookies?	Consequence
1	http://blogger.com	none	redirect to http://www.blogger.com
2	http://www.blogger.com	none	redirect to http://www.blogger.com/home
3	http://www.blogger.com/home	none	redirect to https://www.google.com/accounts/ServiceLogin?service=blogger&passive=1209600&continue=[redirect info]
4	https://www.google.com/accounts/ServiceLogin?service=blogger&passive=1209600&continue=[redirect info]	A couple of accounts.google.com cookies for the session	Page load
5	Other Page Assets from www.google.com	none	Loaded into page
6	Other Page Assets from accounts.google.com	Those two accounts.google.com cookies, valid for the session	Loaded into page

We are presented with the login page now. It is refers to Blogger as an ultimate destination because of a ‘service=blogger’ reference in the URL. When performing login on behalf of other google web-apps, it makes more of an effort to present itself as part of that app, by having more closely copied styling.

Anyway, assume we enter correct Google-Account details, and submit the form:

	Arriving At	Cookies?	Consequence
7	https://accounts.google.com/ServiceLoginAuth (POST)	none	Redirect to: https://accounts.google.com/CheckCookie?continue=[redirect info]&service=blogger
8	https://accounts.google.com/CheckCookie?continue=[redirect info]&service=blogger	blogger_SID=DQ-200CHARS-Ic (for .blogger.com domains)	redirect to http://www.blogger.com/home?auth=DQ-200CHARS-Ic
9	redirect to http://www.blogger.com/home?auth=DQ-200CHARS-Ic	redirect to http://www.blogger.com/home
10	http://www.blogger.com/home	blogger_TID=2344333a78ea3a56, blogger_TID=cedf4444e637f4e0 (both for www.blogger.com) S=blogger=ssS7uFM-btx_UR5555Uo3w (for .blogger.com)	Page loaded

Note that the DQ-200CHARS-Ic ‘value’ is a shorter representation of something that is 204 chars long,

Visiting a Google site with pre-existing cookies.

	Arriving At	Cookies?	Consequence
1	http://blogger.com	S=blogger=ssS7uFM-btx_UR5555Uo3w	redirect to http://www.blogger.com
2	http://www.blogger.com	blogger_SID=DQ-200CHARS-Ic, S=blogger=ssS7uFM-btx_UR5555Uo3w	redirect to http://www.blogger.com/home
3	http://www.blogger.com/home	blogger_SID=DQ-200CHARS-Ic, blogger_TID=cedf4444e637f4e0, S=blogger=ssS7uFM-btx_UR5555Uo3w	Page load

Trying that again after deleting cookies .

Deleting ‘blogger_SID’ (204 chars) while away from blogger.com, then returning causes a redirect through http://accounts.google.com to reacquire the cookie.

Deleting ‘blogger_TID’ while away from blogger.com, then returning doesn’t cause a redirect back to google accounts, but does recreate a new value for blogger_TID (16 chars)

Deleting ‘S’ while away from blogger.com, then returning doesn’t cause a redirect back to google accounts, but does recreate a new value for ‘S’ (23 chars ignoring blogger= prefix)

Deleting all three cookies, then returning to blogger.com, causes a redirect through http://accounts.google.com to reacquire all three.

In terms of values, only blogger_SID (the 204 char one) is consistent. The other ones are regenerated each time.

What we have learned about Google’s auth cookies

The blogger_TID and S cookies contain keys that refer to something internal inside the Blogger platform. The blogger_SID key contains much more information about ‘me’ in the context of google accounts. I think it contains some personal information (my email address, whether I’m part of a “Google Apps For Your Domain” account in this context), and some context about when the authentication occurred, as well as a checksum. That cookie’s expiry is in 2019, so it isn’t part of the authentication (sensibly). The cookie’s payload, which I shortened from 204 chars to ‘DQ-200CHARS-Ic’, is encrypted of course.

As an aside, Google lost their private key for that in a hacking incident (a Windows user innocently installed a back-door or root-kit thing inside their MountainView campus) some years ago. They had to quickly invalidate all auth cookies everywhere, after choosing a new private key, after ensuring their deployment infrastructure was ‘safe’ as well as actually eliminating the back-door. I’d link to the article, but Google’s search isn’t finding it.

Architectural meaning of Google’s auth cookie design.

Blogger.com is going to get all three cookies for each request that comes in. Depending on the URL, that could mean for static assets like Jpegs and JavaScript too [See PS. below]. If the back-end Google application can decrypt the main ‘blogger_SID’ cookie (it can), and the checksum matches, it can trust the details within without referring to any account services internally as part of that transaction. If the backend decides that the cookie is wrong or missing, it will redirect through accounts.google.com. From the user point of view the redirect only happens once in a while. It’s hardly time-consuming anyway as Google apps are tuned for extreme performance for the most part. The redirect can happen when a timeout occurs – the cookie is decrypted and its internal not-valid-after has been exceeded. The redirect can also happen if someone in the security team has hit a “Untrust all auth cookies” button centrally, which probably pushes a signal to all stood-up application servers.

Here is the one-line summary – Google’s secure cookies for an authenticated user allow Google apps to save milliseconds not authenticating every request, while still having a strong centralized accounts service, and the ability to cancel cookies anywhere at any time.

If you are making multiple, separately deployable, sites for your enterprise, you should think about doing the same, instead of replicating a /accounts/Login.do in each application. Authentication is a special case though. Indeed Google is a special case though, as Steve Yegge lamented recently amongst other thoughts in a long piece. I’ll say some more about the general case for shared services in my next blog entry.

PS – I tried to convince the IETF team concerned with the cookie spec, that a useful change could be made. See my failure to do so on their mail-list In the above case, after the page load, Blogger loads 15 static assets all of which receive the same three cookies but do not need to. Google are known for obsessing about unnecessary bytes, so I’m surprise they’ve not approached the IETF cookie steering committee.

Reporting Selenium2 (WebDriver) bugs

Mon, 14 Nov 2011 00:00:00 PST

This has come up again and again on the Selenium user list and when liaising with clients that are using Selenium: How to report bugs to the Selenium team?. The problem is more subtle than you would think. Selenium2 can be driven by a number of languages, has multiple browsers that can be driven, and tastes around build technologies vary (especially within the dev team).

Groovy scripts are the most sharable, terse yet capable in my opinion

Consider this example for using Firefox for searching for “Selenium” on the ThoughtWorks site:

@Grapes([
  @Grab("org.seleniumhq.selenium:selenium-java:2.12.0"),
  @Grab("junit:junit:4.7")
])

import org.openqa.selenium.firefox.FirefoxDriver
import org.openqa.selenium.By
import static org.junit.Assert.assertTrue

def driver = new FirefoxDriver()
driver.get "http://www.thoughtworks.com"

def searchField = driver.findElement(By.id("edit-search-block-form-1"))
searchField.sendKeys("Selenium")

def searchBtn = driver.findElement(By.id("edit-submit"))
searchBtn.click()

def success = driver.getPageSource().contains("to run the same acceptance tests")
assertTrue(success)

println "WebDriver test complete"

driver.quit()

The bug-reporter are able to capture everything about the problem in a single script, with no build-file dependency. They can juggle the version of Selenium quite easily in the @Grab line at the top of the file. That might be important if the bug exists in one version of Selenium2 but not another. If the user is familiar with xUnit style programming they can keep lines similar to the assertTrue(…) line, but if they are not, they can code something rudimentary themselves. As this gets uploaded to the Selenium project issue tracker, it does not matter whether it is an attachment, or cut/paste into the body of the issue. The bug-reporter may also be pleased to know that Groovy itself doesn’t really need to be installed: see later.

From the point of view of the selenium team (who on average are pretty good at Java and hence Groovy too), the script can be run more easily than anything that has a build file, or is just a fragment of logic. The @Grapes business means that only Groovy is a prerequisite install beyond the Java install they already have. Grapes will download the dependencies (somewhat silently) if they are no cached and puts them in [your home directory]/.ivy/ making it easy. Lastly, the Selenium developer will be able to attach a debugger as easily as they would to a Java process.

Proprietary nature of pages?

Some clients don’t want their pages to appear in public before they are ready. In that case you need to consider trusting one or two members of the Selenium dev team with a single problematic page from your being-built application. The MAFF plugin for Firefox will do admirable job of saving a page including state and scripts. You could mount the result of a save on an obscure URL, or send it as a zip to the developer handing your bug report (presumably after a conversation in email). The MAFF trick is a neat one, as it allows you to omit mention of any pages and Selenium operations that are not directly related to the bug in question. Even if your preferred browser is Internet Explorer, this Firefox workflow will save a site that is perfectly navigable by IE.

Getting up to speed with Groovy.

Java is required by Groovy. It is Windows, Mac and Linux compatible. It’s already on Mac versions up to Snow Leopard. For Lion, you have to install it yourself . For Windows and Linux, go to Java.com and click through to the right download. If it’s asking you whether you want “runtime environment” or “developer kit” choose the latter. Test Java on the command line like so:

C:\> java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)

A reboot may be necessary, to get Java’s ‘bin’ directory into the PATH environmental variable. If you’re advanced with such things, you already know about tricks to negate the need for the reboot.

For Groovy itself, things are slightly simpler. Download the “binary release” from the Groovy website Unzip it and make sure the pertinent ‘bin’ directory is in your path. Test it like so:

C:\> groovy --version
Groovy Version: 1.8.4 JVM: 1.6.0_29

Enjoy!

PS – Groovy can be terser still than the test-script represented above – I just didn’t want to scare anyone :)

Locked US iPhones suck

Fri, 21 Oct 2011 00:00:00 PDT

I had the original iPhone via AT&T in the US. I didn’t wait in line, as I didn’t buy on day one, but that’s an aside. Why are these buggers locked in the US? My experience with other phones and other jurisdictions is that they are not, or at least they are not after the contract expires. (Apologies for the ‘first world problem’ blog entry). In other countries, you can bring your unlocked smartphone to a cheaper monthly account (no phone subsidies being paid off).

People who spend more than a week out of the USA in a year (like me)

We want to SIM swap rather than suffer roaming charges. That’s not just for voice, that’s for data too. I like using Google Maps to traverse the UK with. I can’t do that with a locked US iPhone. There mush be a bunch of traveling Europeans like me who’d like to availing of the cheaper temporary plans in their ‘home’ countries before returning to the states to work. A ‘Pay as you go’ account in the UK that works with my iPhone costs me about $100 for a whole year (I’m only there for three separate weeks). That’s with data!

Power travelers are the same. Roy Singham, the ThoughtWorks owner, probably has a SIM for every country he regularly goes to. There are many executive types that are the same. For him it is about data+voice too.

What we know about unlocked iPhones.

They are in a database that Apple keeps. That could even be a spreadsheet for all we know, but all the purchased-as-unlocked iPhones are in it. The ones that are unlocked after purchase are added to it, and there’s a mechanism via iTunes to action the unlock and notify customers that it happens elsewhere.

AT&T’s iPhones.

These have always been locked; Permanently so. At least the subsidized ones are. You can pay premium for an unlocked one, but there’s no cheaper monthly fee from them if your do. They are so from ‘birth’ to when you’ve inevitable sat on it, dropped it, or seen it fade into obsolescence/disfunction after some years. It is rumored that AT&T hardly make any money from the iPhone. More specifically, the monthly fee barely covers the ongoing costs and subsidized hardware.

Me trying to unlock my end-of-contract 3GS.

So as is the custom in other countries for other phones, I’ve tried to call the teleco to get it unlocked. Yesterday I spoke for three AT&T employees via their ‘611’ support service, and one Apple employee who AT&T transferred me to. All people were helpful, and incredibly polite even though I was trying to push my point “please could you unlock my end of contract iPhone 3GS”. I’ve been unsuccessful in that endeavor, but I’ve found out some things that are worth writing about.

In initial call to AT&T I was assured that only Apple can unlock iPhones. That’s true, I said, but only on request from the teleco in question. We didn’t agree, so I was transferred to Apple. That fellow confirmed what I had previously thought, and we completed the call at that point.

My second call to AT&T took be through the same process, but I managed to get bumped to a manager, after thanking the associate for her help and caring about the issue. The manager I reached was very sympathetic to my wish, and ultimately asked a couple of back office techies who’d really know, and the answer came back:

—> AT&T do not know how to initiate the electronic iPhone unlock via Apple

It seems to me that Apple could either unilaterally unlock end of contract iPhones if that were true, or send training materials to AT&T so that they can gain the capability.

Sprint’s iphone (4S)

Just before the launch of the 4S, we thought Sprint may unlock them on request. Sprint has allegedly looked at the $4Bn fee they handed to Apple though, realized they need to double their subscriber pool to cover it. That means they need revenue from the same iPhone after the 24th month that would be the completion of the contract with the customer. Whether that is your extended contract, or your Mom after giving her the phone, they don’t care. As with AT&T you don’t get a reduction in monthly fee, and that is when their real profits start.

Verizon.

They will unlock for customers who have ‘accounts in good standing’. That reasonably means that Verizon think of you as someone who paid your bills on time for 60 days or so.

Value of end-of-contract iPhones.

Given the free 3GS phones available now, and the fact that Apple release a new iPhone every 14 months, the iPhone you buy today is not going to have much resale value. If the phone is locked to AT&T and new customers could add a new one for free, what value is there in an end of contract iPhone? It is zero really for US markets. A silver-lining is that the iPhone could be unlocked for a new life abroad, and for that there’s some value. That unlock is likely to be some unapproved means, and there’s no guarantee that Apple wouldn’t be able to close down the unlock/jailbreak in successive iOS releases. Actually that hypothetical re-closure is more about jail-breaking.

So a legitimately and permanently unlocked iPhone has extra value, once you’re hellbent on upgrading and selling your old iPhone. No-one has to fear upgrading iOS versions and the possibility of it not working as wished for after.

Conclusion.

I think us foreigners living in the USA are fools to go with AT&T iPhones right now. We should go with Verizon and unlock after 60 days. The GSM side will work in UK, France, Germany, Brazil (etc) with some modest SIM swapping. We get the best of both worlds. Verizon should not worry as we will pay our two years of fees for US CDMA usage. It is just that we want to SIM-swap for a few weeks a year, when ‘back home’. As I said before it not just about voice any more; It is data too.

I wonder if people who have already started their AT&T iphone 4S plans two weeks ago shouldn’t cancel (within the 30 days) and reorder a Verizon edition.

Toothless regulators.

The FCC is the regulator here in the USA, and historically they do not step into contractual imbalances. Compare them to OFCOM (UK) and other European bodies who have forced cheapening changes on their country’s telecos.

PS – I do not care about data+voice concurrently feature, or the ‘higher speeds’ that AT&T have (that Verizon do not).

Addendum (Nov 15, 2011)

I’m safely on a Verizon iPhone now. Not being able to surf while making a call is the hypothetical downside. Well that and Verizon’s iPhone4s network speeds being slower than AT&T 3.5G one. Fast enough to watch movies though, so why would I want it faster??

AT&T are charging me a $60 early termination fee given I cancelled ten or so days before my natural end of two-year contract. “Michael” (a helpful fellow in the call center) said they can setup a forward note for themselves to credit me that amount seeing as my normal last month charge should apply, but their system is inflexible (my words) enough to make that a manual human step. I’ll make a calendar note. I also passed on my rationale for canceling [ this whole posting Michael if you’re reading ], and I got to hear again that Apple hasn’t provided a mechanism to unlock iPhones. I’m suspecting that’s a scripted response. Bye bye AT&T, sorry that you feel the need to gouge customers in their 25th month and onwards.

Update (April 22, 2012)

Verizon iPhone 4S unlocked after a simple 5 mins phone call. AT&T change their mind – I hope I helped mount that pressure. Will be testing it soon with the Wife’s 3GS which is out of contract. Also meeting up with buddies recently, I was taunted for having a slower 4S service through Verizon. We did a quick pub-test – watching the same movie clip on Youtube at the same time – there was no difference :)

Mozilla are screwing corporate adopters with their numbering scheme

Wed, 12 Oct 2011 00:00:00 PDT

I cannot be the first to say this, but Mozilla by flipping through Firefox 3, 4, 5, 6, 7 every six weeks or so are really screwing their corporate adoption fans.

Why? For an arbitrary series of Internet Explorer 8.x (or 6.x, or 9.x etc), laggardly corporations could state “our internal web applications will be compatible with all 8.x releases of Internet Explorer”. They can say that with modest certainty because the layout engine (or other major components of browser) will no be rewritten between 8.0 and 8.1. They then choose to fix at the 8.x, and accept patches within that series.

However Mozilla are giving no ability for corporate types to fix at major versions, nor any guarantees around when major components will be rewritten.

At a smaller scale we are already seeing issues with the availability of Firefox plugins for the cutting edge releases. Previously, plugin-makers could mark their products as compatible with a series like 3.x and relax somewhat. Nowadays they have to really watch the release schedule for Firefox and ship accordingly, or aim at a incredibly broad range like 3.x -> 15.x and ignore the fact that that is an egregious claim.

As an aside, it would be nice for Sauce Labs to deploy a ‘daily Firefox’ build, that customers could run test suites against. Corporate types would be able to feel more comfy with Firefox then, and offer greater assurances about compatibility for internal going forward. External apps might also benefit, obviously.

setContext(..) is back for Selenium2 :-P

Fri, 30 Sep 2011 00:00:00 PDT

Of course, Simon objects so it’s not actually in the core API, and it’s only back for Sauce Labs operation, via a hack :)

See a Sauce Labs job that shows it off in use for a BDD/JBehave script.

To use, just do something like this:

JavascriptExecutor js = (JavascriptExecutor) webDriver;
js.executeScript("sauce:context=the sentence you would like to appear in Sauce Labs' job log");

Sauce Labs (and others) can possibly use executeScript() as a generalized control API for their remote cloud instances.

Branchable Continuous Integration

Fri, 30 Sep 2011 00:00:00 PDT

Most folks know that I like trunk-based-development, with branches made in advance of release.

Less well known is that I like my Continuous Integration configuration to be under source control too.

I’ve spent a lot of time in Jenkins recently configuring job, then copy-pasting that (after multiple clicks to navigate around) into similar jobs.

I would really like to be able to do all of that in Intellij, and just commit atomically, possibly in the same commits as regular changes. I’d also like to be able to make branches and push them to within the visibility of the CI master daemon, and it provision the implicit pipelines from that commit automatically.

What that means for me is that the CI tool should watch all commits, and be able to recognize its configuration files amongst the commits (new, changed and deleted) and reconfigure itself before it leaps into action to process the particular commit(s).

Cruise-Control was close in a bygone era. Indeed, I remember working with Paul Julius in 2005 to rework the XML-Include capability to be resilient to botched edits. That last would mean that the reconfig would not take down a whole pipeline if one dev forgot an angle bracket.

I wish the modern CI tools could be configured completely using commits, including new pipelines.

Automatic sequence diagrams from BDD-scenario executions

Fri, 23 Sep 2011 00:00:00 PDT

They would be great – right?

See markup for this on WebSequenceDiagrams.com

Should be easy enough to construct the markup language from instrumented versions of:

JBehave or Cucumber (etc)
Selenium 2.0 (WebDriver)
Your application stack.

Perhaps better if the company behind the graphing technology could adjust the vertical spacing based on actual timings. I’ll email them :)

PS – Colleage Xiao Peng has done something similar that cab be played with on AppEngine

A forgotten aspect of the Facade Pattern

Wed, 21 Sep 2011 00:00:00 PDT

Use of the Facade Pattern in software is about apparent elegance on the ‘user’ side of the facade and the hiding of inelegance on the ‘implementation’ side. The implementation can change or be replaced easily over time, but that applies to many patterns. Thus it is the elegance of the facade that’s making it attractive to use in a layered design. There is a second aspect, thought, that is more subtle and often forgotten: reducing the number of calls needed through it. Hopefully to one.

Dishonored Facade Implementations

Just pretend for a second that a object that straddles a divide has gettable and settable things that are representative of more meaningful business things:

public interface Broken {
  void setGivenName(String gn)
  void setFamilyName(String fn)
  String getGivenName()
  String getFamilyName()
}

What we have in the above is something where two are likely to be called in series for typical interactions. We could see that getGivenName and getFamilyName could often be called together when populating an UI’s fields. Similarly setGivenName and setFamilyName could be called together when pushing the UI’s fields back towards persistence:

// populate UI:

gn = b.getGivenName()
fn = b.getFamilyName()

// (elsewhere) persist UI fields:

b.setGivenName(gn)
b.setFamilyName(fn)

If the facade happens to by hiding inelegance of lower level implementation of things that go over a wire, we will now observe two calls over the a RPC or RESTful divide. There are performance consequences that add up quickly.

Honoring Facade Patterns

When the facade pattern is implemented correctly, more data are channelled though fewer invocations (again these are representative of more meaningful business operations):

public class Name {
  // imagine constructor, getter, setter that
  // are characteristic of Immutable or ValueObject
  String firstName;
  String givenName;
}
public interface BetterFacade {
  void changeName(Name name)
  Name getName()
}

What we have now a single operation through the facade that takes or gives something that could be serialized quite easily. Again it does not matter at this stage whether the underlying implementation is a Remote Procedure Call, or something RESTful.

At this level of object granularity, it should be easy to get right.

The Dark Ages of Enterprise Computing

Sun’s late-90’s J2EE is the largest effort to bring bad practice to enterprise computing. Particularly the EJB stuff in its early incarnations. Sun promoted Session Facade as one of the constructs you should code in to your stack. In so doing, they forced a ton of inelegance on the developer. The page I link to shows the J2EE definition from the 2001-2002 era (Oracle could nix it at any time). It is also in the EJB 2.0 style before EJB 3.0 injected some small amount of sanity in 2005. From 1998 to 2004, Sun told folks to make directed graphs of components while constructing applications, that required a ton of boiler-plate code. In the effort to just get it working many teams forgot the desire to reduce invocations across the divide. Java teams are still perhaps forgetful in the years since the progressive abandonment of EJB in particular and J2EE generally since (the mid noughties).

Clues that you are doing it wrong.

While the above diagram does not show categorically an incorrect architecture, even if the there’s a TCP/IP separation between each of the boxes. It does show the start of a fan-out of connections. What starts with one I/O from the browser to the web-tier, results in three I/Os to lower level services. That could be nine more if each of those three did the same to layers below. We might have a constraint in that these have to be separate boxes, and we have to do I/O in series to them as part of the buyCartContents operation. Incidentally, the Catalog node in the above diagram, could have some caching attached to reduce some of our worries, but that is a different thing.

What I am trying to get across at this level of architecture, is the fact that you could quickly engineer something with a lot of downstream invocations that add up while you are otherwise busy with developing functionality. As you’re making your stack, look at the numbers of invocations across divides that could otherwise be facade-like, and worry of the growth of them is uncontrolled. Look at facade methods that are similar to each other, or always invoked in series and could be rolled into one.

One technique for doing it right from the start.

Simply put, you should rigidly stick to top-down design as you build your application stack over time. This is true for the run up to your initial go live, and the in the incremental releases that follow that first push. Do not let Database Administrators (DBAs) drive the design of your web-app. Nor engineers of lower level services. Both of these are ‘bottom-up’ techniques and classic false economies for enterprise development teams. Instead think about the facade like operations from the UI, and change downstream code as appropriate to support that. If that means that your UI teams are more instructional to service teams, so be it.

While adhering to the above, keep busy with your refactoring agenda , even if you are using a language that does not have an IDE as good as Intellij (Java) is. With a decent Agile perspective the right design will emerge at all times.

What is Dart going to be?

Mon, 12 Sep 2011 00:00:00 PDT

There is not a lot written about what Dart is actually going to look like, so I will speculate a little. ThoughtWorkers are going to cringe as I take the opportunity to bring up something I have been chasing for years.

Convergence of Declarative and Procedural UI markup languages

It has never been described as pleasant jumping from a declarative form like HTML to a turing-complete procedural form like JavaScript. That we use an escape mechanism <script>JS here</script> is only the first clue that the combination is as ugly as hell. JavaScript was defined after HTML, so it is interesting to wonder what the pair would look like if JavaScript had come first. I am sure it would look more like JavaScript and less like HTML. The clue is the huge amount of libraries today that retrofit JavaScript with increasing elegant UI morphing capabilities. JQuery is perhaps the best known of these, but there are others that are compelling too.

Columns of UI setup code – bad bad bad

There is a style of UI programming that is common, where methods and functions separate sections of UI setup code in a classing OO design. Model View Controller (or related patterns) are possible, but the largest visual aspect to actually developing the source for these, is the long columns of ‘object.method(params)’ with the occasional for loop, or if condition. Nesting is only there if the developer has not broken the setup into separate methods.

Swing, SWT and GWT are all examples of this. Elaborate UIs can be constructed using them, but there’s none of the elegance of the original HTML, in that nesting of the UI logic, does not map to visual nesting. That nesting, in HTML 1.0, was the thing that allowed unskilled people to have a go at editing pages, and hitting refresh in the browser.

Some meld of JSON, closures is my hope for Dart.

Here’s something that Duda Dornelles put together using Node.js, that could execute directly in a browser context:

html(
     head(),
     body(
         h1("MJS example"),
         div({id: 'header'},
             ul({id: 'nav'},
                 li(a({href: '/home'}, 'Home')),
                 function() {
                     if(loggedIn) {
                         return li(a({href: '/logout'},"Logout"));
                     }
                 }
             )
         )
     )
 )

I cannort help feeling though that it is too lispy, with the current ugliness of JavaScript interspersed. I would prefer something more like what is possible in Groovy, Ruby or Python with builders/closures:

html {
    head {},
    body{
        h1 "MJS example",
        div { 
	        id: 'header',
            ul {
	            id: 'nav'
                li { 
	                a { href: '/home', txt: 'Home' }
	                ifLoggedIn
	            }
            }
        }
    }
}
function ifloggedIn {
    if(loggedIn) {
        return {
            a { href: '/logout' txt: "Logout" }
        }
    }
}

I have not shown above (because I’m trying to keep close to Duda’s sample) the Turing completeness that could be worked in whether would be best or worse practice with the hypothetical markup language.

With a bootstrap for the browsers shipping today

You would client side code in this style today, but only if you’ve made a bootstrapping JavaScript library that can act as parser and liaison to the inevitable DOM mutations. We know this is possible as Cappuccino has done this for many years – refer 280slides Note that Cappuccino is another column-of-operations style of UI markup. That is because what it is inspired by (Objective-C) is also in that style, notwithstanding InterfaceBuilder and the resulting nib/xib files which are not human readable.

Summary

I asked Martin Fowler what this style is called, and he says that it is ‘Nested Function’ form/style. He mentions in his DSL book, and he reminds me that it is very common in the Lisp world, and uncommon in the OO world.

Incidentally, I remember a markup for an early version of InterfaceBuilder (‘Next’ in the late 80’s, before Apple subsumed it) that used a form of Lisp for the format of the saved UI files. It used Lisp for the WYSIWYG load/save cycle, as well as (I think) the run-time execution. I’ve emailed a bunch of people about that time before the format was changed to ‘nib’ and nobody can corroborate, so I must have dreamed it.

Folks that are interested should read about Erector and Shoes, and Duda’s MJS and something he found that is similar – node.magic_dom

Anyway, I’m hoping that Google Dart stuff is nested functions, interoperating with the DOM, in the sandbox, negating the need to use any HTML, more coffee than ecma, and easy one the eye, with a bootstrap for modern browsers until native support is baked in.

Refactoring Experiment - 'least depending and most depended on' components first

Thu, 01 Sep 2011 00:00:00 PDT

This article follows on from A Spring Shortcoming that had a nice diagram:

The challenge

I now have a github codebase that is an very slim¹ example of that Spring app previously outlined. The code is 100% covered, though that is by accident as you will see on investigation. I have a challenge for folks.

Context: You are a developer who has just committed a change to this codebase. Before you tell your PM that you’ve finished it, you are going to embark on a refactor given you are acclimated to presently. We all love “Red, Green, Refactor” – right ?

You have seen that the controller you’ve been working on has two request-mapped methods ‘placeOrder’ and ‘seeSuggestedProducts’ should not be in that controller class. You want to fix both, by pushing them to new controllers You also suspect that UpSell should be a collaborator (but still request scoped).
Using Intellij or Eclipse Refactoring operations, attack one of the two misplaced request-mapped methods first, try to move it to a new class.
Work out part way through that you should have started in the wrong place² and instead should use the same techniques for ‘findUpsell’ first, and made a collaborator class of that.
Flushed from that success, you should be able to go back to both of the other misplaced methods and pushed them using refactoring operations to new classes.

You are not going to be able to do it all with refactoring operations. Ctrl-Z or abandoning changes in Git will be part of the workflow for the above when you confirm you have taken the wrong direction. Cut/Paste is going to have to happen too sadly in a couple of places, as Intellij can’t do all your intended moves for you.

If you are willing to record and publish a video of the series of refactorings, I will link to it from this posting or a follow up.

The take away?

.. should have been do “least depending and most depended on” components first². Jeez, there’s only one hit for that in Google. Was it mentioned in the Michael Feather’s book?

¹ Yes, the code really isn’t a functional Spring-MVC app, it is an approximation of one, useful only in this context.

² Refer drinking your Guice too Quickly on InfoQ.

Responses

Dec 10, 2011: Danilo Sato has has a blog entry in his series on the differences between Mechanical and Strategic refactorings, and their benefits. He’s got a 14 minute video walkthrough of the refactorings needed as you push towards “most stable” code.

Contrasting Selenium 2 locator styles

Wed, 31 Aug 2011 00:00:00 PDT

Just a quickie - with the Etsy.com tutorial in the JBehave we have three styles of using Selenium 2 (WebDriver) to locate and click on things in the page. The Page objects (which still need some work) illustrate the difference.

Plain Java, canonical WebDriver style
Plain Java, WebDriver in fluent style
Groovy + Geb

Plain Java style:

findElement(xpath("//a[@title = '" + section + "']")).click();
findElement(id("search-query")).sendKeys(thing);
WebElement select = findElement(xpath("//select[@class ='handmade']"));
new Select(select).selectByValue(subCategory);

Fluent style:

link(xpath("@title = '" + section + "'")).click();
input(id("search_query")).sendKeys(thing).submit();
select(className("handmade")).selectByValue(subCategory);

Groovy+Geb style:

$("a", title : section).click()
$("#search-query").sendKeys(thing)
// a small issue in geb presently makes this less elegant
new Select($("select.handmade").getElement(0)).selectByValue(subCategory)
// if it were not for that issues, the code should be:
$("form").category = subCategory

I am not sure if the fluent style is better than the canonical Java way, as the latter’s locators may survive some page refactorings. I wish the WebDriver and WebElement APIs were more fluent though. Not least - I wish things didn’t return void.

Also, Geb’s last release is against Selenium 2.0rc3. We’re at 2.5.0 now, and that’s too far behind for my liking. If you attempt to force an upgrade of dependencies for Geb, you will encounter subtle incompatibilities. The work around for one of those is shown in the new Select(…) logic. Also, I’d wish that Geb did the fluent style too - matching what the WATIR team champion more closely that the fluent selenium thing shown in #2 above.

No One Gives Themselves Bad Ratings (in hiring, apparently)

Tue, 09 Aug 2011 00:00:00 PDT

On the selenium-user mail list, it was hypothesized that candidates rate themselves well in respect of their skills match to the job in question. It was said “No One Gives Themselves Bad Ratings”.

It turns out that this is not universally the case: True experts, when questioned, rate their own skills as lower than they should.

The Dunning Kruger effect

This epiphany is one of a few that comes out the research that Dunning and Kruger have done to explore long-believed tendencies within society. The definitive write-up is on Wikipedia, though that’s not very prose-full.

Easier reads are available: New York Times article from a year ago and a Psychology today on the same timescale. Here is one from the turn of the millennia that I particularly like on the San Francisco Chronicle.

The one-line take away?

The skills to recognize incompetence in others are the same as the skills to recognize incompetence in self.

The slightly longer version?

Candidates who are experts rate themselves lower (typically) than they really are and incompetent ones rate higher. The competent ones only realize the injustice of their position in the rankings, when they are confronted with the self-scores of those that they knew already were less able.

In summary, be wary of rejecting candidates who self-rate to lower scores that a threshold you have set. Always do some independent verification.

Steve Jobs’ view on A, B and C players

Read also some thoughts on A players, B player and C players in a previous blog entry. This in particular suggests that incompetents should be removed from hiring or you will end up lowering the bar. Steve Jobs may have been one of many that was pre-empting the findings of Dunning and Kruger later.

Incidentally, I have to thank Agile industry legends Chris Matts and Andy Pols for first putting me on to this stuff in 2002.

Stitz's Five port rule for enterprise application dev

Sat, 06 Aug 2011 00:00:00 PDT

Five ports only

At a client, ThoughtWorkers were discussing the maximum allowable ports a team should use when configuring an enterprise application they are making. Jeremy Stitz (no blog) held up a hand an contributed an exasperated “five”. That seemed like a good, if arbitrary, number to define the simplicity we are hoping to encourage here.

The problem we’re trying to solve

A production stack composed of: Multiple machines and multiple process. Each is different, though sometimes there could be duplication because one particular permutation appears to be overloaded when live. There is going to be a lot of configuration for the production stack. Sometimes that config is spread around. Sometimes in is centralized in a single properties or XML file that is supplied to all booting processes. Invariably, the configuration is different in production than it is in any of the development, testing or staging environments. Then there are developer workstations where we should be observing developers bring up a microcosm of the stack before they commit each change, that are different again. The company in question, now needs a whole team to manage configuration. There nothing approaching simplicity or uniformity with such deployments.

More than once a client in my past has managed to deploy cookie-cut zip files to a range of machines and bring them all up to serve all service needs for the tiers above. I’m going to return to this topic in future blog entries, but for now am going to concentrate on one aspect – ports and the configuration thereof .

Sockets versus ports.

The enterprise app that you’re making may have hundreds of deployed nodes. Hopefully this is horizontal rather than vertical, and that you’re driven by easier scaling meaning that many nodes are identical. If you look at your whole stack in production, you may observe hundreds if not thousands of sockets concurrently open. Those could be incoming of outgoing. Understand that we’re not trying to restrict sockets here, thus you can still have your multi-process or multi-machine deployment.

Be reminded that sockets are not ports. Sockets are a software instantiation that’s actually shuttling data back and forth, whereas a port is more of an intention to do something. An app could be listening on a port, with the intention of streaming data via a socket in due course. Similarly outgoing port usage results is sockets being allocated. Here is a good stack overflow question on the topic. Note – ports vs sockets is a fun interview question.

Ports 80 and 443, and OS level ports

Most apps we are making these days are web based. Therefore using ports 80 and 443 is going to bring your available count down to three, if you are trying to hit the five-port rule.

Your apps could well be using ports like SNMP (ports 161, 162) and you might not know it. We are not trying to legislate against the ports your app is inadvertently using, just the ones you state in application configuration (XML, YAML, properties files etc)

Sharing multiple services on one port

Having ‘Zip Code service’ (yes that’s a zipcode, here’s some information about it) handled by an web-server node, that also handles authentication is easy to imagine. They would be two different URLs on the same apparent web server:

http://myserver/zipcode
http://myserver/authentication

Imagine you had need in one stack, to support ZipCodeService 1.0, 1.1 and 2.0 though. These were essentially successive releases of the same thing, but you still need to support all there. Perhaps your application’s own code uses those three versions and sadly cannot be upgraded in lockstep. Perhaps they are used by remote customers who are also not upgrading in lockstep. What you really need to do is bake your differences into URLs (assuming HTTP) and construct a deployed process that can handle the multiplicity of that with ease.

http://myserver/zipcode/1.0
http://myserver/zipcode/1.1
http://myserver/zipcode/2.0

Let us assume Java or C# (.Net) for a second, and each of those three versions of the same thing has binary dependencies that are incompatible. This is not so much the diamond dependency problem, but that is an interesting topic anyway. With Java and .Net you cannot construct a single class path which has duplications on it, let alone incompatible duplications. Well you can, but you should not as the results will not be what you wish for. You are going to have to construct something that allows for incompatible jars/dlls in the same deployed app, that is more sophisticated than a simple classpath.

The booter is what constructs this classloader tree. In order it makes the common classloader, the HttpHandler, using the common one as its’s parent, then each of the three zip-code implementations with their respective binary dependencies in the same classloader. Note that there’s a duplicate of an individual jar (it doesn’t matter really).

Once constructed the booter then configures actual endpoints. Think of that stage as more regular Java like so:

handler.register("/zipcode/1.0", new ZipCodeService());
// and the others
handler.startListening();

Except that the booter (which intends itself to be garbage collected as soon as it’s completed its phase) does not see any of the child classloaders directly so will have to use reflection (please imagine the implementation of the fluent helper methods):

Object handler = from(handlerClassloader).make("HttpHandler");
Object zipCode1p0 = from(zipCode1p0Classloader).make("ZipCodeService");
on(handler).invoke("register").with("/zipcode/1.0", zipCode1p0);
Object zipCode1p1 = from(zipCode1p1Classloader).make("ZipCodeService");
on(handler).invoke("register").with("/zipcode/1.1", zipCode1p1);
Object zipCode2p0 = from(zipCode2p0Classloader).make("ZipCodeService");
on(handler).invoke("register").with("/zipcode/2.0", zipCode2p0);
on(handler).invoke("startListening").withNoArgs();

Note also in the diagram that solid lines represent the direction of class visibility. No class in the classloader for ZipCode 1.1 can instantiate any class from its two siblings nor, any class from HttpHandler’s classloader, let alone the one for the booter that should long since be garbage collected by the time the app is running. The booter itself is a special case, as it is the one that constructs the whole classloader tree and temporarily at least has a object reference for all of the classloaders. There are degrees of enthusiasm for hiding the implementation of things that are passed between nodes (like the request)

Java servlet containers (and their capacity to host multiple WAR files) are similar, but we’re defining a common API classloader here that is not itself in amongst Java’s own libraries. In fact our directed graph of classloaders here can be much larger.

Load balancing

Given we’ve now got a single process that can handle everything current and legacy, we might have lost some of the scaling alleged made available from having multiple machines/processes do a single service. The answer is simple – deploy multiple instances of this general purpose services container, and use a regular load balancer to make them appear as one to their clients.

Non-web solutions

Not into RESTfulness? Into CORBA and all that? Sure, this is where you’re going to begin to approach the five ports limit of the title. There are more problems with load-balancing, in that off the shelf solutions for that are tuned for HTTP traffic, but it should be possible engineer a solution where services are somehow named (including versions) within a shared connection of sorts.

Conclusion

I’d much rather compose a directed graph of services that are available for a single cookie cut process deployment, and have a single port number in configuration (80 or 443), than many more ports in play, for many different fine-grained deployments that are listed in some sprawling configuration of machine names and ports. The big-configuration design gets worse when you realize you need one machine for ZipCode service 1.0 and 1.1, but three machines to handle the load of ZipCode service 2.0. Stuff of nightmares; Long live Stitz’s five port rule.

Hiring Selenium QA people

Thu, 04 Aug 2011 00:00:00 PDT

Mixed skills

Ideally people working with Selenium to automate the testing of applications will have some development skills in addition to their QA skills. It could be that they are an experienced developer and they are helping start a QA automation side to their employer’s project. It could be that they are a QA who has used a number of tools (like QTP) and is now trying to apply those skills to a more accessible Selenium solution. Lastly, it could be that they are a manual QA in the past and this is their first foray into QA Automation.

We’re going to try to speak to all of those types in this section of SeleniumHQ. We will phase this as if it were candidate selection for interviews to join a project team.

Levels of experience

Inexperienced

The candidate in question has not used Selenium in the work place, and may only have played with Selenium-IDE in advance of an interview (at the recruitment agent’s suggestion perhaps).

They may have a resume that speaks of Quick Test Professional (QTP), Test Director, or even WinRunner from some years ago. They may alternatively have a resume that details manual testing of web applications, using or not using tools to the cataloging of manual test suites and execution of the same.

When meeting these candidates, you may find programming languages, technologies, libraries, frameworks listed, but be aware that the candidate may be talking of team experience rather than personal use. In order to identify them as inexperience, you’ll have to construct questions that can differentiate between team and personal experience.

Entry Level

The candidate has used Selenium-IDE for recording scripts and playback. These suites may be saved to SCM, but could have been just on a their ‘C drive’ and implicitly not shared between team-mates. The suites could have been highly evolved, relative exhaustive and relied on by the project to validate applications prior to deployment.

The candidate may not qualify for Entry level Selenium skills if they have just used Selenium-IDE sporadically to aid form filling as they manually test.

If you think they qualify for entry level selenium, they should be able to use it in front of you to test a site of your choosing.

One the advanced side, they may be running their suites automatically from Continuous Integration tools, and have a high level of confidence and ability to move forward with confidence. Perhaps they are using Selenium-RC to execute whole sets of ‘html’ Selenium tests.

(There is at least one friend of the Selenium project that has really pushed this space, and we’re not wanting to cast that dev team as beginners with Selenium here)

The nature of HTML, JavaScript and CSS should be understood to a rudimentary degree.

Data driving tests experience may or may not figure at this level.

The candidate has knowledge of the DOM, and for selenium specifically, how XPath, CSS or by-ID selectors are used to test the application.

They are also going to be able to checkout, and commit back to one of a number os SCM tools. Most likely with a UI rather than a from the command line.

When meeting these candidates, they should be able to use Selenium-IDE in front of you to test a site of your choose, and work through tricky issues exposed when play-back doesn’t exactly do what the record function identified. Not jus that, but they should be able to simplify the xpath expressions the recorder has made. Choose a web-site other than your corporate one for them to test.

Intermediate

On top of Entry level …

The candidate has transitioned to Selenium-RC or increasingly WebDriver in the workplace. They will be able to speak to the differences.

They are able to program in one of Java (including Groovy & Scala), Ruby, Python, Perl, C# (or VB.Net), PHP. Or similar 3GL languages. They are also adept with unit-testing (JUnit, NUnit, TestNG) frameworks for their language, as well as appropriate use of assertion technologies.

They may have OO knowledge, or may be beginning to understand it. Without it, they are likely to be using their language in a procedural way (which can also be fine, if its relatively clean from a purists point of view).

If they are procedural (not yet OO savvy) they must be able to demonstrate looping and appropriate conditional aspects of the languages they are experienced with. If they are in the territory of OO in terms of experience, they should be able to give examples of where the facets of OO apply to Selenium test scripts.

The candidate will be able to discern between alternate selector strategies to use the most suitable. They will also use Firebug (or alternatives) to help them, and perhaps also be adept with XPathChecker or XPanther, though hopefully using them far less than Firebug.

That pages are loaded in increasingly strange orders (we’re looking at you Web 2.0) should not be a hurdle to the intermediate Selenium developer. They will be able to demonstrate techniques to wait for page events such that the test scripts they make do not fail when the application is slower than normal.

This level of candidate should be able to respond to broken builds (we’re all using CI right?) find the issues, and liaise with the application developers to fix things.

They can, most likely, arbitrate over merge conflicts with the SCM tool they are experienced with. Hopefully they are as savvy with the command line, as they are with the SCM tool’s GUI.

In terms of testing style, this candidate will be able to separate happy-path tests from regression suites and project related QA.

The candidate, in the interview, should be able to extend an existing Selenium-RC or WebDriver codebase one or two tests more.

If the candidate has not used Selenium-RC, they may still quality for ‘intermediate’ if they have developed large suites of test scripts with Selenium-IDE AND gone as far as using UI-Element and other advanced features of Selenium-IDE.

When meeting these candidates, they should be able to use Selenium-RC or WebDriver to script a test that they may have started in Selenium-IDE. Be lenient on the actual nature of the build-script that kicks off their chosen high-level test class.

Advanced

On top of Intermediate …

This candidate has more programming experience. Specifically:

Closures (Groovy, Python, Ruby) should be familiar to them and they know how to make or use them in a test script, and why to use them. Groovy’s community knows these as ‘Builders’
Functional programming constructs are likely to be known by the candidate, though they are not in position to use them.

The tester-developer should be able to craft large suites from scratch and define architectures for Selenium-RC or WebDriver tests. Their code, indeed because of them the whole style of the codebase should be compositional (this over and above OO). They should be able to train any newbie to the team (from formal development, or from the QA universe) in the tools and techniques needed to grow the test-base, and keep it running.

The storage of test-run results in (and otherwise use of) databases is a skill this candidate will have, as is the integration of related technologies for things that are suitable for them (like Selenium-SoapUI).

Testing architectures that adhere to behavior-driven testing (Cucumber, JBehave, NBehave, Cuke4Duke, RSpec and more) may figure at this ability level. The candidate will know when to use them, and when to stick with Unit-testing type technologies.

The configuration of a cloud capability for their Selenium scripts to use, should be within their skills. These could be Selenium-Grid, Sauce Labs, or BrowserMob (though these are all for different purposes).

The candidate, in the interview, should be able to start a Selenium-RC or WebDriver project, write a build-script, get it all under source-control, hook it up to a pre-existing CI machine and keeps things appropriately separated within the project. Indeed, this level of QA should also be able to pass objective tests for regular developers.

Expert

On top of Advanced …

The candidate will be able to build reusable frameworks and libraries that use Selenium-RC or WebDriver that introduce real value (rather than alleged value) for the for the their team.

They will be able to debug into Selenium itself and provide contributions back to the Selenium team. They are more than likely to be a master of more than three 3GLs.

The candidate’s name might be Simon Stewart.

My Predictive Index

Mon, 25 Jul 2011 00:00:00 PDT

I hope I’m not over sharing here

ThoughtWorks’ Predictive Index Crash Course

ThoughtWorks has a senior fellow, Ed Zaretsky, who has a decade of Predictive Index experience under his belt. He is currently roving around the ThoughtWorks regions running sessions for senior staff. I was lucky enough to invited to to the Dallas event a week ago. There is a lot of value to be had from taking the two-day course from Ed, but one of the biggest take-aways for me was in the understanding of colleagues who are not like me.

The event is preceded by a couple of tasks to complete online. The big one is a series of multi-choice questions about you and how you see yourself that takes about 10 minutes to complete. Ed got our results to us the night before (there were seven attendees). Here is mine in PDF form. At that stage I did not get to see anyone else’s.

My write up was so detailed, and accurate, that I initially suspected that Ed had it pre-written based on the advice from colleagues that had known me for a number of years. It turns out that it solely the product of an application that processes the answers I gave to the online test. The company behind the test must have a huge set of answers for pre-known categories of PI test-taker. Or perhaps AI or something.

Interpreting the Results

About half of the two course is spent in the interpretation of the results, categorizing ourselves, and ultimately the theory of interacting with people after knowing their type. Here’s a social styles model that is similar to the one in the accompanying materials (I can’t share that scan as it is not my copyright) -

My type is ‘Expresser’, though a couple of indicators suggest I am ‘Controller’. That, or “I’m close to the line” (I forget). As I said reading my own writeup did not have a lot of surprises for me. Weaknesses I already knew of (and have mechanical remediations for) were listed. It was the ones that were not listed that was more interesting to me. Specifically it was the strengths that were written for my fellow attendees/colleagues that I wanted to read so that I could envy them.

Thus I purloined the colleague to my right’s writeup. As a group we had already gone into generalities about each of the four quadrants. His was much more opaque to me; Like it was somehow intangible. It turns out he was an Analyzer. I couldn’t grok the way his social model worked after the superficial read of his social style detail. That’s the point I guess; The point why we were all there. When the fellow discovered that I’d read his, he was a little put put out, until he get to read mine.

I’m not going to drill into the types too much. I’d be doing Ed’s two-day session a disservice if I did. Just suffice it to say that insights into the other’s social styles was invaluable. Elsewhere on the web there are lengthier explanations of “Social Styles Model”, with different language for the box labels and axis, go Google for them if you insist!

Agile Epiphany

Here is something that came to me during the session. The Agile methodology that ThoughtWorks helped pioneer, is a highly effective way of making custom software for clients. There’s a lot or reasons for that, but here’s a perspective born from my new understanding of social models, predictive index and all that:

Agile is a tool for software teams that brings Controller, Expresser, Analyzer and Cooperator closer to a uniform behavior. Perhaps a more muted or normalized set of behaviors given the wide range that would be natural. Sure, it is a procedural methodology (time-boxed stand-ups, pairing, TDD, planning game, sustainable pace, etc), but at every turn it is silently neutralizing lead figures subconscious attempts to drag the thinking in their own behavioral direction. The blended team behavior that Agile is best way to eliminate the chance of a death march so characteristic of Waterfall projects.

Lejo contributes some rephrased language that works well for him when getting the point across to managers, that speaks to the same ‘blended’ idea…

“waterfall processes force conformity,
whereas agile processes promote unity”

PS – Meyers-Briggs Types

Myers-Briggs defines sixteen types. We already know that people who understand their own type should not have a strategy to hire the same type, however logical that should seem. It is a useful way to pigeon hole people, but we should understand that all sixteen are good (for a balanced team and workplace). The Myers Briggs stuff seems more applicable to interpersonal relationships, than to vocational things. Predictive Index, seems to be a much more scientific thing.

New Blog Technology and Hosting

Sat, 16 Jul 2011 00:00:00 PDT

Bye bye Subversion

Git is the place to be these days. I have been a happy Subversion patron from version 0.7 onwards, but the last couple of years of Git usage have been a power-up that makes it increasing hard to use Subversion.

Bye bye Sitemesh

Sitemesh is about 11 years old now and at version 3, and still an incredible tool for Java web apps. It requires a constantly running JVM though which is not cheap. There have been to make it work offline like XSite. Jekyll has been made for static site solutions and it is all that I need. Chatting to Joe Walnes (the architect of Sitemesh), I found out that he likes the same tool-chain outlined here for his static sites.

Interestingly, Tom Preston-Warner suggests that Jekyll ‘was primarily inspired by a Ruby project called Webby’, and that he himself had not heard of Sitemesh at the time of Jekyll’s creation. It is funny how things go; Long gone are the days that it was possible to know everything that is relevant in the world of software development.

Bye bye KompoZer (and WYSIWYG HTML editing)

Textile and Markdown are nearly as good as a WYSIWYG editor, for the type of publishing I am concerned with. Though I’m a Junior committer for KompoZer on SourceForge, I am not going to use it for blog editing any more.

Hello Github-pages

Github pages is incredible. I have been a fan for a year now, and my last blog entry was about it. I have managed to preserve all my old article in the migration. I did a git-svn checkout (keeping history is important to me) and a git-push into Github. All my old URLs work, though the ATOM feed has changed. Too many people botch their blog conversions totally (I am looking at you Carlos Villela), or lamely supply a forwarding link.

For twenty or so blog entries, I ran html2textile to make textile. It was not perfect, as the in-line code examples gets chewed up, and I have had to preprocess my HTML with some perl-one-liners to remove evidence my apparent love of   I have cloned a minimal layout by Yuya Saito, and dropped much of the bling that my old Sitemesh decorator added.

Old Blog entries

They are preserved here. Google will index them, and otherwise preserve their pagerank, but they have disappeared from my Atom feed now.

Conclusion

Of course thousands of folks before me have settled on github-pages, so I am not trying to highlight anything new here. I think that anyone technical should look to this platform for hosting of mixed reference and blog materials, including project documentation, small company web sites and where Wiki or CMS solutions might have been used in the past.