Browsers, Clouds & Computing

Agile Planning: Vertical Slices, Done Criteria & Story Points.

2016-08-12T16:18:00.000-07:00

Introduction
one criteria is what this article focuses on.

In a blog post in SD Times, Agile 2016: Scrum is not done yet, and neither are you, Christina Mulligan describes the state of the industry through the eyes of Dave West, CEO & Product Owner of Scrum.org. A nice double entedre in the blog post title!

Dave West notes:

"What we see is that ‘done’ is being fragmented across many organizational boundaries; people don’t really get it, there is not a universal definition."

This fragmented state of affairs gives rise to another phenomenon, capable of giving a bad rap for the Agile industry. Thus, for example, many software practitioners even discuss about disadvantages of agile.

Vertical Slicing
In the rest of the post, I intend to submit a viewpoint that, hopefully, takes us a little closer to a more organized way of agile planning.

A vertical slice is a user story that is crafted by taking a (vertical) slice through the software stack of the system. (The adjective 'vertical' comes about by the manner most macro architectures are depicted -- end user on top of the picture and the various layers of the software system supporting the end user progressively towards the bottom of the picture. See, for example, an illustration in On the Qualitative Importance of Vertical Slicing in Agile Project Executions).

A collection of vertical slices constitutes a release or a minimum viable product (MVP).

It is useful to keep in mind some of the pitfalls while crafting vertical slices. We can describe these by using a simple model of a web application (after Splitting stories into small, vertical slices):

Done Criteria
Once we accept that a vertical slice is a good way to capture end user functionality, the question then becomes: How do we know that we have a good implementation of the vertical slice? This is where the Done criteria becomes relevant. A user story is not completely specified unless it is accompanied with a Done criteria which ensures, when the criteria are satisfied, that the software representing the slice can indeed be made available for a end user (or a proxy) to play with. (Recall one of the tenets of the Agile Manifesto: Working software over comprehensive documentation). If the Done criteria do not assure us that a vertical slice indeed is worthy of end user review, we cannot reliably predict when we can the make the release available! The key thing to note is that we need the Done criteria to assure us that the vertical slice is really done, when the scrum team announces it is done.

Story Points
With the vertical slices needed for a MVP defined, and their corresponding Done criteria identified, it is time to assess the complexity of implementing each one of the vertical slices. Story points are the abstraction used in the Agile community to assess the work involved in creating the implementation of the story by the scrum team. (It is a function of the scrum team). If the stories are crafted by making use of the INVEST mnemonic, it is generally experienced that stories can be moved around to be worked on in different sprints, during planning certainly, to meet with the goal of producing a shippable product increment every sprint.

Concluding Remarks
So, what have seen here so far for a release or an MVP?

Identify the functionality needed by a end user of the system -- user story -- in terms of vertical slices.
Capture the Done criteria for each vertical slice so that, when the criteria are satisfied, the vertical slice can be part of a product increment.
Assess each vertical slice through the abstraction of story points, and using a representative number for the velocity of the team, arrive at the number of sprints needed.

A book that has influenced this post is:
Mike Cohn. Agile Estimating and Planning. Prentice-Hall, 2006. Kindle Edition.

Acknowledgements
Colleagues at Zymr, Inc. are helpful participants in my Agile journey.

Some Thoughts on Modeling in Jira, a Commercial Tool
Jira is an agile project management tool that is quite popular in the industry. This section talks about one way of modeling vertical slices in Jira.

All vertical slices can be modeled as stories in Jira. Sometimes, it is sufficient to model the components of the slice as sub-tasks in Jira; other times, not. In the latter case, it is best to model them as tasks and, since tasks cannot be assigned story points, the task's complexity can be subsumed by a user story that is dependent on the task.

If a requirement is not a vertical slice, it is best modeled as a task that feeds into a story.

On the Qualitative Importance of Vertical Slicing in Agile Project Executions.

2016-05-16T00:04:00.000-07:00

ertical slicing is quite frequently talked about in agile communities. There are good illustrations of this approach using the cake metaphor.

What is the fundamental value of the vertical sliced approach in software development?
First, a brief introduction to vertical slices. A vertical slice is a thin slice — relatively speaking, of course — that cuts across the entire stack of modules in a software system and captures a specific end user interest. For example, in the Netflix cloud architecture chosen below, a vertical slice could be something like "Netflix subscriber wishes to pick a comedy movie, starring Steve Carrell of Office fame, for immediate watching."

Let us look at any end user interaction from the user's experience. Whenever a end user interacts with a system, the end user expects a smooth, near instantaneous, response. That means that the flow of software control through the various layers of software needs to be such that the aggregate behavior, as seen by the end user, is very pleasant.

In general, we have many such end user scenarios in a system; additionally, each end user role would enable definition of different vertical slices. There is another kind of user who'd be interested in a high quality macro behavior; that'd be the administrator. The administrator of a Facebook or a Gmail or an Outlook 365 or a Netflix, for example, would be interested in serving as many subscribers of the service as possible in 100s of millions, if not already in billions, simultaneously.

The diagram below illustrates some representative vertical slices defined through various architectural layers of a system. I have chosen a publicly available architecture, of Netflix' cloud architecture and have illustrated some representative vertical slices through the architecture. We note that Netflix subscribers must feel that the Netflix endpoints — say a Roku player — provides them a very smooth interaction through a variety of functionality — administrative, movie watching, etc.

Since, after all, end user interaction in real usage is always along a (thin) vertical slice through the respective endpoint, it makes sense to consider these vertical slices even during development. The vertical slices represent the manner in which the end user would use the system anyway. This is the fundamental reason why vertical slicing of user stories is important.

An Important Class of IoT Devices

2015-05-03T11:24:00.000-07:00

nternet of Things (IoT) refers to a large class of devices that can be sprinkled throughout a region of activity where we'd like the sensors to capture detailed information about the interesting region. The 'region' could be a farmland, a bridge, railroad tracks, structural parts of aircraft, and almost anything where there is interest to understand any measurable activity. The sensors achieve particular value if they are completely passive, i.e., do not require battery or any power that requires manual replenishment. The sensors are able to provide important site-specific data than can then be transmitted over the Internet, stored, analyzed, and harvested for important macro observations that could be gleaned from such data, sometimes also known as Big Data.

The importance of IoT can be appreciated by knowing, for example, that Cisco's market research of 2011 predicts the worldwide IoT market to be $14.4T in the subsequent 10 years, i.e., by 2020. (As a comparison, recall that the entire US economy is about $17T as of Q1 2015, and the US healthcare market is about $3T today (17.2% for GDP)).

The CardioMEMS™ HF System by St. Jude Medical

The CardioMEMS™ HF System Patient Electronics Unit
by St. Jude Medical

This post is about those class of IoT devices that do not require any battery for their operation. One such device is the CardioMEMS™ HF System by St. Jude Medical. This device is implanted into a heart patient's body and can remain in the body during the patient's life. It is complemented by a bedside Patient Electronics Unit monitor that is able to read the blood pressure information from the implanted device.

Cardiologists at the Sutherland Cardiology Clinic have recently -- April 2015 -- implanted this device into a cardiac patient.

Such battery-less sensors, wherever they are deployed, can provide lots of important site-specific information without the need for any manual intervention for gathering of the information. Once such data is stored in the cloud, the limitations are only in processing the information.

d3.js and the Dining Philosophers' Problem Visualization.

2014-02-16T05:48:00.004-08:00

ecently, I ran into this remarkably versatile tool called d3.js, which is intended to assist JavaScript programmers to visualize various forms of data.

One of the first things I did was to look for any tutorial videos on YouTube, and did I find a treasure trove of d3.js video tutorials! User d3Vienno has around 20 tutorials, many of them in the 10-minute duration, and very easy to follow. (The only thing unusual in the d3Vienno's videos is the narrator's pronunciation of the vowel sound in DOM (Document Object Model) with a \ü\, rather than with a \ä\).

Click on the image to open the HTML source in the browser.

And, an excellent selection of non-video tutorials, particularly the one on Three Little Circles, increased my desire to do a programming exercise of my own.

After I viewed the tutorials, the programmer instinct in me took over and, as I looked around to find a simple visualization problem to program, I recalled the Dining Philosophers problem that is used as a pedagogical tool to discuss concurrency in operating systems.

Of course, the purpose in my solving this problem was merely to see how the table, and the 5 chairs around it, can be programmed using d3.js as the tool. The resulting output is given in the diagram.

The value of this presentation is to be appreciated by looking at the JSON data structure that started it all:

var philosophers = 
 [
  {"value": "Dijkstra", "color": "black"},
  {"value": "Hoare", "color": "red"},
  {"value": "Brinch-Hansen", "color": "blue"},
  {"value": "Wirth", "color": "green"},
  {"value": "Dahl", "color": "yellow"}
 ];

You can view the entire JavaScript source of this visualization here.

Of course, in solving real-world problems, you would find the use of a persistent store as more practical: MongoDB as a persistent store from which to extract JSON structures is such an example.

What if Healthcare.gov Development Were Managed using Scrum?

2013-12-27T15:09:00.000-08:00

In a Forbes' article, HealthCare.gov Diagnosis: The Government Broke Every Rule Of Project Management, the author lists key reasons why the project failed:

Unrealistic Requirements
Technical Complexity
Integration Responsibility
Fragmented Authority
Loose Metrics
Inadequate Testing
Aggressive Schedules
Administrative Blindness

As a Scrum Practitioner, I cannot but help reflect on "What if Healthcare.gov development were managed using Scrum?".

Firstly, due to sheer complexity of the Healthcare.gov web site, you'd need an arrangement of Scrum of Scrums, because we see from the article that:

"A total of 55 contractors were hired to produce the various pieces, and in order for all the steps to work CMS had to involve five federal agencies, 36 states, and 300 private-sector insurers offering well over 4,000 plans."

A "no-brainer" approach would be to define 55 contractor scrum teams, with 55 contractor scrum masters & 55 contractor product owners, even though some of these 55 contractor teams might themselves have to be scrum of scrums. The 36 states and 300 private-sector insurers would take the role of stakeholders in these 55 contractor scrum teams and, working with respective ScrumMaster_i and ProductOwner_i, i = 1, 2, ..., 55, would help create the necessary user stories. And, there would be several user types — subscriber-user, state-user, insurance-user, ... — for which we would have user stories described.

Since the US Government is ultimately the entity answerable to the eventual sponsor, namely the people of the US, you'd have a ScrumMaster_US and a ProductOwner_US within the Government to be answerable to the people of the US¹.

There will be a good number of epic user stories and some epic user stories may indeed reflect the scrum team hierarchy. We'd have to expect that ProductOwner_US² would be the entity responsible for creating this product backlog for HealthCare.gov.

It'd be the responsibility of ScrumMaster_US and ProductOwner_US to ensure a steady output of Healthcare.gov functionality for the end-user. For example, they could say that, at the beginning of every calendar quarter, a new set of functionality would have to be turned online. This is quite doable for, if each of the 55 scrum teams were on a monthly sprint, you would have a chance of having the output of at least three individual sprints from each team for the quarterly output from ScrumMaster_US and ProductOwner_US.

Now, it is important for each scrum team to adopt the scrum framework properly, so that each sprint produces an output that is potentially shippable, meaning potentially usable by relevant user types. And, if necessary, to ensure that only highly-tested output would go online, ScrumTeam_US could form a team that takes the output of the other 55 scrum teams through its own scrum process ... Some could argue that there should be no need to organize ScrumTeam_US in this exclusively quality engineering mode but, somewhere in this whole arrangement, lies the responsibility to ensure that half-baked software is not unleashed on the various user types and, most certainly, one particular user type, namely the public.

Is anyone in HealthCare.gov listening?

¹You now have at least a total of 56 Scrum Masters and Product Owners. If each of the 55 contractor scrum teams is organized as a scrum-of-scrum team, that is an internal complexity that the 55 contractor scrum teams can handle, but the ScrumTeam_US need not worry about it in any routine manner. And, if 55 itself is a large number for ScrumTeam_US to handle, one could consider 5 or 6 scrum-of-scrum teams, each comprising of several of the ScrumTeam_i, i = 1, 2, ..., 55, ...

²ProductOwner_US would work through ProductOwner_i, i = 1, 2, ..., 55, to ensure consistency in project output.

Agile Estimating and Planning: A Book Review - Section I.

2013-10-05T18:11:00.000-07:00

Why write a review in 2013 about a book published in 2005?

Because, Mike Cohn's book, Agile Estimating and Planning is such a book that lays out the philosophy behind agile planning very beautifully, while leading the reader through day-to-day activities.¹

And, as much as I tried to keep the review short, I did not succeed: There are so many nuggets of valuable information throughout the book that I justified making an elaborate review, and decided to publish the review in multiple sections. This post is the first of those.

If your agile development environment already has a set of tools predetermined, such as Rally Software or Jira Agile for example, your day-to-day approach to managing the development may already be somewhat predetermined by the software tool. However, whether you are in such a predetermined environment, or in a pristine environment with possibly no tools other than Microsoft Office, you'll need to recall the basics crisply, and Mike Cohn's book is one excellent reference source.

Part I. The Problem and the Goal

Part I starts with Chapter 1 on The Purpose of Planning by talking about the importance, and purpose, of planning:

"Estimating and planning are not just about determining an appropriate deadline or schedule. Planning—especially an ongoing iterative approach to planning—is a quest for value." (Kindle Locations 573-574).

Chapter 2 on Why Planning Fails is a good set of observations on how traditional approaches tend to fail:

"A critical problem with traditional approaches to planning is that they focus on the completion of activities rather than delivery of features. ... A first problem with activity-based planning is that customers get no value from the completion of activities. Features are the unit of customer value. Planning should, therefore, be at the level of features, not activities." (Kindle Locations 698-702)

This chapter goes on to elaborate many other considerations that distract the team from focusing on value to be delivered to the customer. And, as you reflect on your own experience, I'm sure you will begin to appreciate the wisdom in the chapter.

In Chapter 3, An Agile Approach, we see the author's concern on liberating the creativity of good software designers, and their uniqueness as individual human beings:

"Great software is made by great individuals, and as an industry we have tried too long with too little success to define a development process that relegates individuals to replaceable cogs in the machinery. Agile processes acknowledge the unique strengths (and weaknesses) of individuals and capitalize on these rather than attempting to make everyone homogeneous." (Kindle Locations 843-845)

And, the emphasis on delivering something of business value on each iteration, or sprint, is very well laid out by making use of the concept of Conditions of Satisfaction, both at the story level and at the Release level. The discussion on release planning that proceeds hierarchically from the Release, through Iteration or Sprint, and the Daily scrum, has business value reverberating in every step. (Although the hierarchy can be extended with increasing scope, as Figure 3.1 notes in the The Planning Onion, from the Release, through Product, Portfolio, and Strategy, the book's focus is on the lower levels of the hierarchy: Release, Iteration, Daily Scrum; you can see the planning onion diagram as the slide #8 in Mike Cohn's deck on Agile Planning and Project Management).

Part II. Estimating Size

Estimating Size with Story Points, Chapter 4, takes the reader through the concept of Story Points.

"... a story-point estimate is an amalgamation of the amount of effort involved in developing the feature, the complexity of developing it, the risk inherent in it, and so on." (Kindle Locations 1084-1085).

The concept of story points is well explained, removing doubts as to whether such estimates on incompletely defined stories would even work:

"... However, we need to associate an estimate with each story, even those that are incompletely defined." (Kindle Locations 1107-1108)
"To understand how estimating in unitless story points can possibly work, it is necessary to introduce a new concept: velocity." (Kindle Location 1115)

And, there is persistent emphasis on why the story points idea is valuable because it helps to separate the estimation of effort from the estimation of duration, by using the concept of velocity of the team.

"If we know the team's velocity we can derive [release story-point] size by velocity to arrive at an estimated number of iterations." (Kindle Locations 1126-1127)

There is even assurance that this method works well:

"The beauty of a points-based approach to estimating is that planning errors are self-correcting because of the application of velocity." (Kindle Locations 1137-1138).

And, the example of house painting, where only the relative dimensions of the various rooms are available, is very illuminating on this separation concept.

Chapter 5 introduces the concept of Estimating in Ideal Days, an alternative to story points for estimating the size of stories. Ideal Days is what it would take to complete a user story by the team if worked on uninterruptedly.

Fibonacci number sequence and binary number sequence are two alternatives estimation scales recommended for use in estimating the relative sizes of user stories in Chapter 6, Techniques for Estimating. This chapter also introduces the concept of epic user stories that comprise of simpler user stories that can be considered as being part of the epic, and themes that are a useful collection of user stories not related in any causal manner, but as a thematic collection.

In Chapter 7, Re-Estimating, considerations for when and how to produce re-estimations of story points are examined while noting that, in many cases, the use of velocity in determining the speed of execution obviates the need for re-estimating.

"... velocity is the great equalizer. Because the estimate for each feature is made relative to the estimates for other features, it does not matter if our estimates are correct, a little incorrect, or a lot incorrect. What matters is that they are consistent." (Kindle Locations 1510-1512).

And, there is this reassuring wisdom on not to get excessively concerned about re-estimating:

"Do not become overly concerned with the need to re-estimate. Whenever the team feels one or more stories are misestimated relative to other stories, re-estimate as few stories as possible to bring the relative estimates back in line." (Kindle Locations 1576-1578).
"You should re-estimate only when your opinion of the relative size of one or more stories has changed." (Kindle Location 1581).

Having introduced both story points and ideal days as metrics of story size, Chapter 8, Choosing between Story Points and Ideal Days, discusses pros and cons of using both and concludes with the author's recommendation:

"My preference is for story points." (Kindle Location 1678).

Part III. Planning for Value

Chapter 9 dwells on Prioritizing Themes based on several factors:

Financial value
Cost of development
Amount and significance of new learning and
The amount of risk removed

It is somewhat obvious how the financial value and the cost of developing the value would influence choice of user stories for implementation. However, assessing the value of new knowledge — and the attendant reduction in risk — is a bit trickier, and the book distinguishes between knowledge about the product and knowledge about the project. And, the figure — 9.1 — contrasting the approaches of Waterfall and Agile methodologies is very illuminating. Considerable elaboration of examples based on infrastructure & user interface provides further insight as to how to choose the user stories for development.

Chapter 10 on Financial Prioritization describes, in considerable detail, the various concepts of Net Present Value (NPV), Internal Rate of Return (IRR), Payback Period and Discounted Payback Period.

Prioritizing Desirability is Chapter 11's content. It elaborates on the Kano model of customer satisfaction that separates features into three categories:

Threshold, or must-have, features
Linear features
Exciters and delighters

"Threshold features are those that must be present in the product for it to be successful. They are often referred to as must-have features. ... A linear feature is one for which “the more, the better” holds true. ... Finally, exciters and delighters are those features that provide great satisfaction, often adding a price premium to a product. However, the lack of an exciter or delighter will not decrease customer satisfaction below neutral. ..." (Kindle Locations 2195-2207).

Considerable discussion of assessing themes in the Kano model follows in the chapter.

Splitting User Stories is the topic of Chapter 12. It does take some practice to define user stories in a complete, meaningful, way: Ideally, we want a user story completely implementable in a single iteration or sprint. Splitting user stories along data boundaries, along operational boundaries, based on CRUD operations, by removing cross-cutting concerns, by separating performance concerns, by separating priority considerations are all discussed in this chapter. An important concern that has been quite frequently written about in the web is: Horizontal and Vertical User Stories - Slicing the Cake.

... to be continued in the next post

¹Ken Schwaber's book on Agile Project Management with Scrum is another book that explains the philosophy of the scrum framework.

On Universal Applicability of the Scrum Framework.

2013-07-19T19:36:00.000-07:00

Meetup Announcement. (Click to go to Meetup page).

If you listened to Jeff Sutherland on the July 16 talk on "Scrum: The Future of Work", you cannot but walk away with the feeling that scrum methodology is more universally applicable than software development, but the rest of the world is yet to discover that fact. And, his enthusiasm for Scrum's applicability not just in Software Development, but in other parts of Enterprises, in Government, in non-profit organizations, etc., is contagious. Scrum and the Grameen Bank, the Nobel Prize winning microfinance organization?

You can find the slides and the video of the proceedings at the Silicon Valley Agile Leaders Network Conference by Scrum, Inc.

An interesting statistic presented was the number of Scrum job openings in the US today:

In July of 2012 there were 20,000 Scrum jobs open in the U.S.
Today, there are 600,000 Scrum jobs open in the U.S.

And, there was the anecdote where many of the original scrum ideas were indeed borrowed from the microfinancing operations of Grameen Bank and, 20 years later, a Scrum Master is working with the Grameen Bank to improve their scrum practices. Talk about life coming around full circle!

At about 0:43:00 in the video, Jeff talks about how the Toyota Prius team started using scrum "to build a car that gives twice the mileage of its predecessors, ... to make a greener planet, ... to make the world a better place, and, ... to ... ship in half the time."

You can watch a YouTube video of the presentation here.

To gain a perspective on the universal applicability of the Scrum framework, we can start by reviewing the Scrum Guide. When you read through the guide, which is a succinct representation of the framework, you will realize that the various components of the framework are very general, and can indeed be applied to contexts other than software development. Let's enumerate key concepts from the guide:

The Scrum Team

The Product Owner

The Development Team

Click to open the Guide.

The Scrum Master

The Scrum Events

The Sprint

Sprint Planning Meeting

Daily Scrum

Sprint Review

Sprint Retrospective

The Artifacts

The Product Backlog

The Sprint Backlog

The Definition of "Done"

None of the foregoing concepts have anything specific to software development, although these concepts have gotten formulated and solidified over the years during the use of scrum in software development.

The coming months and years will be interesting to watch as the utilization of the scrum framework spreads.

On the Storage Needed to Capture Human DNA

2013-03-26T18:51:00.000-07:00

Ever since Watson and Crick solved the DNA puzzle, in 1953, and shared a Nobel prize in Physiology or Medicine in 1962 with Wilkins, there has always been interest to store and retrieve DNA information.

However, it was not until the Human Genome Project concluded in 2003, that the entire human genome (of about 3 billion nucleotides) was considered fully decoded ¹.

Now, what would it take to store the entire human genome on persistent storage, such as in the cloud? This type of information may be eminently suited to make use of inexpensive cloud storage such as the Amazon Glacier for about $0.01/GB/month. That is, if you simply stored each nucleotide in one byte, you'd pay $0.03/month. Yes, only 3 cents a month!

Or, of course, you can buy a TB hard disk for less than $100, and be done with it: you'd only use less than about 0.3% of the drive.

Now, since the 3 billion nucleotides are formed from the alphabet space of 4 nucleotides — A, C, G, and T — one could encode these in only 2 bits, thus reducing the storage needed to 1/4 of 3 billion bytes, or about 750 MB. For example:

Nucleotide	Encoding
A(denine)	00
C(ytosine)	01
G(uanine)	10
T(hymine)	11

¹In April 2003, it was believed that 99% of the human genome was indeed captured to an accuracy of 99.99%.

SV-ALN Meetup: Be Agile. Scale Up. Stay Lean.

2013-03-20T23:10:00.000-07:00

Dean Leffingwell

If you are already familiar with the concepts of Agile methodology at the team level, following Dean Leffingwell when he elaborates on the scalability of the Agile framework is a breeze. Such was my experience yesterday when he presented the Scaled Agile Framework™ (SAF) at the SVALN meetup.

The primary ambition of the Scaled Agile Framework is to take the Agile principles to be effective at the enterprise level.

We have already seen several approaches to utilizing hierarchy in using the Agile methodology at the team level. This framework takes the hierarchy, from a team or a project level, to program and portfolio levels, and recognizes the need for non-functional requirements (NFRs), in addition to the user stories that a user would definitely interact with, and recognizes business epics and architectural epics.

Meetup on 19 March 2013.

Several different considerations apply at the project, program and portfolio levels. For instance, at the team level, the recommendation from experience is that the user stories scheduled for an iteration be such that it encourages the development engineer to see the implementation through testing so that, at the end of the iteration, the team has a demonstrable increment in functionality.

At the team level, you may have 2- or 3-week sprints; a release may occur at a Potentially Shippable Increment (PSI) comprising of work amounting to several sprints; at the portfolio level, there may be a need to synchronize of releases of several products.

Scaled Agile Framework™.

Note also, the scaling is chosen to be described as a 'Scaled' framework, rather than a 'Scalable' framework. You'd have to conclude that an impressive array of enterprises that have adopted SAF is what may be behind that choice.

And, if you believe that organizing the enterprise-level needs to be driven hierarchically, you wonder why anyone would not want adopt a framework such as SAF.

On Managing [Complexity in] Systems Development

2013-03-11T19:44:00.002-07:00

Modern society invariably tends to need complex¹ systems. (If a system is too 'simple', there may not be any market for it). Consider, for example, some of the successful products in present times:

Smartphones such as the iPhone, the Samsung Galaxy
Tablets such as the iPad, Google Nexus 7
Electric cars such as the Tesla

Each one of the foregoing has, in addition to software, a collection of parts, in several levels of hierarchy, in creating the end user product. (A blog post makes an interesting observation that mature products tend to have not more than 4 levels of hierarchy in their bill of materials).

Even software-only products exemplified by today's social networks, software-as-a-service (SaaS), etc., have, behind them, a huge cloud computing environment ².

Now, why this elaborate preface? What methodology can be successfully employed in managing the development of such complex systems? A methodology that helps in conquering the system complexity. Hierarchy, it turns out, is a method to conquer the complexity of software. While the concept of hierarchy is not new, its effective utilization in managing software (or software-centric) projects tends to be.

In order to manage a development a software (or a software-centric) project effectively, a hierarchical set of requirements, at whose leaf nodes are the user stories ^3,4, needs to be created. A user story, when fully implemented, i.e., done, is ready for end user consumption. User stories provide a unit of development, and measurement of the value, with a end user in mind.

Agile practitioners can use several concepts of hierarchy in managing the software development that suits their particular environment:

It is my hope that this blog post can be useful in prompting engineering managers to establish the user story hierarchy early on during the project.

¹ It is important in this blog post to distinguish between the words 'complex' and 'complicated'. 'Complex' implies a collection of interacting entities ending up serving a meaningful purpose. 'Complicated' is a collection that either does not produce a meaningful purpose or, if it did, it is very difficult to comprehend.

² Google provides a map of their data centers on their company web site.

³ The concept of user stories is now well entrenched in Agile environments as the Agile methodology experiences its inexorable growth in software development. As a matter of fact, it is even making inroads in hardware-oriented projects.

⁴ A presentation on user stories, titled Invest in Good User Stories, is a good and succinct introduction.

SVForum Meetup: Building a Cloud Platform for Real-Time Video: Learnings and Challenges

2012-02-25T14:46:00.001-08:00

This meetup on Building a Cloud Platform for Real-Time Video: Learnings and Challenges was a very captivating presentation — except in the beginning, during introductory slides, when it was somewhat slow-moving¹ — on how BlueJeans have created a good recipe for videoconferencing service in the cloud.

The speaker, Alagu Periyannan, is Co-founder and CTO of BlueJeans. Basically, they have created a solution for video calls that appears to be as easy as using Cisco Webex audioconferencing, and supports several different video endpoints to participate in those video calls, including those from Cisco, Google, Logitech, Microsoft, Polycom & Skype. To some IT departments' chagrin, the BlueJeans solution can add one more reason to the Bring-Your-Own-Device (BYOD) phenomenon! (See, for example, BYOD: What will work and what won't in your enterprise, by my colleague from HP, Vishwas Manral).

The primary reason for BlueJeans' capability is their ability to transcode a stream of 720p video in 10ms. (This fact was key information of the talk; I don't know when, or whether, the slides would be published). For example, if there are multiple video endpoints in a video call, a composite video can potentially be created within a delay of 20ms: 10 mS for the transcoding each of the individual endpoint streams (in parallel) and 10 ms for transcoding the composite stream. A one-way delay of 0-150 ms is considered acceptable by CCITT G.114 Recommendations [Craig Partridge. Gigabit Networking, 1994]. Thus, there is nearly 130 ms of budget for one-way transmission. If you consider that the core of the video network is built on optical fiber, you can expect a one-way transmission speed of approximately 200m/μs, or a maximum distance of 26,000 km in 130 ms.² This speed seems enough to satisfy across-the-globe transmission.

And, if the architecture of the transcoding and multi-point control units (MCU) is sufficiently parallelized, each transcoding operation can be achieved in the aforesaid 10 ms!

Therefore, it seems that BlueJeans has hit upon a nice solution that can propel them to start-up stardom provided, of course, they execute right! They don't have a freemium pricing model, but services start at $199/month for individual user, and so you'd have to be a serious video call user in order to sign up for their service.

¹Because of historical details on how videoconferencing came about over the years

²Because the index of refraction of most fibre optic cables is about 1.5, the speed of transmission in fiber works out to about 4.9 microseconds of latency for every kilometer. Of course, the last, or first, mile is to be considered too, and it is more frequently copper than fiber.

Is Apple Safari for Windows the best browser for the desktop?

2011-08-08T11:33:00.000-07:00

In a previous post, I explored the JavaScript compatibility, on my laptop, of the 5 browsers I regularly use against the ECMA's test suite. That was my first attempt at the comparison, and I didn't think much of it. I undertook that exercise again a month later, yesterday, and I can say with confidence the following:

Chrome, Firefox and IE fare reasonably along the same lines.
Opera shows a lot of holes in its implementation
Safari is the worst performer: crashes in the middle of the test execution, probably due to memory leak.

You can see all the detailed results in a separate web page.

The Power of CSS (at the Open Web Camp, 2011).

2011-07-17T20:17:00.000-07:00

If you are a novice to CSS, you will like this presentation by Estelle Weyl, which I attended yesterday at the Open Web Camp, 2011. If, on the other hand, you know a little of CSS, you'll love this presentation!

In this presentation, you can learn how to produce nice animation effects, such as producing snow any time you want! This use of CSS, as opposed to an equivalent JavaScript code, shows the power of declarative programming, in comparison with its companion, procedural programming.

I attended a few more presentations on the CSS power at the camp:

Shkrubbel, using web technologies to build a multi-player word game, by Kitt Hodsden. Kitt is a vivacious speaker.

The Future of CSS - Current Experiments and Near-Future Reality, by Tab Atkins. Tab has advanced knowledge of what is happening in the W3C CSS community, because he is a member of the W3C CSS Working Group, among others, while being a software engineer at Google.

Open Web Camp attracts a good community of web developers, thanks to its sponsors for free admission.

A Comparison of Browsers: Popularity and JavaScript Compatibility

2011-07-01T18:28:00.000-07:00

At the beginning of every month, new statistics are published on browser usage: At the onset of July 2011, as you can see from the diagram below, Chrome is closing in on Firefox, even as IE is losing its market share.

Source: StatCounter Global Stats - Browser Market Share

Popularity aside, it is also important to understand how their JavaScript implementations stack up. ECMA has made available a standardized test suite for the latest version of ECMA-262 edition 5.1, published 30 June 2011, known as Test262.

Once I learnt above the availability of these tests, it seemed like a good idea to test the browsers on my laptop. The tests were conducted on July 1, 2011. The results are given below for the browsers I tend to use [Browser Performance is tested against ECMA's Test262 Version 0.7.5.1, 30 June 2011, with 10934 tests]:


Browser	Version	Time to Run (min.)	Pass	Fail	Fail to Load
Mozilla Firefox	5.0	42	10,662	262	0
Microsoft IE	9.0.8112.16421	20	10,634	300	0
Google Chrome	12.0.742.112	55	10,427	507	0
Opera	11.50 Build 1074	46	6,995	3873	66
Apple Safari	5.0.5 (7533.21.1)	Crashes after ~2,500 tests in a little after 5 hours!	~2,500	20	~8,414

The biggest surprise is that some of the browsers do not load all the tests; and, Apple Safari is the worst performer from this perspective, not being able to load more than 75% of the tests even after 5 hours of execution! And, Opera failed more than 35% of the tests!

Firefox, IE and Chrome have had the least hiccups, were able to load 100% of the tests, but failed 1.4%, 1.6% and 2.7% of the tests respectively.

All the tests were run on an ACER Aspire 4730Z, Pentium Dual-core CPU T4200 @ 2.00 GHz with 3GB RAM and Windows Vista Home Premium Service Pack 2, 32-bit. It takes about 50 minutes to run the tests on each browser.

The goal of my testing was primarily to assess JavaScript compatibility, and not execution speed.

SVForum Cloud Services SIG: "The Future of the Data Center"

2011-06-29T11:30:00.000-07:00

By the title of the meetup, namely "The Future of the Data Center", you might conclude that the two scheduled talks would be about how to architect future data centers, racks, power, cooling, etc. But, the talks were a bit more focused on only two, but important, aspects of enterprise clouds:

Brad Stone, Nexenta: Cloud Storage.

Dave Roberts, ServiceMesh: The Big Win.

Both the presentations can be accessed through the SVForum Cloud and Virtualization SIG web page, Resources section.

Nexenta

An interesting thing about the Nexenta storage offering is its ability to manage storage across heterogeneous hypervisor environments -- Citrix, Microsoft, VMware, etc. And, through the use of the 'pool' concept, both SSDs and HDDs appear as uniform abstractions to the storage user. Both file level and block-level storage are provided. There was not much opportunity to go into the specifics of architecture, although it was mentioned that the implementation is based on ZFS and for, parallelism, pNFS. Thanks to ZFS, the scalability of the Nexenta solution is very huge, as seen in the thumbnail above.

There are several storage systems on the market, and the data center designer is well advised to study them all to determine the best choice for a given data center design.

ServiceMesh

ServiceMesh's product offering is the Agility Platform that enables collaboration among the various stakeholders in a cloud environment: product managers, business managers, developers, IT personnel, etc. In addition to the usual identity-based access rights, it also provides for policies of various types that local governmental regulations might require a global enterprise to comply to. As can be seen in the diagram here, the platform connects with various types of clouds, and provides for rich and extensible set of policies driven by meta data associated with various cloud components. Thus, you can set policies such as certain data has to stored within a certain national boundary, certain data has to be SAS 70 compliant, etc.

The Agility Platform itself can be hosted either within the enterprise or in a public cloud. It seems to me that large enterprises would do well to base their cloud implementations on something like the Agility Platform: Otherwise, it can be too expensive to deal with various types of lapses in the cloud. Dave cited the case of a Google Exec that was convicted in Italy as an example of the need for compliance.

Another interesting aspect of ServiceMesh, the company, is that there has been no VC money used, and is completely bootstrapped.

Thus, in one evening, you could get not only a holistic view of how to approach an enterprise cloud creation, but also a look into a focused technology for storage in the cloud.

Cloud Fabrics and Virtual Networks

2011-05-29T21:17:00.000-07:00

In the last couple of years, nearly every networking equipment vendor is touting their new data center fabric aimed at the creation of clouds. Take a look at the [partial] list below, culled from the respective web sites:

Company	Fabric Offering
Arista Networks	????
Brocade Communication Systems, Inc.	Data Center Fabric
Cisco Systems	Unified Data Center Fabric
Extreme Networks	Open Fabric Data Center
Force10 Networks	Open Cloud Networking
HP	FlexFabric
IBM	BladeCenter
Juniper Networks	QFabric
NEC	????

Nearly all of them claim membership, in one form or the other, in the OpenFlow and the OpenStack community. So, how will a cloud provider pick from among these? Some of the obvious considerations are:

Is the physical infrastructure capable of 5 9s¹, or 99.999%, availability?
How many virtual networks can be supported by the physical infrastructure? Or, more specifically, how many virtual links can be supported between any two geographical data centers? The economics for a cloud provider is related to this parameter.

In addition to the foregoing equipment providers, there is a new crop of players as well: Big Switch, Nicira, vCider, ...

It will be interesting to see how this physical infrastructure play will evolve.

¹Actually, in order to provide the 5 9s behavior in virtual networks, it won't be a surprise if 6 9s availability is required at the physical network infrastructure level.

At the Silicon Valley Cloud Computing Group: Zynga presents a Real World Case Study

2011-05-26T23:28:00.000-07:00

If you believe cloud computing is still in its infancy, you have not learnt about Zynga. While Zynga have a hybrid cloud that powers their game servers involving their data center, a private cloud, zCloud, and a public cloud, namely Amazon Web Services (AWS), the more interesting fact is that they expect to be profitable in 2011 with a net margin of 35%, or a net profit of $630m on a $1.8B revenue.

At the Cloud Computing meetup today, Allan Leinwand, Zynga's CTO - Infrastructure, presented a fascinating study of how Zynga has employed cloud computing to power their game servers. It appears we will not see Allan's presentation in a publicly accessible web site any time soon — See Sebastian's talk about the meetup — and what follows is an account of the meeting, transcribed from my notes written out on paper. By the way, Allan's rapid fire delivery is a treat to experience.

Zynga started out renting server power through Amazon Web Services, and managing them by RightScale. By 2009, need for greater flexibility and scalability¹, tipped Zynga to build a private cloud, zCloud, and complement it by using AWS. To smoothen the hybridization effort, they built zCloud as an AWS look-alike, including their security model, etc. In building zCloud, Zynga also avoided the use of inter-rack VLANs, and used an IP-routed network to interconnect racks.

Some interesting details about zCloud:

Zynga uses 1 VM per physical server.
CentOS is the guest operating system.
They use x86-based physical servers.
They have created a highly automated scheme whereby they can provision > 1000 physical servers in < 24 hours.
The traffic in their cloud is more North-South, between tiers, than East-West (across servers).
Growth: In January 2011, the server count was 75 times that in January 2009².
A single orchestration layer manages the three key resources of the cloud: CPUs, Storage and Network.

Allan's wisdom to application designers: "Understand the architecture of the cloud in designing your application."

Allan's recommendation for "Cloud 2.0":

99.999% availability at the IaaS level
Compute power needs to scale to millions of CPUs. (Can the cloud borrow concepts from SETI@home?)
The cloud environment needs to be able to handle any workload

Overall, a very enjoyable and highly educational evening. Box.net — another vendor powered by a cloud — were gracious to provide the venue for the evening and Sebastian & Co. (i.e., whoever else worked in the background to make this event happen) deserve a big round of applause.

¹In addition, the usual OpEx vs. CapEx argument was also a factor.

²For obvious reasons, the absolute server count was not revealed.

At the Cloud and Virtualization SIG on Virtual Networks

2011-05-24T23:32:00.000-07:00

This SDForum Cloud and Virtualization SIG meeting saw a set of very exciting set of presentations by vCider, Big Switch and Cisco. There have been mature commercial availability of CPU and storage clouds, but there have been no rich commercial offerings of [virtual] networks¹.

All of the presenters argued about the need for network virtualization. Chris Marino of vCider visualized a network cloud offering that could straddle multiple cloud service providers. (Amazon and Rackspace mentioned in the slide 14 of his presentation on the right). Guido Appenzeller of Big Switch focused on a network of OpenFlow switches². Ram Durairaj of Cisco described the concept of Network Container, a Cisco proposal to the OpenStack community.

One thing is certainly clear: In the last 25 or so years since the commercial availability of digital networks, thanks to the Ethernet, we have seen an explosion of enterprise networks and data centers. Many of the concepts employed in data centers today would need to be available in cloud form in the coming years ...

¹Amazon Virtual Private Cloud (VPC) is a commercial offering, but is not yet as flexible as what was described this evening.

²The OpenFlow Controller is somewhat similar to the Path Computation Element concept in that they both are focused on offloading of control plane logic to separate computing element(s).

Virtual Networking and Clouds.

2011-05-20T13:44:00.000-07:00

Recently, in a blog post written by Chris Marino, titled Virtual Networking, It’s Not Just a VLAN…, I came across a beautiful definition of what a virtual network is:

"A virtual network is a network you control that runs on top of another network that you don’t control."

Note that the virtual network, in this definition, allows for a rich realization of what the virtual network is. It could be a complete enterprise network replete with multiple IP subnetworks ... or, it could be a small test network with primaily only layer 2 networking involved among the network elements in the cloud. The challenge to cloud service providers is to allow cloud subscribers the highest flexibility in composing latter's virtual networks consistent with the economics of the former's service business. In fact, for some cloud subscribers, there is a need to keep their entire enterprise network assets in the [public or private or hybrid] cloud, and access their enterprise network in the cloud through Internet connections.

Such flexibility seems to be the goal of a new contingent of vendors that have lined up as part of the Open Networking Foundation. If this foundation's work succeeds, the role of networking devices such as the Cisco CRS-3 will be rewritten.

I look forward to listening to Chris and others in the Cloud Computing & Virtualization SIG: Virtual Networks next week!

Cloud Computing according to Yahoo!

2011-05-13T20:06:00.000-07:00

In a presentation yesterday at Cloud Computing at Yahoo! Lessons, Challenges and Futures, Yahoo! Cloud Wrangler Geoff Arnold presented a view of how the Yahoo network needs to evolve in the present day cloud computing environment.
Although the actual extent of the present day Yahoo network could not be divulged for obvious reasons, Geoff argued that public cloud providers like Amazon Web Services (AWS) could not support the functionality that is expected of the Yahoo network. Even if the public network were to be made use of for a 24x7x365 production network, Yahoo would have to rent bare metal CPUs networked and storaged¹ together, and I do not know if the economics for such a deployment is already there. (We know, though, that Netflix uses AWS for streaming movies to its subscribers). This is probably true of other network based services such as Facebook's, Google's, etc.

Yahoo! Cloud Architectural Vision

The Yahoo! cloud architectural vision that was laid out — slide 31 on Slideshare.net — somewhat explains why general purpose public cloud services may not be sufficient: There seems to be substantial functionality that is somewhat Yahoo-specific that is not necessarily readily available in the public cloud. And, Geoff talked about "planet scale" for most of the functionality Yahoo delivers. In addition to the familiar layers of SaaS, PaaS and IaaS, that are probably richer than their run-of-the-mill counterparts, he also sees a Knowledge as a Service (KaaS) layer sandwiched between SaaS and PaaS.

Overall, you come away with the awesome feeling that the Yahoo network is really very big.

The video of the presentation is at the Silicon Valley Cloud Computing Group channel at Ustream.

¹We can regard this kind of journalistic license [of creating new verbs on the fly] as an advantage of the computer science profession.

Proud Recipient of a Google Chrome OS Notebook Cr-48!

2010-12-13T23:21:00.000-08:00

Happiness comes to you in different ways. Today, I was surprised, along with 199 others I believe, at the HTML5 Talk event conducted by Google at their campus in Mountain View. While HTML5 was definitely talked about, the key surprise was kept so until the last few minutes of the presentation by the Chrome OS product manager, when he announced that everyone attending the talk would get a Cr-48, as part of their pilot program!

Cloud computing is certainly taking hold, and you can read about Google CEO Eric Schmidt's thoughts at the announcement. There are several reviews already out on the Cr-48, including this one by IT World.

Here's a first impression of an engineering major, a sophomore — my son — on the Cr-48:

Microsoft and Apple have to rethink their computer strategy!

Nothing but the Web.

Netflix moving to Amazon Cloud by Hien Luu - A Review.

2010-12-09T22:01:00.000-08:00

It is not often that you get to listen to a story that describes how a massively scalable application — as of October 2010, Netflix has nearly 17 million paid subscribers for their streaming solution — has been developed for the cloud, in this case the Amazon cloud, known as Amazon Web Services (AWS). Hien Luu described such a story. He was part of the implementation team at Netflix, and is presently with LinkedIn.
The event was conducted by Software and Architecture Modeling (SAM) Special Interest Group of the SDForum.

Netflix Moving To Cloud

View more presentations from hluu.

He made the presentation livelier by sprinkling interesting questions about Netflix throughout his presentation, and awarding a copy of the book titled Host Your Web Site in the Cloud to whoever in the audience answered a question with the greatest accuracy¹.

Hien's presentation involved four sections (Slide #2):

Why Amazon cloud? The answer, as you would expect, is that "Netflix is not in the business of creating and operating data centers", even though they currently have 2 data centers for their private data, and the Amazon cloud proved to be an effective platform for hosting the Netflix streaming solution.
Tour of AWS. This tour was a very useful introduction to AWS, particularly since the later portion of the talk makes use of several of the AWS components in creating the streaming solution.
How Netflix leverages [the cloud]. Netflix uses ~4000 EC2² instances during encoding movies into digital format for streaming, typically a 15-day exercise, while ~1000 EC2 are normally deployed for servicing subscribers. The storage is of the order of petabytes. The storage for movies, built on AWS S3, has a rather large latency — 300 mS — and hence the S3 is front-ended by Memcached, and the resulting latency is brought down to about 5 mS. (Slide #30). Slide #32 highlights the storage usage: e.g., ~800M objects in SimpleDB, ~17M objects in S3. Slide #33 brings out the importance of some key features that would be desired in a cloud, but were missing from AWS, as Netflix prepared their implementation; Memcached is one of the missing services.
Best practices. In this section, Hien highlighted the need for automation — Can you imagine setting up 4000 machines by hand? —, minimizing latency, ensuring storage persistence, instrumenting every step of the call, sharding, etc.

Overall, a lively evening, as you got to know how Netflix has prepared itself for paid subscriber growth for the streaming solution.

There were several announcements made at the beginning of the session:

SAM SIG & Java SIG will be combined beginning January 2011
Jian Zhang of GridX announced they have openings for software engineers — Hadoop, Java, Front-end UI programmers — in building a cloud based financial application for the Smart Grid.

¹One example question was "How many paid subscribers does Netflix currently have?". The answer is 17 million. There were more books than there were questions, and some lucky folks in the audience got the rest of them in the end.

²Elastic Compute Cloud.

The Action behind Clouds: Cisco's Acquisition of LineSider.

2010-12-04T14:28:00.000-08:00

Clouds are good for cloud subscribers, but how can cloud providers offer on-the-fly provisioning such as is expected by the subscribers? Clearly, requests¹ for Infrastructure as a Service (IaaS) at a subscriber level — a certain combination of CPU, networking and storage — must be translated down to device-level, actionable, commands to both physical and virtual devices. And, these commands need to be effected as a single transaction in the cloud provider system. Cisco's intent last week, to acquire LineSider Technologies, brings this matter to the fore.

LineSider's white paper on Network Services Virtualization elaborates on the challenges by bringing out the limitations of point² products of the following types:

Configuration Management and Automation Engines
Inventory and Capacity Management
Network Alarm and Correlation Management

Their OverDrive product "... automatically initiates the creation of the required virtual machines (VMs). As the VMs are coming online the OverDrive virtualization engine defines and deploys the network access and security models across all required infrastructure devices (routers, switches, firewalls) as needed, to deliver the Cloud service to the defined end-users. The entire process is completed in seconds ...". Moreover, "Once the Business Policy is implemented and the Cloud service is active, the access and security models are bound to the end-point resources and persisted in the OverDrive engine."

What is the moral of the story? While it is important to create point technologies for the cloud, it is also important to think about their manageability in an 'end-to-end' scenario: The cloud subscriber is one of the ends, and the cloud provider is the other end, with all the devices in between — physical or virtual — associated with the cloud service in a persistent, auditable, manner.

¹Requests for cloud services are equally well applicable to both private enterprise cloud subscribers and public cloud subscribers.

²What were once 'end-to-end' solutions become 'point' solutions in an evolving system.

HTML5: Up and Running by Mark Pilgrim - A Book Review.

2010-12-02T22:18:00.000-08:00

For quite some time, I have felt that good programmers can be good storytellers; this book confirms my belief. The conversational style of presentation is particularly appealing.

This is an excellent introductory HTML5 book for someone who has prior knowledge of HTML, CSS, DOM & JavaScript. This is a book that is useful even to experienced web programmers who may think that they already know what the book is going to say; a good author can tie together various nuances into beautiful prose.

Chapter 2 on Detecting HTML5 Features is a typical example of how a good programmer can be a good storyteller. It introduces the Modernizr, an open source MIT-licensed JavaScript library, in a very easy way. The reader is led slowly, but inexorably, to the conclusions the author wishes to establish regarding how to ensure proper web programming in the context of HTML5.

Chapter 3 provides a very nice introduction to the semantic elements of HTML5 by evolving a HTML4 document, also a syntactically valid HTML5 document, into a more desirable HTML5 document, making use of new semantic elements.

Chapter 5 is a rather thorough explanation of the semantics of the <audio> and <video> elements. There is ample detail about containers and tracks — of audio and video packaging — that it completely demystifies the handling of audio and video in HTML5.

Other chapters delve into various other aspects of HTML5, and this is a book you want to have so that you can refer to it whenever you want to recall the thought behind a certain HTML5 feature.

There is humor in the book too. e.g., on p. 147,

"Everybody knows about web forms, right? ... You don't know the half of it ... HTML5 defines a dozen new input types ... I don't mean to say that all of these exciting features are actually supported in every browser ... In modern browsers, yes ... In legacy browsers ... all of these features degrade gracefully in every browser. Even IE 6."

The italics are mine, for emphasis.

The online version of the book is made available free under a Creative Commons License.

There are many other reviews of the book on the web, and I found this one by Ben Nadel particularly enjoyable to read.

This is definitely a book for your bookshelf, physical or virtual.