After three fascinating years shepherding development for Songbird, the popular open media player platform, I’m excited to announce that I’ve recently left the nest and embarked on a new adventure.

Throughout my career, I’ve spent many years at startups backed by top VCs (Sequoia Capital, KPCB, Atlas Venture and Sutter Hill Ventures to name a few), building engineering teams from the ground up. During those years, I discovered the best way to manage development organizations and keep them at peak performance. I’ve elaborated a unique and pragmatic approach to the balancing act of managing process, team and technology, yielding great results.

It’s been challenging, humbling and rewarding and I would do it all over again in a heart beat. I’ve come to a point where I have the desire to share that knowledge and expertise more broadly and mentor many more upcoming technical talent along the way. To that end, I’ve started a boutique software engineering consulting practice called Ternary Labs.

I’m offering a service I call “Engineering Coaching” which focuses on providing technical guidance, establishing processes and coaching engineering teams. Startups immediately benefit from an experienced VP Engineering, at a fraction of the cost and without the hassle of recruiting a full time senior executive.

Having experienced how difficult it is to find a reliable partner to outsource projects to, I also build custom software under contract. We specialize in Ruby on Rails and iPhone development. More information can be found at:

http://ternarylabs.com

If you’re trying to figure out how to organize your engineering team, experiencing growing pain, lacking good process, having problems with your technology or just need something built, give us a shout.

Over the years, we’ve taken pride in making our tools, product and development process, as open and transparent as possible. Our tools (Bugzilla, Litmus, Wiki) are publicly accessible, our source code open (svn) and we’ve blogged on many occasions about our Agile practices.

Today we’re taking a step further by publishing our Development Survival Guide. This presentation is an internal step-by-step guide that we take new engineers thru during their orientation. It’s a good summary of what to expect on a day-to-day basis as an engineer working at Songbird (other than daily Fussball tournament and unfettered access to the beer stocked mini-fridge). It’s your opportunity to take a peek from within.

Want to get even closer? Apply for one of our openings.

Download Songbird Development Survival Guide 1.2 in pdf.

Fortunately, the metrics gathered during each Agile release cycle can be very helpful for that purpose. Once we gain a good understanding on what is being worked on, for how long and by how many people, we should be able to extrapolate this to forecast future releases.

Let’s take a look at what activities take place during a typical release cycle:

1) Plan release
2) Write code
3) Test
4) Fix bugs

Then repeat ad nauseam.

In that context, the programming tasks can be categorized as follow:

1. Planned work

This is the body of work identified during the planning stage. This is the raison d’être of the release. For the most part, this covers new features or less tangible things such as performance improvements. This is what we’ll want to talk about when the product gets released. It’s an easy planning trap to think that this is the only work required.

2. Unplanned work

As the name implies, this is work that was unforeseen at the beginning of the release but is required to be completed before the product can ship. It can be further refined as follow:

a) Change in requirements
This should not be unexpected. In fact, any Agile methodology assumes that there will be changes down the road. This is not a problem per se as long as there is a mechanism to trade features, extend duration or increase resources.

b) Omissions in planning
Because the planning period is relatively short, it is assumed that not everything is fully specified or researched upfront. As development proceeds, new pieces are discovered and introduced into the plan.

3. Bugs

Defects can be considered a side effect of software development. They can affect the product in different ways:

a) Regression
These are defects introduced when working on new code. They usually impact unrelated functionality that used to work before.

b) Defect in new feature
These are problems in a newly coded feature. The feature does not work quite as expected.

c) Existing bug
These are bugs present in the previous version of the software. They are either known or newly discovered during the course of the release and prioritized to be fixed now because they affect existing users.

Learning from the past to project in the future

A believable long term plan needs to layout new features on a timeframe against a set of resources. Because there is only a limited amount of time and people allocated to thinking through the issues upfront, the initial estimates are always very rough. It is then very difficult to predict how much time certain things will take or how many people will be required. To increase accuracy, you’d have to invest more time and resources, which is not practical, mostly because these are the very same resources that are critical to deliver against the current release plan.

You need a model to help forecast based on imperfect data. You could decide to simply pad all the current estimates, but you’d still need to figure out a factor that’s sufficient.

A better model is to find a way to anticipate the additional work that will be generated by the introduction of a brand new feature. With that model, we can layout a plan that should allow sufficient room in every release to accommodate for all the work, planned, unplanned and bugs.

Let’s take a look at historical data from previous Songbird releases:

The first row in the table contains the budgeted points for each release. That number represents the sum of all planned work as determined at the end of the planning phase for that release. The release schedule was set based on that number and the historical team velocity.

The second row represents the total actual points completed during the release, including features and bugs. This represents that amount of effort to get the release out.

The third row represents the actual completed feature points only. Next row computes ratio of planned vs. actual work.

It’s very tempting to compute a velocity based on total completed points and use it as a predictor for how much work the team can accomplish. While it’s true that it is a measure of total effort, it can’t be used against planned work only.

What we want to determine is that given 1 point of planned feature work, how much unplanned work and bugs will be generated and thus how long will it take to complete. Then normalize this for team size and you get a factor that can be used for long term planning.

To get there, I’ve computed a velocity for each release based on planned work divided by actual release duration (in days) and number of engineers. I then use the median value across release as a factor to turn estimated planned points into calendar days. With that in hand, I created a simple model where scope, duration and resources are linked. Set two values and the third one gets computed automatically. This model becomes very handy to run through what-if scenario during roadmap sessions. It provides answers to business questions such as: “How long would it take to do feature X?”, “Can you meet deadline Y?”, “How many more engineers would it take to do Z in Y time frame?”.

Hope you enjoyed learning more about our approach to the difficult task of long term planning. I’d be interested to hear about how you’ve approached the problem, so feel free to share your story.

Lastly, let’s remember the words of the famous Danish physicist Niels Bohr:

“Prediction is very difficult, especially if it’s about the future.”

Songbird Release Trains

We concluded that a dashboard that was more dynamic and worked in real time with Bugzilla would provide a more accurate picture of development progress. This is an overview of the couple of extensions we added to the tool.

Dashboard

The dashboard focuses on the current iteration (all P1 stories/tasks/bugs for a release target milestone). It provides a transparent and dynamic view of what’s under active development for the week.

Items are grouped by developer so we can easily review what each person is working on in this iteration. There are few additions that make it more useful than a plain Bugzilla query. When an item is carried over from a previous iteration, it gets highlighted, providing visibility on something that may require more work than anticipated or might be blocked by some dependencies. That kind of view was lacking before and it made it difficult to figure out what was happening when tasks got carried over. Similarly, when a new task gets added to the plan mid-iteration, it’s identified as such, making the tracking of intake very easy.

Because of our code review system, often time tasks are completed from a programmer standpoint but can’t be checked in until a positive review (R+) is performed by someone else. Additionally, some code requires the re-compilation of large amounts of 3rd party libraries before the patch is available. This kind of information is now bubbled up on the tracking page so the exact status of a particular issue is visible at a quick glance.

Iteration Dashboard

This dashboard is helping us to know what everyone is doing at any given time. It fosters a sense of accountability and ownership of the issues and provides a sense of accomplishment when the work is completed.

Below is a view of the legend for the Dashboard. Most attributes for an item are extracted from Bugzilla and represented via an icon.

Dashboard Legend

Punchlist

Once a release reaches QA, a release branch is cut and the focus of the team switches from working on planned features to fixing bugs. We found that this period is best managed by using a punch list which contains all the issues remaining to be resolved before the release can ship as opposed to a weekly plan of what gets accomplished for the period. It keeps the team focused on the entirety of what’s on our collective plate. During that period, issues are being promoted to Blocker status to represent what is blocking the next milestone (e.g. Beta, RC, Final). It also signals what can be landed on the branch automatically and what needs further clearance. This allows us to throttle the rate of change being introduced into the source tree so that QA can be ensured stable builds.

Release Punchlist

When in punchlist mode, it’s critical that all issues are being assigned. The list reflects that by highlighting items that have no owner yet.

Pretty Graphs

Two frequently tracked Agile metrics are Burndown and Velocity. Those are now being graphed automatically for each release. The Burndown helps forecast when the work will be completed. The Velocity gives us a historical perspective on what the team was able to accomplish and thus helps us plan better.

The Burndown graph breaks down tasks (what we refer to as “planned work”) from bugs (unplanned consequences of software development). When the Burndown line cross the horizontal axis, we’re done. Notice in the graph below the bug points going up during the QA period.

Burndown Graph

The Velocity graph plots the team velocity, normalized per day. Actual Velocity is computed at the end of each iteration and plotted along side the Planned Velocity for easy comparison.

Velocity Graph

These new tools provide a dynamic view of the state of the release. They bring transparency into the development process for everyone involved.

This is a repost of a series of article I originally published on Songbird’s blog

In the previous two installments of this series on Agile development at Songbird, I’ve covered our move from waterfall to Agile and provided an in-depth look at some actual release cycles. In this last post, I’m going to introduce a tool – which I gave the uninspiring name sdpbot – built internally to help facilitate the tracking of our releases.

Wrestling Bugzilla into shape

Because so much of our existing workflow occurred in Bugzilla, we’ve decided to use it as a central database to drive our process. Every actionable project artifact lives in Bugzilla, a Feature, Story, Task or Bug. From a release standpoint, the only actionable items are Story, Tasks or Bug and hence, we only track these. Here is how we’ve organized our bugzilla.songbirdnest.com to help us track each release.

Release train

Every release train has a name and is used to create a target milestone in Bugzilla. This allows us to put items in release buckets. See some examples of what was included in the Fugazi and Genesis releases.

Priority field

As we create stories, tasks and bugs we assign them to the release milestone and place them in a priority bucket. Each priority level indicates how the item is being treated during the release:

• P2 – must have for the release, next candidates for promotion to P1
• P3 – nice to have for the release, we may ship without it
• P4 - nice to have and low risk. This represent the kind of work that we probably will undertake during the QA phase as it is low risk and won’t likely cause a regression.
• P1 has a special meaning. It is used as a flag to represent what is being committed by the team for the current weekly iteration. Everything in P1 signals our tool that this is our “plan” for the week and is under active development (We are considering changing this in the future to use the assigned bug functionality since Bugzilla 3.2 will now allow you to accept and assign a bug in one operation).

Costing

Weekly iterations

Every Monday, we plan our work for the week. Every member of the team elects items to P1 status. We then tally up the total points and determine if the plan is sound. Once we are all in agreement, the plan gets locked. This simply means that the tool takes a snapshot of all the items marked as P1 and saves their ids.

Daily reports

The main function of the bot is to compute delta and generate a daily report. Let’s take a look at every sections of the report.

Iteration

A simple number to represent the iteration (week)

Plan

This section represent the plan of records as snapshot at the beginning of each iteration. Once the plan is locked, it becomes immutable.

Duration

Duration of the iteration. Usually 5 working days, but it automatically adjusts for holidays.

Remaining

Total points for P1, P2 and P3 items which status is open (e.g. UNCONFIRMED, NEW, ASSIGNED or REOPENED)

Planned

Total points for P1 items that are opened.

Planned Velocity

Normalized points per work day.

Actual

This section represent the actual metrics computed at the end of each iteration. The iteration ends on Sunday at midnight.

Intake

Represent everything that was either added or removed to the P1 list (the plan) during the iteration.

Intake Velocity

Normalized intake points per work day.

Completed

Total point of everything that was marked fixed during that iteration.

Team Velocity

Normalized completed points per work day.

Carried over

Everything planned at the beginning or added during the week and not completed by the end of the iteration.

Environment

Originally, this was introduce to track distractions outside of the core release that could impact the team velocity. The idea being that it would give us a gauge of what the “environment” was like during that period. This columns would capture everything open or closed in the special ASAP Bugzilla milestone. We used this classification to track urgent issues, infrastructure work, etc. As our process matured, the need for this diminished and we no longer use this.

Here is a sample of such report for Fugazi

What to look for

Every day an email report is being generated allowing us to track progress to date. For every column described above, a link is provided. The link points back to Bugzilla and contains a list of ids for the items included in the list. Every item can then be reviwed independently.

At our weekly planning meeting, we usually start by reviewing what was carried over from last week. Ideally, we want this to be a null set, but the reality is that it often contains a fair amount of items. It’s primarily due to intake during the iteration. Items are being added because of regression introduced and discovered, new tasks being identified or slight adjustment to the plan. This is definitely an area we need to improve on.

The next view is the planned items for the new iteration. This allows us to make some tweak to our plan before we commit.

The intake views are very useful to analyze what is being added or removed during the cycle.

The team velocity gives us a measure on how well we are doing and how much load we think we can carry during the next iteration.

Finally, the remaining points tally gives us our burn down number. This is how much work is left and by dividing this with our historical velocity gives us a sense of how long it will be until we are done.

Open source library

SDPBOT is built as a Ruby library. The source code has been open sourced under the MIT license. You can download it here: http://code.google.com/p/sdpbot/

We have made no attempt to make it generic, so do expect to do some work to customize it for your own needs. If you’re using Bugzilla and want to generate some type of reports you will hopefully find the library to scrape and instantiate objects from Bugzilla very valuable. Enjoy!

]]> http://auberger.com/archives/2008/12/songbird-path-to-agility-part-iii/feed 0 http://auberger.com/archives/2008/12/songbird-path-to-agility-part-iii http://feedproxy.google.com/~r/auberger_com/~3/Zkal1rtLbc4/riding-up-and-down-haleakala http://auberger.com/archives/2008/10/riding-up-and-down-haleakala#comments Thu, 23 Oct 2008 04:09:42 +0000 Georges http://auberger.com/?p=360 Why do people climb mountains? Because it’s there. When it comes to mountain, Haleakala is not your average hill. It’s a dormant volcano and you can ride it from sea level to summit, 10,023 feet (3′055 m) of elevation gain in one uninterrupted climb. Some claim it’s the highest paved road on earth. Here’s my account of attempting such climb.

I rented a bike from South Maui Bike. They now feature Trek 5000, all carbon for $250/week. You can call in advance to reserve it and make sure you get the proper frame size. The bike was in decent condition, but don’t expect too much. I started from Paia as planned, and got rolling shortly after 6:30 am. Plenty of daylight at that time of the year. Baldwin Beach Park is the perfect location to take a picture and a toe dip in the water before starting the ascent.

Baldwin Beach Park, Paia HI

Here is what I carried with me:

• Arm warmers
• Knee warmers
• Rain jacket
• Long finger gloves
• 2 extra portions of cytomax in ziplock bags
• 4 energy bars
• 4 dates
• Some chocolate covered coffee beans
• Digital camera
• Cell phone (reception can be a bit spotty up top)
• Cash to pay for the park entrance ($5) and for emergency ($60)
• iPod shuffle
• Additional sunscreen
• Trek seat 75-cubic-inch seat pack
• 2 spare tubes
• Tire levers, compact multi-tool, patches

I ended up eating everything and wore all the clothes I brought. The weather was pretty good, slightly overcast. The arm warmer came handy once I reached 8,000 feet. I wore everything for the descent and I was glad that I did. Unless you like more abuse than necessary, don’t attempt a descent without long finger gloves and a wind jacket at a minimum.

You should check out the weather forecast and pick the best day to ride. You will notice a big difference between weather at the summit and forecast at the base.

• Haleakala Summit weather forecast
• Windward Haleakala weather forecast

The route is pretty straight forward. The only tricky part is the right turn off Baldwin or else you’ll be climbing an extra 1,700 feet for nothing like Mike did. When you see the horse arena on the left, it’s your cue to make a right.

Turn right after the horse arena

The first 3,500 feet of the ride are pretty easy. I stopped at Sunrise market and indulged on Maui potato chips and fresh cut pineapple chunks. Apparently, you get good karma points for going up judging by my short exchange with the cashier:

Cashier: Are you going up hon?
Me: Yes.
Cashier: You’re a good man.

After a short rest, downhill riders in full gears including motorcycle helmets and rain coveralls started to appear. Expect to see a lot of them. As of end of 2007, the tour operators are no longer allowed in the park proper because of too many accidents resulting in death. As a result, their tour start right below the entrance of the park at around the 6,500 feet mark.

Make sure to refill on water at the Sunrise market. Your next opportunity will be at the visitor center in the park at 7,000 feet. The next leg of the climb is pretty cool. Lots of switch back with pretty vistas.

One of many switchbacks

Along the way, you won’t see many elevation signs but if you pay attention, there are marking on the asphalt. A blue number below a yellow sun. I’m assuming that they are for the Cycle to the sun race.

Elevation markers

The marking stops as you enter the park.

Park entrance

I payed the $5 entrance fee and off I went. Next stop is the visitor center several hundred feet up. There is a water fountain by the restroom and plenty of space to sit, relax and recharge.

As I continued the ascent, I started to feel the effect of the elevation. I don’t know if it was the lack of oxygen in my brain but the sight of the 9,000 feet sign felt exhilarating and I could not suppress reading it out loud with the mandatory expletive: nine f#&*ing thousand feet!

The last 1,000 feet go by pretty quickly. Once you see the summit and Science City, you know you’re home.

Summit and Science city

The steepest portion of the ride is on the road joining the lower parking lot to the upper obeservation deck. Or maybe it just felt like it.

Mandatory summit picture

Although the ride is pretty long, once I reached the 10,023 feet mark I was surprise that it was over already. I took the time to look around and take pictures of the crater. I had fun watching people’s reaction of finding a lonely rider that far up. A nice couple took my picture and confessed they thought about taking a picture of themselves in front of the bike so they could pretend they rode up.

The ascent took a little over 6 hours with plenty of time to stop, enjoy the vista and take pictures. Although the climb is never very steep, I found the ride to be somewhat difficult because of its length and elevation. I also think that I screwed up the adjustment of the seat and had it too high. As a result, I developed a pain on my left glute that made it pretty uncomfortable to stay seated. I also did not train much prior to it, relying on miles accumulated previously in the season. Ok, that should be enough excuses to ignore the crushing realization that the best rider in the Cycle to the sun race made it in 2 hours and 51 minute, wow.

After dressing up with all the clothes I brought, I started the descent. I was surprise how easy and pleasant the downhill turned out to be. As I lost elevation, the weather warmed up gradually and I passed a couple riders on their way up.

I made it back to Paia in a couple of hours, in time for a celebratory Gianduia Gelato at Ono Gelato.

Overall, the ride went very well. The only problem was a total failure of my Garmin Edge 305 computer. After I turn it off at Sunrise Market to preserve batteries, it would not re-acquire satellite signals. Because the bike was a rental, there were no sensor mounted so without GPS, I lost elevation and distance tracking. So, no route and no stats to share for posterity. I guess I’ll have to do it again.

Complete picture gallery

Route

• 0: Start on Baldwin Ave (downtown Paia).
• 6.5: Pass Makawao Park and town. Baldwin change into Olinda Rd.
• 8: RIGHT to Hanamu Road (as you’re passing the horse stables).
• 9: LEFT at intersection.
• 9.1: LEFT on Haleakala Hwy.
• 13.5: LEFT on Haleakala Hwy.
• 14: Last chance for food STOP at Sunrise Market (3,500 feet)
• 23: Park entrance (7,000 feet)
• 34: Summit (10,000 feet)

Google map or Cycle to the sun map