Through the four stories I shared in my QCon presentation yesterday, the broad accomplishment of Shopzilla’s greenfield site redesign, was incredible business agility. It’s true that we’re very fast from board room to back office.

I’ll extrapolate a little here on how our continued investment in our engineering and deployment infrastructure has been the biggest enabler for our agility.

Archetypes

Perhaps our biggest enabler has been our Maven archetype. In fact, we no longer have only one archetype, but a myriad archetypes for different purposes; web-services, spring-batch, Java-based command-line jobs.

Our archetypes allow our engineering teams to focus more immediately on solving business problems. Engineers of all levels of seniority, focus less on boilerplate, project structure, and deployment concerns. Our Maven archetype encapsulates;

• JMX MBean exposure using Spring
• Metrics exposure, through JMX using JAMon
• Sample Spring configuration and wiring
• Patterns forcing separation of configuration from application binaries

Business Agility, Cross Pollination, DevOps

As a result project structures look mostly the same from team to team, making cross pollination easy. Importantly too, our applications are homogeneous from a deployment / DevOps perspective. Distilled, performance of applications deployed to our any environment, can be visualized on our internal dashboards (permanently) in a matter of minutes.

I’ve written about this topic previously on the Shopzilla Tech Blog. The hindsight here is particularly interesting, in seeing how much the archetype approach has allowed agility from a business perspective. The proof: beso.com and more recently, tada.com.

War Stories from Shopzilla’s Site Redesign

When I say ‘team’, I’m not referring to engineering and QA alone. I’m referring to the entire team consisting of scrum master, business, engineering and QA.

Shifting Priorities

Our business team has a worthy challenge. They’re armed with an ultra responsive engineering team and a raft of features which are aimed squarely at improving our site SEO and traffic monetization.

We have two week iterations, and oftentimes by the time we’re attacking the final couple of stories, the business priorities have shifted.

Features for the Short Term

With two releases going out per iteration, there’s a tendency to want to ‘squeeze’ stories into the coming release. Stories arriving late into the release branch are often rushed, trend high with defects, and create churn and uncertainty as we’re trying to ship our product. With regularity, it would seem, defects find their way to production.

It’s true that as the team addresses production defects and releases hot-fixes, our velocity for the iteration is impacted. Of course the overarching goal is to be responsive to change while minimizing atrophy through defect churn.

Dev Complete

In our retrospective today, the team engaged in a discussion about ‘dev complete’. It was interesting to observe the discussion, particularly as focus shifted to reviewing test cases, and considering the announcement of ‘dev complete’ to mean production ready.

Mature Code-Base

It’s true our code-base is mature. Over time, it can be difficult to maintain a focus on craftsmanship and constant improvement through refactoring. There were great retrospective items today, where we identified opportunities for refactoring in the context of user stories which had come to pass.

Our Defect Trend

Where To?

• I’m not sure how it only struck me now: should we consider one week iterations? What with those shifting priorities?
• Monitoring our defect trend will continue to be critical. We’ve only just started watching this metric in earnest, but surely it can only aid our focus on quality.
• Can added focus on the pivotal, story by story announcement of ‘dev complete!’ help? The team is sure hoping so.
• We’re planning to identify brittle parts of our code-base, and prioritize clean up accordingly. We in fact have one story in this vain in the current iteration. Will this help for the long term?
• Can we shift our expectations from a ‘feature squeeze’ mentality to a releasable feature set (planned in advance) mentality?

What’s the optimal approach for performance testing an application with;

• 200 million domain objects
• 90 JVMs, constituting an Oracle Coherence data-grid, and 12 load-balanced Tomcats
• RESTful web-services, using FastInfoset, JAXB and streaming over HTTP
• A core requirement to deliver 100′s of gigabytes of data within a short time frame

In Rudi’s article, “The Web-Service May Not be the Bottleneck!“, he discusses the key challenges inherent with performance testing at such scale, and how scaling at the client layer helped achieve higher network utilization and throughput.

One core goal of the Inventory LIVE! project was to deliver a ‘snapshot’ of our 200 million domain objects to our clients in a short time period. Once the functional testing effort was out of the way, we focused squarely on the complexity of performance testing. I was personally thrilled and excited about our new and unique approach to shipping such an inordinate amount of data to our clients!

With our Coherence grid fully loaded at 200 million ‘mock’ domain objects (we use a mock data loader), we launched multiple clients against the web-service to test the performance and timeliness of our HTTP streaming approach. An hour into the test, I was puzzled and had A LOT of questions for the team. Why had we only streamed 2MM offers so far? Why does it seem so much slower at scale?

Read on!

Twitter post from @uclajug inquiring about the yield of the Fire Chief role

It’s been some months since the inception of the Fire Chief role, and I’d like to share further information about how the Fire Chief has helped the Inventory Engineering team realize their maximum potential. The Fire Chief role has proven that where legacy systems cause distractions, removing a single person’s capacity from the team’s total can actual increase the team’s velocity: team members are more focused, and have fewer distractions.

Before the Fire Chief

As I joined the Inventory Team, I was amazed to learn that the daily stand-up was more than 50% focused on discussion about fires. Any net-new user stories included in our iterations saw the attention equivalent to a second class citizen. Fires: important. New, strategic, foundational engineering projects: unimportant.

The Inventory Engineering Team's languishing velocity prior to the Fire Chief

After the Fire Chief

I still remember the day when the Fire Chief hat arrived in a wrinkled brown envelope at the office. I must admit to being pretty excited as I sat down with the team to unveil our brand new, bright red, Fire Chief hat. For a brief second there, the perspective that legacy systems and fires make for less glamorous work melted away.

If I had to point to the coolest part of the Fire Chief role? Overhearing people around the office, “Have you seen the Fire Chief?”. It’s so true that our Fire Chief is incredibly popular. What else is cool? Our burndown…

The Inventory Engineering Team's improved velocity with the Fire Chief

It’s also worth noting, that in a team where legacy systems meant distracted engineers, and bred more legacy systems, the Fire Chief role has bred focus, a passionate (popular?) team, and excellence. I’m pretty confident when I describe the Inventory Engineering team as “high performing”. Our stand-up meetings (which the Fire Chief still attends) are focused around new projects, with approximately 1 minute of the 15 minute stand-up dedicated to the Fire Chief’s daily report.

Proof in the Pudding

Our strategic, foundational engineering project, “Inventory LIVE!” is currently being launched by the team. Inventory LIVE! is focused squarely the incredibly challenging problem of scale; 200 million domain objects in an Oracle Coherence data-grid, 90 JVMs, 12 Tomcats, and RESTful web-services, using FastInfoset and JAXB, streaming over HTTP.

I should mention that Inventory LIVE! is a net-new engineering project, owned solely by the Inventory Engineering team. I’m confident that such an undertaking would not have succeeded without our Fire Chief concept.

• Improve our legacy systems and services
• Maintain our production uptime
• Build a high performing team, focused solely on net-new systems and services
• Create a fun, light-hearted but professional atmosphere for the above
• Promote passion for our team and what we do within the organization

My teams deliver high-performing solutions to serious business problems, but they have a ton of fun too! From the article;

On a team with myriad legacy systems, production problems will often be a significant burden for the team. In my experience, without a strategy for managing the team’s approach to tackling these production ‘fires’, the team’s yield for new value creation will be far below it’s potential.

…

Our approach has been to define a new role – the Engineering Fire Chief. Simply put, our Fire Chief is an engineer (or two) who actively accepts the role of providing distraction-free “cover” for their team. (yes, we actually bought a Fire Chief hat.)

(Keep reading!)

The Fire Chief concept, and hat are popular.

A fire chief wearing multiple hats, and proud to fight fires!

It’s been a thoroughly eye-opening experience to frame our roles at Shopzilla. Petter and I were charged with defining the Lead Engineer role at Shopzilla.

I am a Lead Engineer:

• I hold “excellence” as my yardstick.
• I am a software craftsman, I am proud of our code, and I promote this craftsmanship and pride amongst other engineers.
• I lead by example, and I love getting my hands dirty.
• I love to learn and teach through conversations about code.
• I favor reuse over reinventing the wheel.
• I am passionate about performance and know there is no such thing as “perf spray”.
• I engage other engineers, and in seeking them out I foster communication.
• I heart QA.
• I ensure my decisions are based on not just my own competence, but that of the whole team.
• I am a mentor, and I understand the difference between posing questions and mandating solutions.
• I am approachable.
• I encourage my teams to develop their own technical voice.
• I ensure my engineering team is not an island – our discoveries are shared; our struggles are too.
• I help strike the balance between quality and getting functionality out the door.
• I help business owners translate ideas into technical solutions.
• I find the bumps in the road before we hit them.

I am a lead engineer and my success is the sum of the success of the people around me.