mtomlins

It's time for PerfBytes!

noreply@blogger.com (Mark Tomlinson) — Wed, 21 Nov 2012 08:07:12 PST

What is PerfBytes? It's a live internet radio show about performance started by myself and James Pulley over a series of early-morning breakfast meetings at the IHOP in Beaverton, Oregon. While consuming copious amounts of pancakes, omeletes (made-to-order) we engaged in a serious debate about: what's the difference between load test and stress test? It took about an hour and a half to agree that we hadn't come to an agreement on how to answer that question, so we scheduled another breakfast meeting for the next cold and rainy Tuesday morning. And it wasn't just James and I at the table. We had not just a few of the top minds in performance and load testing, including Rex Black, Steve Oulighan, Howard Chorney and Carlos Chidiac (just to name a few). This group of performance gurus kept meeting for breakfast (although Carlos was usually late every time!) hashing over the contemporary practices, learnings, techniques, pitfalls and tools used by load testers. It was actually the most engaging part of the rare and precious time we spent together in the Spring of 2012.

But, let me tell you this show "PerfBytes" and you might still be wondering wondering how two highly-entrenched performance geeks like us could end up having an internet radio show. The truth is, we just created one. No hassle - thanks to the folks at BlogTalkRadio.com - all we had to do was signup and voila! - PerfBytes was born.

The name: PerfBytes. What's in a name? Not unobviously the word "perf" stands for performance - including testing, engineering, tuning, monitoring and all that good stuff we do professionally. The "bytes" part was more important because we wanted to share information about performance in digestible units of information, akin to a short-stack of blueberry pancakes. The format was also inspired from another hit TV show "Good Eats" by Alton Brown. We thought - why not have a show like "Good Eats" only on the subject of performance testing and load testing? I mean, really - who wouldn't love that show?!?!

PerfBytes bi-weekly listeners and callers will learn:

to increase their awareness of performance subjects
about the Science! behind performance
to implement performance practices in tools
the real stories from industry practitioners, vendors and consultants
to translate practices into value for their organizations

You can find out more by visiting the PerfBytes website at www.perfbytes.com!

Or how about following us on Twitter @PerfBytes?

And of course every 2 weeks, call us during the live show http://www.blogtalkradio.com/perfbytes!

What is Realistic Performance Testing?

noreply@blogger.com (Mark Tomlinson) — Mon, 10 Sep 2012 07:50:29 PDT

Today I received a comment from Dzmitry on an article we posted over at www.softwaretestpro.com, asking the following question:

Dzmitry Kashlach
9/10/12 5:31:25 AM

Thank you, Mark, for the article. I agree, that in many cases performance testing engineer cannot choose right tool because of limits in budget, company's policy and so on and so forth. BTW, do you agree, that testing in cloud is more realistic and gives us more opportunities than testing in isolated test-lab, as it is described in the following article?(http://blazemeter.com/blog/top-ten-reasons-run-load-and-performance-testing-cloud)

Essentially, I agree - testing in the real world (e.g. production, or the cloud, or the internet, or in real end-user situations) is more "realistic" than *not* testing in those contexts. But let's be very clear about the use of the word "realistic" in regards to performance testing.

Let's start by defining an unrealistic test. An unrealistic test could refer to any test case or condition that absolutely would not happen in the real world. Example: testing 5,000 concurrent users hitting a system that will be installed for a small workgroup team of only 10 people maximum. The truth is - this is still a valid test if it is based on a story hypothesis like:

“In order to determine the maximum capacity of the system beyond a team of 10 end users, as the Product Owner and System Operator, the system should not handle more than 10 users concurrently."

You might assume that running a test at 5,000 would obviously fail, right? But what if it doesn't fail? You see, unrealistic tests can lead you to new understandings and capacity for the performance of the system – even if that test was not conducted in the “real world” configuration.

By contrast, a realistic test could refer to any test case or condition that plausibly could happen in the real world. Example: during the peak rush time all 10 expected users on the system submit 5,000 records per hour. The performance story hypothesis being:

“In order to successfully process all transactions during peak times, as the Customer and End User the system should handle more than 5,000 records per hour without failing or crashing."

This is a realistic test and it can be setup and tested in the test lab sufficiently. In this test even if you run the user load from outside the firewall against production it doesn't make the test condition any more realistic than before. It is valid to test performance in the lab if you are very smart about the physical configuration of the transactional paths employed in your test. You can learn about performance and scalability, you can find bottlenecks and defects. It is valid, realistic testing.

But often times we aren't rich enough to setup and configure the test lab to be just like the real world. So, it is also valid to test performance in "the real world" be it in production, or the cloud, or at the install site where the entire system is configured exactly as it will be for the customer and end-user. Be aware that the risks of something going wrong or bad associated with testing in production are higher because you might not take the time to analyze and think through all of the implications of the transactions you are throwing at the production system. Some of those risks include managing privacy and security in the test data, preventing external downstream processing, backing-out data and changes from test traffic, setting-up a kill switch, blocking real production use during the test time - these are just a few.

By contrast, when we do take the time to set up an internal test lab we usually perceive that as an unusual and extraordinary (and more costly) initiative. It demands more attention to detail and configuration. Because it boosts our attention and consciousness to those details - as humans, we are more likely to be cautious and explicit in our efforts. That helps to reduce risk.

Quality Investors: NASA vs Wall Street

noreply@blogger.com (Mark Tomlinson) — Fri, 24 Aug 2012 07:51:40 PDT

You know something’s wrong when a Mars spacecraft lands without any software glitches or crashes, while at the same time here on Earth a company’s financial trading system fails disastrously costing nearly half-a-billion dollars. I expected the opposite, didn’t you? These two separate events demonstrate the brutally obvious outcomes of an organization’s level of commitment to quality. And the differences couldn’t be more stark.

The Mars landing (a.k.a. the 7 minutes of terror) required about 500,000 lines of code to execute with perfect timing, slowing Curiosity’s descent from 13,000 miles per hour to less than three miles per hour in a hovering maneuver to touch down on the surface of Mars. This system’s execution flawlessly triggered 76 explosive charges without damaging the sensitive electronics of Curiosity’s scientific lab equipment.

Alternatively, the high-frequency low-latency trading software used by Knight Capital Group might involve exponentially more code when you consider all of the end-to-end components that comprise the system’s architecture. Knight’s software is responsible for routing trades across the various exchanges with extremely quick response times. There are typically no explosive charges in Knight’s trading system; that is if you leave out the $440 million charge to rescue the company from a financial crash.

With similar levels of system complexity both of these organizations had a common goal: avoiding a crash. But the difference between them is their attitude about quality. Some think that quality is just a characteristic that must be technically validated. But there aren’t just knobs to twist, buttons to press and formulas to compute for quality. Quality is not just a technical aspect of software. Quality is the result of human attention to detail and critical thinking about the outcomes and impacts of our actions. An attitude that supports high quality comes from people who are engaged in and supported in the pursuit of excellence in their work.

The commitment to quality software starts with the attitudes of the people at the very top, through investment in resources and support for culture that rewards the individual’s behavior to prioritize and ensure quality outcomes. There’s an expectation that if individuals are given time and respect to deliver high quality, they will be committed to doing so. As an example, with regard to the engineers working on the Curiosity program NASA Administrator Charles Bolden was quoted as saying: "This is an amazing achievement, made possible by a team of scientists and engineers from around the world and led by the extraordinary men and women of NASA and our Jet Propulsion Laboratory." Contrast that with this statement from Knight’s CEO Thomas Joyce: "You cannot keep people from doing stupid things…that is what happens when you have a culture of risk." Joyce’s company lost nearly 70% of its value in 2 days.

(PHOTO CREDIT: NASA JPL/CALTECH)

The “culture of risk” Joyce refers to is about taking financial risks for gain, which includes hedging against poor quality financial decisions. This attitude of risk-taking is counter-productive to quality. It can permeate the culture of a company even when it comes to decisions about technology. You can see this in the form of extreme cost-cutting on tools and expertise, a lack of training and professional development and also in the prioritization of project completion over quality. When it comes to technology systems the most effective approach to hedging risk is to test thoroughly everything involved with the system and to not simply rationalize impending failure.

As evidence to their attitude towards quality, the engineers at NASA know how to manage technical risk.

They test.

---
Sources: Bloomberg.com & Bloomberg News, USA Today, NASA JPL & MSL, Eric Pitsenbarger

UPDATE: this article was re-posted in modified form on the STPCON website as “Quality Risk Takers: NASA vs. Wall Street” (which is totally cool!)

Performance Tool Selection and Hyperspace

noreply@blogger.com (Mark Tomlinson) — Mon, 10 Sep 2012 08:44:34 PDT

For most performance testers, our first few years of work in the industry were focused on learning the basics of automated testing tools and how to conduct and develop load tests. But what I used to hear often from testers is that even though they developed these skills, they were not included in the decision about what testing tool they must learn first -- the performance tooling solution was usually pre-determined. This may have been due to a corporate licensing decision or the specific market domination of one particular tool.

Which means that as a tester, someone above you in your organization was traditionally engaged (and probably completely overwhelmed) by the task of selecting the right tools for the job you would be performing. The reason usually given for this is that the majority of performance testing solutions have been almost exclusively commercial products, some of which required formal evaluation and validation before your company would be willing to make an investment in the vendor’s solution.

Ah, how times have changed.

Today we have a much larger array of performance testing solutions with a range of costs -- from completely free open source tools, to cloud-based tools with incremental operational licensing, and sophisticated enterprise-class tools. With more choices (in different pricing classes) the perception is that you have more freedom to choose and now may use many different tools to get the job done right.

There has also been an increased focus in recent years on automation. Although automation is not a new idea for performance testing, it is becoming a dominant topic of conversation, so much so that the expectation now is that performance testers have a comprehensive understanding of all tools on the market, can conduct a full feature set comparison, and really know the benefit of one tool over another.

And today performance testers can be engaged much more directly with vendors than in years past; this can be challenging for an engineer with no practical experience in vendor relationship management. There are often complex policies and processes when it comes to working with your management and procurement departments. Even if you are selecting free or open-source tools, navigating all this can become a full time job!

So nowadays you are expected, as a performance tester, to know everything about all the tools, to make an objective choice for the best one to use, to decipher all of the licensing constructs, to compute and justify the costs/benefit calculations and then manage the implementation for your entire team. With little time or education on the subject, you typically end up just “shooting from the hip” when selecting performance tools. Thus, typically, you often shoot yourself in the foot.

And this is why Scott Barber and I have put together an STP Online Summit virtual conference series, to help you learn how to do all this. The “Survey of Performance Testing Tools” (August 21st-23rd) is a great opportunity to learn how to conduct tool evaluation and vendor management in less than three days. During the summit you will be guided by Scott and I through interviews with actual testing tool vendors, engage them with your questions, and learn how to bring all the information back together at the end to drive your decision. It’s a crash-course on tool surveying and necessary navigation skills that you will not find anywhere else.

In closing, what does this have to do with Hyperspace? Well, there is a great scene in Star Wars Episode IV where Han Solo turns to the inexperienced Luke Skywalker and states:

"Traveling through hyperspace ain't like dusting crops, boy! Without precise calculations we could fly right through a star or bounce too close to a supernova, and that'd end your trip real quick, wouldn't it?”

In an alternate universe (and alternate galaxy…far, far away) where Luke is a fledgling performance tester, Han Solo might have instead given the strict warning:

"Selecting the proper performance testing tool ain't like dusting crops, boy! Without precise calculations from the tools we could fly right past a false positive or come too close to overloading the wrong system component, and that'd end your testing career real quick, wouldn't it?”

Testing as an essential dependency for DevOps

noreply@blogger.com (Mark Tomlinson) — Mon, 09 Jul 2012 07:54:40 PDT

This post is a response to and a comment on my colleague Paul Peissner’s blog: Expanding QA and Tester Role in the DevOps IT Era? One of Paul’s primary points in that blog post is:

“To enable business agility, IT needs to grow QA's value, role and function, this will help with better job estimating but more importantly it will drive the right long-term response by IT when surprises hit.”

In my observation, the mission and purpose for QA professionals is far more important in the DevOps model -- QA professionals strive to proactively improve the probability of delivering a higher quality product, a more innovative product. Also, I want to correctly label DevOps as not a cultural movement; DevOps is simply a change in approach or method to delivering a product.
Part of the impetus to adopt DevOps is the pressure on business leaders to be more profitable and more responsive to market demand, through innovation. Thanks to IDEO and David Kelley, those business leaders have been sold on an idea of innovation as a product, even though I think most of them actually have a preference for profitability over innovation. But we all might agree that improving the quality of our innovative thinking is an integral part of how we build successful products. From my experience as both a tester and a product manager, I know the benefits of improved quality in products and the correlated reduction in risk to failure of the product. And I know the importance and pressure to deliver high quality innovations and differentiation, if you hope avoid failure of the idea.
What does this have to do with DevOps? To become more innovative, the Web Operations sub-team in IT Ops started picking up more and more “innovative” skills and recruiting web developers and super-administrators to merge onto the same team so they could move faster and be more innovative. Just being a system administrator (who keeps the monthly release train wheels turning and the gears oiled) isn’t good enough enough anymore – the business needs “more responsiveness” from you. And just being a developer engineer waiting for requirements and specifications isn’t responsive enough either – the business needs “more innovative problem solving” from you. What happens when you start to ask a bunch of developers and system administrators to stop focusing on the product development machinery and start contributing to innovation, ideas, and responsiveness? The wheels can come off.
If this self-proclaimed DevOps movement is to provide any positive change for our industry and to the businesses we serve, then I think it’s important to recognize three underlying origins for DevOps and reflect on how QA can get involved:

Vendor Agnosticism: as a web site or system administrator being pressured to innovate working with new ideas, tools, and stuff – you might turn first to the vendors you already work with. In some cases, if there is resistance to DevOps “cultural immaturity” and a lack of DIY skill some folks might cling too tightly to their vendors as a safe-haven. As a tester, you might log a bug against an incomplete process or limitation of benefit caused by vendor-bias or preference. If the quality of your product is going to suffer because there was a top-down mandate to use ‘Vendor Product X’ or ‘Vendor Product Y’ then QA should log a bug against that biased decision or perceived mandate.

Organic Collaboration: as a tester who tries to conserve the practices that successfully find bugs and secure the high-quality standard for product development, it’s natural to question anything new and “innovative” (quotes emphasized). The idea of new can also mean higher risk of the unknown impacts on quality. This means testers are perceived as conservatives or “keepers of the old” while this new DevOps-y adoption may be perceived as risky or “leading the wrong way.” No matter the perceptions we have, we need to work together to deliver something. If the quality of the product is going to suffer because certain groups "just don't get along" then QA should log a bug against the people who aren't collaborating. Yes, it's a defect to be willfully uncooperative.

Rationalization of Failure: as a business leader investing in technology as a future for the corporation, failure is definitely not preferred -- and as shareholders, we really don’t like failure either. However, failure and recovery is now becoming an accepted and even preferred mode for operating. But if the quality of the product is at risk because we are over-accepting failure in any part of the process from inception, definition, stories, coding, automation, builds, performance, security, tools - then we should log a bug against the policy (or lack of a policy) around failure.

I agree with Paul that with the adoption of DevOps practices, testers have new kinds of bugs to find, new bugs around policies on failure, organizational cooperation and unethical vendor biases.
The opportunity for testers is to expand our scope to include both products and ideas. We can bring a professional rigor to testing ideas and innovations prior to their instantiation and forecasting into product revenue or application. We can re-emphasize our role as testers of ideas and testers of innovation, testers of risk and value.
Unfortunately, what I often see is that testers are regarded as just technical bug hunters, as servants to the code or system. This treatment stems from a limited consideration for the mind's capability to engage holistically in the creative process for the very innovation we seek. In this new world where QA is a part of DevOps, then, if you limit the role of the tester, you are inherently limiting the benefits, the innovations, and the responsiveness of DevOps.

Goin’ Mobile with an STP Online Summit

noreply@blogger.com (Mark Tomlinson) — Mon, 11 Jun 2012 21:10:28 PDT

Pete Townshend wrote Goin’ Mobile (for the Who’s 'Next' album) back about the time I was born, but I've been hearing it my head lately. One lyric from the song is: “When I'm drivin' free, the world's my home” -- a reference to travelling in an RV (in the UK they might call it a caravan), your own house on wheels. The modern re-interpretation of this gypsy fad from the 1970’s is paralleled in how we now take our electronic “homes” with us on the road. Our computing experiences are increasingly goin’ mobile.

This past week I supported and presented at the Software Test Professional’s STP Online Summit: Insights on Deploying a Mobile Testing Strategy. If you’ve never attended an online STP Summit, you really should. This is different from casual webinars or online lunch-n-learns -- it’s more like a virtual one-on-one training, with dedicated panel of professionals, highly interactive discussions in the “STP Crew” site, and access to evolving content as the summit proceeds.

On Tuesday, Matt Johnston from UTest gave an introduction to SOLOMO (social, local, mobile) and overview of that market landscape, and presented a robust understanding of the pressure that testers are under to adapt their skills. The main thing that I heard was that marketing professionals don’t make good testers, even though they are increasingly becoming the business sponsors for IT projects when it comes to mobile applications. These SOLOMO business sponsors are pushing developers and testers so hard and so fast to deliver apps that testing can easily get overlooked. Matt encouraged testers to be proactive and be protective about testing mobile.

Karen Johnson reviewed in expansive detail how the entire environment for a mobile application is completely different from the normal applications we have tested historically. This is especially true for how we interact with mobile applications, as UX designs have a different context for usage and interaction. She covered some great suggested books like Theresa Neil’s “Mobile Design Pattern Gallery” as a way to upgrade your testing skills to the latest mobile UX techniques.

On Wednesday Dan Bartow from SOASTA offered up something different from his usual performance presentations. This time the focus was on SOASTA's new mobile functional automation solution -- which is very unobtrusive to the mobile application running live on the device. He showed a live demo connecting SOASTA’s CloudTest Lite to the mobile device over wifi and made it look easy! Another key takeaway from Dan’s presentation was that automation tools are all scrambling to support more and more gestures (touching the device screen/interface) in an easy way, which reminded me of old GUI automation (which used to be the race to support both analog recording and context-sensitive, object-based record and replay.) Mobile touch-sensitive interfaces have brought a whole new challenge to automation tools.

I presented with Srikanth Gullapalli from Cognizant on Wednesday as well, and we dove as deeply as we could into performance testing, diagnostics and profiling for mobile applications…in less than an hour. Our presentation focused on the application’s performance optimization for the local device platform, which needs to be conducted together with load testing of the back-end server and network load testing.

Dan Cornell jumped into mobile application security testing, and Dan’s presentation was the most comprehensive presentation I’ve ever encountered on the subject – amazing that he got through all of it in just one hour. One thing I got from the presentation is that with the increased use of mobile applications comes an increase in mobile data. Given that any mobile device is physically easier to steal and compromise (typically unlocked/insecure), the mobile applications we use absolutely should be more secure than desktops. He covered vulnerabilities at the service layer in use by mobile applications requiring a combination of static, dynamic and forensic analysis and testing.

On Thursday there were presentations from Peter Dimitopoulos giving a retrospective on the mobile evolution culminating in the modern mobile revolution from 2008-onward. He pointed out that fragmentation (e.g. unmanaged diversity of devices, os, apps, ux) presents the biggest challenge to testers who seek to make a solid and lasting determination about quality for a mobile application. (I thought: yeah, nothing about mobile testing is solid or lasting). In light of the demand for faster turn around time on testing feedback and defects, testers are under more pressure than ever due to mobile.

Then the biggest tweeter of Summit topics, JeanAnn Harrison, led the Summit deep into mobile GUI functional testing into the hardware; into the firmware, battery, temperature sensors, peripherals. One suggestion for testers needing to dig this deep into a mobile test is to purchase a volt meter, and maybe pickup a temperature sensor. For normal software-only testers this might seem like overkill, but consider an application that requires high power (battery) or CPU to function properly. It would be good to know which top five devices would work best to recommend to customers and which five to avoid completely. To wrap things up, Scott and the crew had an open panel discussion with the top ten tips for mobile testing.

This is my second experience participating and presenting in an STP Online Summit and there are two great things that stick with me at this point. First, it’s completely convenient for busy people who can’t take off from home or work for a week-long conference. And I like virtual conferences because I can really tune-in to the entire set of content in the presentations, crew discussion forums, twitter and other side chats. It’s a multi-media party! Second, it’s a cost-effective means for professional development in an age where most employers have limited funding or permission to send testers to get additional training in the craft.

The Summit consists of 9 sessions that are design to dig deeper into the topic than you could get in a normal conference. If you feel like you’re stagnating or outdated in your profession as a tester, the online summit can give you that leap back into the game and moving ahead.

Or, as Pete Townshend wrote, “Keep me movin', groovin', groovin', yeah, Movin', Yeah”

Testing with Professional Integrity

noreply@blogger.com (Mark Tomlinson) — Tue, 29 May 2012 07:11:55 PDT

Last week I witnessed what can only be described as a completely integrity-challenged moment by a performance consultant I know, who said, in reference to my current customer: “You’ve done a nice job educating them -- or should I say, brainwashing them!”

The consultant was cheering on what they thought was an effective strategy, that of persuading the customer into an overly elaborate view about what they need to succeed in order to compete with lesser experienced consultants. In actuality my work had been focused on helping my customer and their other consultants fully understand the value and importance of doing performance testing and engineering in the best possible way.

Later, that same person went on to claim that “there are a lot of stupid people in the world” and that consultants like us need not commit ourselves to excellence in delivery of our consulting services, that lesser quality would suffice. This person was advocating that it might be just fine to waste the customer’s time and money by delivering less than what was required by or expected by the customer. And sadly, this far from the first time I have witnessed this kind of unethical approach to consulting.

If you haven’t yet read Matt Heusser’s excellent posts on integrity in IT (Part 1 and Part 2), I encourage you to do so. He highlights a specific example from New York City (CityTime/TechnoDyne), an appalling tale of a team who decided to shakedown the city for $700M by taking advantage of the customer’s lack of technological acumen.

This is on an entirely different level from the snakeoil-selling forwarned by technology guru Clifford Stoll -- the CityTime scam went beyond the simple seductions of technology, and right off the deep end into egregious illegality. For a consultant with no ethics, it seems, any rule will get bent just to make few bucks.

So, after considering my interaction with the performance consultant who actively encouraged customer manipulation, and after reading Matt's great posts, and after reflecting on the many ethical lapses I have been witness to while working in this field over the last 20 years, I’d like to put on the table the three key practices in technology consulting which are perfect examples of a lack of professional integrity:

#1 – Unscrupulous persuasion. The consultant engages in dishonest tactics to “win the customer” by leaving out important details about their lack of talent, lack of legal licensing, lack of real experience or lack of professional certification. Conversely, embellishing on any of these subjects dishonestly is equally damaging to customers.

To operate with integrity, most professional testers will recommend that customers obtain second or third opinions, consult with other sources or professionals, and also thoroughly check consultant references.

#2 – Deliberate under-delivery. The consultant delivers less than excellent quality, when they know that they should be producing higher quality deliverables, but they leave out some of the required work in favor of a work extension or to ensure continued dependency on their own work product (e.g. leaving comments out of code, inadequate documentation).

To operate with integrity, a professional tester will provide industry-standard examples of the deliverables to the customer and commit to the same or better level of quality.

#3 – Willful perpetuation of ignorance. The consultant withholds important details about the work or the subject area from the customer so that the consultant may gain favor or financial gain. In the example above, the performance consultant expressed support for the idea of under-educating the customer about all aspects of the subject or situational context.

To operate with integrity, a consultant must provide as much information as possible to assist the customer, as well as admit with honesty the limits of their own scope of knowledge.

Consulting companies often have mechanisms to ensure that they are operating with professional integrity. Customers should always ask to see the employee code of ethics, standards of business conduct, or the legal statement of integrity in their contracts.

As an independent consultant, there are several ways I demonstrate my commitment to professional integrity. My contributions to and membership in industry forums shows that I am aware of and accountable to many other individuals in my own profession. As a member of the Software Test Professionals organization I am committed to learning and keeping up to speed on the state of the art practices for my profession. I consider myself a representative both of this profession and of its professional ethics; I encourage all of us to do the same.

I also just wrote this blog entry -- to put it on the record.

A Theory of Optimistic Performance

noreply@blogger.com (Mark Tomlinson) — Mon, 30 Apr 2012 15:31:11 PDT

About 15 years ago my manager was quoted in an article where he said that software testers are inherently pessimistic people. The idea has resonated with me over the years, and it has shifted in my mind as I have shifted in my career from being a full-time quality analyst to becoming a test consultant specializing in performance testing.

Initially, I had agreed with him, based solely on my experience in enterprise IT departments (and relatively little worldly experience.) Starting out as an internal quality analyst I spent much of my time learning how to break software and processes, learning how to find bugs. That’s what I got paid to do. But I also spend quite a bit of time being confused and frustrated in that role.

When I became a consultant, things flipped around. I spent more of my time focusing on the optimistic side of testing -- the benefits, the value, and the return on investment. Testing had to be a positive investment for my customers. And I realized that becoming a testing evangelist distanced me from those old feelings of frustration or disappointment, because in a consultative role, I had to be inspiring and persuasive.

Then I went back to working as a tester again, at Microsoft, and I spent time every week with real customers engaged in the real world, one full of frustrating bugs, disappointing circumstance, and looming failures. This dose of reality brought me back down to earth. There it was again, that frustration and pessimism. But this time, during those short-cycled and intense performance optimization engagements, I saw customers change from a state of frustration to one of celebration. Reaching new peaks for performance with customers was exciting and positive, and often the result of tireless efforts.

And this new view on the experience led me to create my own theory about testers.

We are not inherently pessimistic people. Instead, we are actually relentlessly hopeful that we won't be disappointed. What I mean is, we aren't frustrated with the world as it is around us, and we also aren't longing for a more traditional approach to something. Testers don't lament about the past. Instead, when we are testing, we are all just temporarily frustrated, because our hopeful vision for an improved future just isn’t quite here. Yet.

This is especially true for performance testers, where we attempt to manifest success for a system that’s limping along slowly and wrapped-up around itself, and we think “I know it’s just a few more tweaks and we’ll uncover the real power of this system!”

So, the next time you get some complaint or push back on a performance defect where the developer is growing tired of your presumed negative or pessimistic attitude, be the optimistic tester you actually are. Remind them of your feelings about the future. Tell them about how you envision this bug -- and so many more -- being fixed. Let them in on the secret that you are not complaining about what they did in their code at some point in the past, but that you are thinking about how to celebrate the success that lies just around the corner.

Bucket full of Bottlenecks

noreply@blogger.com (Mark Tomlinson) — Mon, 23 Apr 2012 11:18:00 PDT

For years I've seen a few super-guru performance engineers impress everyone by walking into the lab and with just 2-3 clicks through the code casually proclaim: “Sure, you just need to tweak the buffer setting in the max_connections_kernel_init.xml file to match the incoming parameters for the extension listing sequencer” with an “everybody-knows-that-one” attitude. I figured my dream of becoming that cool could never happen. Or could it?

At one time or another in my own career as a performance engineer and tester, my self-perception was that I needed to be the guy who knows a huge list of all bottlenecks, and that would be my secret to success. But I've come to see there are 2 major flaws with this objective:

1) there are a whole lotta of bottlenecks out there, and
2) the list of bottlenecks is always changing and evolving

What I’ve learned from spending my working life encountering massive techno-churn, new platforms, and new usage patterns is that knowing where bottlenecks come from is far more important than knowing the bottlenecks after they exist. Understanding the engineering patterns that result in a performance bottleneck means that you can help to prevent performance bottlenecks from getting into the system in the first place.

In one of the performance engineering workshops I teach (“Performance Engineering for Product Owners”), I explain the importance of writing the specific performance requirements into the user stories or specifications. Ideally, this means as the user story is evolving to become dev-ready there’s an opportunity to engineer performance into the thinking of the design, and that will impact how a developer responds to writing code or constructing the system to deliver on that story. And product owners can get ahead of bottlenecks, too.

So, the next time you are tirelessly tracking down the root cause of a performance bottleneck and some super-guru performance engineer comes walking in to pull a "miracle" solution from their bucket full of bottlenecks – remember the relative longevity of that fix, and the limits on such specific solutions.

STPCON 2012 Reflections on Excellence

noreply@blogger.com (Mark Tomlinson) — Mon, 09 Apr 2012 07:11:18 PDT

If you’re like me, when you get a chance to attend a professional conference you look forward to meeting up with other people in “your tribe” (as some say). I love these opportunities to connect with colleagues and peers who I’ve not seen for months or years sometimes. Sometimes it’s people whom I have never had the pleasure of meeting in-person. And always I’m meeting new people – with new ideas and new solutions that open my mind to new things. My experience in the Spring of 2012 at the Software Testing Professionals conference (STPCON) in New Orleans (NOLA) was an excellent example of all these reasons. Here’s a rapid-fire reflection on the folks I met and why it’s so valuable to be at this conference with our tribe!

For starters, I met up with Matt Heusser right away and that’s like diving into the deep end of the STPCON pool – he’s so intense, passionate and excited about testing it’s hard to believe. During our late-night dining at Daisy Dukes, he invented an alcoholic milkshake made of warmed milk and Kahlua. (Apparently not bad!) Oh yeah, we also got caught up on work for future conferences and also giving a referral to a customer near his family in Salem, OR. I got to chat with Jon Bach for 10 minutes before his presentation and introduced myself as a fellow responder to Alberto Savoia’s “Test is Dead” industry-bashing scandal. Jon’s treatment of the subject in his keynote was a very professional response to the naiveté’ of Alberto and James Whittaker.

Then, in my first track presentation on Conscientious Testing I noticed a familiar face, Ben Simo (@QualityFrog) in the back – lurking, but also smiling a nodding quite a bit at the review of how to be an ethically aware software tester. At the same time my excellent friend Scott Moore was also in the session bringing up points of reference from his experience in performance test consulting from the last 20 years. I stayed on for Scott’s session on Peak Performance Planning which was excellent!! Later that day I had my second session on Performance Testing Metrics and Measures – which was exhausting, packed, standing-room only – and I got to sync-up with my mentor and peer Scott Barber. Basically, if you have a job today in performance testing or as a load tester or performance engineer of any kind…you probably owe some thanks to Scott for his undying commitment to our discipline and industry. That evening I got to hang out with the team from HP Software (John, Sylvia, Heather and Kristin) - chatting about the latest & greatest news from our favorite old ex-Mercury product vendor. Scott Moore also taught us a new way to properly taste single-malt scotch.

The next morning I participated in Breakfast Bytes – taking an interactive survey of participants on the topic of Performance Engineering vs. Performance Testing titles and careers (to be summarized in a later blog post, coming soon). I got to meet up with Lanette Creamer, who I recalled from the last STPCON in Fall of 2011 in Dallas, TX. Lanette is a very sharp tester and presenter with a great sense of humor to keep you going! At the reception that evening I got to have a marvelous chat with Fiona Charles about her own writing on ethics in software testing. She’s also worked closely with Jerry Weinberg and sent me a few references and readings to consider.

I finally met Dan Bartow from SOASTA in-the-flesh – and he’s really a great guy; enthusiasm for performance is not his weakness in any way. Brad Johnson and Dan went with a group of us out to the Old Absinthe hotel and like true testing geeks we just hung-out and told stories about testing. Sure we had some absinthe (the less-potent, legal kind) which was tasty actually. As the conference continued, there were presentations from several other excellent testing guru’s out there like Kerry Fields, Griffin Jones and Brian Lynch – each of whom gave marvelous presentations, with depth and experience.

But there was a moment that really struck me at the very last day when I saw Ben Simo again and he responded to me: “I realized – I need to do this.” [referencing the idea of getting out and presenting, speaking and sharing and participating in the community]. I smiled and said, “Absolutely you should!“

What Ben was experiencing is the kind of buzz at the STPCON event, which is highly positive and uplifting, and that is being surrounded and immersed in testing excellence. It’s so different from the conflict-ridden existence we typically face each day in our jobs as testers. It’s different from the politics or the negativity we sometimes are forced to deal with in our work, in business or on our teams. At STPCON all of that noise just fades into the background and you are surrounded by testers! You’ll find (like me) that every conversation is filled with opportunity to learn and to share something excellent about software testing.

See you in Miami – STPCON FALL 2012!!

PS – special thanks to Abbie, Fiona, Rich, Matt and the STP Crew! This couldn’t happen with your excellent support.

Slow-motion Performance Diagnosis

noreply@blogger.com (Mark Tomlinson) — Mon, 13 Feb 2012 13:19:00 PST

As a performance engineer, you know that one goal is to diagnose the client's sensitivity and vulnerability to failure due to high-latency and limited bandwidth. And if you've read Steve Souders books, you know that behind every browser is a whole-lotta loading activity going on, activity which obeys a somewhat bizarre set of logical rules, depending on the browser type and version. As you are investigating a client-side performance issue, all that activity behind the client is rendering so quickly that it's very difficult to see each of the steps in the process. Why not try to slow it down?

Actually, you can. Using a network throttling tool like Shunra's vCat or Charles proxy - you can slow the back-end calls down and put the entire rendering sequence in slow-motion. It's like watching the instant replay from the 2012 Super Bowl of Mario Manningham's spectacular catch in the 4th quarter at the edge of the sideline. In a split second, he took 2 steps just within bounds, before being hit and pushed out. At normal speed you would have missed it.

What's important here is to try and correlate the visual sequence steps on the front end of the client with the activities happening behind the client -- like the number of active connections and the resource types in-flight. This technique can help to reveal the application's sensitivity or vulnerability to latency -- which may affect the functionality in the real world.

Three specific tips I have for you are:
- in a Charles proxy, you can switch to a window that shows Active Connections and the resources that are being downloaded
- with Shunra's vCat, in addition to the throttling behavior, there is more sophisticated reporting and deep dive into the client behavior
- and remember that the purpose of the network throttling is just for diagnosis, which not to be confused with real-world simulation.

Questioning the High-Performance Holiday Report

noreply@blogger.com (Mark Tomlinson) — Tue, 31 Jan 2012 15:17:44 PST

Recently the engineers at Strangeloop Networks repeated their annual performance benchmarking of the top-ranked E-Commerce websites during the 2011 holiday online-shopping season (in November). This is a very popular and influential report in the E-Commerce industry. This is also an interesting report for me to read because I worked with a few of these companies last year to improve their website’s performance just to be prepared for the holiday load. The report is filled with commentary on trends in website development and performance characteristics of over 1000 websites, and while it may not appear to be controversial on first reading, there’s an interesting bullet-point in the Strangeloop conclusions:

“Top-ranked sites are slower, not faster, than the rest of the pack. The primary reason: bigger pages containing more objects (files such as images, CSS, and JavaScript).”

This is a surprise, especially if you assume that the top websites to have the most priority, attention (and investment) to ensure high performance and scalability. But then you see newcomer Nike.com (ranked 78th) entering the report at #1 with page load times under 3 seconds on average. You might suspect, as I did, that this report is just evidence of an underdog effect for these lesser-ranked E-tailers ambitiously striving to compete, using every tactic available. Or it may indeed be proof that applying the web performance optimization (#wpo) rules are a very cost-effective competitive advantage to any company’s online initiatives.

But before you crack out the champagne to toast the winners, let’s take a closer look at what’s not revealed in the report:

The report includes no disclosure of the financial investment in the systems delivering these super-fast performance and response times. We all know that it costs money to throw hardware at a performance problem or bottleneck. Perhaps these companies just spent loads of money to make it into the top-ten. Or did they employ #wpo practices to optimize the sites?
Although the report doesn’t include directly observed qualitative assessment of the websites, it does reference a 2010 study of a 57% abandonment rate for sites with over 3-second page load time. Specifically absent is any reference to the conversion rate measured by these top-ten websites during the timeframe of Strangeloop’s testing in November. Is slower more profitable?
Aside from pointing out that the page sizes are getting bigger and more complex, it would be helpful to understand the efficacy of the end-user experience. Was the extra size worth it? Is bigger better? Did customers notice the upgraded experience?
The “repeat view load time is 20% slower” means that while #wpo techniques work to improve the the initial loading of a web page, there is less attention to the repeated loading of the same page. Does this indicate a misaligned optimization for browser caching? Did any websites require repeated loading as part of their shopping cart processing?
“The average page load time is 10.0 seconds. (Median load time is 8.4 seconds.)” According to the 3-second threshold listed above, our E-commerce industry’s average conversion rate is considerably less than 30% – nowhere near our full potential.

Email the guys at Strangeloop for the detailed test results: info@strangeloopnetworks.com.

Web Performance Escalation (#WPE)

noreply@blogger.com (Mark Tomlinson) — Tue, 31 Jan 2012 05:03:55 PST

In 2007 Steve Souders authored a book called High Performance Websites which ignited an entire movement in web engineering called Web Performance Optimization (#WPO). The WPO practices and techniques are focused on the optimization of client-side performance for modern web applications; establishing a set of rules or guidelines for the design and engineering of a web application.

Now that I’m jumping back into performance consulting, the interesting thing for me is learning as many of these newer techniques and new tools as I can. My recent work at Shunra gave me more respect for the impacts of network limitations on client-side performance, so wouldn't you know it: in less than 1 week into a consulting project, I'm already hunting down the root-cause of a client-side browser performance issue.

On this project the performance testing team reported that the average response times for a critical transaction had grown to just over a 3-second page complete time using HP LoadRunner. The team immediately got into a huddle to consider whether we could give a GO or NO-GO on this release, with significant pressure to be defending the business requirement for conversion (e-commerce) and end-user experience. A NO-GO on this release would be a serious show stopper.

Here's a rundown of the observations during the escalation:

We escalated a NO-GO vote on the release, because the response time was unacceptable on the cart checkout process; it was slow enough for us to raise the issue with the teams.
The testing data was taken from LoadRunner's web virtual user measuring at the transport layer, which wasn't accurate for a rich web presentation layer including asynchronous calls.
The root cause of the performance issue was related to in-line loading of resources (JavaScript and CSS) which we diagnosed and cross-checked in Chrome, FireBug and Charles proxy. The problem was worst on IE7 - which was known to have limitations.
The investigation revealed that we need only to debug the client-side performance for the release, but that any load-related scalability issues would only make things worse. Single-user baselines don't include multi-user latency or bottlenecks.
We learned the real response time measurement was not "page load complete" but instead a point in the script where a load mask (grey transparent cover with animated spinner) was lifted so that an end-user could continue with their shopping purchase and checkout.
The checkout business process was extremely important to the business (no surprise), so if we even have a slight 500ms increase in end-user flow through the cart, the stakeholders would want to know.
Development already knew much of the technical root cause for this issue, but they were relying on the performance team to accurately give an official measurement from testing in the performance environment.
Once we checked the website analytics, we noticed that less than 7% of the end-users were still on IE7 and even fewer of them were on slow routes to the website. It was a very anticlimactic realization.

That's what we explored and discovered and learned through the 5 days of escalation with the teams. There were three different groups (development, test, and business) all digging into this issue, and so a little miscommunication and some chaos was to be expected. But we found there were also some discrepancies in the testing tools and environmental configurations. Here are a few lessons learned and observed:

Don't use a multi-user load testing tool to validate a single-user client side bottleneck. Better yet - combine load and UI performance tools in your approach.
The load testing engineer should have also been using LoadRunner’s Ajax TruClient virtual user to accurately measure the client-side performance in FireFox (which was also the developer's standard).
Always have your testing completed and test results compiled prior to escalating a major NO-GO vote to a release. Do your homework – it will pay off.
Remember to analyze and calculate risk according to the real-world evidence - like the % of users on an old browser or slow link speed.
Performance results form “lower environments” used by development can lead to inconclusive evidence about real-world performance. Be aware of and allow for factoring these inaccuracies.
To improve the breakdown of client response times measurements in LoadRunner and Gomez, we need to measure all the points in the rendering of the client experience.
Get savvy with client-side performance tools like FireBug and Charles Proxy, otherwise your developers will know more about performance than you do.

A Big Yellow Truck

noreply@blogger.com (Mark Tomlinson) — Sun, 22 Jan 2012 17:24:21 PST

In February of 2011 we relocated back to the East Coast; we packed everything we own into a big yellow truck and moved it all the way across the country. We made it a road trip, and wound our way from Mountain View, CA to Philadelphia over the course of about two weeks. In the process I ended up with about 4,000 miles worth of time, which is a lot of time to think and reflect about a great many things...including performance testing.

In my mind I translated our cross-country move into a metaphor for how information systems work. I thought about the supposition that if all of our worldly possessions in the back of the truck were actually just data (instead of our stuff), how could the experience of moving all this stuff be relevant to how we process data in computing systems?

Storage: Let’s be clear, our stuff was just fine sitting in our house in California. It was happy there. It was at a state of rest. Or at least, most of it wasn't moving about very much. Each item had it's own place and we knew where most things were located. Some of our things were in use, some on display, and other, infrequently used items were packed away. And some of our things were set in places where we could manipulate them, use them or move them around quite a lot, like our car keys, our clothes, or our dishes. This also holds with computing systems: data is stored across many mechanisms as a means to enable different types of access, depending on frequency of use and also the speed or convenience of retrieval.

Temporary Storage: After we decided to make a move back east, the first effort started with packing everything up into boxes. Some of our things didn't need to be changed very much to be packed and prepared for the move, but they did need to be pulled from their in-house storage state and put into some type of a temporary storage state. The act of packing was tough work - it took processing power to manipulate our stuff. And when we go to retrieve data from a disk volume or database table, the storage system or database must first put our data into some type of container or dataset object.

Manipulation: As we were packing up our stuff, some of it had to be drastically dismantled to fit into the truck; it had to be changed or manipulated in order to be moved into a temporary state (a box) into another temporary state (the truck). This manipulation of our stuff took a lot of time, just as in computing systems that must transform or change the data format. Even doing a small aggregation or conversion for a lot of data (stuff) takes a long time. Manipulation of our stuff also included the packing, the compression of stuff into smaller spaces, much like how we compress data before it moves between systems. On the other end of the trip in Philadelphia we had to reverse all the manipulations – assembly, decompression, un-packing – which again takes a lot of time and processing power.

Movement: Even though this stage of the move was easier than packing and loading, while we were moving we didn't take anything out of the truck or boxes - it was left as-is for the long journey to Philadelphia. There were limitations on how fast we could drive and how much stuff would fit into the truck. And not only did we have a truck, we had a car trailer on the back, had the car on the trailer, and inside the car we had a disassembled motorcycle! It was quite a rig. So, we had limitations on capacity and speed and there were only certain routes we could take because of the size of our truck/trailer combo. One option we considered was that our stuff would have travelled faster had we divided all of it into smaller, individual containers and shipped them all separately to our destination.

Display: As we put our stuff into boxes and then, once we landed, took it back out of the boxes, we had to examine many of the things to make decisions what to do with them. We had to see things in order to make a decision about them - while they were being manipulated in packing or un-packing, we had to display them to ourselves to decide where they belonged in our new place. You might know this feeling when you look at an unmarked cardboard box that is taped shut, but you can’t remember what’s inside it. The only way to know what “data” is inside is to open the box and display the contents.

Reflection on the long cross-country drive led me to the recognition of the four natural states for stuff: storage, manipulation, movement, and display. And as I examined information moving between these different states, I noted that we have the same base concepts in performance engineering; assessing costs of storage, movement, manipulation, and display of information.

How many times is information manipulated into temporary storage on it's way to/from a display state? And is there a conceptual model (that does not involve trucks, trailers, or boxes) which could help us formulate new language for transactional analysis? Applied to the analysis of end-to-end distributed system performance, data must flow through the system entering-and-exiting the four states. “Awesome!” I thought.

But my next thought was: how much did it cost for our stuff to change state? How much did it cost in dollars, and how much did it cost in terms of energy or effort? (e.g. muscles and ibuprofen...lots of ibuprofen...)

My New Year's Resolutions and Goals for 2012*

noreply@blogger.com (Mark Tomlinson) — Fri, 06 Jan 2012 14:51:00 PST

As I enter the twentieth year of my career in technology, software testing, and performance engineering, it's an exciting time and it's a time of change and evolution. Here are the areas that I’m consciously making an effort to change and improve in 2012.

	Projects, Writing, Work:
	- the blog (this blog)…I will write a minimum of 50 blog posts, that’s one per week, leaving a little bit of room for vacation...
	- guest posting on HP LoadRunner Blog…I will be submitting more how-to videos and product tips-and-tricks on the HP Community website
	- posting on STP…based on my experience in software testing beyond just performance, I will be submitting additional articles to Software Test Professionals online publishing
	- projects in 2012…two initiatives I will be working on are BandChemistry and Socrates, both of which are start-up ideas

	Professional Growth:
	- in January I will be going out on my own as an independent consultant, selling advisory services, training, and performance consulting
	- later in the year, I hope to launch the “Performance Radio” weekly 1-hour online show with my colleague Howard Chorney
	- I will read 10 new books in 2012, and stop re-reading the old books I love about software testing, performance and technology... I’m open to recommendations of course, to read on my Kindle

	Community:
	- Vivit Philly…I owe you! I hope to continue my voluntary leadership of the Philadelphia Vivit Chapter with the support of colleagues and friends who live local here in the Delaware Valley
	- Software Test Professionals…in addition to presenting at the Spring conference in March in New Orleans and submitting more articles I’d like to facilitate a crew around something – perhaps current performance testing tools and techniques
	- SQA Forums…this is my favorite forum for independent experts and practitioners for testing tools and practices and also for sage advice – I’d like to join my peers in voluntary moderating of the forums

	Great People:
	- old friends…working as an independent consultant means I will have a chance to spend time with some old friends (and on some new subjects) like Julio Rojas, Richard Bishop, James Pulley, Scott Moore, Scott Barber, Sharon Bjeletich, Holly Casper and Dan Downing
	- new friends…at the STPCON last October I found a kinship with a new generation of testing professionals, so thanks to Theresa Lanowitz, Matt Heusser, Lanette Creamer, Dawn Haynes, Steve Sturtevant, Shane Evans, Catherine Powell and Pete Walen – I hope to share and collaborate more with you all in 2012

Happy New Year everyone!

* – all plans contingent upon some end of the world event actually not happening in 2012, because, hey, let’s not forget that there might be the rapture or a meteor hit the earth or some other unforeseen end to all of life on our planet...

What’s your Future in Performance?

noreply@blogger.com (Mark Tomlinson) — Tue, 03 Jan 2012 13:40:12 PST

Today is a good day. Is it a good day because it’s a whole new year? Is it because I’ve spent 2 weeks on vacation and feel rested and prepared for new challenges? Nope. Today is a good one because I was published for the first time in Software Test Professionals’ mag/e-zine. No matter how many people try to downgrade online publishing I still think this experience of having a professional editor take on your writing and agree to post your words is fun and cool. But most of all it’s the contribution that makes me happy – I really enjoy being helpful to my peers and colleagues in testing.
Read it when you get a chance: “Mentoring, Self-Study, Fearlessness and Other Practical Tips for Taking the Next Step in Your Performance Career” I hope you find something useful and inspiring for your career in performance.

Also, thanks Rich Hand for the opportunity and to Scott Barber for giving an opportunity to join his STP Summit last year and inspiring me.

Happy New Year!

Test is Dead They Say, Long Live Test

noreply@blogger.com (Mark Tomlinson) — Sat, 31 Dec 2011 14:14:30 PST

I first came to know Alberto Savoia's work back in 2008 when I was at Microsoft and collaborating with Scott Barber on his book “Performance Testing Guidance for Web Applications.” Alberto had just published a paper which demystified the definition of “concurrent users” in load testing terminology, a paper I use as a reference whenever I am talking about concurrency.

More recently, Alberto has gained notoriety with his "test is dead" keynote speech at GTAC 2011, a presentation which led to some incorrect understandings of his remarks. Let's face it, today's business language and Zen presentation style can be filled with lots of loaded language and codified speech that can leave your head spinning.

For Alberto’s sake, let’s try and break it down.

- Alberto gave this presentation as the motivational keynote speaker at a Google software testing automation conference, so we can assume he's talking about software testing, and also that his remarks are intentionally thought provoking and self-promoting. In both his presentation and his blog, this opinion is conveyed:

"Alberto Savoia suggests that software testing is dead."

- Alberto is loosely borrowing from Nietzsche here, so his perspective (as he eventually explains) is not to be taken literally. Instead this is a philosophical statement which he encourages testers to take seriously. That brings our understanding of the statement above to:

"Alberto Savoia suggests (philosophically) that software testing is dead."

- When we jump forward to the first question after his presentation, where Alberto is respectfully asked whether his philosophical statement applies equally to testing "enterprise grade software" (mission-critical) and also "artificial hearts" (life-critical). Alberto's response is that his “New Testmentality” doesn't necessarily apply to traditional testing practices at companies “like Boeing or Pfizer (life-critical) testing...especially like testing an artificial heart.” So now an understanding of the statement might be:

"Alberto Savoia suggests (philosophically) that traditional software testing is dead, except for life-critical or mission-critical software testing practices."

- Now, consider the context for the location and setting of this presentation: a Google conference on software testing where Alberto responds to his first question from his experience working at start-ups. "If you want to work at start-up" he says, "you need to consider this [new] approach." Now he has identified the intended target audience for his statements about testing:

"Alberto Savoia suggests (philosophically) that start-ups should consider that traditional software testing is dead, except for life-critical or mission-critical software testing practices."

- Alberto uses the word "traditional" in his explanations which we should consider code language. If you are an entrepreneur in Silicon Valley, you spend all your time being hell-bent on innovating or inventing "the NEXT BIG THING." During the VC pitches that I've witnessed, many presenters will get dinged on their use of the word "traditional" because it is seen as a weak competitive reference to the "dominantly successful products or practices" in the marketplace. So, that changes the context for Alberto's statement:

"Alberto Savoia suggests (philosophically) that innovative start-ups should consider that the dominantly successful products and practices for software testing are dead, except for life-critical or mission-critical software testing practices."

- Also, let's not forget that Alberto is pitching his own practice or approach, called pretotyping. So the further-demystified statement should now read:

"Alberto Savoia suggests (philosophically) that innovative start-ups should consider that the dominantly successful products and practices for software testing are dead, except for life-critical or mission-critical software testing practices and they should adopt the pretotyping approach as an alternative."

- Let's dig a little deeper into pretotyping, because it's the essence of what Alberto is suggesting here. Pretotyping is the "it" that Alberto is building. The definition for pretotyping from his website is "verb: Testing the initial appeal and actual usage of a potential new product by simulating its core experience with the smallest possible investment of time and money." The definition includes testing for "initial appeal and actual usage," which are the requirements for evaluation of a new product's market viability. There is no mention of testing software code: boundary testing, calculation testing, scalability testing, exploratory testing or even quality requirements. The focus here is on launching a start-up and testing the market viability by "building the right it" before doing rigorous software testing. So let's replace all references to "software testing" in his statement with the appropriate phrase:

"Alberto Savoia suggests (philosophically) that innovative start-ups should consider that the dominantly successful products and practices for testing a new product's market viability are dead and that they should adopt the pretotyping approach as an alternative."

- In the end, if we consider who Alberto Savoia is as an author, a start-up entrepreneur, and an employee at Sun and Google (when they were still considered start-ups), and that he admits to receiving "significant revenue, recognition and numerous awards" throughout his career, we can finally decode the entire statement to it's most native and direct form:

"Through the promotion of his own book, a founder of multiple Silicon Valley start-ups who has received significant revenue, recognition and numerous awards suggests (philosophically) that innovative start-ups should consider that the dominantly successful products and practices for testing a new product's market viability are dead and that they should adopt the pretotyping approach as an alternative."

Now why didn't Alberto just say that?

That is great advice and very encouraging to any software entrepreneur who is trying to decide which people and practices to employ just to get off the ground with their idea. Pretotyping is perfect for the 2-person startup (a technical genius + a business-minded guru) where you are trying to get angel funding to sow seeds of an idea, and it might work for non-critical web applications where failure is well-tolerated.

Unfortunately, this advice isn't very useful or valuable to the majority of IT departments in the real-world outside the start-up bubble of Silicon Valley. The corporations who provide the backbone for the global economy aren't start-ups. They know exactly what they care about and what their customers care about so they have little need for market research when it comes to IT. They test their software and they will continue to test their software to ensure their success. And here is where I can end with my own statement, one that needs no demystification: Long Live Test!

Enterprise Performance: over budget,over extended, under prepared. Sound familiar?

noreply@blogger.com (Mark Tomlinson) — Sat, 24 Dec 2011 07:30:30 PST

Rafal Los and I are always referring to one another as the step-brothers of the non-functional testing world, meaning that Performance and Security most often don't fare well when compared to our more popular sibling, Functional Testing. I challenged Raf once that we could take just about any of his blog entries and replace the word "security" with the word "performance" and it would have almost exactly the same meaning, call-to-action and value as his original blog post. (And if the title of this blog entry rings a bell then you’re probably also following Raf’s blog The Wh1t3 Rabbit.)

Here is one of Raf's recent blogs on Enterprise Security which I have changed just slightly to be read as a blog entry about Enterprise Performance:

__________________________________________________
Earlier today at the Cleveland, Ohio Information Performance Summit my friend and colleague Jim Smith (the CPO for Diebold Corp) presented on a topic that seemed to validate many of the things I've been saying lately ...quite frankly that was a bit of a relief. You see, Jim’s a bit of a superstar in the System Performance world ...one of the youngest executives I know, and razor sharp wit. You see, Jim's presentation today was on how the majority of enterprises are simply not built to thwart performance outages.

While the average Enterprise Performance organization keeps increasing capital and operational spending for performance... all those blinking lights in the closets and vast trove of tools all generate lots of interesting data, but what gets done with that data? Furthermore, the more agents we drop onto desktops, laptops and mobile devices the slower they get and the louder the user backlash. In reality - how much additional benefit does an enterprise get with each added new installable agent deployment? It's tough to tell ...but the value doesn't increase like we'd like.

So now we're faced with a precarious situation. The technology spend performance organizations are making isn't returning the kind of benefit they expect, and the user backlash is growing all while it's getting harder and harder to manage all these consoles, dashboards, boxes and tools ...what to do?!

I think we can all agree the answer isn't more randomly placed technology, for sure. So what then?

I've been talking and starting conversations about Enterprise Performance and exactly what it means to organizations in this position, and concepts like EPI (Enterprise Performance Intelligence) ... all of which contribute to what I think is a higher state of performance awareness, and responsiveness. It's not how well you fortify the (virtual) castle walls anymore, since those walls have all but disappeared ...but rather how prepared you are for when the performance issue shows up inside your keep, and starts tarnishing your precious response times.

Jim and I diverge slightly though, I think that post-outage is one of the worst times to think about how you're going to build up a performance program for your enterprise ...as the decisions that are made at that time tend to be hasty, poor, and often forced. When your organization's house is on fire, the pressure's on to put out the fire immediately, rather than to worry about long-term sustainability and strategic thinking. I think the best time to formulate a strategy is pre-outage when you've got a rational, clear-thinking head on your shoulders. Unfortunately, this is often the time when you probably won't have the funds ...details, details.

So let's get back to your enterprise, and what you can actually do to protect the company customers and partners from inferior performance ...

As with any battle plan, segment your defensive strategy into risk categories. General performance issues (some of us call this background noise) should get one type of defensive strategy, while focused, targeted attacks need to have their own. If you don't have both you're only defending against someone running a background job ...or that "cloud based compliance scanning service" which runs crawls against your IP space during a regularly scheduled maintenance window. These aren't serious threats, much like the 'random batch job' you're stopping with that ancient-old piece of APM software.

Treat your peak-hour performance issues differently by focusing more of your attention there. The mitigation strategy for the peak outage (or spike issue) is quite different than against nominal threats, and involves first off knowing your assets, then being able to clearly understand data movements, users and business processes. This is often quite difficult - but necessary.

Prepare for failure in the face of the inevitable performance issue. Once you've made your peace with the fact that the outage will eventually bring down your critical assets you actually feel better and can think clearly about what happens next. How does your performance incident response organization mobilize, what resources and data do you have available to you on a moments' notice, and how prepared are you to disrupt critical business services to stop the bleeding? These are all things that you not only must plan out, but try at regular intervals like disaster recovery drills. Anyone who's been through this will agree that planning for incident response, and actually surviving through a critical situation are completely different animals.

Collaborate your performance technologies, as much as possible, as often as possible. Taking information from your application logs, server diagnostics, network monitoring, anti-malware, and other systems and plugging them into a central system to perform advanced analytics - I'm talking way, way beyond traditional PIEM here - is what will save you. Think outside the performance and scalability bubble and incorporate things like access control, application logging and behavioral analysis, system and network logging to track how users and systems interact with each other, and when deviations occur which warrant your immediate mobilization.

Remember, bad things will happen. You will experience an outage, catastrophic performance incident, or failure of some type ...unless you're simply too ignorant to know the difference, in which case I can't help you. The incident is important, but more important than the cause of the performance issue is how you, your organization, and your business respond. Be prepared for the outage, test your response plan, and get real with your Enterprise Performance. ___________________________________________________

Out of respect for the original author and composition, here is the link to Raf’s excellent blog post on Enterprise Security: ”Enterprise Security - Over Budget, Over Extended, Under Prepared”.

Predictions for IT Performance in 2012

noreply@blogger.com (Mark Tomlinson) — Tue, 20 Dec 2011 20:40:18 PST

We're down to the wire in 2011, bringing this crazy year to a close in the last few weeks of Q4. All of us are preparing budgets and predicting costs, risks, expenses, headcount, growth, servers, cloud, mobile, networking, telecommuting and even how much we spend on printer toner cartridges. Of course, there’s another way to look at all this frenzied preparation for the upcoming year.

Gary Jackson, the CEO of Shunra joined me in an interview to share ten of his predictions for IT in 2012, specifically targeting the things that he thought would be on the top of mind for many IT executives.

Photo 1: an “IT Decision Maker” who did not listen to our predictions for 2012.

Great to conduct an interview this way, which is different from what I’m typically doing in an online presentation -- typically, I’m the guy going on-and-on about system performance, load testing, performance engineering and offering technical advice, support and tips around IT Performance. It was a very different experience here, asking questions, challenging some premises, and having Gary elaborate on his thinking.

Before you “install updates and shut down” for the holiday, take a listen to this webinar and you might be a little more prepared in the new year.

Ten Predictions for IT Decision Makers in 2012

Wrap-up Video from HP Discover

noreply@blogger.com (Mark Tomlinson) — Wed, 14 Dec 2011 18:05:54 PST

Over the past three years I have been hanging around with the social media team at HP, a team which grew from just a few bloggers in 2009 to nearly TWENTY bloggers and journalists at the conferences in 2012. And aside from a great team, they have some very cool capabilities at the HP Discover shows for video blogging and shooting video from the Discover Zone expo hall. This year I just happened to show up and Paul Muller just pulled me into his team’s wrap-up video, along with Christian and Rafal.

Here are four guys with HP Discover experiences that overlapped somewhat and who offered a few choice take-aways:

1 – When geeks go to a party, they may be having fun, but they are still talking about geek stuff like Cloud and Technology and Security and Performance...
2 – If you are presenting at a conference and they give you the session right after the previous night’s party, it might not be a bad thing -- your audience might be actually listening and passionate about what you’re saying!
3 – When you go to the booths or the primary vendor demos at a conference and they DON’T have actual expertise to share, or they are just “booth workers” who are clueless, then you probably aren’t at an HP show.

What I learned from Artur Bergman in less than 18 minutes

noreply@blogger.com (Mark Tomlinson) — Tue, 13 Dec 2011 15:46:00 PST

Thanks to @lounibos retweet from OreillyMedia on Twitter, I recently came across this presentation from Artur Bergman given at Velocity EU Berlin last month. To be totally honest, I had previously never heard of Artur Bergman, but I had heard of his company Fastly (www.fastly.com).

Artur Bergman presents “Full Stack Awareness” at Velocity EU

In his presentation (above) you learn how to consider that good system performance engineering requires you to have a comprehensive understanding of the complete system, end-to-end, from “kernel to continents” as he says. Watch the video to glean a whole set of tips and tricks on Linux performance...and you will also hear a nearly flawless use of vernacular language which I found quite refreshing. Some of my reflections:

People still don’t understand how computers work. This is great news for me as a performance engineer and tester, because that means I will continue to be employed. But this is bad news because we aren’t making any headway on improving the state of the industry’s maturity around system performance.
There are a few new cool tools to use for Linux performance work, like latencytop, which might be even better than powertop, and “sar –B” or “sar –r” for memory diagnosis -- and also don’t forget that you might need to dig into the source code to find the root cause of a performance limitation.
Don’t forget that Linux kernels were created first as desktop operating systems and there is still a tremendous amount of code that still isn’t optimized for massively scalable server optimization – look for NUMA limitations, single-threaded I/O drivers and virtual memory management assumptions.
Correlation is not causation: you might not be aware that just because you correlate activities between system performance metrics or measures graphed on a mutual timeline, doesn’t absolutely mean that they are actually linked causally. This includes misleading bottlenecks: like increased disk I/O is not a disk bottleneck, if there’s a swap condition due to limited memory.

Thank you Artur for your contributions to performance, your honesty, integrity and f’ing serious commitment to f’ing performance! You can follow him at @crucially and find the rest of the crew at Fastly.

A new kind of protest: “Occupy Web Site”

noreply@blogger.com (Mark Tomlinson) — Tue, 13 Dec 2011 15:50:58 PST

My good friend Rafal Los (Wh1t3 Rabbit) and I have recently spent a good amount of time at HP Discover chatting about IT security, performance, and new stories from the real world of IT. Of course I had to share some new things I have learned from a few customer engagements – in particular, a new twist on Denial of Service attacks that you might not normally associate with a performance vulnerability. Here is the blog post we created together:
__________________________________________________________
Many of the vulnerabilities we hear about in technology systems are exploited for the purpose of financial gain, competitive tactic, or simply for the challenge of doing it. One especially common vulnerability can cripple the infrastructure of your website from what’s called a denial of service or DOS attack, and a more sinister version called a distributed denial of service (DDOS) attack. In these attacks, your website is bombarded with SYN-flood or other low-level network activity that overloads the physical infrastructure of the system. Recent variants of this attack targeted faulty web server handling of requests, enabling even tiny, slow carefully crafted packets to completely stop a website's function. But there’s another type of DOS attack which exploits a vulnerability at higher levels of the architecture. The vulnerability is the persistent cart object of your e-commerce web application.

Recently during a testing engagement, a test engineer engaged in root-cause analysis on a strange anomaly in the website. They were observing an occasionally-blocking, long garbage collection in the app layer causing extended stop-the-world state on processing – in essence, “gracefully pausing” all activity on the system. The results of his investigation showed that there were a few persistent carts that stored more than 100,000 items in the cart. Here’s the logical root-cause of the performance issue:

an individual persistent cart grows to 100,000 items (objects) in the cart object
when an end-user opens the cart, the app server must re-populate the cart with those objects, loading them into the heap
when the session ends after closing the client (aborting), those objects age on the server until they must be marked and cleaned-up in GC (garbage collection)
when that GC time comes and you have enabled many threads for parallel GC (very common), then you hit an STW (stop-the-world) condition on the server
no processing is allowed, everything is “paused” and end-user response times come to a stand-still

No real human being added 100,000 items to an online shopping cart, or at least not with any reasonable purpose. Of course if we we consider the Occupy movements across the globe, demonstrating and protesting against income inequality and inequitable policies around commerce and taxation, this persistent cart vulnerability could become a seemingly benign form of occupation that could develop into a serious threat: Occupy Wall Street could become Occupy Web Site.

Most likely this anomaly was the result of some mechanical automation (and how hard is it for most of us to 'mechanically automate' 100,000 connections to a web server?), but it still showed us the impact of what could happen if someone (or several people) loaded up carts on a website but never checked out and never cleared them out. It would be a slow trickle of traffic over a longer period of time that would never be prevented by your IDS or Firewall protection.

Imagine you work for an online retailer that has a vested interest in doing well over the holiday season which is upon us. Imagine also that you're like many online retailers and make a large chunk of your yearly revenue over the brief, but extremely busy, holiday gift-buying rush. You and I can probably name at least 2 or 3 of these e-tailers right now...If you work for these e-tailers you know there is a "holiday freeze" that happens from the week before US Thanksgiving, until right after New Years' holiday...where no code changes are allowed to occur. OK, so combine those two things, with the fact that many companies rush out code for holiday promotions, site upgrades, etc right before the holiday rush and often forget (or neglect?) to test performance to the degree they should. So how is this a problem?

Effectively, if one of your competitors wasn't as ethical as you are they could pay a couple of (unethical, black-hat) hackers a bit of money to make sure you don't have a good shopping season. Those hackers would then script up an attack -- using presumably one of the hundreds of millions of compromised computers world-wide -- to make sure your site and e-commerce system was completely offline during the times where your company needs to be making money and selling your wares. This attack will cost little, but have a potentially devastating effect on your organization's yearly revenue...believe me this is not FUD. From a Washington Post article "...November results offer an important benchmark for retailers and economists. During the holiday shopping season, merchants can make up to 40 percent of their annual revenue." (Washington Post, 12/1/11)

How to avoid or protect against this “Occupy Web Site” condition:

limit the number of items that can be added to a cart
sweep through persistent cart objects in the database and clean out any of them with over 100 items
configure parallel GC appropriately, leaving enough threads open for normal processing to avoid STW

__________________________________________________________
The original post on Raf’s website is here: http://h30499.www3.hp.com/t5/Following-the-White-Rabbit-A/OWWWS-The-Other-Form-of-Occupy-Occupy-World-Wide-Web-Site/ba-p/5408553
Note: please take note that references to the Occupy movement are made with full awareness of the fact that the majority of network and system administrators working in the trenches of a datacenter in the IT department are members of the 99%.

HP Discover 2011 - Vienna Wrap-up

noreply@blogger.com (Mark Tomlinson) — Thu, 15 Dec 2011 21:44:57 PST

On this last day of HP Discover 20011 – Vienna as the weather passes and the Clouds clear, here are some of my reflections:

Conversation with Christian Verstraete on the latest cloud developments from a cross-group collaboration between many different business units inside HP. I have a sincere respect for this successful effort, knowing the hard work we endured a few years ago in our attempt to host HP LoadRunner on the Amazon EC2 Cloud.
The mobile performance session with my friends and colleagues John Jeremiah, Todd DeCapua and Stephen Feloney, which had the best ending punch line from Steve: “You have to start testing differently. Mobile means you have to change the way you test.”
The Vivit Worldwide team -- especially Susan Russel and Stephanie Konkoy -- are just awesome at these events, openly connecting disparate partners, customers and HP teams into collaborative and fun events. They are really making a difference at these shows and I heard some good plans for the future!
Application Lifecycle Intelligence is now an actual product, instead of how we historically use customized queries into QC. I watched a seriously cool demo of ALI and how to leverage more value out of the repository and gain insight into the lifecycle.

Check out the backstage coverage from the show

Live from HP Discover

noreply@blogger.com (Mark Tomlinson) — Tue, 29 Nov 2011 06:29:23 PST

Here are a bunch of links to Qik videos I\'m taking at HP Discover 2011 Vienna.

This is the first one with my Vivit colleague Richard Bishop in the Discover Zone:

Catching-up with Ed Julson at HP Discover Vienna

noreply@blogger.com (Mark Tomlinson) — Thu, 15 Dec 2011 21:42:57 PST

While I’m here in Vienna, Austria at HP Discover EMEA I’m staying at a really nice, newly remodeled hotel next door to the Schonbrunn Park – it’s a lovely setting, albeit outside the center of Wien. I headed down to the Hotel Café for some coffee and dinner and around the corner comes my old friend Ed Julson from the Product Management team of HP Software. You might recall Ed’s background in the products with HP SOA Systinet and developer integrations with the HP Quality Management & ALM suites – working to extend quality and SV HP "removing constraints"governance earlier in the lifcycle. Cool stuff, but it certainly has proven to be an uphill (upstream) battle.

In catching up, Ed told me about some of the latest items he is working on including his technical breakout session ‘Embracing the Cloud Gracefully’ (on Tuesday, November 29th at 4:30 PM in Session Room #4)
Here are a few of the interesting things that Ed shared with me about his session:

1) much of the functionality of Systinet has already been integrated into/with the HP ALM platform with regards to functionality and value – which perhaps comes with a little less governance and a little more lifecycle enablement.

2) when it comes to cloud, the old “removing constraints” virtualization value proposition are really hitting home for the black-box of REST-based public services; the early work in the application lifecycle absolutely requires virtualization.

3) emulation of the real-world latencies of those services (real or virtualized) is critical considering the platform for the end-point is now mobile and highly latent; which might require some Shunra Software for complete and proper emulation.

Just the tip of the iceberg when it comes to Ed’s experience and vision for the future, but I’m sure he’ll have more to share on Tuesday!