Unfortunately, thinking of it this way underestimates two other variables: user behavior and system configuration.

The outputs of a load test are only as reliable as its inputs. If your test scripts simulate different user behaviors than what actually happens in production, your results will be invalid. And of course, if the test system is configured differently from what’s running in production, you’ll need to take any result with a huge grain of salt.

I’ll dive more into each of these factors, and some techniques to minimize the risk of them skewing your test results, in my upcoming post.

You’d be surprised how many teams get started on load testing without a clear, quantifiable question.

Binary outcomes

The most complete and straightforward performance requirements are those with binary outcomes:

• Can my application handle at least 4200 shipments per hour, with all pages loading in 3.0 seconds or less?
• Can this system support 3000 concurrent users at peak load, with 90% of response times in 2.5 seconds or less?
• Does the application support 10,000 logins per day, and at least 2,000 logins per hour at peak load?

These are great because they’re easily restated as yes/no questions, and after testing, there’s little ambiguity whether the system satisfied the requirement.

Quantifiable outcomes

The next most straightforward type of question is one that has a quantifiable, numeric outcome:

• How many concurrent users can this web app handle, while keeping response times at 2.0 seconds or less?
• How many total transactions per minute can this system process, before errors start to happen?
• What is the average response time of this website with 500 concurrent users?

This type of question is not really the same thing as having a concrete performance/scalability requirement, but it does at least allow for a single outcome with little wiggle room.

Vague outcomes

There are some questions that often lead to vague or useless outcomes:

• How many users can my website handle?
• What’s the average response time under load?
• How many concurrent users until my app becomes unresponsive?

Granted, you might learn a lot along the way, but if you set out to answer such vague questions you won’t have a meaningful result unless you refine the question as you go.

Lots of exciting features on the way, including long-awaited Mac support. Each release candidate build will reveal more stuff, but there are a couple things we’re holding out until the final release, so add your name to the list by emailing beta@loadsterperformance.com. Looking forward to your valuable feedback.

I’ve been invited by our partners at Polarion Software to present an introduction to load testing. We’ll cover the rationale for load testing, the basics of how it fits into a typical software product lifecycle, and more. Also stay tuned for a sneak peek at our upcoming integration with Polarion QA.

The webinar is totally free, so register now to reserve a spot.

It used to be that our proxy failed to record HTTPS traffic because it was encrypted, so my advice was to temporarily disable HTTPS on your application while recording your scripts, then re-enable it for playback. For most apps this worked fine, but doubtless it was an inconvenience for many of our users.

As of Loadster 2.6.5, the recording proxy uses a “man-in-the-middle” approach to decrypt, record, and re-encrypt HTTPS traffic. Your browser will generate a warning about the certificate, which you can often bypass on a case-by-case basis. To globally trust ALL of Loadster’s certificates, follow these instructions to install our root certificate in your browser.

Good news! You can now deploy multiple load engines and control them from a single workstation. The Loadster Engine comes with its own Windows installer just like the workbench, and can be downloaded from our download page.

No user limit, so buy what your infrastructure requires

Our single fixed price makes us fairly unique among commercial load testing tools. Loadster doesn’t have tiered pricing based on the number of concurrent users you need to simulate. Just one price for the software license and that’s it.

We have continued that tradition with the Loadster Engine pricing. At $229 per engine, they are a straightforward and affordable way to extend your load testing capabilities to the tens of thousands of users and beyond.

How it works

First of all, install and run the Loadster Engine. The UI is predictably simple, since everything is still controlled from the Loadster workbench.

Now, back on your workstation, you’ll need to associate the load engine so that Loadster knows about it. Open up the Preferences… dialog and add it.

Next, set up a test scenario or edit one of your existing ones. Each virtual user population can be assigned to a different load engine: in this case, either the remote one you just added, or the built-in load engine that’s part of Loadster workbench. Naturally, it’s best to split the load fairly evenly when possible, so none of the engines is overloaded.

Running a test with distributed load engines is exactly like running any other test. However, you’ll now see a small performance graph for each load engine, that gives you a snapshot of how that load engine is using its resources.

Make sure your load engines stay in a healthy range, because if the bars start to hit the top it means they are overloaded and their measurements may become less accurate. Sometimes if you’re running an unusually large virtual user population, it makes sense to split it into two or more identical populations so you can maintain accurate measurements across all engines.

Your feedback

Let us know what you think! It may sound cliché, but your detailed feedback is totally indispensable as we continue to refine Loadster and make improvements. We would love to hear your stories about load testing experiences, challenges, feature requests, and any other comments or ideas you’re willing to share.

Polarion’s customers already benefit greatly from a platform that brings together requirements, design artifacts, source code, test planning, and issue tracking in a single system. This partnership is a great way to add load testing to that mix so that more organizations, large and small, can make it an integral part of their development lifecycle. By using Polarion’s software in conjunction with Loadster, teams can release applications with the confidence they will perform and scale admirably along with their business.

We already have some exciting joint developments planned for the next few months to make our applications work more closely together. More information to follow!

In the meantime, a quick update on a smallish but very useful new feature that debuted in Loadster 2.4: custom HTTP headers.

Custom HTTP Headers

A lot of apps, particularly web services, expect custom headers in the request. These can be used by the client for passing authentication info, special caching/content instructions, and so forth. While Loadster has taken care of the standard headers automatically (for cookies, HTTP BASIC authentication, etc), there was previously no way to handle these custom headers as part of your test scripts.

To use custom headers in your test scripts, just double-click a command and look under the “Custom Headers” tab.

Tip: you can also use variables in custom header values.

Now the bad news: it’s ugly! Well, not terrible, but certainly not up to Mac visual standards. The truth is, it will take a decent amount of time to polish up the Mac version to where it fits in with all the smooth glass and machined aluminum bevels we adore so much. Then there’s all that extra work required for packaging it in an app bundle, etc.

Releasing a long-in-the-tooth Mac version is not an option. So I figured it’s best to gauge interest first. If you want a Mac version, please drop us a line, send us a tweet, or comment below!

Variable Capture from HTTP Headers

Loadster lets you capture values from an HTTP response (typically a dynamic page). Now, it supports capturing from HTTP response headers too, not just the body!

Why is this important? Well, lots of web applications use a “save-and-redirect” pattern. When you submit a web form (let’s say) to create a new entity in the database, that new object will likely have an ID. After receiving the POST, many web apps will then send an HTTP 302 redirect so the user can view the newly created entity. That way if the user refreshes the page, it won’t resubmit the post and create a duplicate entry.

When creating automated load test scripts in Loadster, it’s really essential that you be able to capture this ID and use it later on in the script. Sometimes it’s also important to capture other dynamic values, such as cookies or session IDs.

Setting up a capturing rule in the script editor is simple:

After that, you can use the variable you captured in subsequent requests, like so:

Every time that variable is encountered, Loadster will dynamically substitute the value that was most recently captured. Remember, variables are unique per virtual user, so each virtual user can have their own dynamic session data just like real users would.

Virtual User Population Details

Another little bonus in Loadster 2.1 is the addition of more details on virtual user populations in a test report. Loadster previously let you configure these variables for each population, but they were missing from the final report and trying to recall them afterwards was no fun.

As you can see, Loadster virtual users have a lot in common with real web browsers. They have configurable timeouts, and can make use of multiple parallel download threads for loading page resources (images/CSS/JavaScript). The number of threads per v-user is configurable and defaults to 6.

What’s next?

We have a really aggressive roadmap over the next couple months that is going to be really exciting, including some much bigger stuff than what I just talked about. I can’t tell you exactly what just yet because, well, you know…

And if you read all this and still haven’t tried Loadster, download it here! It’s totally free up to 25 v-users which works fine for small-medium web apps. For larger load tests, we offer competitively priced licenses.