The BrowserMob Blog

BrowserMob Joins Neustar Webmetrics Family of Services

2010-07-08T23:21:51Z

I’ve got great news to share with you – BrowserMob has been acquired and is joining the Neustar family of services, which includes UltraDNS and Webmetrics!

I started BrowserMob in 2008 with the belief that a combination of cloud computing and real browsers wrapped up in a self-service model could dramatically change how people used load testing. Neustar shares that same philosophy and has built a world-class portfolio of cloud-based performance services that currently has over 3,000 customers. I’m most excited by this partnership because it means that we can now offer our unique services to an even greater audience.

A few months ago Neustar approached us and it was immediately obvious that our visions for website testing, development, and operations were very similar. For example, just like BrowserMob, the team at Webmetrics utilizes real browsers and supports open source and cloud computing initiatives. Even better, upon closer look I found that Webmetrics Load Testing and Monitoring solutions actually complement our own services much more than they compete. Joining forces made perfect sense!

While in the short term nothing will be changing, the long term benefits of this partnership will be an even better experience for all our users. For example, Webmetrics Load Testing provides high quality, hands-on professional services that BrowserMob customers will be able to utilize if they need extra assistance. And for monitoring, BrowserMob customers will be able to benefit from the massive infrastructure and capabilities that Webmetrics already has in place including a global infrastructure with agents in over 100 major cities. Similarly, we expect to bring to the table many innovations of our own over the coming months and years.

Of course our highest priority is you – our customer. I want to assure you that the quality of service you’ve been receiving from us will not change, and that 100% of our team will remain intact and dedicated to making you successful. In fact, effective today both Ian White and I are officially Neustar employees and we are committed to enhancing our solutions and your satisfaction!

Thank you for showing your continued support to BrowserMob. I’m excited to continue working with you and updating you on our plans for a fully integrated BrowserMob + Neustar offering. If you have any questions, you’re of course always welcome to contact me directly.

Patrick
+1 (503) 828-9003 x 101
patrick.lightbody@neustar.biz

Tweet This Post

Estimating concurrent users based on past traffic

2010-06-16T05:10:57Z

Today we received an excellent question from a customer of ours:

We were wondering if you all have any information that says “X Unique visitors per day translates into Y simultaneous users at any given time.”

Essentially, we’re looking for a way to determine how many simultaneous users we should load test with if we know the sites normal daily traffic.

While every site is different, we recommend following this line of reasoning to help you find the answer. Suppose your site gets 100K unique visitors per day, with peak traffic in the mornings and afternoons. Assume that 40% of the traffic comes between 7AM and 11AM, 40% at 4PM to 9PM, and 20% at other times. This means during your peak hours (7AM to 11AM and 4PM to 9PM) you’ll get ~10% of your unique visitors per hour, or 10K uniques in our example.

Now that you know how many unique visitors you’ll get in an hour, you can start turning that in to concurrent users. To do that, it’s important to understand that a unique visitor is roughly equivalent of a transaction. So really you want to figure out how many users you need to reach 10K transactions in an hour.

Suppose your script (or scripts) take an average of 2 minutes to complete. That means a single user will execute 30 transactions in an hour. So to reach 10K transactions, you’d need 334 users (10K divided by 30). If you decide you want to create realistic scripts that include human think time, then the scripts will take that much longer and you’ll need that many more concurrent users. For example, if the script gets 5X think time added and now takes 10 minutes to run, then you’ll need 1,667 users (10K divided by 6).

Of course, this calculation will only get you the load on a typical day (assuming a single hour sees 10% of traffic). Your traffic patterns may vary, or you may want to prepare for a larger surge. For example, if you want to test what happens when 60% of the daily traffic visits in an hour, then you’d need 2,000 users (60K divided by 30).

Tweet This Post

Eliminating concurrent access to sensitive data

2010-06-14T18:36:17Z

We recently had a customer from a large clothing retailer ask us if there was any way to ensure that data, such as a username/password combination, could be restricted such that it was “checked out” and available only for a specific concurrent user. This is very common with logins, where systems often will prevent concurrent logins from multiple IP addresses.

While BrowserMob does not have a concept in which data rows can be “checked out”, some simple scripting can achieve the same results. The key is in creative use of the browserMob.getUserNum() and browserMob.getTxCount() APIs. You can learn more about them by reading up on the BrowserMob APIs.

The getUserNum function returns 0, 1, 2, etc based on the concurrent user in your load test. So if you have a 100 user test, getUserNum will return between 0 and 99. It’s important to understand that it will return the same value for the same user throughout the test.

The getTxCount function returns 1, 2, 3, etc based on the number of cycles for that specific user. This number will effectively be a counter of the unique number of transactions that that particular user has executed. So user 1 and user 100 will both have a getTxCount of 1 returned, but by the time user 100 sees it, user 1 might already be on transaction 50.

Now suppose you want to run a 1000 user test in which you never concurrently log in with the same user. All you need to do is pre-create 1000 user accounts and then write your script like so:

var userId = browserMob.getUserNum();
var username = "test-" + userId;
var password = "password";
selenium.type("username", username);
selenium.type("password", password);

This works great, but what if you want to use more than 1000 logins? Suppose you want to use up to 10,000 logins among the 1000 user test? This is where the getTxCount function comes in to:

var loginsPerUser = 10;
var userNum = browserMob.getUserNum();
var txCount = browserMob.getTxCount();
var userId = userNum * loginsPerUser + txCount % loginsPerUser;
var username = "test-" + userId;
var password = "password";
selenium.type("username", username);
selenium.type("password", password);

What this does is allocate 10 logins per concurrent user. So user 1 will get usernames test-0, test-1, …, test-9 while user 8 will get usernames test-80, test-81, …, test-89, etc. Because of the mod call (%) the ten usernames will simply wrap around once they’ve been used.

Tweet This Post

Advanced handling of page timeouts in Selenium

2010-06-13T21:45:11Z

Because both our load testing and website monitoring services are based on Selenium, we have a unique ability to measure the performance of things like page load times, AJAX timings, and other in-browser interactions.

Selenium has both a setTimeout command and a waitForPageToLoad command. Both can be given a timeout value, which will control how long Selenium waits for a given page to load or element to appear. When it comes to using our services, most people stick with the default time of 30 seconds. If the timeout is reached, an error is thrown, the script aborts, and the transaction is recorded.

However, sometimes people want to know when pages take more than X seconds to load, but don’t want to necessarily interrupt the flow of the script. In fact, just last week we got this request from a customer:

Ability to trigger an alert based on a set threshold (by the user) – not using the timeout. This basically came from the performance issues we are experiencing. Here’s the scenario:

We have a specific page that takes 2+ minutes to load.

The page load timeout in the Selenium script was set to 60s.

BMob properly reported the “timeout” error but when this error happens, BMob quits the script.

This is not ideal for me since I want to still be able to see how long the page takes to load.

Increasing the page load timeout for the page in question works, but now I don’t have a way (that I know of) to still trigger an alert after 60s.

I should be able to set a threshold for any page I choose that would then send a notification alert.

In other words, this customer wanted for a way to still report a transaction as a failure and receive an alert, but also still allow the script to continue. Fortunately, our support for JavaScript as a scripting language provides the answer:

var timeout = 90000;
...
// start of script
...
var start = new Date().getTime();
selenium.waitForPageToLoad(timeout);
var end = new Date().getTime();
...
// rest of script
...
if ((end - start) > 45000) {
    throw "An important page took longer than 45 seconds to load";
}

What this script does is sets the timeout to a very long amount (1.5 minutes) but will still report an error if a specific page takes longer than 45 seconds. This allows the remainder of the script to execute even when the page takes more than a specified 45-second threshold.

The only problem with this script is that if the page takes longer than 90 seconds, then the rest of the script will still not run because waitForPageToLoad will throw an exception. You can solve that too with a little code:

var start = new Date().getTime();
try {
    selenium.waitForPageToLoad(timeout);
} catch (e) {
    // this will happen after 90 seconds
    // todo: recover and send the browser to the the next URL
}
var end = new Date().getTime();

The only thing that is important to remember with this use of try/catch is that you’ll need to properly recover from the error. Simply catching the error and trying to continue may not work. For example, if the next Selenium command requires clicking on a button that should have loaded from the last page, there may be no way to recover. However, if the next step is simply visiting a new URL, you could possibly get away with a simple open() command in the catch block.

Tweet This Post

dynaTrace shows how to track AJAX performance with Selenium

2010-05-25T18:26:24Z

dynaTrace just posted a really nice tutorial showing how to track client-side performance (JavaScript, DOM, render times, etc) automatically using Selenium. They do this by using their super-cool free AJAX edition of their product.

Tweet This Post

Monitoring Maintenance Windows

2010-05-19T19:07:05Z

When setting up monitoring jobs, there are often predictable time periods in which you want to change the behavior of a script or prevent it from running at all, without having to manually stop/start the monitoring job each time. For instance, you might want to prevent errors and alert emails during routine maintenance windows, or perhaps you’re only interested in site performance on weekdays during regular business hours. We’ve come up with solutions to a few common situations that will help jump start your scripts.

Overview
At the core of all these examples is a small configuration object that sits at the top of the script for quick changes, and a single function that returns true or false depending on whether or not the current time is within a blackout period. The following is some pseudo-code outlining one way in which you might choose to implement these examples in your own scripts. Don’t forget to add in the appropriate maintenance object and isInMaintenance() function from the examples below to make this work.

// The configuration object for the maintenance window
var maintenance = {...}
 
// The maintenance function (plug in the appropriate function)
var isInMaintenance = function() {
    ...
}
 
var selenium = browserMob.openBrowser();
 
if(!browserMob.isValidation() && isInMaintenance()) {
    // Active maintenance window
    browserMob.log("Is in maintenance!");
} else {
    // Run normal script
    browserMob.beginTransaction();
 
    browserMob.beginStep("Step 1");
    selenium.open("http://google.com/search?q=website+monitoring");
    ...
    browserMob.endStep();
 
    browserMob.endTransaction();
}

Web Page Check
The most straight forward way to check for a maintenance window is to have Selenium load a web page and look for an element that’s only present during scheduled down time. You’ll want to be careful not to choose something that is also present when the site is really in trouble or you will never get your alerts!

var maintenance = {
    check: {
        url: "http://example.com/home",
        element: "//img[@title=\"maintenance\"]"
    }
}
 
var isInMaintenance = function() {
    if(maintenance.check && maintenance.check.url != null && maintenance.check.element != null) {
        selenium.open(maintenance.url);
        return selenium.isElementPresent(maintenance.element);
    }
 
    return false;
}

Multipe Date Ranges
One of the most common requests we get is for a way to handle scheduled maintenance windows. These windows tend to be one-off date ranges decided well in advance of the actual event. Given a set of start/end dates, you could prevent monitoring errors and email alerts during the blackout periods. When setting the dates, don’t forget to include the time zone abbreviation in the argument string or the script will use the local time of the machine it’s running on.

var maintenance = {
    dates: [
        {start: new Date("05/12/2010 23:00 PDT"), end: new Date("05/13/2010 02:30 PDT")},
        {start: new Date("05/24/2010 23:00 PDT"), end: new Date("05/25/2010 02:30 PDT")}
    ]
}
 
var isInMaintenance = function() {
	var now = new Date(),
	    nowTime = now.getTime(),
	    maintenanceWindows = [];
 
    // Check for maintenance date ranges
    if(maintenance.dates && maintenance.dates.length) {
        maintenanceWindows = maintenance.dates;
    }
 
    for(var i in maintenanceWindows) {
        var m = maintenanceWindows[i];
 
        if(nowTime >= m.start.getTime() && nowTime < m.end.getTime()) {
            return true;
        }
    }
 
    return false;
}

Days of the Week
Here’s an example for blacking out specific days of the week. This will allow you to create a script that, for example, monitors on Mondays, Wednesdays, and Fridays, or only on business days. Simply provide a time zone offset, which is the number of minutes your time zone deviates from coordinated universal time (a.k.a. UTC, GMT, Zulu time), and the days of the week you don’t want to run the script (from 0-6, where 0 is Sunday, 1 is Monday, etc.).

var maintenance = {
    timezoneOffset: 240, // UTC offset in minutes (eg: NYC (GMT-4) is 4 * 60
    days: [0, 6]    // Sunday = 0, Saturday = 6
}
 
var isInMaintenance = function(){
    var now = new Date(),
        nowTime = now.getTime(),
        timezoneOffset = maintenance.timezoneOffset || 0,
        localTimezoneOffset = now.getTimezoneOffset(),
        timezoneDiff = localTimezoneOffset - timezoneOffset;
 
    // Check for full blackout days (such as weekends)
    if(maintenance.days != null && maintenance.days.length) {
        var targetDate = new Date(nowTime + (timezoneDiff * 60 * 1000)),
            targetDay = targetDate.getDay();
 
        for(var i in maintenance.days) {
            if(maintenance.days[i] == targetDay) {
                return true;
            }
        }
    }
 
    return false;
}

Time of Day
This example shows how you can have a daily blackout range. This code will parse the start/end strings into a daily time range adjusted to the target time zone. Use this if you only want to monitor during banker’s hours, or if you want to prevent a script from running while your database is backing up.

var maintenance = {
    timezoneOffset: 240, // UTC offset in minutes (eg: NYC (GMT-4) is 4 * 60
    hours: {start: "23:30", end: "07:30"}
}
 
var isInMaintenance = function(){
    var now = new Date(),
        nowTime = now.getTime(),
        timezoneOffset = maintenance.timezoneOffset || 0,
        localTimezoneOffset = now.getTimezoneOffset(),
        timezoneDiff = localTimezoneOffset - timezoneOffset,
        maintenanceWindows = [];
 
    if(maintenance.hours != null && maintenance.hours.start != null && maintenance.hours.end != null) {
        var sH, eH, sM, eM;
 
        var chunks = maintenance.hours.start.toString().split(":");
        sH = parseInt(chunks[0]);
        sM = parseInt(chunks[1]) || 0;
 
        chunks = maintenance.hours.end.toString().split(":");
        eH = parseInt(chunks[0]);
        eM = parseInt(chunks[1]) || 0;
 
        var startDate = new Date(now.getFullYear(), now.getMonth(), now.getDate(), sH, sM, 0, 0);
            endDate = new Date(now.getFullYear(), now.getMonth(), now.getDate(), eH, eM, 0, 0);
 
        // Convert to target timezone
        startDate = new Date(startDate.getTime() - (timezoneDiff * 60 * 1000));
        endDate = new Date(endDate.getTime() - (timezoneDiff * 60 * 1000));
 
        if(startDate.getTime() <= endDate.getTime()) {
            // Single range mid-day
            maintenanceWindows.push({start: startDate, end: endDate});
        } else {
            // Date wraps, which means two blackout ranges in a single day
            var morning = new Date(now.getFullYear(), now.getMonth(), now.getDate(), 0, 0, 0, 0),
                midnight = new Date(morning.getTime() + (24 * 60 * 60 * 1000));
 
            maintenanceWindows.push({
                start: morning,
                end: endDate
            });
 
            maintenanceWindows.push({
                start: startDate,
                end: midnight
            });
        }
    }
 
    for(var i in maintenanceWindows) {
        var m = maintenanceWindows[i];
 
        if(nowTime >= m.start.getTime() && nowTime < m.end.getTime()) {
            return true;
        }
    }
 
    return false;
}

Now, a few of you might have noticed that the above examples are compatible with one another; you can mix-and-match them as you see fit. Just combine the various maintenance object fields into one config object, combine the contents of all the maintenance methods you want to use (just watch not to overwrite the variable declarations), and you can create something that fits just about any situation.

Tweet This Post

Integrating BrowserMob with Nagios

2010-05-06T20:50:54Z

To demonstrate how you can use our new API to leverage deeper integration with your internal monitoring tools, we’re going to guide you through an example that connects Nagios (a ubiquitous open source monitoring tool) with BrowserMob’s web site monitoring service.

Getting it all setup:

Since this is intended to be a quick how-to, we’re going to assume that you already have a working Nagios installation behind your firewall and an active monitoring job with BrowserMob. Here at the BMob we eat our own dog food, utilizing our own monitoring service to monitor BrowserMob.com. For this example, we’ll use a monitoring job called “BMOB” for our Nagios integration.

What you will need:

API Key & API Secret. Log on to browsermob.com with your account and under Home > My Profile you will find your API key and secret.
CLI BrowserMob API tool. Download our nifty API command line tool from http://github.com/rferreira/bmob-python
Nagios 3.06. We have it installed on Ubuntu Jaunty for this example.
jsonpretty. This just makes JSON strings human-readable in the CLI. Get it here: http://github.com/nicksieger/jsonpretty

Getting to know your monitoring job:

Before we can actually wire anything into Nagios we need to find out the internal id of a monitoring job. In the BrowserMob web interface monitoring jobs have friendly names like “BMOB”, but internally they are referenced by a unique id such as “fe33b13dc0764588b5eabf747a96a48b”.

You will need the monitoring job’s unique id. You can get it by running the following query (replacing the credentials with your own):

$ ./bmob.py -c XXX:XXX http://browsermob.com/a/m/all | jsonpretty
[
{
"browsers": [
"FF3"
],
"name": "BMOB",
"lastBilled": null,
"alertEmail": "support@browsermob.com",
"preferenceId": "7ea6d3b47c8c4a1c8d30db7fef9d5fd1",
"lastRun": 1273007522255,
"deleted": false,
"enabled": true,
"id": "fe33b13dc0764588b5eabf747a96a48b",
"lastUpdated": 1273005121243,
"frequency": 15,
"locations": [
"DALLAS",
"SING",
"AMS",
"NY",
"DC",
"DUBLIN",
"SF"
],
"scriptId": "21312a2da1834f2aa02db73f32037619",
"accountId": 2,
"email": "raf@browsermob.com"
}
]

Look for the “id” property in the JSON response.

Wiring it all up:

Now we need to write a wrapper script to query and parse the API results. Create a file called “check_bmob.sh” with the following contents:

#!/usr/bin/env bash
 
# RETURN CODES:
OK=0
WARNING=1
CRITICAL=2
UNKNOWN=3
 
WARNING_T=2000
CRITICAL_T=5000
 
ID="fe33b13dc0764588b5eabf747a96a48b"
 
# dates
START=$(date -u  --date="5 mins ago" +%s)000
END=$(date -u  +%s)000
RESP=$(/opt/bmob-python/bmob.py -d metric=responseTime,start=$START,end=$END,resolution=hour –c XXX:XXX http://browsermob.com/a/m/$ID | tr "," "\n" | grep responseTime | awk -F ":" '{print $2}')
 
for t in $RESP
do
   if [ $t -gt $CRITICAL_T ]
      then
         echo $t
         exit $CRITICAL
   elif [ $t -gt $WARNING_T ]
      then
         echo $t
         exit $WARNING
   fi
done
 
exit 0

What this script will do is look up the response time for the last 5 minutes and trigger an alert if the response time is greater than the threshold for WARNING_T or CRITICAL_T.

Now, we just need to tell Nagios to use it; the sample config below should do the trick:

# defining the command
define command {
    command_name    check_bmob
    command_line    /usr/lib/nagios/plugins/check_bmob.sh
}
 
define hostgroup {
    hostgroup_name website
    alias web site
}
 
define host {
    host_name browsermob.com
    check_command check_http
    alias login-check
    use generic-host
}
 
define service {
    hostgroup_name                  website
    service_description             response_time
    check_command                   check_bmob
    host_name                       browsermob.com
    use generic-service
}

Please keep in mind that Nagios’ configuration files can be–and usually are–fairly complex. The example above is purposely simple and may need to be modified to work in your environment.

If everything works as planned you should be able to log in to the Nagios web interface and see:

Pudding

Official BrowserMob API documentation:

http://cdn.browsermob.com/api.html

Tweet This Post

Introducing the BrowserMob API

2010-05-06T15:54:43Z

The BrowserMob team is excited to launch our new API, giving customers access to all their data (including load test results and monitoring reports) through a simple programmatic interface. The API is still in beta and will likely change over the next few weeks, but we welcome everyone to take a look, kick the tires, and send us any feedback.

Currently the API is “read only”, allowing access to load test and monitoring data. We will be adding new services in the coming weeks, including endpoints for creating load tests, monitoring jobs, and working toward complete programmatic access to anything you can do on the BrowserMob website today.

To get started, check out the API documentation. We will be updating the docs as things progress and more endpoints are added. Included in the documentation are links to some simple tools to help you get started quickly, and as always you can email the BrowserMob team if you have any questions.

- The BrowserMob Team

Tweet This Post

Identifying bottlenecks with New Relic’s Java profiling tool

2010-04-29T20:19:51Z

A couple weeks ago, New Relic launched a new hosted, on-demand Java profiling tool. We’re big fans of New Relic (great for Ruby or Java), dynaTrace (great for .NET or Java), and other application performance monitoring tools because they complement our website load testing and website monitoring services so well.

At BrowserMob, we’re a Java shop, and so it didn’t take long for us to have an opportunity to try out New Relic’s Java profiler. We were getting reports from some customers that viewing the response time charts of their monitoring data was occasionally slow (don’t worry, we didn’t miss the irony.)

To investigate the problem, we turned on New Relic’s profiler and selected 5 minutes of profiler time. You can choose between 1 and 10 minutes as the length of time you want to gather data. During that time, there is a small but potentially noticeable hit in performance on your site, but the upside is that the profiler is able to inspect deep in to our codebase and expose bottlenecks.

Five minutes later, we instantly saw the problem:

As you can see, during that 5 minute period, 55% of the time was spent on java.net.SocketInputStream.socketRead. Tracing back a bit, we were able to see that was from a call by MonitoringController.selectRangeForObject. It turned out we were calling a cloud-based data server that was having performance issues due to the way we constructed our API call. It didn’t take more than a few minutes to optimize the call and roll out the fix.

Such is the power of application performance monitoring tools. If you’re doing any sort of load testing, website monitoring, or page optimizations, you’re going to be very blind unless you have access to information like this. That’s why we jumped at the opportunity to partner with New Relic and dynaTrace and encourage all our customers to look at their products.

Tweet This Post

Watir and Selenium joining forces for the benefit of all testers and developers

2010-04-27T14:39:53Z

We’re a few weeks late to the story, but we think it’s still important to call attention to: Watir and Selenium are joining forces. For those unfamiliar with Watir or Selenium, both of them are open source browser automation frameworks. For years they have been “competing” in the open source community, with Watir winning favor among the Ruby community and those who needed strong IE support, while Selenium won favor among the Java and C# crowds and those who really valued cross-browser support.

While both projects had their pluses and minuses, it’s great to see them finally working together. On behalf of BrowserMob (which builds on top of Selenium) and as a Selenium contributor, it pleases me to know end to know that the Selenium community will now gain two huge contributions: a first-class Ruby API and all the benefits of Watir’s fantastic IE support.

Note: this is not a merger. Watir will continue to run as an independent project and it will even still work with it’s own browser backend. The important thing here is that Watir will now offer an option to wrap around Selenium WebDriver, the core cross-browser automation library that all the other Selenium projects also wrap around. This means that while the projects will remain independent there will likely be a lot more cooperation moving forward.

Tweet This Post