var selenium = browserMob.openBrowser(); var tx = browserMob.beginTransaction(); var step = browserMob.beginStep("Home Page"); selenium.open("http://example.com"); // record how long it took for some text to appear var start = new Date().getTime(); selenium.waitForTextPresent("dynamic text"); var end = new Date().getTime(); // get a special cookie out and store it var cookie = selenium.getCookieByName("myCookie"); tx.put("cookie_value", cookie); // and store the "text" timing, associated with the step step.put("text_timing", end - start); browserMob.endStep(); browserMob.endTransaction();
The important things to notice here are that both beginTransaction() and beginStep() actually return objects that represent the transaction or step, respectively. You can then call put() on those objects to associate arbitrary data, such as custom timings or cookie values, with them.
Then when the transaction is stored in the load test database, you’ll be able to look them up in the name_value_pairs table. For more information on the test database schema, consult the documentation.
]]>The project, called the Repower America Wall, involved a massive, Flash-based interface in which thousands of user-created images and videos could be scanned through at high speed.
In the case study, they describe how they calculated the estimated site traffic and decided to use cloud services, such as Amazon and YouTube, to help the site scale. However, before launch they still needed to prove to the customer that the site was ready:
While this stress-testing was valuable, and optimizations were made based on them, none of these techniques truly simulated real-world use. Typically, a testing environment such as Selenium IDE might be used. But since the UI of the wall was built in Flash, things got tricky.
Because of the rich Flash interface, traditional load testing services based on simple HTTP protocol simulation weren’t well-suited. For that reason, they came to BrowserMob. Using our real browser technology, we were able to load the entire UI, allow the background AJAX and AMF calls to take place, and even stream the various videos:
BrowserMob’s team helped us craft some tests which we ran on the dev and production servers. The information was invaluable — based on the test results, we decided to upgrade the production server, and doubled its memory.
Below is some of the load test charts SignalFive generated using our software:
This post explains script blocking, and then shows how to safely setup a tracking script or any external script, such as Google Analytics or Quantcast, to not block page loads or other javascript handlers on your site.
It’s a great read and provides a clear explanation on how third party components (everything from advertisements to analytics to content widgets) can slow your site down. JavaScript includes have been commonly used in websites for coming on a decade, but there are still many people who don’t understand the destruction a rogue script can cause to user experience.
One such example of that kind of destruction recently happened to customer of ours who ran an advertisement during the Super Bowl. In the weeks and months leading to the event, they had done massive amounts of tests and verified that their data center could easily handle 2.5 gigabits per second or more.
They were more than prepared and, overall, the event went off without a hitch. However, there was one minor problem that did plague their launch: a third party that served a component on their site couldn’t keep up with the load.
It turns out that one page (unfortunately, an important one that allowed visitors to search for their products) included an inline JavaScript file hosted externally. This included file was added at the last minute and didn’t get subject to the intensive testing that the rest of the site had gone through.
When the external site went down, visitors’ browsers would timeout after 1-2 minutes of trying to fetch that content before finally firing the important onload event in the browser. Nothing hosted by the company had a problem, but yet it still affected the user experience.
To make matters worse, the customer did have certain JavaScript that fired on the onload event. This JavaScript added rich functionality, such as making a calendar pop up when clicking on a date field. Because of the 1-2 minute delay, this functionality wasn’t available immediately, causing the experience to deteriorate.
In addition to all the guidance given in the aforementioned article, there are two things they can do to avoid this next time:
The “on content ready” psuedo-event doesn’t have an official name, but a lot of people also refer to it as “on DOM ready”. YUI, Dojo, Prototype, jQuery, and many others support it, so be on the lookout.
]]>We call these errors “soft timeouts”, and they require a little bit of code to ignore them:
var selenium = browserMob.openBrowser(); selenium.setTimeout(2500); function ignoreIfSafeTimeout(e, step, minObjects, acceptableThreshold) { if (e.message.indexOf("Timed out after ") == -1) { throw e; // this isn't a timeout, so throw it } var objects = step.getObjects(); var objectCount = objects.size(); if (objectCount < minObjects) { throw "Expected at least " + minObjects + " objects but only saw " + objectCount; } var errors = 0; for (var i = 0; i < objectCount; i++) { var object = objects.get(i); var code = object.getStatusCode(); if (code < 200 || code >= 400) { errors++; } } var errorRatio = errors / objectCount; if (errorRatio > acceptableThreshold) { throw "Expected no more than " + acceptableThreshold + " error ratio but saw " + errorRatio; } } browserMob.beginTransaction(); var step1 = browserMob.beginStep("Home Page"); try { selenium.open("http://marketwatch.com"); } catch (e) { ignoreIfSafeTimeout(e, step1, 130, 0.1); } browserMob.endStep(); browserMob.endTransaction();
What this code does is catch the automatic timeout exception thrown when visiting a website (in this case http://marketwatch.com) and decide whether it ignore it or not enough of the page content had loaded. There are a few things to note:
The most important thing in here is the call to ignoreIfSafeTimeout() in the catch block. When called, we pass in the exception that we just caught along with the step object. The last two arguments are the important ones.
The third argument, 130, means that if there is a timeout and less than 130 objects have been requested, an error will be thrown. The fourth argument, 0.1, means that if more than 10% of the requests are error codes, then an error will be thrown.
Feel free to use this code as a starting point for reducing the number of “soft timeouts” in your load tests and monitoring jobs.
]]>By any metric, this a price reduction. We’re extremely excited to have found a model that controls costs for our customers while actually making the revenue flow for our company healthier.
Not only did we reduce the overall pricing for monitoring and/or give a better value for the price, we created a new way to buy load testing from us that is significantly less expensive for most projects.
But even if you stick with the old model of paying for load tests, we’ve reduced the price because the first 25 RBUs or 100 VUs are always sold “at cost”. This means that a 100 RBU test that used to cost $300 now costs $15 under the new plan and $228.75 when using the old plan. To understand how you can choose between either plan, please see the Can I still use my old pay-as-you-go plan? question.
The old pricing model for monitoring used a complex formula in which the time of the script being used to monitor your site heavily influenced the overall cost. For example, a job that averaged 3 seconds per run might cost half of that of a script that took 7 seconds to run. The problem with this model was that if your site slowed down, your costs would go up, making your monitoring budget unpredictable.
The new pricing model eliminates that unpredictability. Instead, there is a constant number of monthly page views that you’ll used, based entirely on your script, number of locations you check from, and frequency at which you check. No matter how fast or how slow your site is, the price will remain constant.
For example, if you wanted to test a script that hit your home page, visited the login page by clicking the login link, and then logged in, that would be 3 page views. If your monitoring job ran that script every 15 minutes from two locations, you’d use 17,280 page views per month:
![[Post to Twitter]](http://blog.browsermob.com/wp-content/plugins/tweet-this/icons/tt-twitter.png){kind=icon}
17,280 = 3 page views x 60 mins / 15 min freq.
x 24 hours x 30 days x 2 locationsOur pricing packages start at 3,000 monthly page views per month (FREE) and go all the way up to 350K monthly page views.
If you were paying for a monthly, recurring plan, then you have already been converted over to the new model and are now paying less than you were before. That’s because we’ve lowered our pricing for all monitoring packages. For example, what was $300/mo is now $249/mo, and what was $175/mo is now $149/mo.
However, if you were not using a monthly credit package to pay for your monitoring, you may need to take some action. Your monitoring jobs have not been disabled, but they are currently running free of charge. Over the next few weeks, we will be contacting each customer who is in this situation and work with them on finding a monitoring package that fits their needs.
In some cases, a customer may want to transfer credits previously purchased and apply them toward a monitoring job. We are more than happy to convert purchases over this way and will work with each customer that is interested in doing this.
In our previous pricing model, each concurrent RBU cost 10 credits/hour while each concurrent VU cost 1 credit/hour. Customers could buy credits at various rates ranging from 30 cents/credit down to 7.5 cents/credit, depending on the size of the purchase. Purchases of $5,000 or larger were eligible to purchase additional credits of any amount at the original price (usually 7.5 or 10 cents).
Under the new model, there are no more credits. Rather, we have converted them all to “cloud dollars” by multiplying the remaining credits in your account by the price paid for the credits. For example, if you purchased 1000 credits for $300 and had 300 at the time of the switch, you would be left with $90 in cloud dollars.
These cloud dollars can be used to pay for load tests. The price for each load test will be quoted at the bottom of the “schedule load test” page. If the test is larger than the load test package that you currently have, we will quote a higher price that is based on the old pay-as-you-go pricing plan.
If you’re used to the old credit system, the easiest way to think of cloud dollars is that they reflect a credit price of just 1.5 cents – a 20X reduction from the 30 cent price most of our customers were using.
This means that a 100 RBU test that used to cost $300 now costs only $15 when you are subscribed to a qualifying load test package. For customers who purchased a larger credit package and therefore bought in to a smaller credit price (ie: 15 or 10 cents/credit), the savings are not as large but are still very significant (ie: 10X or 6.67X, respectively.)
Absolutely.
Simply keep buying cloud dollars just like if they were the old credit system. When you schedule a test, you’ll be quoted a price to run the test. You may be presented with a note explaining that the per-test fee would be lower if you were using a load test package, but you are under no obligation to sign up for such a package. This is especially ideal for anyone who does infrequent load testing.
The price of test will be based on what your original credit price was. This means that a 100 RBU test might cost $228.75 for one customer at the 30 cent price point and $78.75 for another customer at the 10 cent price point. However, both would only pay $15 if they were subscribed to a load test package.
Yes. We can apply any unused cloud dollars towards load testing or monitoring packages. Just contact us and we can get the process started.
For monitoring, you can do up to 3,000 page views per month for free.
For load testing, you can run up to 25 RBUs and 100 VUs at any time without paying for a load testing package. However, you will have to pay the raw cloud fees for each test. These fees are 15 cents per RBU hour and 1.5 cents per VU hour (ie: 25 RBUs = $3.75 and 100 VUs = $1.50). Each account comes with 10 cloud dollars to get you started, but you can purchase more at any time.
]]>Our load testing service now has two types of fees:
The base fee is where BrowserMob makes money as a company. We offer a weekly, monthly, or annual price – with significant discounts for annual purchases. We also offer three levels of service: Basic, Advanced, and Professional. Each package includes enough “cloud dollars” (see below) to complete most testing projects.
The cloud fee represents our raw costs to run in the cloud. This means after you pay the base fee, you are free to run as many tests as you’d like with no extra markup. For example, $15 in “cloud dollars” gets you a 100 browser test or a 1,000 virtual user test.
The most exciting thing about this model is that if your tests require no more than 25 browsers or 100 virtual users, there is no base fee. This means that after you use up the complementary 10 cloud dollars with each signup, you can keep using the service by simply covering the raw fees incurred with each test.
We’re also excited to offer any of our Basic, Advanced, or Professional packages to non-profits and open source organizations free of charge.
You can review all the load testing packages here.
Our monitoring pricing is now based on a very predictable number of “monthly page views”. For example, if you want to check a page every 15 minutes from one location, you would require 2,880 page views:
2,880 = 1 page view x 60 min / 15 min x 24 hrs x 30 days
The best part about this new model is that if you need 3,000 or fewer monthly page views, the service is 100% free. For more frequent monitoring, our monthly plans start at $49. You can review the monitoring pricing here.
We will be posting a follow-up Q&A for all of our existing customers shortly. It will explain the transition process and what options are available. At a high level, we expect 99% of our customers to immediately get a better value. However, if any existing customer prefers to stick with the old model, we will gladly honor it.
Update: Please see our FAQ entry that discusses the changes for existing customers.
]]>While this release doesn’t have many new user-facing features, it does clean up several bugs. More importantly, however, is that Adam Goucher and Jérémy Hérault did some amazing work to lay the foundation for a plugin framework. This means that soon you’ll see Selenium IDE plugins that further expand the Selenium IDE capability.
Jérémy is working on one such plugin, called Helenium (see proof of concept in action), that will allow you to do text matching against images and PDF files using optical character recognition (OCR). We’re also working on a plugin that will make it easier to upload scripts from the IDE to BrowserMob.
If you’re interested in how to build your own plugin, I recommend reading Adam’s blog, which has a series of recent posts on how the plugin framework works.
]]>Hot off the heels of Amazon’s announcement of a new US West Coast cloud data center, we are happy to report that you can now schedule load tests and monitoring jobs from this new location. Simply select the “San Jose, CA” location when scheduling tests.
In December, Selenium 2.0 alpha 1 was released. This release dramatically improves the realism and reliability of Selenium scripts. We’re proud to say that you can try out Selenium 2.0 support (but keep in mind it’s still in alpha) by simply changing your selenium script from this:
var selenium = browserMob.openBrowser();
To this:
var selenium = browserMob.openBrowser(true);
We will continue to keep BrowserMob up-to-date with all the latest happenings in the Selenium world, as well as donate our time and code back to the Selenium project. We also upgraded all the BrowserMob browsers to have the latest version of Firefox (3.5.7) and Flash.
We’ve also made scheduling a load test a lot easier. We now give you a realtime estimate of what the test will cost you, changing dynamically based on your test configuration. We also display tooltips explaining things like “location”, “ramp”, and “constant”. Finally, we’re really excited to have rolled out a “Run ASAP” option that will kick off the test as quickly as it can, usually within 10 minutes.
Our users love that they can write their scripts using JavaScript, so we decided to make working on that JavaScript code even easier. By utilizing the Bespin open source project from Mozilla, you now will see a rich text editor with code syntax highlighting. If for some reason you’re having trouble with it, you can always switch back to the plain text editor.
We’ve always been proud of our realtime load test charts, but some users had recently complained that they were too heavy-weight and were slowing down their browser. Of course, this is a perfect example of why performance in the browser is starting to matter just as much as performance on the server.
Responding to this complaint, we rewrote the charts from scratch, moving from YUI Charts (Flash-based) to Flot (Canvas-based). We hope you like them!
If you do advanced scripting, especially with virtual users, you’ll definitely want to take a look out the BrowserMob scripting API. We added a whole bunch of useful functions, including:
This makes sense: the faster the web, the more people will use it and be more likely to search and click on ads. In fact, in an interview with Web Pro News in November last year, Matt Cutts (head of Google’s webspam team) casually mentioned that in 2010 Google may give search ranking preference to sites that load faster. Om Malik went on to discuss whether this was a good idea or not.
More recently, Google opened up a new “Labs” feature in their Google Webmaster Tools that actually shows you the performance of your site as measured by the Googlebot. This move is further evidence that Google is indeed going to shift to making performance part of it’s ranking routine if it hasn’t already. See a screenshot of what you can expect to see:
In addition to the above chart, there is also some basic analysis and recommendations on how to improve performance, similar to what Page Speed and YSlow recommend. While it is a good start and provides the clearest view of how Google perceives your site’s performance, the data is still rather basic.
As such, when measuring and optimizing page load times we’d still recommend you work with a more in-depth tool such as Page Speed and YSlow. And, of course, for website monitoring and website load testing, we’d have to recommend BrowserMob 