Latest Telerik Test Studio Blog Posts

Webinar Series: Getting Started With Web UI Automation

Jim Holmes — Tue, 22 May 2012 14:20:53 GMT

I mentioned earlier that we’ll be hosting a three day “lunch and learn” format webinar designed to help folks new to web UI automation get off on the right foot. We’ll be covering lots of fundamental concepts to help you and your team succeed with automation instead of falling in to the pit of despair.

You can start signing up now for the webinar! You’ll only need to sign up once, and you can join us for any or all of the sessions. The webinar is a three day series, with each day’s sessions running somewhere between 30 and 45 minutes. Lots of Q&A is encouraged!

I’ve got a great lineup for content already (you can see some high-level points in my previous post), but as always, I’d love to address specific problems you and your team(s) have been seeing in web automation.

What sorts of topics would you find helpful to have addressed in this series?

Load Testing Webinar is Live on Telerik TV!

Jim Holmes — Fri, 18 May 2012 13:30:02 GMT

Did you miss our first-ever webinar on the new Load Test functionality we just released in Test Studio 2012 R1?

Fear not! We’ve just uploaded a recording of the webinar to Telerik TV. I hope you find it useful and informative!

If you’re not already using Test Studio, why not go grab a fully functional trial download?

Load and Performance Testing; What’s the Difference?

Phil Sams — Wed, 16 May 2012 22:35:55 GMT

The difference turns out to be a lot! To start with load and performance testing goals are very different.

Load testing goals usually seek to answer questions regarding system capacity, throughput, and client response times for multiple concurrent users. For example, how many concurrent users can the system successfully accommodate? Or how many transactions per second can the system successfully process? Or can the system successfully scale to 1,000 users? Whereas performance testing goals seek to answer questions regarding server processing, and client response times for a single user. The primary goal of performance testing is to tune the system to achieve maximum performance, [e.g. execute a given amount of work (transactions) in the least amount of time (transactions per second (TPS)].

Load and performance testing are both challenging; however, performance testing is the easier of the two because only one user is tested. Whereas in load testing there are hundreds or thousands of concurrent users tested.

Additionally the implementation of performance testing is much less complicated because, again of the fact that only one user is being tested versus hundreds or thousands for load testing. Also, each user type in load testing must have a scenario designed, implemented, and modified, to execute and simulate live users as accurately as possible.

When modifying and enhancing test code for load testing the test engineer must consider concurrent users and adjust the test code accordingly. This will mean modification of session IDs, user form data, and input data, whereas in performance testing there is only one user and typically no session IDs, etc., need to be handled.

Because the performance test only involves one user, it is typically a much shorter test than a load test. Load testing involves hundreds or thousands of concurrent users and is usually a much longer test and thus much more data to be collected and saved.

When it comes to results analysis and correlation of test data, the performance test data volume is much less then the volume of load test data. Again, because performance testing executes only one user, thus there is much less data to be analyzed for performance vs. load testing, even though the same or similar metrics are collected.

During the remediation phase of testing, performance testing is a much smaller problem space than load testing. With remediation of performance tests, typically metric ‘outliers’ are identified, then the engineer drills down to determine root cause. Load testing remediation is not that straight forward or easy. Load testing results may show errors. However, there may be bottlenecks which are more difficult to identify and more difficult yet to remediate. So, it turns out there is a great deal of difference between load and performance testing.

Test Studio, from Telerik, has long been capable of performance testing. Further, any functional test can also be used as a performance test. To execute a performance test in Test Studio, there are just three steps: 1) Create or select an existing web functional test that satisfies the test scenario criteria, (e.g. user actions, and or transactions to be executed), and open that test, 2) configure performance test properties such as location to store results then select the computer to collect performance metrics from, 3) execute (run) the performance test.

With the latest Test Studio 2012 R1 release in April, load testing has been added to the capabilities of Test Studio!

Through load and/or performance vulnerabilities of software comes financial vulnerabilities of the company, which both directly affects company financials and customer satisfaction. Test Studio can help mediate these risks and reduce vulnerabilities.

Testing Summit Highlights Video Posted!

Jim Holmes — Wed, 16 May 2012 17:00:58 GMT

Last week I wrote a quick summary of the Telerik Testing Summit.

I’m happy to announce we’ve got a short video posted up showing some of the highlights of the discussions during the first day of the conference:

If you’re not already, why not follow our Test Studio YouTube channel?

Hold The Dates! June 26-28: Getting Started with Web Automated Testing

Jim Holmes — Tue, 15 May 2012 17:48:20 GMT

Are you completely new to functional web testing? Don’t know where to get started? If so, please mark off your calendar for a three day series of webinars June 26th, 27th, and 28th. I’ll be hosting these webinars from 11AM – 12PM Eastern US.

This series is targeted to individuals or teams just getting started with web automation—although more experienced folks may learn a few things too!

I’m still fleshing out the agenda and content, but here’s a few of the things you can look forward to learning about:

Automation’s purpose/fit in your overall quality and testing strategy
What a web page looks like under the covers, and how Test Studio interacts with it
What a page’s lifecycle is, and why you need to know about it
Understanding element locators, and why it’s so important to understand them well
Dealing with dynamic content like AJAX
Designing and implementing great test cases
Why you’ll need close collaboration with your entire team
Learning to build a backing framework
Keeping your tests reliable and lean
Learning why it’s important to not test everything—and learning the sorts of things to not test!
Data driving tests to flesh out coverage
Working with baseline data sets
Understanding teardown and setup actions
Working with source control
Surviving and keeping your sanity as your test suite grows in size

These webinars are intended to be “Lunch and Learn” sessions: informal but with a lot of information flowing. I’ll be leaving plenty of time for questions, too!

More information, and registration links, will follow soon. For now, just block off those dates if you’re interested!

Got a Bug? File a Report

Jim Holmes — Tue, 15 May 2012 17:30:54 GMT

Telerik’s TeamPulse is a terrific project management tool for boosting communication in and between projects. We’re constantly looking for ways to better integrate Test Studio with TeamPulse to clearly communicate quality metrics around your automated tests back to the projects.

We’ve had integration tying tests to acceptance criteria in TeamPulse for some time, but with Test Studio’s 2012 R1 release we’ve kicked up the integration another notch: You can now push bug reports from Test Studio directly to your TeamPulse projects from anywhere you can examine a failing test: the test explorer, test results calendar, or test list details.

(BY THE WAY: We’ve also got the same bug reporting functionality for Team Foundation Server. I’ll post on that one later!)

The user interface is the same at any of these places in Test Studio:

From the failure details screen you click the Submit Bug icon at the bottom, select where you want to submit the bug report to, edit the title and description as desired. Checking the “Attach failure details” checkbox will create a zip file with the exception details, screenshot of the UI at the point of failure, and enclose the page DOM at the moment of failure. That zip gets attached to the bug report in the target system—folks working triage and bugfixes will have everything right at their fingertips!

You can also use Test Studio Explore to file bugs during your exploratory testing sessions! (Read more about the Explore tool in a previous post.)

I’ve just posted up a short video walking through all of this. Please have a look at it. I think you’ll find it extremely useful!

Load Testing and Performance Webinar 5/16

Jim Holmes — Fri, 11 May 2012 19:38:51 GMT

Join us Wednesday, 16 May, at 11am US Eastern time, to learn how Test Studio’s performance testing and just-released load testing features can help you detect bottlenecks in your system’s performance and answer business-critical questions about your site’s ability to handle large amounts of traffic.

This webinar will focus solely on Test Studio’s Performance and Load Testing features. We’ll be drilling down in to some great specifics of how these features can help you respond to questions from management like “Will our site be able to support traffic after this new product’s launch?” and “Are our customers happy when they’re using our site during normal and high traffic times?”

You’ll learn how you can use one functional test and run it as a performance test to give you great breakdowns for information about specific use cases – and tie in detailed metrics from Windows Performance Monitor counters.

You’ll also discover how easy it is to construct complex, realistic load test scenarios. We’ll also walk through analyzing results from those tests using Test Studio’s clear, understandable reports!

Interested? Why not head over and register for the webinar right now?

Want to Learn More About Test Studio for iOS?

Jim Holmes — Tue, 08 May 2012 19:13:08 GMT

Are you working with iOS applications? Do you need to get some automated tests around your native (UIKit), web, or hybrid applications?

Have a look at Test Studio for iOS! I’ve written about it beforehand, and tomorrow I’ll be hosting a webinar giving an introduction to our great testing tool for apps on iPhones, iPads, and iPod Touches.

I’ll show you how to set up your Xcode projects, and run through a few highlights of working with Test Studio for iOS. The webinar is at 11AM Eastern US.

Interested? Why not go register for the webinar?

Vote for us in Code Project Reader Choice Awards 2012

What's Hot — Tue, 08 May 2012 10:32:55 GMT

Telerik Test Studio has been nominated in the Debugging & QA category in this year’s Code Project Readers’ Choice Awards. If you enjoy using Test Studio; appreciate your experience with our customer & technical support, please, do cast your vote for us!

Telerik has been nominated in 14 categories. If some of your favorite products are in that list you can vote for them until Friday, May 11.

Thank you for your support!

What does a good test case look like?

Phil Sams — Mon, 07 May 2012 18:55:08 GMT

A good test case is neither too small nor too big. A good test case stands alone, sets up pre-test conditions, exercises the application under test (AUT), verifies one thing, reports the results of the verification, then if applicable, cleans up (usually data) and reports the overall test case result; pass or fail.

Telerik Testing Summit Wrapup

Jim Holmes — Mon, 07 May 2012 14:18:14 GMT

If you were following Twitter last this last weekend (4 and 5 May) you may have seen a number of Tweets tagged with #TelSum. That was the Telerik Testing Summit, a peer conference held in Austin, Texas. Telerik welcomed a number of thought leaders in the testing and software industry to Austin, Texas, for two days of discussions on the state of software development. More importantly, we as a group were focused on what we could do to try and improve things.

Telerik was represented by myself and Phil Sams, plus Holly Bertoncini, our event organizer. Joining us were Charley Baker, Jeff Morgan, Lanette Creamer, Lisa Crispin, Marlena Compton, Dawn Test Code (yes, that’s her real legal name!), Matt Barcomb, Adam Goucher, and Chris McMahon. Austin locals Brett Pettichord and @rubytester joined us for various parts of the conference as well.

The summit was completely free-form in nature. We built the agenda Friday morning and adjusted it on the fly as we progressed. We had several hours of focused discussion Friday morning and afternoon, a site visit to Telerik’s Test Studio office in Austin, open space discussions Saturday morning, lightning talks, two hours working on the Weekend Testing project for Wikipedia, and a few hours closing out to summarize and create action items. We also had some practical sessions on things like using GitHub (fork, clone, fix, commit, push, pull), and a number of folks paired up to solve practical problems for their “day” jobs.

I mention action items, because one thing we felt critical was to ensure we left the summit with some concrete things we could do to improve, change, or elevate testing and software development.

I’ll be posting up more details on various things over the next few days, but here’s the rough agenda as it shook out:

Brainstorming topics

Discussions

Whole Teams (Benefits of, how to get teams working better, etc.)
UI Automation 101 (why so many don’t understand, what to do)
Big Frameworks (Why so many, how to understand differences)

Open Spaces

Data for automation
Visualizing QA activities
Page Objects
Spurring ideas for testing

Weekend Testing #27

We took two hours to test and debrief as part of the Weekend Testing Americas project.

Cheesy’s GitHub walkthrough

Lightning Talks

Lanette on Agile/cultural antipatterns. Great humor!
Me on It’s Not About You
Adam on Selenium v1 vs v2
Chris (and his ukulele) on getting folks moving as part of an icebreaker
Cheesy on requests for his Relish tool he’s evolving to handle reporting for Cucumber suites

As I mentioned, I’ll be posting up more details on all this over the next week or so. In the meantime, I thought it would be interesting to post up images of the sheets we used for brainstorming discussion topics. We took 20 minutes or so where each person wrote down items they were interested in discussing. Everyone had a few moments to describe their topics to the group, then we did a bit of consolidation. The group voted on topics to discuss, and off we went!

Without further ado, here’s the pictures of our lists.

Page 1

Page 2

Pages 3 and 4

Pages 5, 6, and 7

Page 8

And finally, page 9, the last one.

Chris McMahon was kind enough to post up his initial thoughts on the event here on his blog. He hits one thing that’s central to why I came to work at Telerik: The company’s culture of understanding we gain so much by focusing on industry-wide issues instead of just trying to push our products.

The conference/summit turned out extraordinarily well. We had some amazing discussions, everyone learned something, and we came out if it with many concrete things to work on over the next year. It’s going to be very exciting to see how everyone approaches their various action items!

Check out the new Exploratory Tool!

Jim Holmes — Fri, 27 Apr 2012 13:20:26 GMT

UPDATED: I goofed and listed Quality Center as an option for pushing bug reports to—but we only integrate bugs with TeamPulse and Team Foundation Server.

As much as I love them, automated tests (be they functional, integration, or unit) aren’t the end-all-be-all for your testing approach. Exploratory testing, particularly for any system of any consequence, is a critical part of your overall “Deliver Great Stuff!” strategy.

Getting your team freed up from repetitive regression testing empowers them to put their brains to use as they poke around the corners of your system. Exploratory testing gives you tremendous value as your team uses their knowledge of the overall problem domain and the system.

We’ve shipped a small, lightweight feedback tool with Test Studio 2012 R1 in order to help you out with quickly filing bugs during your exploratory sessions! Installing Test Studio adds the Test Studio Explore launcher to each of your installed browsers' toolbars. Here’s what it looks like in Firefox:

Click that button and you’ll find the Explore app appears – and it’s nicely done to stay out of your way while you’re doing your testing! Drag it around your screen via the grab handle. (That’s the part with all the dots on the left of the tool.)

With Test Studio Explore you can capture and annotate screenshots. Give it a good title and great description, too!

Once that’s done, push it to your bug tracking system. We now link directly to TeamPulse, and Team Foundation Server!

Test Studio Explore is a great lightweight tool to help you quickly document issues as you’re doing your exploratory. I’ve always been adamant that automation’s only part of your overall testing strategy. We’re here to help with your exploratory testing, too!

One Second ~ 7% Customer Conversion Loss

Phil Sams — Sun, 22 Apr 2012 20:58:45 GMT

What does one second of performance degradation in web and e-commerce applications cost?

It depends, but according to the Aberdeen Group (June, 2008 Benchmark report), customers are won or lost in just one second, based solely on the performance of the web or e-commerce application. Just this single second of delay for an online retail site equates to a “seven percent reduction in customer conversion” and thus lost revenue, seven percent!

Telerik Test Studio Performance test can help customers achieve a higher level of web or e-commerce performance, and mediate this risk.

StarEast 2012 Wrap Up

Jim Holmes — Fri, 20 Apr 2012 18:31:15 GMT

I and four other Telerikers (Holly, May, Dan, and Jasper) just returned back to our respective homes after a few days at StarEast. Our booth at StarEast was busy the entire time!

We all had an extremely productive trip in a great many ways. Holly’s our organizer extraordinaire, May (our UX dev) got a lot of insight to how our users see the product, Jasper had many conversations with users about our feature set and potential new features, and Dan spent a lot of time talking with folks to introduce them to the awesomeness that is Test Studio.

We spent a lot of time covering our brand new feature releases: iOS testing, load, exploratory testing feedback tool. Folks seemed very excited, and we got some great feedback on things, too.

I had two different talks to give: Making Distributed Test Teams Work, a regular breakout session I had selected for StarEast, and Real World Problems, Real World Load Testing, an Industry Technical Presentation (read: “vendor session”).

We also raffled off an iPad 3 at our booth. It drew a bit of a crowd…

Here’s our winner: Sathya Betha!

Thanks to the many attendees who stopped by and spoke with us. We really enjoyed chatting with you!

Test Studio 2012 R1 is Here (Soon!)

Jim Holmes — Tue, 10 Apr 2012 04:46:45 GMT

It’s that time of the year: the latest, greatest release of Telerik’s Test Studio is due out Wednesday, 11 April!

We’ve got a number of great features lined up for this release: integrated bug tracking, exploratory testing tooling, better support for HTML controls, and of course, the big feature: load testing!

Grab the release bits tomorrow, but be sure to stop by the Release webinar on Wednesday, 25 April at 11AM Eastern US. You’ll need to register first, but there are plenty of spaces!

Load testing can be a complex, intimidating subject. If you’d like some more information, I encourage you to check out the following resources:

My series on Performance/Load Testing
Steve Smith’s blog (lots of good development stuff, some great performance content)
Scott Barber’s blog. Prolific Tweeter & blogger.
Scott’s book Web Load Testing for Dummies.
Microsoft’s Performance Testing Guide for Web Applications

I look forward to seeing you at the release webinar!

PS: Working with developing apps on the iOS platform? Have you signed up for our Test Studio for iOS beta program yet?