Sauce Labs

A bit of sugar and parallelism for Rails and RSpec

Sean Grove — Wed, 10 Mar 2010 01:14:05 +0000

Even though we focus very heavily on full-stack acceptance testing for the rails world, we know other forms of automated tests are critical as well. Our rails developers here make pretty heavy use of RSpec unit tests, and it’s nice to understand how to run those in parallel as well.

If you’re looking at how to setup a rails and selenium testing environment, check out our last post.

Parallelize the specs

We’ll use the excellent parallel_specs to beat a bit of parallelism into our specs. It prepares a separate database for each test environment, groups the specs to divide amongst processes, and then starts up a rails environment with a separate database for each group of processes.

I’ll paraphrase the installation instructions for convenience.

Install the required plug-in/gem:

sudo gem install parallel
script/plugin install git://github.com/grosser/parallel_specs.git

Here’s the semi-ingenious point – yaml can interpret ERB, so we can pass in an environment variable to the database.yml specifying at launch which database we want it to connect to.

Open config/database.yml and add the following:

test:
  adapter: sqlite3
  database: db/xxx_test<% ENV['TEST_ENV_NUMBER'] %>.sqlite3
  pool: 5
  timeout: 5000

(You can of course replace xxx_ with your project name)
So for example, to have our tests run against the xxx_test2 database, we would use:

export TEST_ENV_NUMBER=2; rake db:test:prepare

But it doesn’t make much sense to invoke it manually. That’s what plug-ins are for! Let’s go head and create/migrate a few test databases:

export TEST_ENV_NUMBER=0; rake db:test:create; rake db:test:migrate;
export TEST_ENV_NUMBER=1; rake db:test:create; rake db:test:migrate;
export TEST_ENV_NUMBER=2; rake db:test:create; rake db:test:migrate;

Great, now you’re able to run your non-Selenium tests in parallel!

But what about Selenium tests?

Stay tuned for our article on Sauce Labs’ SpecStorm plugin, that allows you to run your Selenium tests in true parallel fashion with Selenium Grid or our very own Sauce OnDemand service.

Selenium IDE 1.0.5 is released!

Adam Goucher — Fri, 19 Feb 2010 22:22:30 +0000

One month (to the day!) after the release of Selenium IDE 1.0.4 was released, 1.0.5 is now available.

Download Selenium-IDE 1.0.5 now

This release fixes a couple really annoying bugs that were introduced in 1.0.4 and starts to build towards the future. Here are the highlights.

Self-hosting – Se-IDE will not longer be relying on addons.mozilla.org to be hosting updates. This gives the project more flexibility around when releases are available to users.
Back into main project – The code for Se-IDE was in a separate subversion branch than the rest of the current Selenium project which meant its visibility to other developers was lower and that changes were not easily propagated. Se-IDE now lives in the main project so will reap the benefits of the rapidly changing core.
User Formats work again – In 1.0.4 there was a bug which meant that while you could add a custom format it would not appear in the list. This has been fixed.
Version Display – The version of Se-IDE now appears in the title bar
Reload Extensions – A tweak was made to the ‘reload user extensions’ functionality introduced in 1.0.4 so the toolbar button only appears when you have turned it on.
Bool Preferences – Preferences used to only be string values, even when a boolean would have been a better choice. Well, now you have a choice about whether you want your preference to be a string or bool.
Plugins Pane – There is a new pane in the Options screen called ‘Plugins’. This will become the central means of managing plugins in future releases, but for now it doesn’t do much aside from listing which plugins have registered themselves with the (also new) addPlugin(id) API function.
Icons – You will see a couple icons now in Firefox for Se-IDE rather than the default ones. Such as in the main Extensions window and in the Tools menu

For the full release notes and who contributed what, see the Google Code wiki. Also on the Google Code site is the Issue Tracker for any problems you find. (Just make sure you tag them as ide.)

And as before, thanks to Sauce Labs for sponsoring my work on this release.

-Adam Goucher

Sauce RC 1.0 Released with Web UI for Selenium RC!

Santiago Suarez Ordoñez — Fri, 12 Feb 2010 17:25:25 +0000

We’re pleased to announce we just released the new Sauce RC!

Hoping you all find this useful, we created a web interface for anyone to configure Sauce RC from the browser, on the same machine or even remotely!

Here’s a video of Sauce RC in action. We hope you download and enjoy it!

Running Selenium RSpec tests for Rails 2.3.5

Sean Grove — Tue, 09 Feb 2010 20:03:24 +0000

This article will get you setup with the bare-minimum environment to run Selenium tests with RSpec, for automated, full-stack testing of Rails apps. Once that’s done, we’ll work on polishing it for a nicer experience. In this post you find amalgamations of mostly outdated articles from around the internet updated to work with a modern rails system.

It’s written on a clean environment courtesy of rvm, so you may have some of the gems already setup.

A new Rails app, RSpec, and rspec-rails

First off, we’ll start with a new rails project:

gem install rails --no-ri --no-rdoc
rails rspec_saucerc
gem install rspec
gem install rspec-rails

Edit config/environment.rb and add:

  config.gem "rspec", :lib => false, :version => ">= 1.2.9"
  config.gem "rspec-rails", :lib => false, :version => ">= 1.2.9"

This sets up all the required files for RSpec to get off the ground, but we need to integrate it into Rails:

script/generate rspec

That will add the rake tasks, create the appropriate directories, and basic files. Now let’s get Selenium up and running!

Selenium

We’ll need to create a separate selenium database. in config/database.yml, let’s add:

selenium:
  adapter: sqlite3
  database: db/selenium.sqlite3
  encoding: utf8
  timeout: 5000

To get Selenium running under our RSpec stories, we’ll need the Selenium gem (note the capital “S” — it’s case-sensitive). Let’s also make sure we have the sqlite3 gem installed, and then prepare an appropriate environment for our selenium tests:

gem install Selenium
gem install sqlite3-ruby
cp config/environments/test.rb config/environments/selenium.rb

Open config/environments/selenium.rb and remove the last line that reads:

config.gem 'rspec-rails', :version => '>= 1.3.2', :lib => false unless File.directory?(File.join(Rails.root, 'vendor/plugins/rspec-rails'))

Sauce RC and Selenium RC

Sauce RC will broker all communication between rails and any browsers you might be driving. Get it from our downloads page for Windows or Mac, or use Selenium RC for Linux, and start it up so we can run our tests.

Testables

We’ll need some fodder to test. Use the RSpec generator, then create and migrate the database:

script/generate rspec_scaffold person name:string age:string language:string
rake RAILS_ENV=selenium db:create
rake RAILS_ENV=selenium db:migrate

Startup our test server

Rails tests do not normally bind to a webserver, so there is no way for Selenium to access the frontend. As a temporary workaround, we’ll manually invoke a Rails server using the selenium environment:

script/server -e selenium

Our spec helper

We will be using the Selenium gem, so let’s add it to the spec helper:

gem "selenium-client"
require "selenium/client"
require "selenium/rspec/spec_helper"

Example story

We’ll gloss over which types of tests should include selenium front-end testing for now, and just say it belongs in integration tests. Let’s run an example story that will fail the first time through, and we’ll then fix it.

Put the following in spec/integration/people_spec.rb:

require 'spec_helper'
describe "People" do
  before(:all) do
    @verification_errors = []

    @browser = Selenium::Client::Driver.new(
      :host => "localhost",
      :port => 4444,
      :browser => "*firefox"
      :url => "http://localhost:3000",
      :timeout_in_second => 90)

    @browser.start
  end

  before(:each) do
    @browser.start_new_browser_session
  end

  append_after(:each) do
    @browser.close_current_browser_session
    @verification_errors.should == []
  end

  it "should create a new Person with valid input" do
    @browser.open "/people"
    @browser.click "link=New person"
    @browser.wait_for_page_to_load "2000"
    @browser.type "person_name", "Jason Huggins"
    @browser.type "person_age", "26"
    @browser.type "person_language", "Albanian"
    @browser.click "person_submit"
    @browser.wait_for_page_to_load "30000"
    @browser.is_alert_present.should be_true
  end
end

Let’s try it out:

rake spec:integration
F
1) 'People should create a new Person with valid input' FAILED
expected false to be true
./spec/integration/people_spec.rb:22:

Finished in 7.443904 seconds

1 example, 1 failure

It’s failing as we expected it to (there shouldn’t be a javascript alert on submit). Let’s change the last line:

    @browser.is_alert_present.should be_false

… and try our test again:

rake spec:integration
.

Finished in 7.079062 seconds

1 example, 0 failures

Looks good! We finally have a working rails project with rspec runners and Selenium.

Weaknesses

Although we’ve got Rails, RSpec, and Selenium all working together, the relationship is not harmonious.

Automated tests in rails are not meant to serve the outside world, and as such they don’t bind to a webserver/port. Tests are run within the same process, which makes them quite fast, but prevents us from using browser-based frontend tools such as Selenium. To get around that, we manually fired up a rails server instance with the selenium environment, but this is messy for a number of reasons:

We have to manually start/stop the test server that selenium wants to access
We have to manually create the selenium environment’s database
We have to manually migrate the selenium environment each time there’s a schema change
Worse, we have to manually reset the database after each run
Running Selenium tests in serial is slow

We‘ve developed a plugin to automate some of these issues, called SpecStorm. We’ll go over installing it to get the most out of your tests (including running them in parallel) in the next post.

Notes:

Selenium matchers are case-sensitive: @browser.click “link=New Person” won’t match a link with “New person”

Sauce OnDemand supports Firefox 3.6

Miki Tebeka — Wed, 03 Feb 2010 19:55:57 +0000

Following our zero-day support for Firefox 3.6 in Sauce RC, we are pleased to announce that Sauce OnDemand now supports Firefox 3.6 as well.

Use the following settings in your JSON configuration:

{
    "username" : "SAUCE-USER-NAME",
    "access-key" : "SAUCE-API-KEY",
    "os" : "Windows 2003",
    "browser" : "firefox",
    "browser-version" : "3.6."
}

Death to Internet Explorer! Long Live Internet Explorer!

Jason Huggins — Tue, 02 Feb 2010 23:29:10 +0000

There has been a lot of negative press surrounding Internet Explorer. This week, Google sent an email to Google Apps administrators explaining their intentions to phase out support for Microsoft Internet Explorer 6.0. Just last month, French and German governments asked citizens to stop using the Internet Explorer browser (no matter the version). Yes, there is even a blog titled “Why IE Sucks.” (Parental discretion advised).

We’re of mixed opinion on the matter. Do we believe people should dump Internet Explorer? Yes! In fact, I outline three reasons why below. At the same time, due to subpar browsers like Internet Explorer, the value of cross-browser functional testing using Selenium is that much more important and relevant. Call it job security for testers.

I have no secret reason to pan Internet Explorer. I work closely with all browser flavors including Internet Explorer, Firefox, Safari, Chrome and Opera. Though I am a former Googler, I have no built-in bias to push one browser over another. And even though I recommend users not use IE, Sauce Labs supports and will continue to support IE in our downloadable products, Sauce IDE and Sauce RC, and our cloud testing service, Sauce OnDemand. With that context, here are three reasons why I feel you should dump Internet Explorer.

1) IE is a virus magnet
Virus authors target IE/Windows users because IE still has the majority market-share. If they targeted Opera users, even if they had 100% success rate, they’d still not have a massive impact. In this case, following the herd can get you more hurt. Meanwhile, creators of Firefox and Chrome argue that their browsers are more secure than IE in that their open source nature makes it easier for security analysts to inspect the source code and fix security holes. Of course, the best way to test that argument will be when Firefox or Chrome reaches majority market share (already true in some places) and virus authors target those browsers, too.

2) IE is slower
IE 6,7,8 are all significantly slower compared to Firefox, Chrome, and Safari when viewing very dynamic sites (like Google Maps or Gmail). Google Chrome is arguably the fastest. Among the “alpha geek” set, Google Chrome (and the non-Google branded Chromium version) is becoming their every-day browser because it is so fast.

3) Web-standards and innovation
MS is becoming irrelevant to the conversation. Microsoft is not keeping up, so users are missing out on new features. HTML5 brings offline storage, and new kinds of applications that leverage the new

Selenium ToTW: CSS Selectors in Selenium Demystified

Santiago Suarez Ordoñez — Fri, 29 Jan 2010 22:42:42 +0000

Following my previous TOTW about improving your locators, this blog post will show you some advanced CSS rules and pseudo-classes that will help you move your XPATH locators to CSS, a native approach on all browsers.

Next sibling

Our first example is useful for navigating lists of elements, such as forms or ul items. The next sibling will tell selenium to find the next adjacent element on the page that’s inside the same parent. Let’s show an example using a form to select the field after username.

Let’s write a css selector that will choose the input field after “username”. This will select the “alias” input, or will select a different element if the form is reordered.

css=form input.username + input

Attribute values

If you don’t care about the ordering of child elements, you can use an attribute selector in selenium to choose elements based on any attribute value. A good example would be choosing the ‘username’ element of the form without adding a class.

We can easily select the username element without adding a class or an id to the element.

css=form input[name='username']

We can even chain filters to be more specific with our selections.

css=input[name='continue'][type='button']

Here Selenium will act on the input field with name=”continue” and type=”button”

Choosing a specific match

CSS selectors in Selenium allow us to navigate lists with more finess that the above methods. If we have a ul and we want to select its fourth li element without regard to any other elements, we should use nth-child or nth-of-type.


Heading
Cat
Dog
Car
Goat

If we want to select the fourth li element (Goat) in this list, we can use the nth-of-type, which will find the fourth li in the list.

css=ul#recordlist li:nth-of-type(4)

On the other hand, if we want to get the fourth element only if it is a li element, we can use a filtered nth-child which will select (Car) in this case.

css=ul#recordlist li:nth-child(4)

Note, if you don’t specify a child type for nth-child it will allow you to select the fourth child without regard to type. This may be useful in testing css layout in selenium.

css=ul#recordlist *:nth-child(4)

Sub-string matches

CSS in Selenium has an interesting feature of allowing partial string matches using ^=, $=, or *=. I’ll define them, then show an example of each:

^=	Match a prefix
$=	Match a suffix
*=	Match a substring

css=a[id^='id_prefix_']

A link with an “id” that starts with the text “id_prefix_”

css=a[id$='_id_sufix']

A link with an “id” that ends with the text “_id_sufix”

css=a[id*='id_pattern']

A link with an “id” that contains the text “id_pattern”

Matching by inner text

And last, one of the more useful pseudo-classes, :contains() will match elements with the desired text block:

css=a:contains('Log Out')

This will find the log out button on your page no matter where it’s located. This is by far my favorite CSS selector and I find it greatly simplifies a lot of my test code.

Tune in next week for more Selenium Tips from Sauce Labs.

Sauce Labs created Sauce OnDemand, a Selenium-based testing service that allows you to test across multiple browsers in the cloud. With Selenium IDE and Selenium RC compatibilities, you can get complete cross-platform browser testing today.

Selenium ToTW: HTTPS and self-signed certificate exceptions

Santiago Suarez Ordoñez — Fri, 22 Jan 2010 18:02:32 +0000

This week in our support email, help@saucelabs.com, we’ve seen a considerable number of cases of problems with Firefox and the Invalid Certificate warning thrown when a development environment is using the production’s certificate for HTTPS URLs, which causes the browser to wonder about the website’s identity.

Warning displayed by Firefox 3.5 in this situation

If you do much browser-based testing, you have surely dealt with this situation more than once. In the manual testing world, you just tell your browser to add an exception for that certificate and ignore further errors, and you can forget about seeing that annoying warning anymore.

But when you move on to automation in the Selenium world, things are not always that simple. In most browsers, adding a certificate exception will work, as Selenium shares the same session as the user and will find the exception as you do. In Firefox, though, Selenium RC creates a special profile each time the browser is started and there’s no trail of the user settings in it.

The workaround for this problem is to create your own Firefox profile, with the specific certificate added on it by hand, and then tell Selenium to launch the browser based on that profile.

Another interesting approach, the one we take at Sauce Labs, where we can’t do this kind of trick, because we just don’t know which certificate the user will need before their test starts, is the use of RCE (Remember Certificate Exception), which is a plugin that will automatically detect the warning and make the browser go through it, returning the control to Selenium after 4 or 5 seconds. Notice that if you use this approach, you will need to make sure your tests will tolerate this 5 secs additional delay to open the page.

Note: We currently support RCE in our Firefox 3.0 machines, and will be porting this extension to Firefox 3.5 soon.

You can find more info about RCE on its plugin page or in the author’s blog post.

Update: Adam Goucher, one of the big minds in the testing world just wrote a blog post that we couldn’t have written better ourselves about the first and most important advice regarding HTTPS and testing: Do yourself a favour and don’t test using HTTPS

Zero day Firefox 3.6 support in Sauce RC, Selenium RC

Sean McQuillan — Fri, 22 Jan 2010 02:23:49 +0000

Today Sauce Labs released a new version of Sauce RC supporting Firefox 3.6, which was released by Mozilla this morning. Sauce Labs also sponsored work on Selenium RC to support Firefox 3.6. In under 12 hours, Sauce RC has added support for a brand new browser, allowing customers to keep their websites on the cutting edge of browser support.

Sauce RC is Sauce Labs’ commercially supported version of Selenium RC that includes simplified install and administration. Sauce RC is backed by a dedicated team of engineers to ensure bugs are fixed promptly and new browsers are supported.

Congratulations and thanks to Adam Goucher for his effort in making this happen in record time.

Ridiculously easy cross-browser testing with Sauce IDE

Jason Huggins — Wed, 20 Jan 2010 15:17:07 +0000

Riding on the coattails of yesterday’s official Selenium IDE 1.0.4 release, we’re very excited to announce Sauce IDE 1.0 (beta 1). Download it here. Sauce IDE is Sauce Labs’ certified, enhanced and commercially-supported version of Selenium IDE.

With Sauce IDE, we’ve made cross-browser cloud testing ridiculously easy. After you record a test in Sauce IDE, you’re only one click away from replaying that test on any major browser on Windows or Linux. Sauce IDE allows users to tap directly into our Sauce OnDemand cloud-hosted Selenium service while getting access to a sophisticated Selenium test infrastructure without having to build and maintain the infrastructure yourself.

Another really cool feature is video recording and playback. With every browser test you run with Sauce IDE on the Sauce OnDemand cloud, you can view or download a full screen video of your test.

Sauce IDE is a plugin for Firefox, and we’ve also included Firefox 3.6 support in our latest release.

Here’s a quick screencast showing how easy it can be to start running cross-browser tests in the cloud.

- Jason Huggins