Anyhow. The workshop was ok, and the main thing I would change was entirely my fault. The original idea was to show some code then have people work on their own services on their laptops. But I didn’t state that; at all, let alone not clearly enough. As a result, it was just me talking about code for the whole time. It still ‘worked’ but could have been much better I think as more of a workshop.

There were no slides, but we started talking about HTTP. If you are testing anything on the web, you really need to have a handle on how information flows back and forth between the bits.

• The HTTP Standard at the IETF
• HTTP Response Codes as cats

The response codes are especially important when doing web services since they tend to be how information is communicated. Especially with REST based ones.

And speaking of REST, it is one of the two types of web services that I talked about. For this I part I used the Flickr Photo Search API.

• Request format
• Response format

For automating this web service I used Py.Test with (so incredibly) awesome Requests module. Web service automation is one of those places where data driving becomes an excellent strategy. Because every runner does runtime data injection differently, one needs to look at Py.Test’s parametrize documentation. Once you have your brain wrapped around how to data drive it, I suggest you look at driving it via an external means so you can update the test data without having to touch the script itself; csv, Python Database API Specification v2.0. When driving from a database, I try to reach into the actual database being used for testing to grab random data that meets the characteristics that I am trying to use. If I am using something like Hexawise to reduce my combinatorial complexity down I would store that in a csv.

Whether your data-drive it or not, you need to parse the response and decide whether or not you got the information you expected. Python comes with XML parsing tools, but are not quite a joy to work with. Beautiful Soup may not be a complete joy, but it is better than the built-in stuff and is pretty pythonic in its implementation.

All the scripts are available on Github but here is the ‘basic’ example.

import py
import pytest
import requests
from bs4 import BeautifulSoup
 
key = "994124a9812d7a63972e89fc2145b9bc"
secret = "d8761d47cb245888"
 
url = "http://api.flickr.com/services/rest/"
 
class TestPhotoSearch(object):
    def setup_method(self, method):
        self.payload = {
            # required for everything
            "method": "flickr.photos.search",
            # required for this method
            "api_key": key
        }
 
    def test_thumbtack(self):
        self.payload["text"] = "thumbtack" 
        print(self.payload)
        r = requests.get(url, params=self.payload)
        print(r)
        print(r.text)
        if r.status_code == requests.codes.ok:
            soup = BeautifulSoup(r.text, features="xml")
            if soup.rsp['stat'] == 'ok':
                assert(soup.photos["total"] > 0)
            else:
                pytest.fail(msg = 'Did not get OK from Flickr')
        else:
            pytest.fail(msg = 'Did not get OK from server')

Oh. And I disabled my API key so it won’t work by just cloning the repo without first getting your own key. Were these actually part of a ‘real’ project I would externalize the keys to some common file so all your scripts don’t have to change if the keys change.

The other category of web service we did was the SOAP/WSDL-y type. These are not nearly as fun to interact with but have a bunch of features like encryption and digital payload signatures which are appealing in certain situations. And it occurred to me that their popularity in Java/.NET is not unsurprising since they are static typed languages and WSDLs enforce the type of parameters passed around in the request/response.

My original idea was to use the same API for for both types of web service but that didn’t really pan out. So for this I used the Bing API, Version 2. The trick for dealing with this type of service is to limit how much SOAP Envelope construction you have to do by hand. To hopefully zero. In Python the module I use for this is Suds which actually has some tragically [newbie] unfriendly documentation. But seeing an example is useful. So here is the example.

import pytest
from suds.client import Client
 
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger('suds.client').setLevel(logging.DEBUG)
 
application_id = "0E100A06A5C6822E953B7F954C568BA6437FA918"
 
class TestBingSearch(object):
    def setup_method(self, method):
        self.client = Client('http://api.bing.net/search.wsdl?AppID=%s&Version=2.2' % application_id)
        self.SearchRequest = self.client.factory.create('SearchRequest').parameters
        # print(self.SearchRequest)
        self.SearchRequest["AppId"]= application_id
 
    def test_thumbtack(self):
        self.SearchRequest["Query"] = "thumbtack"
 
        adult = self.client.factory.create('AdultOption')
        adult = "Strict"
        self.SearchRequest["Adult"] = adult
 
        source_types = self.client.factory.create('ArrayOfSourceType')
        source_types.SourceType.append("Web")
        self.SearchRequest["Sources"] = source_types
        # print(self.SearchRequest)
        SearchResponse = self.client.service.Search(self.SearchRequest)
        # print(SearchResponse)
        assert(SearchResponse.Web.Total > 0)

Data driving this type of service is exactly the same as with a REST one so I didn’t write that.

At this point it is just about practice. And even if your application doesn’t have web services now, it will likely grow them at some point so learning how to do it useful. The programmable web is a useful resource for free APIs that you can mess about with.

• The video tag is a Media Element
• So is the audio tag
• Actually, the implementation of an audio tag is exactly the same as the base media element type
• The video tag has a viewable section so has dimensions aside from those specified as media elements
• Interaction with the video and audio element is done entirely through JavaScript
• Which means that there is bound to be differences between browsers as they interpret the standards slightly different
• There are also a growing number of non-native implementations of players which override the native implementations and will need their own WebDriver implementation

Rather than paste the entire code for dealing with HTML5 media elements, I’ll just explain some of the important parts.

I hope to convince David Burns to include this as part of the Python WebDriver bindings as part of the support package (I suspect I need to write tests for it and document it first) so I took a queue from the Select class in creating the constructor.

class Video(HTML5Media):
    def __init__(self, webelement):
        """
        Constructor. A check is made that the given element is, indeed, a VIDEO tag. If it is not,
        then an UnexpectedTagNameException is thrown.
 
        :Args:
         - webelement - element VIDEO element to wrap
 
        Example:
            from selenium.webdriver.support.ui import Select \n
            Video(driver.find_element_by_tag_name("video")).play()
        """
        if webelement.tag_name.lower() != "video":
            raise UnexpectedTagNameException(
                "Video only works on <video> elements, not on <%s>" % 
                webelement.tag_name)
        self._el = webelement
        self.changed = False

My initial thought was to include this sort of tag inclusion in the find_by_* methods but the diversity of third party players made me rethink this.

Next, remember than control of these elements is through JavaScript. This means that that any WebDriver control is going to be through the JavaScript Executor interface. One feature of the executor is that you can pass in a WebDriver WebElement object as an argument and it is available [somehow] to the JavaScript being run.

@property
@stale
def loop(self):
    return self._el._parent.execute_script("return arguments[0].loop", self._el)
 
@loop.setter
def loop(self, value):
    if isinstance(value, bool):
        self._el._parent.execute_script("arguments[0].loop = arguments[1]", self._el, value)
        self.changed = True
    else:
        raise ValueError("value needs to be a boolean")

A few things to point out…

• Script execution is done at the page, not element level so have to use _parent which is the actual driver instance
• When you manipulate (or have manipulated) an element in WebDriver, normally it will trigger a Stale Element Exception, but the manipulations this way were not triggering it, so I had to cobble it in with the self.changed stuff
• Rather than have a check in every property I created a @stale decorator to do it

So what does it look like in a script?

from selenium import webdriver
 
driver = webdriver.Firefox()
 
driver.get("http://html5demos.com/video")
e = driver.find_element_by_css_selector("video")
v = Video(e)
v.play()
print("Ended?: {0}".format(v.ended))
 
driver.quit()

Easy-peasy.

As I mentioned, I hope to get this into the official distro, but until then it is up on github

More for myself than anyone else. If you want to see the Selenium IDE debug log set extensions.selenium-ide.internalLogThreshold to DEBUG

— Dave Hunt (@davehunt82) February 9, 2012

If you are developing a plugin getting the messages to appear is only half the story. You of course have to get them into the log in the first place. This is made slightly more complicated by the fact there are two different logging implementations in Selenium IDE. (here and here).

What I have done in a customer plugin I’m developing is take the first implementation and extracted it to its own file. With that file included in your project somewhere, here is how you use it.

var loader = Components.classes["@mozilla.org/moz/jssubscript-loader;1"].getService(Components.interfaces.mozIJSSubScriptLoader);
loader.loadSubScript("chrome://your_plugin/content/js/log.js", this);
 
function Foo() {
    this.logger = new Log("Foo");
}
 
Foo.prototype.do_something = function() {
  // do somethng
  this.logger.log(this.logger.DEBUG, 'a message!');
};

It is on my ‘to do’ list to back port this extraction into the main Selenium IDE project, but its down the list a bit.

As with when dealing with any file format, the first thing when parsing it is to make sure that it is in the right format. For XML we use DTDs or XML Schemas and for JSON we use JSON Schemas.

Determining that we want to do that is a lot easier than actually doing it. Especially in PHP. As usual, there are a handful of projects, all with a different level of compatibility/completeness with with the JSON Schema RFC, missing license information and not packaged for easy inclusion in projects. Ah. The joys of PHP.

One thing I didn’t want to do was write a schema from scratch; that is just no fun at all. So I looked to Jari Bakken’s HAR project for help. There he has schema for HAR files in JavaScript and in raw JSON. Awesome. I can recycle the latter one.

Except.

Remember I mentioned the varying degree of completeness in PHP validation projects? Ya. The use of $ref is the area they all seem to be lacking. Which meant a couple hours of copy-paste and debugging to remove those and make one huge schema file the results of which can be seen here.

Armed with a working schema I decided to use the php-json-schema project to validate my har file.

<?php
namespace PHPHAR;
 
require_once(dirname(__FILE__) . '/../src/Json/Validator.php');
 
class HARTest extends \PHPUnit_Framework_TestCase {
  public function testGoodHar() {
    $validator = new \Json\Validator(dirname(__FILE__) . '/../src/har_schema.json');
    $validator->validate(json_decode(file_get_contents('hars/good.har')));
  }
 
  /**
   * @expectedException \Json\ValidationException
   */
   public function testBadHar() {
     $validator = new \Json\Validator(dirname(__FILE__) . '/../src/har_schema.json');
     $validator->validate(json_decode(file_get_contents('hars/bad.har')));
   }
 
}
?>

What I like about this particular schema project is that you load the schema once and then validate any number of time with it. Of course, it uses a different license than the one I use for my stuff and is distributed by composer and not pear, but those can be worked around. In the meantime, I hope the schema is useful to people.

Implicit waits are where an upper bounded polling action happens looking for an element in the page before throwing an exception saying it wasn’t there. This would seem like a good thing as you will have less scripts failing due to synchronization problems which in turn means that you have to spend less time maintaining and repairing scripts.

But.

What it is actually doing is masking synchronization problems.

When I am writing scripts I want the synchronization to be as tight as absolutely possible. Taking a line from the Tao of Python, Explicit is better than implicit. Disabling implicit waits in an existing script set often leads to two things

• Wholesale breakage due to places you were being lazy with the synch
• Some bug reports with the application because you now notice which things are appearing when, in what order and in how long

One of the knocks against automation is that, unlike humans, it doesn’t have emotions. Confusion, frustration, etc. What you can teach your scripts to be though is impatient. And you do that by disabling implicit waits.

Writing automation with explicit waits is without question more work and forces you to not take convenient shortcuts. The resulting code however will be tighter, run more efficiently and be half a step closer to the end-user’s perspective than without.

All that said, when automating against a moving target implicit waits can save the day. The trick though is to continually lower the wait duration until it is off. Fixing problems as you go.

[1] It was actually billed as a vendor talk and is not a slag on Jim’s other talk at the conference which was very un-vendor-like

• 1:40 – Why RC? – Now, the decision between RC and WebDriver is much easier. Use WebDriver unless you have a large investment in RC. And, well, even then, just stop writing RC scripts and start using WebDriver.
• 2:40 – Dropping Perl and PHP – It wasn’t for anything malicious; more that no one stood up and said that they wanted to maintain drivers for it. No maintainer, no driver.
• 6:05 – Flags – I personally would also add in either a ‘shallow’ or ‘deep’ tag. She uses ‘smoke’ or ‘regression’ to indication the same, but I like things that are logical opposites.
• 11:11 – Why the modules directory? – Why not? I could have called it ‘lib’ or ‘stuff’ or something else. But it holds modules (and packages…) so modules it is
• 12:30 – Asserting in loops – For a script like this, you really want a soft assert (verify) rather than a hard assert. The way the script is written here, as soon as a key is missing the script will end and the remainder will not be checked
• 15:08 – Link locators – See death to visible content as locators in automated tests for what Mary Ann is talking about
• 16:32 – Locator Generation – Re Eliza’s question, you the scripter need to generate the locators. One of the knocks on Se-IDE is that it tries to generate locators but they can be pretty darn brittle.
• 18:04 – files dir – I would put this in support/*
• 19:42 – Screenshots – The commands that get snapshots taken in them can be seen in SaunterSelenium.py
• 21:30 – Viewing Screenshots – An idea would be to use a photo album tool and point its root directory at the logs directory
• 22:55 – Can’t find things its not looking for – People should remember that automation will happily pass something that is ‘wrong’ that it isn’t expecting
• 27:40 – Directories – Yes. If you have 1000 tests and are taking screenshots you will end up with 1000 directories. Run it in ci; then zip up the directory and store it as an run artifact
• 31:40 – TestLink integration – It’s manual, but baby-steps. I could see a situation where it launches things and reports failures back to the system automatically
• 34:50 – FogBugz integration – FogBugz has a nice API, so things could get logged automatically that fail
• 41:15 – Testing the Test – Nice trick! pdb is black magic…
• 45:50 – Mailing List – Ask and you shall receive (first post moderation is enabled for spam trapping…)
• 45:55 – Future of Py.Saunter – I don’t expect it to go away since I use it. But its also open source…
• 46:00 – Future of RC – Details of this are starting to
• 46:30 – Docs no good? – Where do I need better information? These are the official ones

Thanks Mary Ann! Oh and I’ll be speaking at the July meetup.

Yes, the Singleton pattern has a poor reputation in some circles but it worked well for almost two years since my scripts tend to be built around ideas that include not running things in parallel, not using the runner to manage execution distribution (through the use of something like Se-Grid) and using a single browser per stack.

A couple different conversations with customers [and potential ones] have made me rethink the first idea. It turns out that running a single browser, in parallel is perfectly in my ‘acceptable’ idea set. In fact, its a pretty good idea. [I know. Duh.] You can still have a single browser (controlled from a config file) that is distributed through a CI server — or better still into something like Sauce Labs’ OnDemand service.

But.

That doesn’t work in a world where you get a browser from a Singleton. For that, you need to use Dependency Injection. That is where you pass into the constructor of an object what it needs to know in order to behave. In the Page Object context we pass in the reference that this particular script has to the Se server.

And while a lot of this can be change can be hidden at the framework level, there is some user-side code changes as well.

If you are using Py.Saunter 0.39 or newer with WebDriver, you will have to make the following changes. The Se-RC implementation remains using the Singleton approach. SaunterPHP users will notice that this is how their WebDriver Page Objects have always been written.

1. update conftest.py – conftest.py is a ‘magic’ file that gets used by Py.Saunter’s underlying runner (py.test) which can augment py.test’s behaviour at various points in the execution lifecycle. Remove the contents that came with the installation and replace them with the following.

   import py
    
   def pytest_runtest_makereport(__multicall__, item, call):
       if call.when == "call":
           try:
               assert([] == item._testcase.verificationErrors)
           except AssertionError:
               call.excinfo = py.code.ExceptionInfo()
       rep = __multicall__.execute()
       return rep

2. modules/tailored/webdriver.py – While I was reworking how browsers get launched I made it a lot easier to add or customize WebDriver methods by putting in proper inheritance rather than just having static class methods.

   Create a new file called modules/tailored/webdriver.py with the following.

   """"
   =========
   WebDriver
   =========
   """
   from saunter.SaunterWebDriver import SaunterWebDriver
    
   class WebDriver(SaunterWebDriver):
       """
       Modifications to the core WebDriver API are done in this class
       """
       def __init__(self, **kwargs):
           super(WebDriver, self).__init__(**kwargs)

   This class is where you would add methods to handle your special control behaviours.

3. de-static – As a result of SaunterWebDriver no longer being an object with a bunch of static methods on it, you need to change references from it to self.driver. So change

   SaunterWebDriver.is_element_present(locator):

   to

   self.driver.is_element_present(locator):

4. self.driver (1) – Each Page Object should now be created with with a driver instance as it starting parameter. self.driver is set on a script when the browser is opened.

   @pytest.marks('shallow', 'ebay', 'shirts')
   def test_collar_style(self):
       s = ShirtPage()
       s.go_to_mens_dress_shirts()
       s.change_collar_style("Banded (Collarless)")
       assert(s.is_collar_selected("Banded (Collarless)"))

   becomes

   @pytest.marks('shallow', 'ebay', 'shirts')
   def test_collar_style(self):
       s = ShirtPage(self.driver)
       s.go_to_mens_dress_shirts()
       s.change_collar_style("Banded (Collarless)")
       assert(s.is_collar_selected("Banded (Collarless)"))

   If you miss a location is will throw and exception about an incorrect number of arguments.

5. self.driver (2) – Page Objects likely already have self.driver being set in the __init__ method, but make sure that it now uses the passed in driver information.

   def __init__(self):
       self.driver = se_wrapper().connection
       self.config = cfg_wrapper().config

   vs.

   def __init__(self, driver):
       self.driver = driver
       self.config = cfg_wrapper().config

6. elements – since Element classes are created at object instantiation time they don’t have access in their constructor to the session information. But they do have access to it through their ‘obj’ parameter. Which means to access the browser in an Element class you need obj.driver. Here is the before and after from an Element that gets clicked.

   def __set__(self, obj, val):
       e = SaunterWebDriver.find_element_by_locator(self.locator)
       e.click()

   def __set__(self, obj, val):
       e = obj.driver.find_element_by_locator(self.locator)
       e.click()

This is unfortunately not a trivial amount of code to change, but it is pretty straight forward and puts things into a nicer place.

I expect to publish the version of Py.Saunter with this change on Tuesday morning and I converted two projects in order to get the steps figured out but if you run into any problems, please email me and I’ll see if I can help you out.

def pytest_runtest_makereport(__multicall__, item, call):
    if call.when == "call":
        try:
            assert([] == item._testcase.verificationErrors)
        except AssertionError:
            call.excinfo = py.code.ExceptionInfo()
    rep = __multicall__.execute()
    return rep

This ‘clever’ function took me almost two days to figure out how to do. And meant that I had to learn waaay too much about how py.test behaves behind the scenes so this post is partly to share those findings with others and partly [largely] for my own reference later.

First, background.

I first came across the notion or hard and soft asserts exporting code from Se-IDE. A hard assert is your normal one where if the condition fails the script stops immediately whereas a soft assert will collect failures and ultimately fail the script but will proceed as if nothing happened.

Normally this soft assert pattern is implemented like this where you create a list in the setup, catch any hard assert fails and append them to the list, then check that the list is empty in the teardown.

class SomeTest():
  def setUp(self):
    self.verification_errors = []
 
  def testSomething(self):
    try:
      assert(False)
    except AssertionError:
      self.verification_errors.append("Some soft assert failed")
 
  def tearDown(self):
    assert([] == self.verification_errors)

py.test’s workflow is a bit, erm, unique though so that doesn’t really work.

py.test treats each of these three methods as completely separate and atomic calls. Each call is implemented as a hook that can be overridden (pytest_runtest_setup, pytest_runtest_call, pytest_runtest_teardown). Due to implementation, the executing code knows only about its locally scoped information and not some of the meta information you might have access to in a pure UnitTest2 script — like current script status. (Likely explained wrong, but that’s what it ‘feels’ like.) That useful bit of information is calculated in the reporting step (pytest_runtest_makereport) which is called after each of the other three steps which means that we can’t make the pass/fail determination in teardown.

But we also cannot make the determination after teardown is complete, because it then looks like the failure is happening in the teardown — which is wasn’t; the ‘call’ step was what failed. Thus the ‘if call.when == “call”‘ condition. Once in there, we can check if any of our soft asserts failed (the list isn’t empty) and then add it to the ‘item’ (everything in py.test is an item) as if it had happened during the script execution.

(The __multicall__ argument is a bit of deep, dark py.test black magic I borrowed from their mailing list archives.)

If you drop this function in your conftest.py you will be able to make use of the soft assert pattern with py.test and have it report correctly. Just remember not to put the check in the teardown as well…

class SomeTest():
  def setUp(self):
    self.verification_errors = []
 
  def testSomething(self):
    try:
      assert(False)
    except AssertionError:
      self.verification_errors.append("Some soft assert failed")

For a lot of teams, getting started and getting all the infrastructure in place for automation is often harder and more time consuming than the actual automation. SaunterPHP addresses this problem for PHP through convention-over-configuration. These conventions include

• Page Objects
• Easy CI integration
• Sauce Labs OnDemand integration
• Dynamic suite generation through groups

SaunterPHP is Open Source and free to install and use. Support and Feature Requests are available for a cost.

But.

Since SaunterPHP is just launching I am looking for 1 (maybe 2) beta sites for the month of March. In exchange for bug reports and feedback I’ll waive bug fixing and support fees. Email adam@element34.ca for details.

x ='//td[@class="ql-account-requests-title_author"]/a[contains(translate(text(), "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"), translate("' +  material_row.get("author") + '", "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"))]/../../td[@class="ql-account-requests-pickup_location" and contains(text(), "' + branch_row.get("branch") + '")]/..//input[@class="ql-account-select-checkbox"]';

Yes. That is a 408 character XPath. And it is awesome.

Context.

This is from a BrowserMob script for a library site revamp. At the end of the script I needed to find the material I had reserved and cancel that reservation. The list of reservations is a tabular data and so was logically in a table. But the accounts are reused during the run so I needed to find the material by the author and title that was randomly chosen for this run. This meant that I needed to find a cell, go back up and over, find a cell, got back up and over and click a checkbox.

This is not a time machine; its a locator. It can’t go back.[1]

When you need a plain top-to-bottom structural locator then CSS Selectors should be your default approach, but it cannot go back up the DOM. XPath can.

CSS cannot also deal with the inevitable matching problems that come from using real-life data. And unfortunately Firefox’s implementation of XPath does not have the lower-case function so you have to use the nasty looking translate. CSS Selectors do not also have substring matching for of descendent text in the standard. Sizzle solves that, but that is not standard. XPath has contains though.

Circling back to this particular XPath.

• Does it solve the problem? Yes
• Is it more inherently brittle than another structural locator? No
• Does it provide guards around case differences in data? Yes
• Would it be pretty easy to modify if the way that the identifying bits change? Yes

Awesome.

The point of this little rant is to not be afraid of large XPath when it is the proper locator for the task and reserve your judgemental sneers for those use of bad used XPath poorly.

Oh, and remember, your ‘but it relies on the structure of the HTML’ hatred should really be directed at both XPath and CSS Selectors. They are both structural location schemes and therefore suffer the same problems.

[1] Stolen blatantly from Back To The Future by Pagano