blog. Ivan De Marino

Me @ Selenium Camp 2013

2013-05-11T03:09:25Z

Selenium Camp 2013 has been a great experience. Not only I hanged out with great people, but, for the first time in my life, I also got to savour a little bit of Ukraine in Kiev.

VERY friendly people (really friendly). GREAT food. WELL-ORGANISED conference.

What else could I ask for?

My talk

The title of my talk is Getting Started with GhostDriver (and my experience working on it!).

UPDATE: here is the video:

Videos of the other talks can be found here. Here are the slides:

You can find them also on Google Drive and Speaker Deck.

Video will be available soon: the organisers have recorded every session and I guess it's just a matter of time. Added. Scroll above.

I'm off. See you in 2 weeks.

2013-03-29T01:53:06Z

This message is for the few souls that care. This is for the few people that maybe are waiting for me to fix/implement stuff on GhostDriver or PhantomJS (for the latter you have coverage, as the team on that is rich of geniuses - so you won't miss my average-hood).

I'm getting married (crazy, right?) and will be mostly off-line. I should be back from the second week of April.

Surely there will be the usual casual-tweet but nothing major.

See ya!

BST traversal without stack or recursion (?)

2013-03-24T01:40:23Z

SPOILER: this article assumes quite a lot of things about your knowledge of binary trees and data structure. I won't spend much time introducing concepts. Deal with it.

Some time ago I had the pleasure to meet with Chris Lord, who introduced me to the concept of Binary Tree Traversal without using neither a stack/queue or recursion.

The list of requirements that such algorithm has to fulfil:

Constant Space: no extra data structure to do the traversal
Non-destructive: once finished, the tree needs to be exactly how it was before the visit

Sounded crazy initially, but than I looked into it more deeply and discovered, thanks to some pointers from Luca Colantonio, that there is an algorithm in literature that does just that: Morris In-Order Traversal.

Morris does just that - and a very good description of how it works can be found on Stack Overflow. I suggest you have a good read and come back: pointless of me to repeat it here.

...

You back? Great!

So, before "discovering" the Morris algorithm, I was trying to write a solution myself. My basic idea was to:

Move downwards
Manipulate pointers to the branch we are about to explore, to point backward so we have "a way back"
Restore once done with that branch

At that stage, it had not occur to me to try to manipulate the entire tree structure like Morris does: I wanted to be very subtle, with the data structure resembling the original tree (though, with backward pointers).

The key issue I hit? Going Backward. Once I'm done visiting the branches of a node, I need to go back to the parent (to which I have a pointer too), and restore the structure.

But HOW am I going to reconstruct the tree? How do I know if the Branch I just visited was the left or the right branch of the parent I want to move back to?

Downsize the problem

In a general Binary Tree, there is no guarantee of the order and distribution of nodes. It's just a tree where every node has 0, 1 or 2 child nodes. I didn't have any "rule" on which to build a backward-restoring logic.

But then I had an idea: what about BSTs (Binary Search Tree)? In those trees, there is a strong order relationship between a node and its childs.

This means that at any moment in time, once I'm done visiting a branch, I can reconstruct where I'm coming from, looking at the values of the parent and the child:

parent->value > child->value ? is_left_child : is_right_child

And, so I wrote the De Marino BST Post-Order Traversal algorithm:

void de_marino_bst_post_order_traversal(node *root) {
    node *curr = root;  //< current node
    node *par = NULL;   //< parent of current node
    node *next = root;  //< next node to traversal (but sometimes, just temp variable)
    int dir = DOWN;     //< direction in which we are moving

    do {
        if (dir == DOWN && curr->left != NULL) {
            // Go down left
            next = curr->left;
            curr->left = par;
            par = curr;
            curr = next;
        } else if (dir == DOWN && curr->right != NULL) {
            // Go down right
            next = curr->right;
            curr->right = par;
            par = curr;
            curr = next;
        } else {
            printf("%d\n", curr != NULL ? curr->value : -1);

            if (par->value > curr->value && par->right != NULL) {
                // Go down right, coming from left
                // NOTE: order gives us a hint about where we come from
                dir = DOWN;

                next = par->left;   // hold the way back
                par->left = curr;
                curr = par->right;
                par->right = next;  // store the way back in the right child of par now
                next = curr;
            } else {
                dir = UP;

                if (par->value > curr->value) {
                    // Go up, coming from left
                    // NOTE: order gives us a hint about where we come from
                    next = par->left;
                    par->left = curr;
                } else {
                    // Go up, coming from right
                    // NOTE: order gives us a hint about where we come from
                    next = par->right;
                    par->right = curr;
                }
                curr = par;
                par = next;
            }
        }
    } while(next != NULL && par != NULL);
    printf("%d\n", curr != NULL ? curr->value : -1);
}

Now, I'm almost 100% sure that I'm not the first one to stumble into something like this, but for now I'm unable to find anything in literature - cough ... Google ... cough .

An advantage over Morris

Yes, this algorithm requires the strong condition of BST while Morris works with any Binary Tree. That's a given. But this algorithm is also faster: it doesn't go deep into the tree to find the rightmost left child of a node, like Morris. From a quick look the computation is linear - O(n): every node is visited only once. But maybe I'm overlooking something in Morris.

Far from perfect

I'm also sure that this algorithm is far from bug-less: I have spent on it less than 24h between coding and thinking. So, please, to all the Computer Scientists people out there: let me know your thoughts and critics. This is a great way to spend some of that theory that we have in our heads.

I have added this code to my coding exercises repo on GitHub, if you want to try it out (check latest commits).

Now, as I expect, feel free to ignore my warnings and start your opinionated trashing - I'm pretty confident I'd deserve it. :)

[UPDATED - PhantomJS 1.9.0] GhostDriver 1.0.3 is out!

2013-03-22T23:55:50Z

After few months of endless nights trying to figure out what I screwed up and where, I'm finally happy (enough) to release GhostDriver 1.0.3 into the wild.

This release is an hardening release: this means that it's focused on bug-fixing and very little new features have been added.

GhostDriver 1.0.3 ~~will be part of the upcoming PhantomJS 1.9 release, do in few days~~ is now part of PhantomJS 1.9.0, and you can download binaries for various platform from https://code.google.com/p/phantomjs/downloads/list.

If you want to know what made it for the 1.0.3 cut, take a look at the Changelog.

I apologise in advance for whom have seen his "pet issue" not being implemented/fixed/addressed for this release, but it's only one of me.

I'd like to take the time to thank the few brave souls that have contributed to this release with their code. In chronological contribution order:

Thanks guys! Keep being awesome!

Console++ - enhance your console

2013-02-18T09:05:48Z

I need a logger for GhostDriver but I'm too lazy to figure out how to use a nice, proper logging library. Plus I'm a BIG fun of old-school-debugging: I just substituted all the printf() calls with console.log.

So I decide to make every JavaScript developer shiver in disgust and wrote a small library that enhances the global console object.

This tiny library is called Console++ and works with NodeJS, PhantomJS and any decent JavaScript runtime.

It does output colouring, adds timestamps and log level to the lines, and can be used to redirect the output to other media than the console (just register a callback). On node, just type npm install consoleplusplus to get started.

It's so simple that the README is more than enough to use it.

It's under BSD license, so you can have fun with it.

Spin-down your USB HD and make it last longer

2013-02-10T21:27:49Z

I have built for myself a simple Media Center, based on Zotac ZBOX ID41 Plus, XBMC-Live, Transmission, Flirc and other useful stuff. The idea is having a low-energy consumption machine that I control with a remote. And it has to be always on.

One thing I was concerned about is how to deal with my external HD connected to it: I'm of course expecting those to eventually fail if they are always spinning.

Well, a bit of scripting and cron can go a long way. Here is my solution.

Step 1: dependencies

Just install sdparm and you should be all set, at least on a "normal" Linux box where Cron is already configured.

Step 2: scripting `FTW`

Put a script like this somewhere in your system. I have it in my ~/bin:

# !/bin/sh

# Only ROOT can run this
if [ "$(id -u)" != "0" ]; then
        echo “This script must be run as root” 1>&2
        exit 1
fi

STATE_FILENAME=state_$1.spindown
PREV_STATE_FILENAME=prev_state_$1.spindown

# Get new state from diskstats
NEWstate=$(cat /proc/diskstats | grep $1)
echo $NEWstate > "$STATE_FILENAME"

# compare md5 sums
md5new=$(md5sum $STATE_FILENAME | sed 's/ .*//')
md5old=$(md5sum $PREV_STATE_FILENAME | sed 's/ .*//')

# if no changes, power down
if [ "$md5new" = "$md5old" ]; then
        sdparm --flexible --command=stop /dev/$1 &>/dev/null
fi

# Write current state to file
echo $NEWstate > "$PREV_STATE_FILENAME"

A very basic and simple one, isn't it? What it does is just probe /proc/diskstats virtual file to get info about the recent activity of a disk. If it has not changed in the last 30 minutes, than it's time to use sdparm to send a stop command --command=stop. That's it.

Step 3: Cron-it-up!

Add the following lines to your /etc/crontab:

*/30 *   * * *   root   sh /home/xbmc/bin/spindown/spindown.sh sda
*/30 *   * * *   root   sh /home/xbmc/bin/spindown/spindown.sh sdb
*/30 *   * * *   root   sh /home/xbmc/bin/spindown/spindown.sh sdc
*/30 *   * * *   root   sh /home/xbmc/bin/spindown/spindown.sh sdd
*/30 *   * * *   root   sh /home/xbmc/bin/spindown/spindown.sh sde

The above lines execute the script against sda, sdb, sdc, sdd and sde.

That's it. I have been using this method for almost 2 years not, and thought it might be helpful to others.

Finally, GhostDriver 1.0.0

2013-01-29T16:25:52Z

After ~10 months of work, I finally managed to push the ball forward enough to reach GhostDriver 1.0.0. I won't lie: it does feel great. It always is to see so much effort mutate into an achievement, even it is such only for me.

Before I go any further though, I owe a massive THANK-YOU to the people that through contribution and / or issue reporting have helped me to build a releasable 1.0.0. In particular to Jim Evans, an awesome guy and a bit of a mentor for me: you ROCK Jim, and I think that is true, literally!

Explaining `1.0.0`

In 1.0.0 I wanted to have the right amount of Wire Protocol implemented, but still be able to deliver something stable before Christmas 2012!

So, what's in GhostDriver 1.0.0 (changelog)?

Navigation
Page content extraction
Arbitrary JS execution
Window handling
Frame handling
Screen-shot generation
Cookies
Element Search, Localization & Manipulation
Mouse interaction (even though doubeClick and rightClick seems to be a bit flaky)
Keyboard interaction

It's not the entire protocol yet, but it feels like a good start: I do think you can cover a lot of ground in your tests with those already.

To monitor progress, and to know in details what's there and what's not, check this spreadsheet out.

You said "explaining"...

OK. So, GhostDriver is developed as a plain PhantomJS script that makes use of basically 2 PhantomJS built-in modules:

require("webserver"): to receive the Wire Protocol REST commands HTTP Requests
require("webpage"): to execute those commands into Browser Windows (i.e. 1 webpage corresponds to 1 window)

You can download the source code from the repository (see README.md), and launch it with the following:

1	$ phantomjs ghostdriver/src/main.js [LOCAL_IP:]PORT

This approach was and is for development only though: GhostDriver target, since the beginning, was to embed the driver into PhantomJS - a feature demanded since February 2011(!!!).

So, from next release of PhantomJS (1.8 should come just before Christmas) you will be able to do instead:

1	$ phantomjs --webdriver=[LOCAL_IP:]PORT

without the need to download anything but your favourite single-binary-headless-browser. If you are impatient though, just download latest PhantomJS master branch, compile with ./build.sh and enjoy.

Binding

To make it easier to get started, Luke Inman-Semerau, Jim Evans and I have provided binding classes that will make even easier to get up and running with PhantomJS/GhostDriver in your Selenium/WebDriver tests. Luke is providing Python binding, Jim covers C#/.Net and I wrote the Java ones.

The next Selenium version is 2.27.0, and it's going to contain those bindings (hurray!). As I write this, David Burns is taking matters into his hands to prepare the release.

But I'm sure you want to see some code. Here is a Java snippet:

import org.openqa.selenium.phantomjs.PhantomJSDriver;

public class TestPhantomJSDriver {

    public static void main(String[] argv) {
        // prepare capabilities
        Capabilities caps = new DesiredCapabilities();
        caps.setJavascriptEnabled(true);                //< not really needed: JS enabled by default
        caps.setCapability("takesScreenshot", true);    //< yeah, GhostDriver haz screenshotz!
        caps.setCapability(
            PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,
            "/path/to/your/phantomjs/binary/greater/than/1.7"
        );

        // Launch driver (will take care and ownership of the phantomjs process)
        WebDriver driver = new PhantomJSDriver(caps);

        // Load Google.com
        driver.get("http://www.google.com");
        // Locate the Search field on the Google page
        WebElement element = driver.findElement(By.name("q"));
        // Type Cheese
        String strToSearchFor = "Cheese!";
        element.sendKeys(strToSearchFor);
        // Submit form
        element.submit();

        // Check results contains the term we searched for
        assertTrue(driver.getTitle().toLowerCase().contains(strToSearchFor.toLowerCase()));

        // done
        driver.quit();
    }
}

It's also important to highlight that GhostDriver implements the standard WebDriver WireProtocol. This means that you can connect to a running instance of PhantomJS in WebDriver mode by simply using any RemoteWebDriver class available in your favourite language. No strict need to use those bindings.

Where do I get it?

Because GhostDriver is now part of PhantomJS, you can get it in the next PhantomJS binary, stable release. It should be PhantomJS 1.8, and should be out by the 21st of December (don't tell Ariya I told you that - it's still cooking :-) ).

In the meantime, as briefly mentioned above, you have to do a bit of leg work: check out latest PhantomJS master branch (from either Ariya's (official upstream) or my own (usually in sync with upstream)), and compile it (takes 30 minutes on average specs machine - much less if you have got SSD).

The nice thing is that to build this you apply the same "set-up" required to compile any PhantomJS: nothing special is required. Just follow the instructions from here if you have never compiled it before.

After `1.0.0`: what's next?

There is a lot left to do. To push for 1.0.0 I had to make compromises, particularly in the number of Wire Protocol commands I had to implement. Few key ones haven't made it, but work on them it's already started (mostly in my head for now).

`1.0.1`: bug fixes

I plan to have a 1.0.1 to address all the immediate issues. The work for this will be limited fixing bugs though. I already got a couple of issues / comments reported and I'll act on those.

So, don't expect this release to be anything BUT bug-fixes: new stuff has to go in "at the right time" (see next).

A quick list of issues that will be address in this release:

Issue #123: Wait for potential "Page Load" when "Element.click()" is used
Issue #119: Command /shutdown arrives via HTTP HEAD in some language bindings (by Jari Bakken)
Issue #124: Invalid response when checking element equality
Issue #126: Crash on get + execute_async_script

Track the work for 1.0.1 here.

`1.1.0`: new features

This is "the right time" to add new features. This will be about pushing towards a full, complete WireProtocol implementation. So far I have picked the following from the issue tracker:

Issue #108: "Timeouts default value should be INFINITE"
Issue #120: Improve command line interface: support for just --webdriver without parameters
Issue #121: Better check for command line parameter: enforce LOCAL_IP:PORT or localhost:PORT
Issue #20: Implement support for JavaScript Alert/Prompt/Confirm
Issue #84: Rewrite the Request Routing code to be more like Node.js's Express
Issue #57: Implement "Drag & Drop" action builder

Track the work for 1.1.0 here.

`1.x`: finishing up

Well, this is quite simple. I can't plan too much ahead, but the idea is to push toward 100% of the whole WireProtocol. Maybe putting on top of that some Mobile User Agent simulation, but that requires some thinking first.

Anything else?

Developing a WebDriver has been quite a journey so far, and it's not yet finished. If you listen to some of the Selenium devs, apparently will never actually end, as issues, feature requests, criticism and pure simple trolling should keep flowing. At least that should happen if this is successful.

That might turn out to be tiring (or a failure), but I don't hide that knowing this is something that few people actually want to use makes me a tiny bit proud of my self.

Priorities

Before I go deep into another cycle of development though, my personal life is to regain some priority: I'm getting married in April 2013. That, as you might expect, will take time and energy. So, if you want to contribute to a small Open Source project that has some potential to be awesome, please join me and the other few awesome souls that helped me so far. And this leads me to my last thoughts and conclusions...

Lesson learned (still learning actually!)

As any small Open Source project, I need contributors. People that are willing to either write code, or kind enough to report issues in a proper and reproducible manner. One thing I had to learn is that, if I want to keep this project going, I need to be very strict in what I allow myself to get involved into.

And, worst of all, while I got a bunch of guys really supportive of this work (I'm looking at you, Selenium-ers), there is always the kind of jerk that prefers to spend time telling you why you are going to fail, instead of trying to help you contributing improvements or fixes.

Those "encounters" taught me some very useful things:

Open Source it's not Democracy: You have to keep the ship sailing in the right direction
Open Source it's a "doers" culture: ignore who talks and does nothing - embrace who makes things happen
Lots of people ignore README files: they just don't read - but don't get too upset by it
Kill unhelpful issues: they pollute the bug-tracker and lower morale, but reopen and discuss if the reporter "comes back to his senses"

I have recently started to read Team Geek, and it matches perfectly what I have experienced so far. Read it if you can!

Conclusions

This post was long overdue, and it took me few days to finish it up. I'm sure I could add more details here, but it's now the 4th night in a row and I want to move on.

What else can I say? Just that I hope this to be the first of other, even better releases!

Happy Driving! :)

GhostDriver, 90% and priorities

2012-10-23T22:41:38Z

Last few weeks have been HECTIC to say the least. Work on GhostDriver and related PhantomJS improvements has been very intense.

We just shipped PhantomJS 1.7, and work for 1.8 is already started (check out the ghostdriver-dev branch I linked above).

GhostDriver has reached almost 90% completion for 1.0, and the code is being hardened, thanks also to the amazing work that Jim Evans is doing at spotting issues and, a lot of times, providing fixes. GhostDriver is getting better and better thanks to you Jim.

Yes, but in short, what's left to do?

Based on my plan, to release 1.0 I'm missing "only":

Handling of window.alert, window.confirm and window.prompt
Handling of LocalStorage and SessionStorage
Handling fo /log commands

Recently though I have been thinking and discussing with others and I decided that I'd delay the support for those 3 things in favour of:

Fix issues related to Mouse interaction
Integrate GhostDriver into PhantomJS itself (I need to prepare a plan with Ariya)
Providing a Java Binding (Jim Evans already pushed a .Net binding into the Selenium codebase)
Code hardening

WHAAAAT? Are you crazy? No "alert()"?

Yeah, I know. While LocalStorage can be seen as nice-to-have features for 1.0, being able to interact with alert() (and relata) is a big deal.

The issue there is that the current PhantomJS doesn't provide an easy way to implement those commands: while PhantomJS does provide support for alert, prompt and confirm (I wrote them myself!), it will probably require quite some work to "adapt" PhantomJS/GhostDriver to the way the WebDriver API work for those things.

In short: PhantomJS allows you to register a onAlert/onConfirm/onPrompt callback to handle those events - WebDriver treats those as a "state" in which the Window is. Maybe I could write a never-returning callback? I don't know, it requires time to think about the best solution.

And that the point: I think I got no time now.

The best thing I can do right now

Time is key here.

I think the best thing I can do right now for the PhantomJS WebDriver project future, is to release a 1.0 version, with the hope it will receive enough road test, so that 1.1 and later can be better, comprehensive, more WebDriver-worthy.

Ah, and did I mention that I'm moving to Amazon LoveFilm? But that's for another post...

Look mummy, I'm 80% done!

2012-08-13T19:58:10Z

Let me give you a quick update about GhostDriver, the PhantomJS WebDriver implementation.

Yes, it's 80% ready for prime time (bugs aside)!!!

How did I get from 40% to 80% so quickly? Because I decided that, for version 1.0, the commands that relate to the mobile web driver are out of scope. 1.1 or 1.2 will have them.

For details, I suggest you check out the spreadsheet I'm using to track progress closely.

Some might call it "cheating". I call it "focusing".

The Stupid Olympics

2012-08-01T17:02:08Z

I'm baffled on how stupid the British officials and politicians are. The entire show, is as bad in practice as it looks good on TV.

Business starving for customers in Central London
No one has tickets: either because are too expensive or because buying tickets was a Olympic sport on it's own
Empty stations and entire city areas because of TfL end-of-the-world-is-night emails
Trains, somehow, still managed to get fucked up and cancelled

Was all this on purpose?