Corey Goldberg

Python - Using Chrome Extensions With Selenium WebDriver

2018-09-07T12:07:00.001-04:00

When you launch a Chrome browser from Selenium, a fresh temporary profile is created for the browser session. This means that by default, Chrome extensions are not loaded. To use an extension during a Selenium test, you must install the extension each time you launch a new browser.

Example 1: Installing a packed extension

A packed extension comes in the form of a single .crx file.

You can install a packed extension like this:

Example 2: Installing an unpacked extension

An unpacked extension is a directory containing extension code.

You can install an unpacked extension like this:

examples tested with: chromium 68, chromedriver 2.41, selenium 3.14

Python - Fixing My Photo Library Dates (Exif Metadata)

2014-01-11T11:02:00.000-05:00

I have a large image library of photos I've taken or downloaded over the years. They are from various cameras and sources, many with missing or incomplete Exif metadata.

This is problematic because some image viewing programs and galleries use metadata to sort images into timelines. For example, when I view my library in Dropbox Photos timeline, images with missing Exif date tags are not displayed.

To remedy this, I wrote a Python script to fix the dates in my photo library. It uses gexiv2, which is a wrapper around the Exiv2 photo metadata library.

The scipt will:

recursively scan a directory tree for jpg and png files
get each file's creation time
convert it to a timestamp string
set Exif.Image.DateTime tag to timestamp
set Exif.Photo.DateTimeDigitized tag to timestamp
set Exif.Photo.DateTimeOriginal tag to timestamp
save file with modified metadata
set file access and modified times to file creation time

* Note: it does modifications in-place.

The Code:

deadsnakes - Using Old Versions of Python on Ubuntu

2013-10-22T10:38:00.000-04:00

How do you install an older version of Python on Ubuntu without building it yourself?

The Python packages in the official Ubuntu archives generally don't go back all that far, but people might still need to develop and test against these old Python interpreters. Felix Krull maintains a PPA (package archive) of older Python versions that are easy to install on Ubuntu.

see: https://launchpad.net/~fkrull/+archive/deadsnakes

Currently supported Python releases: 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, 3.3

Instructions:

Add the deadsnakes repository:

$ sudo add-apt-repository ppa:fkrull/deadsnakes

Run Update:

$ sudo apt-get update

Install an older version of Python:

$ sudo apt-get install python2.6 python2.6-dev

Generating Audio Spectrograms in Python

2013-06-22T15:44:00.000-04:00

A spectrogram is a visual representation of the spectrum of frequencies in a sound sample.

more info: wikipedia spectrogram

Spectrogram code in Python, using Matplotlib:
(source on GitHub)

"""Generate a Spectrogram image for a given WAV audio sample.

A spectrogram, or sonogram, is a visual representation of the spectrum
of frequencies in a sound.  Horizontal axis represents time, Vertical axis
represents frequency, and color represents amplitude.
"""


import os
import wave

import pylab


def graph_spectrogram(wav_file):
    sound_info, frame_rate = get_wav_info(wav_file)
    pylab.figure(num=None, figsize=(19, 12))
    pylab.subplot(111)
    pylab.title('spectrogram of %r' % wav_file)
    pylab.specgram(sound_info, Fs=frame_rate)
    pylab.savefig('spectrogram.png')


def get_wav_info(wav_file):
    wav = wave.open(wav_file, 'r')
    frames = wav.readframes(-1)
    sound_info = pylab.fromstring(frames, 'Int16')
    frame_rate = wav.getframerate()
    wav.close()
    return sound_info, frame_rate


if __name__ == '__main__':
    wav_file = 'sample.wav'
    graph_spectrogram(wav_file)

Spectrogram code in Python, using timeside:
(source on GitHub)

"""Generate a Spectrogram image for a given audio sample.

Compatible with several audio formats: wav, flac, mp3, etc.
Requires: https://code.google.com/p/timeside/

A spectrogram, or sonogram, is a visual representation of the spectrum
of frequencies in a sound.  Horizontal axis represents time, Vertical axis
represents frequency, and color represents amplitude.
"""


import timeside


audio_file = 'sample.wav'

decoder = timeside.decoder.FileDecoder(audio_file)
grapher = timeside.grapher.Spectrogram(width=1920, height=1080)
(decoder | grapher).run()
grapher.render('spectrogram.png')

happy audio hacking.

Python - concurrencytest: Running Concurrent Tests

2013-06-10T09:27:00.000-04:00

Add parallel testing to your unit test framework.

In my previous post, I described running concurrent tests using nose as a loader and runner.

On a similar note, let's look at building concurrency into your own test framework built on Python's unittest.

Have a look at this module: concurrencytest

(Thanks to bits and concepts taken from testtools and bzrlib)

An Example:

Say you have a 'TestSuite' of tests loaded. You could run them with the standard 'TextTestRunner' like this:

runner = unittest.TextTestRunner()
runner.run(suite)

That would run the tests in your suite sequentially in a single process.

By adding the concurrencytest module, you can use a 'ConcurrentTestSuite' instead, by adding:

from concurrencytest import ConcurrentTestSuite, fork_for_tests

concurrent_suite = ConcurrentTestSuite(suite, fork_for_tests(4))
runner.run(concurrent_suite)

That would run the same tests split across 4 processes (workers).

Note: this relies on 'os.fork()' which only works on Unix systems.

There's no way to understand this better than looking at some contrived examples!

This first example is totally unrealistic, but shows off concurrency perfectly. The test cases it loads each sleep for 0.5 seconds and then exit.

The Code:

Output:

Loaded 50 test cases...

Run tests sequentially:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 25.031s

OK

Run same tests across 50 processes:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 0.525s

OK

nice!

Now another example that shows concurrency with CPU-bound test cases. The test cases it loads each calculate fibonacci of 31 (recursively!) and then exit. We can see how it performs on my 8-core machine (Core2 i7 quad, hyperthreaded).

The Code:

Output:

Loaded 50 test cases...

Run tests sequentially:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 21.941s

OK

Run same tests with 2 processes:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 11.081s

OK

Run same tests with 4 processes:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 5.862s

OK

Run same tests with 8 processes:
..................................................
----------------------------------------------------------------------
Ran 50 tests in 4.743s

OK

happy hacking.

Python - Nose: Running Concurrent Tests

2013-06-09T19:17:00.000-04:00

TLDR:
To enable multiprocessing with N workers,
run nose with:

$ nosetests --processes=N

When writing tests in Python, I start with TestCase's derived from unittest.TestCase, and standard test discovery. When I need more complex test discovery/loading or output reports, I often use nose and its assortment of plugins as my test loader/runner.

One nice feature of nose is the multiprocess plugin. It allows you to run your tests suites concurrently rather than sequentially, spread across a number of worker processes. Running tests in parallel like this can potentially give you a large speedup in your test run times.

from the nose multiprocess docs:

"You can parallelize a test run across a configurable number of worker processes. While this can speed up CPU-bound test runs, it is mainly useful for IO-bound tests that spend most of their time waiting for data to arrive from someplace else and can benefit from parallelization."

Normally, you run tests from nose with:

$ nosetests

To run the same tests split across 4 processes (workers), you would just do:

$ nosetests --processes=4

Assuming your tests are properly isolated, everything should run normally, and you can benefit from a speedup on a multiprocessor machine.

However, Beware.

"Not all test suites will benefit from, or even operate correctly using, this plugin. For example, CPU-bound tests will run more slowly if you don't have multiple processors."

"But the biggest issue you will face is probably concurrency. Unless you have kept your tests as religiously pure unit tests, with no side-effects, no ordering issues, and no external dependencies, chances are you will experience odd, intermittent and unexplainable failures and errors when using this plugin. This doesn't necessarily mean the plugin is broken; it may mean that your test suite is not safe for concurrency."

Python - Re-tag FLAC Audio Files (Update Metadata)

2013-03-28T18:35:00.000-04:00

I had a bunch of FLAC (.flac) audio files together in a directory. They are from various sources and their metadata (tags) were somewhat incomplete or incorrect.

I managed to manually get all of the files standardized in "%Artist% - %Title%.flac" file name format. However, What I really wanted was to clear their metadata and just save "Artist" and "Title" tags, pulled from file names.

I looked at a few audio tagging tools in the Ubuntu repos, and came up short finding something simple that covered my needs. (I use Audio Tag Tool for MP3's, but it has no FLAC file support.)

So, I figured the easiest way to get this done was a quick Python script.

I grabbed Mutagen, a Python module to handle audio metadata with FLAC support.

This is essentially the task I was looking to do:

#!/usr/bin/env python

import glob
import os
from mutagen.flac import FLAC

for filename in glob.glob('*.flac'):
    artist, title = os.path.splitext(filename)[0].split(' - ', 1)
    audio = FLAC(filename)
    audio.clear()
    audio['artist'] = artist
    audio['title'] = title
    audio.save()

It iterates over .flac files in the current directory, clearing the metadata and rewriting only the artist/title tags based on each file name.

I created a repository with a slightly more full-featured version, used to re-tag single FLAC files:
https://github.com/cgoldberg/audioscripts/blob/master/flac_retag.py

Python - verify a PNG file and get image dimensions

2013-01-28T17:09:00.000-05:00

useful snippet for getting .png image dimensions without using an external imaging library.

#!/usr/bin/env python

import struct


def get_image_info(data):
    if is_png(data):
        w, h = struct.unpack('>LL', data[16:24])
        width = int(w)
        height = int(h)
    else:
        raise Exception('not a png image')
    return width, height


def is_png(data):
    return (data[:8] == '\211PNG\r\n\032\n'and (data[12:16] == 'IHDR'))


if __name__ == '__main__':
    with open('foo.png', 'rb') as f:
        data = f.read()

    print is_png(data)
    print get_image_info(data)

/headnods:
getimageinfo.py source, Portable_Network_Graphics (Wikipedia)

Python Unit Testing Tutorial (PyMOTW unittest update)

2013-01-23T18:06:00.000-05:00

tl;dr: an update to PyMOTW for `unittest` in Python 3: Python Unit Testing Tutorial.

When I was learning programming in Python, Doug Hellmann's 'PyMOTW' (Python Module Of The Week) blog-series was one of the best resources to learn Python's standard library.

His series later culminated in the book: 'The Python Standard Library By Example'.

The PyMOTW entry on `unittest` was a great introduction to unit testing in Python. Since the PyMOTW version is getting quite outdated, I updated the `unittest` module entry.

This new version includes some edits and updates to the text, and all code and examples have been updated to reflect Python 3.3.

Have a look at my updated Python 3.3 version:
'Python Unit Testing Tutorial'

license: Creative Commons Attribution, Non-commercial, Share-alike 3.0

say no to bugs...

Python - "The Matrix" in your terminal

2013-01-14T09:42:00.001-05:00

Linux console program in Python, that scrolls binary numbers vertically in your terminal. Inspired by the movie: The Matrix

a screenshot:

in action:

the Code:

https://gist.github.com/4530348

Python Testing - PhantomJS with Selenium WebDriver

2013-01-06T16:35:00.000-05:00

PhantomJS is a headless WebKit with JavaScript API. It can be used for headless website testing.

PhantomJS has a lot of different uses. The interesting bit for me is to use PhantomJS as a lighter-weight replacement for a browser when running web acceptance tests. This enables faster testing, without a display or the overhead of full-browser startup/shutdown.

I write my web automation using Selenium WebDriver, in Python.

In future versions of PhantomJS, the GhostDriver component will be included.

GhostDriver is a pure JavaScript implementation of the WebDriver Wire Protocol for PhantomJS. It's a Remote WebDriver that uses PhantomJS as back-end.

So, Ghostdriver is the bridge we need to use Selenium WebDriver with Phantom.JS.

Since it is not available in the current PhantomJS release, you can try it yourself by compiling a special version of PhantomJS:

It wes pretty trvial to setup on Ubuntu (12.04):

$ sudo apt-get install build-essential chrpath git-core libssl-dev libfontconfig1-dev
$ git clone git://github.com/ariya/phantomjs.git
$ cd phantomjs
$ git checkout 1.8
$ ./build.sh
$ git remote add detro https://github.com/detro/phantomjs.git
$ git fetch detro && git checkout -b detro-ghostdriver-dev remotes/detro/ghostdriver-dev
$ ./build.sh

Then grab the `phantomjs` binary it produced (look inside `phantomjs/bin`). This is a self-contained executable, it can be moved to a different directory or another machine. Make sure it is located somewhere on your PATH, or declare it's location when creating your PhantomJS driver like the example below.

for these examples, `phantomjs` binary is located in same directory as test script.

Example: Python Using PhantomJS and Selenium WebDriver.

#!/usr/bin/env python

driver = webdriver.PhantomJS('./phantomjs')
# do webdriver stuff here
driver.quit()

Example: Python Unit Test Using PhantomJS and Selenium WebDriver.

#!/usr/bin/env python

import unittest
from selenium import webdriver


class TestUbuntuHomepage(unittest.TestCase):
    
    def setUp(self):
        self.driver = webdriver.PhantomJS('./phantomjs')
        
    def testTitle(self):
        self.driver.get('http://www.ubuntu.com/')
        self.assertIn('Ubuntu', self.driver.title)
        
    def tearDown(self):
        self.driver.quit()


if __name__ == '__main__':
    unittest.main(verbosity=2)

resources:

Python Timer Class - Context Manager for Timing Code Blocks

2012-06-17T10:39:00.000-04:00

Here is a handy Python Timer class. It creates a context manager object, used for timing a block of code.

from timeit import default_timer


class Timer(object):
    def __init__(self, verbose=False):
        self.verbose = verbose
        self.timer = default_timer
        
    def __enter__(self):
        self.start = self.timer()
        return self
        
    def __exit__(self, *args):
        end = self.timer()
        self.elapsed_secs = end - self.start
        self.elapsed = self.elapsed_secs * 1000  # millisecs
        if self.verbose:
            print 'elapsed time: %f ms' % self.elapsed

To use the Timer (context manager object), invoke it using Python's `with` statement. The duration of the context (code inside your `with` block) will be timed. It uses the appropriate timer for your platform, via the `timeit` module.

Timer is used like this:

with Timer() as target:
    # block of code goes here.
    # result (elapsed time) is stored in `target` properties.

Example script:
timing a web request (HTTP GET), using the `requests` module.

#!/usr/bin/env python

import requests
from timer import Timer

url = 'https://github.com/timeline.json'

with Timer() as t:
    r = requests.get(url)
    
print 'fetched %r in %.2f millisecs' % (url, t.elapsed)

Output:

fetched 'https://github.com/timeline.json' in 458.76 millisecs

`timer.py` in GitHub Gist form, with more examples:

History of Python - Development Visualization - Gource

2012-06-04T15:45:00.002-04:00

I made a new visualization. Have a look!

History of Python - Gource - development visualization (august 1990 - june 2012)
[HD video, encoded at 1080p. watch on YouTube in highest resolution possible.]

What is it?

This is a visualization of Python core development. It shows growth of the Python project's source code over time (August 1990 - June 2012). Nearly 22 years! The source code history and relations are displayed by Gource as an animated tree, tracking commits over time. Directories appear as branches with files as leaves. Developers can be seen working on the tree at the times they contributed to the Python project.

Video:
Rendered with Gource v0.37 on Ubuntu 12.04

Music:
Chris Zabriskie - The Life and Death of a Certain K Zabriskie Patriarch

Repository:
cpython 3.3.0 alpha, retrieved from mercurial on June 2 2012

for more visualizations and other videos, check out my YouTube channel.

SST 0.2.1 Release Announcement (selenium-simple-test)

2012-04-23T14:02:00.000-04:00

SST version 0.2.1 has been released.

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

SST version 0.2.1 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

SST downloads | (Jan 1 2012 - April 23 2012)

1600+ downloads from PyPI since initial release.

Python - Getting Data Into Graphite - Code Examples

2012-04-09T12:16:00.000-04:00

This post shows code examples in Python (2.7) for sending data to Graphite.

Once you have a Graphite server setup, with Carbon running/collecting, you need to send it data for graphing.

Basically, you write a program to collect numeric values and send them to Graphite's backend aggregator (Carbon).

To send data, you create a socket connection to the graphite/carbon server and send a message (string) in the format:

"metric_path value timestamp\n"

`metric_path`: arbitrary namespace containing substrings delimited by dots. The most general name is at the left and the most specific is at the right.
`value`: numeric value to store.
`timestamp`: epoch time.
messages must end with a trailing newline.
multiple messages maybe be batched and sent in a single socket operation. each message is delimited by a newline, with a trailing newline at the end of the message batch.

Example message:

"foo.bar.baz 42 74857843\n"

Let's look at some (Python 2.7) code for sending data to graphite...

Here is a simple client that sends a single message to graphite.

Code:

#!/usr/bin/env python

import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003

message = 'foo.bar.baz 42 %d\n' % int(time.time())

print 'sending message:\n%s' % message
sock = socket.socket()
sock.connect((CARBON_SERVER, CARBON_PORT))
sock.sendall(message)
sock.close()

Here is a command line client that sends a single message to graphite:

Usage:

$ python client-cli.py metric_path value

Code:

#!/usr/bin/env python

import argparse
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003


parser = argparse.ArgumentParser()
parser.add_argument('metric_path')
parser.add_argument('value')
args = parser.parse_args()


if __name__ == '__main__':
    timestamp = int(time.time())
    message = '%s %s %d\n' % (args.metric_path, args.value, timestamp)
    
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()

Here is a client that collects load average (Linux-only) and sends a batch of 3 messages (1min/5min/15min loadavg) to graphite. It will run continuously in a loop until killed. (adjust the delay for faster/slower collection interval):

#!/usr/bin/env python
 
import platform
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003
DELAY = 15  # secs


def get_loadavgs():
    with open('/proc/loadavg') as f:
        return f.read().strip().split()[:3]


def send_msg(message):
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()


if __name__ == '__main__':
    node = platform.node().replace('.', '-')
    while True:
        timestamp = int(time.time())
        loadavgs = get_loadavgs()
        lines = [
            'system.%s.loadavg_1min %s %d' % (node, loadavgs[0], timestamp),
            'system.%s.loadavg_5min %s %d' % (node, loadavgs[1], timestamp),
            'system.%s.loadavg_15min %s %d' % (node, loadavgs[2], timestamp)
        ]
        message = '\n'.join(lines) + '\n'
        send_msg(message)
        time.sleep(DELAY)

Resources:

Installing Graphite 0.9.9 on Ubuntu 12.04 LTS

2012-04-07T13:21:00.001-04:00

I just setup a Graphite server on Ubuntu 12.04 (Precise).

Here are some instructions for getting it all working (using Apache as web server).

It follows these steps:

install system dependencies (apache, django, dev libs, etc)
install Whisper (db lib)
install and configure Carbon (data aggregator)
install Graphite (django webapp)
configure Apache (http server)
create initial database
start Carbon (data aggregator)

Once that is done, you should be able to visit the host in your web browser and see the Graphite UI.

Setup Instructions:

#############################
# INSTALL SYSTEM DEPENDENCIES
#############################

$ sudo apt-get install apache2 libapache2 libapache2-mod-wsgi /
    libapache2-mod-python memcached python-dev python-cairo-dev /
    python-django python-ldap python-memcache python-pysqlite2 /
    python-pip sqlite3 erlang-os-mon erlang-snmp rabbitmq-server
    
$ sudo pip install django-tagging

#################
# INSTALL WHISPER
#################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz

################################################
# INSTALL AND CONFIGURE CARBON (data aggregator)
################################################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz
$ cd /opt/graphite/conf/
$ sudo cp carbon.conf.example carbon.conf
$ sudo cp storage-schemas.conf.example storage-schemas.conf

###########################
# INSTALL GRAPHITE (webapp)
###########################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz

or

$ wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz
$ tar -zxvf graphite-web-0.9.9.tar.gz
$ mv graphite-web-0.9.9 graphite
$ cd graphite
$ sudo python check-dependencies.py
$ sudo python setup.py install

##################
# CONFIGURE APACHE
##################

$ cd graphite/examples
$ sudo cp example-graphite-vhost.conf /etc/apache2/sites-available/default
$ sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
$ sudo mkdir /etc/httpd
$ sudo mkdir /etc/httpd/wsgi
$ sudo /etc/init.d/apache2 reload

#########################
# CREATE INITIAL DATABASE 
#########################

$ cd /opt/graphite/webapp/graphite/
$ sudo python manage.py syncdb
$ sudo chown -R www-data:www-data /opt/graphite/storage/
$ sudo /etc/init.d/apache2 restart
$ sudo cp local_settings.py.example local_settings.py

################################
# START CARBON (data aggregator)
################################

$ cd /opt/graphite/
$ sudo ./bin/carbon-cache.py start

Resources:

* works on my machine, Ubuntu 12.04

Python - Graphite: Storage and Visualization of Time-series Data

2012-04-07T07:45:00.000-04:00

I'm doing some work with Graphite in Python. Here is a quick overview of what Graphite is...

Graphite provides real-time visualization and storage of numeric time-series data.

Links:

Project: https://launchpad.net/graphite
Docs: http://graphite.readthedocs.org

Graphite does two things:

Store numeric time-series data
Render graphs of this data on demand

Graphite consists of a storage backend and a web-based visualization frontend. Client applications send streams of numeric time-series data to the Graphite backend (called carbon), where it gets stored in fixed-size database files similar in design to RRD. The web frontend provides 2 distinct user interfaces for visualizing this data in graphs as well as a simple URL-based API for direct graph generation.

Graphite consists of 3 software components:

carbon - a Twisted daemon that listens for time-series data
whisper - a simple database library for storing time-series data (similar in design to RRD)
graphite webapp - A Django webapp that renders graphs on-demand using Cairo

Python Book Giveaway - Boston Python User Group - April 12, 2012

2012-04-05T09:17:00.001-04:00

I am bringing some of my [lightly read] Python books to give away at the next Boston Python User Group meetup.

Boston Python User Group - April Project Night
When: Thursday, April 12, 2012, 6:30 PM
Where: Microsoft NERD, Cambridge

Books:

Pro Python (Alchin)
Head First Python (Barry)
Programming in Python 3 (Summerfield)
Python Programming Patterns (Christopher)
Hello World! (Sande)

If you are interested in getting a free book, come to #bostonpython Project Night on April 12!

Python Screencast: Install/Setup "SST Web Test Framework" on Ubuntu 12.04

2012-03-24T12:59:00.000-04:00

I uploaded a 5 minute video/screencast showing how to install and setup SST Web Test Framework on Ubuntu (Precise Pangolin 12.04).

I step through: creating a virtualenv, installing SST from PyPI, and creating a basic automated web test:

http://www.youtube.com/watch?v=LpSvGmglZPI

the following steps are essentially a transcript of what I did...

install system package dependencies:

$ sudo apt-get install python-virtualenv xvfb

create a "virtualenv":

$ virtualenv ENV

active the virtualenv:

$ cd ENV
$ source bin/activate
(ENV)$

* notice your prompt changed, signifying the virtualenv is active

install SST using `pip`:

(ENV)$ pip install sst

Now SST is installed. You can check the version of SST:

$ sst-run -V

Codeswarm - Python Core Development Visualization

2012-03-08T12:14:00.000-05:00

particle visualization of Python core development commits: Jan 1, 2010 - Mar 06, 2012

http://www.youtube.com/watch?v=IQPuU_YtN8Q

data source is the commit log from cpython mercurial trunk:

$ hg clone http://hg.python.org/cpython

... trimmed to show development activity since Jan 1, 2010.

(images making up this video were rendered with: Codeswarm)

SST - Automated Web Page Profiling (Python)

2012-03-05T09:04:00.001-05:00

SST - Web Test Framework: http://testutils.org/sst

The latest release of SST (0.2.0) adds the ability to capture HAR (HTTP Archive format) output for pageload performance tracing/profiling.

New SST doc section: Performance tracing with Browsermob Proxy (HAR)

The HAR format is based on JSON, and is used by tools that consume/produce data collected by monitoring HTTP communication. These files contain a log of HTTP client/server conversation and can be used for additional analysis of page load performance.

The capture is achieved by routing browser requests through BrowserMob Proxy, which records web page loads while your tests run. SST will launch the proxy and save output to .har files if you enable the `--browsermob` command line option. HAR files are saved in the results directory for each page load.

HAR files can be viewed/analyzed with various tools, such as `harviewer`:

I created a screencast demo showing it all together:

Automated Web Page Profiling : SST + BrowserMob

SST 0.2.0 Release Announcement (and codebase visualization)

2012-02-26T15:05:00.000-05:00

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

Version 0.2.0 Released!

SST version 0.2.0 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

Development Update

So... what's up with SST development? is it active?

I ran Gource against the trunk branch and created an awesome visualization (with soundtrack!) to show off development activity:

video: link (with soundtrack)
video: link (no soundtrack)

The magic incantation of `gource` to produce the video:

$ gource \
    -s .3 \
    -1280x720 \
    --auto-skip-seconds .3 \
    --multi-sampling \
    --stop-at-end \
    --hide mouse,progress \
    --file-idle-time 0 \
    --max-files 0  \
    --background-colour 222222 \
    --font-size 20 \
    --logo docs/assets/sst-logo_small.png \
    --title "SST Development - lp:selenium-simple-test" \
    --output-ppm-stream - \
    --output-framerate 30 \
    | ffmpeg -y -r 30 -f image2pipe -vcodec ppm -i - -b 2048K movie.mp4

(rendered on Ubuntu 11.10, audio mixed with Pitivi)

Python - Matplotlib and Numpy on Debian/Ubuntu

2012-01-29T15:38:00.000-05:00

There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.

However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:

$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv

Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work:

$ virtualenv env
$ cd env
$ source bin/activate
(env)$ pip install numpy matplotlib

...
...
Successfully installed numpy matplotlib
Cleaning up...

(tested on Ubuntu Oneiric 11.10 and Ubuntu Precise 12.04 alpha)

Officially Introducing "SST" (Python Web Test Framework)

2012-01-01T18:55:00.000-05:00

"SST (selenium-simple-test) is a framework built on Selenium WebDriver, using Python to make writing functional web tests easier with code."

Since early 2011, I have been working for Canonical on the Infrastructure Systems Development team (Core Dev Ops).

[pic of canonical-isd-hackers at UDS-P-Orlando]

A by-product of our recent development efforts is a web testing framework. It has been available on Launchpad for a while, but I've never really announced it in public. We are using SST internally, and I want to expose it to a wider audience.

What is Selenium WebDriver?

Selenium (WebDriver) is a popular open-source library for automating browsers. It can be used to create functional/acceptance tests of a web application. The Selenium client bindings provide API's that allow you to programatically drive a browser and access web content/elements. The bindings are available and supported for many languages and platforms.

While working directly with Selenium API's from code is fine for ad-hoc browser interaction, it is rather low-level and lacks things necessary for creating suites of automated web tests. For larger-scale testing, you will soon want to use a framework to help organize, execute, and report.

Introducing SST...

SST aims to keep things simple.

Tests are made up of scripts, created by composing actions that drive a browser and assert conditions. You have the flexibility of the full Python language, along with a convenient set of functions to simplify web testing.

SST framework consists of:

user actions and assertions (API) in Python
test case loader (generates/compiles scripts to unittest cases)
console test runner
data parameterization/injection
selectable output reports
selectable browsers
headless (xvfb) mode
screenshots on errors

Test output can be displayed to the console, saved as an HTML report, or JUnit-compatible XML for compatibility with CI systems.

SST is free open source software (Apache Licensed). SST is primarily being developed on Linux, specifically Ubuntu. It should work fine on other platforms, but any issues (or even better - patches) should be reported on the Launchpad project:

https://launchpad.net/selenium-simple-test

I just uploaded SST 0.1.0 to PyPI:

http://pypi.python.org/pypi/sst

go ahead, give it a try:

`$ [sudo] pip install sst`

documentation and more info:

http://testutils.org/sst

a sample test script in SST:

from sst.actions import *

go_to('http://www.ubuntu.com/')
assert_title_contains('Ubuntu homepage')

Here is the development progress of SST (shown as a code_swarm visualization) over the past 8 months:

Special thanks to all the SST code committers so far:

Danny Tamez
Julien Funk
Kenneth Koontz
Leo Arias
Lukasz Czyzykowkski
Rick McBride
Sidnei da Silva

Extra special thanks to SST's initial creator:

Michael Foord

Happy New Year!

-Corey Goldberg

"Web Performance Testing night" in Boston/Cambridge - Python Meetup Dec. 19

2011-12-12T09:22:00.001-05:00

I will be speaking at: "Web Performance Testing, lightning talks, and beers" at the Boston Python User Group.

Event Info: http://meetup.bostonpython.com/events/36664122/
Date: Monday, December 19, 2011, 7:00 PM
Location: Microsoft NERD, Cambridge, MA

"Corey Goldberg of Canonical and Dan Kuebrich of Tracelytics will tag-team to tell us about web performance testing, and a few interesting tools they've built."

Free pizza and beer!
Come Join!

Edit/Update:
Thanks to all who attended! The night was a big success.

The slides from my portion of the presentation are posted here:
webperf.html