Corey Goldberg

SST 0.2.1 Release Announcement (selenium-simple-test)

2012-04-23T11:02:26-07:00

SST version 0.2.1 has been released.

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

SST version 0.2.1 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

SST downloads | (Jan 1 2012 - April 23 2012)

1600+ downloads from PyPI since initial release.

Python - Getting Data Into Graphite - Code Examples

2012-04-09T09:16:49-07:00

This post shows code examples in Python (2.7) for sending data to Graphite.

Once you have a Graphite server setup, with Carbon running/collecting, you need to send it data for graphing.

Basically, you write a program to collect numeric values and send them to Graphite's backend aggregator (Carbon).

To send data, you create a socket connection to the graphite/carbon server and send a message (string) in the format:

"metric_path value timestamp\n"

`metric_path`: arbitrary namespace containing substrings delimited by dots. The most general name is at the left and the most specific is at the right.
`value`: numeric value to store.
`timestamp`: epoch time.
messages must end with a trailing newline.
multiple messages maybe be batched and sent in a single socket operation. each message is delimited by a newline, with a trailing newline at the end of the message batch.

Example message:

"foo.bar.baz 42 74857843\n"

Let's look at some (Python 2.7) code for sending data to graphite...

Here is a simple client that sends a single message to graphite.

Code:

#!/usr/bin/env python

import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003

message = 'foo.bar.baz 42 %d\n' % int(time.time())

print 'sending message:\n%s' % message
sock = socket.socket()
sock.connect((CARBON_SERVER, CARBON_PORT))
sock.sendall(message)
sock.close()

Here is a command line client that sends a single message to graphite:

Usage:

$ python client-cli.py metric_path value

Code:

#!/usr/bin/env python

import argparse
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003


parser = argparse.ArgumentParser()
parser.add_argument('metric_path')
parser.add_argument('value')
args = parser.parse_args()


if __name__ == '__main__':
    timestamp = int(time.time())
    message = '%s %s %d\n' % (args.metric_path, args.value, timestamp)
    
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()

Here is a client that collects load average (Linux-only) and sends a batch of 3 messages (1min/5min/15min loadavg) to graphite. It will run continuously in a loop until killed. (adjust the delay for faster/slower collection interval):

#!/usr/bin/env python
 
import platform
import socket
import time


CARBON_SERVER = '0.0.0.0'
CARBON_PORT = 2003
DELAY = 15  # secs


def get_loadavgs():
    with open('/proc/loadavg') as f:
        return f.read().strip().split()[:3]


def send_msg(message):
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()


if __name__ == '__main__':
    node = platform.node().replace('.', '-')
    while True:
        timestamp = int(time.time())
        loadavgs = get_loadavgs()
        lines = [
            'system.%s.loadavg_1min %s %d' % (node, loadavgs[0], timestamp),
            'system.%s.loadavg_5min %s %d' % (node, loadavgs[1], timestamp),
            'system.%s.loadavg_15min %s %d' % (node, loadavgs[2], timestamp)
        ]
        message = '\n'.join(lines) + '\n'
        send_msg(message)
        time.sleep(DELAY)

Resources:

Installing Graphite 0.9.9 on Ubuntu 12.04 LTS

2012-04-07T10:26:30-07:00

I just setup a Graphite server on Ubuntu 12.04 (Precise).

Here are some instructions for getting it all working (using Apache as web server).

It follows these steps:

install system dependencies (apache, django, dev libs, etc)
install Whisper (db lib)
install and configure Carbon (data aggregator)
install Graphite (django webapp)
configure Apache (http server)
create initial database
start Carbon (data aggregator)

Once that is done, you should be able to visit the host in your web browser and see the Graphite UI.

Setup Instructions:

#############################
# INSTALL SYSTEM DEPENDENCIES
#############################

$ sudo apt-get install apache2 libapache2 libapache2-mod-wsgi /
    libapache2-mod-python memcached python-dev python-cairo-dev /
    python-django python-ldap python-memcache python-pysqlite2 /
    python-pip sqlite3 erlang-os-mon erlang-snmp rabbitmq-server
    
$ sudo pip install django-tagging

#################
# INSTALL WHISPER
#################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz

################################################
# INSTALL AND CONFIGURE CARBON (data aggregator)
################################################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz
$ cd /opt/graphite/conf/
$ sudo cp carbon.conf.example carbon.conf
$ sudo cp storage-schemas.conf.example storage-schemas.conf

###########################
# INSTALL GRAPHITE (webapp)
###########################

$ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz

or

$ wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz
$ tar -zxvf graphite-web-0.9.9.tar.gz
$ mv graphite-web-0.9.9 graphite
$ cd graphite
$ sudo python check-dependencies.py
$ sudo python setup.py install

##################
# CONFIGURE APACHE
##################

$ cd graphite/examples
$ sudo cp example-graphite-vhost.conf /etc/apache2/sites-available/default
$ sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
$ sudo mkdir /etc/httpd
$ sudo mkdir /etc/httpd/wsgi
$ sudo /etc/init.d/apache2 reload

#########################
# CREATE INITIAL DATABASE 
#########################

$ cd /opt/graphite/webapp/graphite/
$ sudo python manage.py syncdb
$ sudo chown -R www-data:www-data /opt/graphite/storage/
$ sudo /etc/init.d/apache2 restart
$ sudo cp local_settings.py.example local_settings.py

################################
# START CARBON (data aggregator)
################################

$ cd /opt/graphite/
$ sudo ./bin/carbon-cache.py start

Resources:

* works on my machine, Ubuntu 12.04

Python - Graphite: Storage and Visualization of Time-series Data

2012-04-07T04:45:28-07:00

I'm doing some work with Graphite in Python. Here is a quick overview of what Graphite is...

Graphite provides real-time visualization and storage of numeric time-series data.

Links:

Project: https://launchpad.net/graphite
Docs: http://graphite.readthedocs.org

Graphite does two things:

Store numeric time-series data
Render graphs of this data on demand

Graphite consists of a storage backend and a web-based visualization frontend. Client applications send streams of numeric time-series data to the Graphite backend (called carbon), where it gets stored in fixed-size database files similar in design to RRD. The web frontend provides 2 distinct user interfaces for visualizing this data in graphs as well as a simple URL-based API for direct graph generation.

Graphite consists of 3 software components:

carbon - a Twisted daemon that listens for time-series data
whisper - a simple database library for storing time-series data (similar in design to RRD)
graphite webapp - A Django webapp that renders graphs on-demand using Cairo

Python Book Giveaway - Boston Python User Group - April 12, 2012

2012-04-05T06:17:52-07:00

I am bringing some of my [lightly read] Python books to give away at the next Boston Python User Group meetup.

Boston Python User Group - April Project Night
When: Thursday, April 12, 2012, 6:30 PM
Where: Microsoft NERD, Cambridge

Books:

Pro Python (Alchin)
Head First Python (Barry)
Programming in Python 3 (Summerfield)
Python Programming Patterns (Christopher)
Hello World! (Sande)

If you are interested in getting a free book, come to #bostonpython Project Night on April 12!

Python Screencast: Install/Setup "SST Web Test Framework" on Ubuntu 12.04

2012-03-24T09:59:24-07:00

I uploaded a 5 minute video/screencast showing how to install and setup SST Web Test Framework on Ubuntu (Precise Pangolin 12.04).

I step through: creating a virtualenv, installing SST from PyPI, and creating a basic automated web test:

http://www.youtube.com/watch?v=LpSvGmglZPI

the following steps are essentially a transcript of what I did...

install system package dependencies:

$ sudo apt-get install python-virtualenv xvfb

create a "virtualenv":

$ virtualenv ENV

active the virtualenv:

$ cd ENV
$ source bin/activate
(ENV)$

* notice your prompt changed, signifying the virtualenv is active

install SST using `pip`:

(ENV)$ pip install sst

Now SST is installed. You can check the version of SST:

$ sst-run -V

Codeswarm - Python Core Development Visualization

2012-03-08T09:14:52-08:00

particle visualization of Python core development commits: Jan 1, 2010 - Mar 06, 2012

http://www.youtube.com/watch?v=IQPuU_YtN8Q

data source is the commit log from cpython mercurial trunk:

$ hg clone http://hg.python.org/cpython

... trimmed to show development activity since Jan 1, 2010.

(images making up this video were rendered with: Codeswarm)

SST - Automated Web Page Profiling (Python)

2012-03-05T06:04:10-08:00

SST - Web Test Framework: http://testutils.org/sst

The latest release of SST (0.2.0) adds the ability to capture HAR (HTTP Archive format) output for pageload performance tracing/profiling.

New SST doc section: Performance tracing with Browsermob Proxy (HAR)

The HAR format is based on JSON, and is used by tools that consume/produce data collected by monitoring HTTP communication. These files contain a log of HTTP client/server conversation and can be used for additional analysis of page load performance.

The capture is achieved by routing browser requests through BrowserMob Proxy, which records web page loads while your tests run. SST will launch the proxy and save output to .har files if you enable the `--browsermob` command line option. HAR files are saved in the results directory for each page load.

HAR files can be viewed/analyzed with various tools, such as `harviewer`:

I created a screencast demo showing it all together:

Automated Web Page Profiling : SST + BrowserMob

SST 0.2.0 Release Announcement (and codebase visualization)

2012-02-27T09:47:52-08:00

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

Version 0.2.0 Released!

SST version 0.2.0 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

Development Update

So... what's up with SST development? is it active?

I ran Gource against the trunk branch and created an awesome visualization (with soundtrack!) to show off development activity:

video: link (with soundtrack)
video: link (no soundtrack)

The magic incantation of `gource` to produce the video:

$ gource \
    -s .3 \
    -1280x720 \
    --auto-skip-seconds .3 \
    --multi-sampling \
    --stop-at-end \
    --hide mouse,progress \
    --file-idle-time 0 \
    --max-files 0  \
    --background-colour 222222 \
    --font-size 20 \
    --logo docs/assets/sst-logo_small.png \
    --title "SST Development - lp:selenium-simple-test" \
    --output-ppm-stream - \
    --output-framerate 30 \
    | ffmpeg -y -r 30 -f image2pipe -vcodec ppm -i - -b 2048K movie.mp4

(rendered on Ubuntu 11.10, audio mixed with Pitivi)

Python - Matplotlib and Numpy on Debian/Ubuntu

2012-01-29T12:38:01-08:00

There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.

However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:

$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv

Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work:

$ virtualenv env
$ cd env
$ source bin/activate
(env)$ pip install numpy matplotlib

...
...
Successfully installed numpy matplotlib
Cleaning up...

(tested on Ubuntu Oneiric 11.10 and Ubuntu Precise 12.04 alpha)

Officially Introducing "SST" (Python Web Test Framework)

2012-01-03T06:52:24-08:00

"SST (selenium-simple-test) is a framework built on Selenium WebDriver, using Python to make writing functional web tests easier with code."

Since early 2011, I have been working for Canonical on the Infrastructure Systems Development team (Core Dev Ops).

[pic of canonical-isd-hackers at UDS-P-Orlando]

A by-product of our recent development efforts is a web testing framework. It has been available on Launchpad for a while, but I've never really announced it in public. We are using SST internally, and I want to expose it to a wider audience.

What is Selenium WebDriver?

Selenium (WebDriver) is a popular open-source library for automating browsers. It can be used to create functional/acceptance tests of a web application. The Selenium client bindings provide API's that allow you to programatically drive a browser and access web content/elements. The bindings are available and supported for many languages and platforms.

While working directly with Selenium API's from code is fine for ad-hoc browser interaction, it is rather low-level and lacks things necessary for creating suites of automated web tests. For larger-scale testing, you will soon want to use a framework to help organize, execute, and report.

Introducing SST...

SST aims to keep things simple.

Tests are made up of scripts, created by composing actions that drive a browser and assert conditions. You have the flexibility of the full Python language, along with a convenient set of functions to simplify web testing.

SST framework consists of:

user actions and assertions (API) in Python
test case loader (generates/compiles scripts to unittest cases)
console test runner
data parameterization/injection
selectable output reports
selectable browsers
headless (xvfb) mode
screenshots on errors

Test output can be displayed to the console, saved as an HTML report, or JUnit-compatible XML for compatibility with CI systems.

SST is free open source software (Apache Licensed). SST is primarily being developed on Linux, specifically Ubuntu. It should work fine on other platforms, but any issues (or even better - patches) should be reported on the Launchpad project:

https://launchpad.net/selenium-simple-test

I just uploaded SST 0.1.0 to PyPI:

http://pypi.python.org/pypi/sst

go ahead, give it a try:

`$ [sudo] pip install sst`

documentation and more info:

http://testutils.org/sst

a sample test script in SST:

from sst.actions import *

go_to('http://www.ubuntu.com/')
assert_title_contains('Ubuntu homepage')

Here is the development progress of SST (shown as a code_swarm visualization) over the past 8 months:

Special thanks to all the SST code committers so far:

Danny Tamez
Julien Funk
Kenneth Koontz
Leo Arias
Lukasz Czyzykowkski
Rick McBride
Sidnei da Silva

Extra special thanks to SST's initial creator:

Michael Foord

Happy New Year!

-Corey Goldberg

"Web Performance Testing night" in Boston/Cambridge - Python Meetup Dec. 19

2011-12-21T08:46:12-08:00

I will be speaking at: "Web Performance Testing, lightning talks, and beers" at the Boston Python User Group.

Event Info: http://meetup.bostonpython.com/events/36664122/
Date: Monday, December 19, 2011, 7:00 PM
Location: Microsoft NERD, Cambridge, MA

"Corey Goldberg of Canonical and Dan Kuebrich of Tracelytics will tag-team to tell us about web performance testing, and a few interesting tools they've built."

Free pizza and beer!
Come Join!

Edit/Update:
Thanks to all who attended! The night was a big success.

The slides from my portion of the presentation are posted here:
webperf.html

Python - Stock Quotes From Google Finance

2011-09-27T05:46:17-07:00

Quick example of retrieving stock quotes from Google Finance in Python:

#!/usr/bin/env python

import json
import pprint
import urllib2


def get_stock_quote(ticker_symbol):   
    url = 'http://finance.google.com/finance/info?q=%s' % ticker_symbol
    lines = urllib2.urlopen(url).read().splitlines()
    return json.loads(''.join([x for x in lines if x not in ('// [', ']')]))


if __name__ == '__main__':
    quote = get_stock_quote('IBM')
    print 'ticker: %s' % quote['t']
    print 'current price: %s' % quote['l_cur']
    print 'last trade: %s' % quote['lt']
    print 'full quote:'
    pprint.pprint(quote)

* note: all values in the returned dict object are Unicode strings.

Output:

ticker: IBM
current price: 174.51
last trade: Sep 26, 4:00PM EDT
full quote:
{u'c': u'+5.17',
 u'ccol': u'chg',
 u'cp': u'3.05',
 u'div': u'0.75',
 u'e': u'NYSE',
 u'ec': u'0.00',
 u'eccol': u'chb',
 u'ecp': u'0.00',
 u'el': u'174.51',
 u'el_cur': u'174.51',
 u'elt': u'Sep 26, 6:07PM EDT',
 u'id': u'18241',
 u'l': u'174.51',
 u'l_cur': u'174.51',
 u'lt': u'Sep 26, 4:00PM EDT',
 u'ltt': u'4:00PM EDT',
 u's': u'2',
 u't': u'IBM',
 u'yld': u'1.72'}

Python - Getting Started With Selenium WebDriver on Ubuntu/Debian

2011-07-11T09:48:33-07:00

This is a quick introduction to Selenium WebDriver in Python on Ubuntu/Debian systems.

WebDriver (part of Selenium 2) is a library for automating browsers, and can be used from a variety of language bindings. It allows you to programmatically drive a browser and interact with web elements. It is most often used for test automation, but can be adapted to a variety of web scraping or automation tasks.

To use the WebDriver API in Python, you must first install the Selenium Python bindings. This will give you access to your browser from Python code. The easiest way to install the bindings is via pip.

On Ubuntu/Debian systems, this will install pip (and dependencies) and then install the Selenium Python bindings from PyPI:

 
$ sudo apt-get install python-pip
$ sudo pip install selenium

After the installation, the following code should work:

 
#!/usr/bin/env python

from selenium import webdriver

browser = webdriver.Firefox()
browser.get('http://www.ubuntu.com/')

This should open a Firefox browser sessions and navigate to http://www.ubuntu.com/

Here is a simple functional test in Python, using Selenium WebDriver and the unittest framework:

 
#!/usr/bin/env python

import unittest
from selenium import webdriver


class TestUbuntuHomepage(unittest.TestCase):
    
    def setUp(self):
        self.browser = webdriver.Firefox()
        
    def testTitle(self):
        self.browser.get('http://www.ubuntu.com/')
        self.assertIn('Ubuntu', self.browser.title)
        
    def tearDown(self):
        self.browser.quit()


if __name__ == '__main__':
    unittest.main(verbosity=2)

Output:

 
testTitle (__main__.TestUbuntuHomepage) ... ok

----------------------------------------------------------------------
Ran 1 test in 5.931s

OK

Python - Reading MP3 Meta Information with 'mpeg1audio'

2011-07-09T10:25:26-07:00

'mpeg1audio' is a Pure Python MPEG Audio Layer 1, 2 and 3 meta information retrieval package. It is capable of retrieving duration, bitrate, average bitrate, sample count, etc.

Here is an example of using mpeg1audio for getting meta data from a directory of MP3 files.

Code:

 
#!/usr/bin/env python

import glob
import mpeg1audio  # (https://github.com/Ciantic/mpeg1audio/)

for f in sorted(glob.glob('*.mp3')):
    mp3 = mpeg1audio.MPEGAudio(f)
    mb = '%.2f' % (mp3.size / 1048576.0)
    fn = f.replace('.mp3', '')
    print '%s (%s) [%dk] %s MB' % (fn, mp3.duration, mp3.bitrate, mb)

Output:

 
Eminem - Buffalo Bill (0:03:56) [253k] 7.15 MB
Minor Threat - Betray (0:03:02) [180k] 3.92 MB
Social Distortion - Bakersfield (0:06:24) [320k] 14.68 MB
Social Distortion - Diamond In The Rough (0:04:34) [320k] 10.49 MB
Social Distortion - Prison Bound (0:05:24) [227k] 8.81 MB
Social Distortion - When She Begins (0:05:02) [320k] 11.54 MB

Python - processing GMail IMAP email

2011-07-07T14:34:40-07:00

Here is an example of processing your GMail IMAP email in Python.

The script below will:

login to GMail account using IMAP
open your Inbox
retrieve and print all messages
close mailbox
logout

#!/usr/bin/env python

import imaplib

USER = 'username@gmail.com'
PASSWORD = 'xxx'

mail = imaplib.IMAP4_SSL('imap.gmail.com', 993)
mail.login(USER, PASSWORD)
mail.select('Inbox')

status, data = mail.search(None, 'ALL')
for num in data[0].split():
    status, data = mail.fetch(num, '(RFC822)')
    print 'Message %s\n%s\n' % (num, data[0][1])
   
mail.close()
mail.logout()

Python - Taking Browser Screenshots With No Display (Selenium/Xvfb)

2011-07-04T10:41:18-07:00

In my last two blog posts, I showed examples of using Selenium WebDriver to capture screenshots, and running in a headless (no X-server) mode.

This example combines the two solutions to capture screenshots inside a virtual display.

To achieve this, I use a combination of Selenium WebDriver and pyvirtualdisplay (which uses xvfb) to run a browser in a virtual display and capture screenshots.

the setup you need is:

Selenium 2 Python bindings: PyPI
pyvirtualdisplay Python package (depends on xvfb): PyPI

On Debian/Ubuntu Linux systems, you can install everything with:

$ sudo apt-get install python-pip xvfb xserver-xephyr
$ sudo pip install selenium

once you have it setup, the following code example should work:

#!/usr/bin/env python

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox()
browser.get('http://www.google.com')
browser.save_screenshot('screenie.png')
browser.quit()

display.stop()

this will:

launch a virtual display
launch Firefox browser inside the virtual display
navigate to google.com
capture and save a screenshot
close the browser
stop the virtual display

Python - Headless Selenium WebDriver Tests using PyVirtualDisplay

2012-01-02T07:25:19-08:00

I need to integrate my functional UI tests (Selenium/WebDriver) with my Jenkins CI system. The problem is that my Jenkins CI server has no display, so I must run my GUI tests in a headless X-server.

A colleague pointed me to PyVirtualDisplay, a Python wrapper for Xvfb and Xephyr.

This makes running headless Python Selenium/WebDriver tests very easy.

Here is some Python code showing WebDriver with a virtual display provided by Xvfb:

    
#!/usr/bin/env python

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

# now Firefox will run in a virtual display. 
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

install PyVirtualDisplay on Ubuntu/Debian:

$ sudo apt-get install xvfb python-pip
$ sudo pip install pyvirtualdisplay

Python - Selenium WebDriver - Capture Screenshot

2011-07-12T08:30:48-07:00

Example of capturing a screenshot from Selenium WebDriver in Python:

 
#!/usr/bin/env python
from selenium import webdriver

browser = webdriver.Firefox()
browser.get('http://www.ubuntu.com/')
browser.save_screenshot('screenie.png')
browser.quit()

selenium/webdriver bindings for Python: http://pypi.python.org/pypi/selenium

Performance and Scalability Testing with Python and Multi-Mechanize

2011-03-31T06:28:03-07:00

I put together a slide-deck to help introduce Multi-Mechanize. I wanted something a little friendlier and easier to digest than "go read the project wiki". (It is also the basis of a presentation/talk I might give someday).

Hopefully, this will help people better understand what the project is all about.

Performance and Scalability Testing with Python and Multi-Mechanize

linux-metrics - Python Package - System Metrics/Stats for Linux

2011-03-22T06:20:24-07:00

I just released my Python package: 'linux-metrics'

linux-metrics contains Python modules for getting OS metrics on systems running the Linux (2.6) kernel. It is a pure python library with no external dependencies.

This project is under development, and nowhere near comprehensive. Only basic stats for major subsystems are provided (Processor/CPU, Disk, Memory, Network). Hopefully, more are coming.

Project Home: http://linux-metrics.googlecode.com
PyPI Entry: http://pypi.python.org/pypi/linux-metrics

Going to PyCon 2011!

2011-01-25T14:04:40-08:00

This year I am attending my first PyCon (the annual Python community conference).

I will be in Atlanta: March 10-13.

If anyone is interested in meeting up or collaborating while I'm there, get in touch:

Twitter: @cgoldberg
Homepage/Info: goldb.org

Python - Search a Local or Remote Splunk Server

2010-12-01T06:40:24-08:00

Some basic instructions for searching Splunk from Python...

First, you must install Splunk on the machine you will run the Python script from. Splunk installs its own Python interpreter that you can use to run your code. I am using Splunk 4.14, which includes Python 2.6.

(It looks like you can set some environment variables and install a few Python dependencies along with the Python SDK and get this going "outside" of Splunk. But the easiest option is just to run on their interpreter).

To run your own Python scripts on Splunk's interpreter:
- save script into Splunk's "bin" directory
(usually "/opt/splunk/bin" or "C:\Program Files\Splunk\bin")
- go to the "bin" directory and run:
splunk cmd python your_script.py

So...
What goes in your Python code?

First, import the modules you will need:

 
import time
import splunk.auth
import splunk.search

Next, authenticate and get a session key.

For the local splunk host:

 
key = splunk.auth.getSessionKey('user', 'password')

If you are going to search a remote splunk host, you must authenticate against it by adding the "hostPath" parameter:

 
key = splunk.auth.getSessionKey('user', 'password', hostPath='https://mysplunk:8089')

Tips:
- use https, even if you are not using ssl in your splunk web interface
- 'admin' user doesn't seem to work. user a normal user/password.

Next, submit a search job.

For a local search:

 
job = splunk.search.dispatch('search index="os" *', earliest_time='-15m')

For a remote search, use the "hostPath" parameter again:

 
job = splunk.search.dispatch('search index="os" *', earliest_time='-15m', hostPath='https://mysplunk:8089')

print the job details:

 
print job

wait for the results:

 
while not job.isDone:
    time.sleep(.25)

print results

 
for result in job.results:
    print result

Altogether in a Python script:

 
#!/usr/bin/env python
# Corey Goldberg - 2010
# 
#  search a remote splunk server
#
#  instructions:
#   - save script into splunk's "bin" directory
#     (usually "/opt/splunk/bin" or "C:\Program Files\Splunk\bin")
#   - go to the "bin" directory and run: 
#     $ splunk cmd python my_script.py
#



import time
import splunk.auth
import splunk.search



SPLUNK_SERVER = '192.168.12.173'
USER_NAME = 'foo'
PASSWORD = 'secret'
SEARCH_STRING = 'search index="os"'
EARLIEST_TIME = '-15m'



def main():
    # authenticate
    key = splunk.auth.getSessionKey(USER_NAME, PASSWORD, hostPath='https://%s:8089' % SPLUNK_SERVER)
    print 'auth key:\n%s' % key
    
    # submit a search job
    job = splunk.search.dispatch(SEARCH_STRING, earliest_time=EARLIEST_TIME, hostPath='https://%s:8089' % SPLUNK_SERVER)
    print 'job details:\n%s' % job

    # wait for results
    while not job.isDone:
        time.sleep(.25)
    
    print 'results:'    
    for result in job.results:
        print result
          
          

if __name__== '__main__':
    main()

Python - Shorten a URL Using Google's Shortening Service (goo.gl)

2010-10-08T09:02:44-07:00

Using Python to shorten a URL with Google's shortening service (goo.gl):

 
#!/usr/bin/python 
#  Corey Goldberg - 2010

import json
import urllib
import urllib2


def shorten(url):
    gurl = 'http://goo.gl/api/url?url=%s' % urllib.quote(url)
    req = urllib2.Request(gurl, data='')
    req.add_header('User-Agent', 'toolbar')
    results = json.load(urllib2.urlopen(req))
    return results['short_url']


if __name__ == '__main__':
    print shorten('http://www.goldb.org/')
    print shorten('www.yahoo.com')

You give it a URL to shorten: shorten('http://www.goldb.org/long_url')

... and it returns a shortened URL for you: 'http://goo.gl/jh4W'

Python - Linux: Parse Network Stats From ifconfig

2010-09-24T15:04:16-07:00

I needed to get some Linux networking stats in my Python program today. Specifically, I needed 'bytes sent' and 'bytes received' counts since last reboot from the local machine.

ifconfig is a network configuration utility for Linux that you run from the command line:

corey@studio17:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:22:19:e5:07:31  
          inet addr:10.0.0.5  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::222:19ff:fee5:731/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3353822 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3052408 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3476310326 (3.4 GB)  TX bytes:256706611 (256.7 MB)
          Interrupt:17

The following function parses output from ifconfig to get the network stats I was after:

import re
import subprocess

def get_network_bytes(interface):
    output = subprocess.Popen(['ifconfig', interface], stdout=subprocess.PIPE).communicate()[0]
    rx_bytes = re.findall('RX bytes:([0-9]*) ', output)[0]
    tx_bytes = re.findall('TX bytes:([0-9]*) ', output)[0]
    return (rx_bytes, tx_bytes)

Example usage:

import re
import subprocess

def main():
    rx_bytes, tx_bytes = get_network_bytes('eth0')
    print '%s bytes received' % rx_bytes
    print '%s bytes sent' % tx_bytes
      
def get_network_bytes(interface):
    output = subprocess.Popen(['ifconfig', interface], stdout=subprocess.PIPE).communicate()[0]
    rx_bytes = re.findall('RX bytes:([0-9]*) ', output)[0]
    tx_bytes = re.findall('TX bytes:([0-9]*) ', output)[0]
    return (rx_bytes, tx_bytes)

if __name__ == '__main__':
    main()

Update: someone left an anonymous comment and mentioned you can just read from proc/net/dev rather than using ifconfig. I modified his code sample and came up with this:

def get_network_bytes(interface):
    for line in open('/proc/net/dev', 'r'):
        if interface in line:
            data = line.split('%s:' % interface)[1].split()
            rx_bytes, tx_bytes = (data[0], data[8])
            return (rx_bytes, tx_bytes)

Example Usage:

def main():
    rx_bytes, tx_bytes = get_network_bytes('eth0')
    print '%s bytes received' % rx_bytes
    print '%s bytes sent' % tx_bytes
      
def get_network_bytes(interface):
    for line in open('/proc/net/dev', 'r'):
        if interface in line:
            data = line.split('%s:' % interface)[1].split()
            rx_bytes, tx_bytes = (data[0], data[8])
            return (rx_bytes, tx_bytes)

if __name__ == '__main__':
    main()