tail -f findings.out

A better way to search for methods of Python objects

Samuel Huckins — Sun, 08 Nov 2009 22:08:10 +0000

Python’s introspection abilities are quite extensive and useful. They are also well-documented, so I won’t go into the basics here. Check out this article if you need a good overview. N.B.: discussion and code below applies to both methods and attributes. I will simply refer to “methods” for simplicity.

Beyond simply listing the methods of an object, however, I often find that I want to search through them for something in particular. And eyeballing the output of dir(obj) is only efficient in the simplest of cases. hasattr(obj, “method”) won’t get you far either, as you need to match the “method” name exactly. What if you just have a good guess about the name of a method based on what you need to do? What if you want to know everything you can do with directories in the os module or ISO related methods in datetime.date? I haven’t found anything to help with this sort of problem yet. Approaches like writing a loop to do the search every time or perusing the pertinent API docs are too circuitous for such quick questions. So let’s make a tool to do this more easily!

We start with knowing some object and some string containing all or part of the method(s) we’re interested in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def mf(obj, term):
"""
Searches through the methods and attributes defined for obj,
looks for those containing the term passed.
Returns all matches or a warning if none found.
"""
meths = dir(obj)
match_meths = []
for meth in meths:
if meth.rfind(term) != -1:
match_meths.append(meth)
if match_meths:
print match_meths
else:
print "No matches!"

This simply iterates over the methods and attributes defined for the object passed and looks for the term passed within the name of each. rfind returns the highest index of the substring passed in the string it’s called on and returns -1 if no match is found. Once the matches are collected they are printed out. An example:

1
2
3
4

>>>from method_finder import mf
>>>import os
>>>mf(os, "dir")
['chdir', 'curdir', 'fchdir', 'listdir', 'makedirs', 'mkdir', 'pardir', 'removedirs', 'rmdir']

For convenience let’s make it available on the Python interactive prompt at every load without any extra effort. This is easy enough using a .pythonstartup file. This file can be used to load various useful items like tab completion for interactive Python sessions. If you haven’t used it before, you’ll need to add the following to your .bashrc:

1	export PYTHONSTARTUP="$HOME/.pythonstartup"

Then in your home directory create a .pythonstartup file containing something like the following:

1
2
3
4
5
6
7
8
9
10
11

import os
# Adds mf(obj, "str") allowing search for methods matching 'str' on obj
# as well as obinfo(obj) returning lots of info on obj
util_loc = "/home/shuckins/code/code_homerepo/python-programming/utilities"
if os.path.isdir(util_loc):
import sys
sys.path.append(util_loc)
from utils import mf, obinfo
sys.path.remove(util_loc)
del sys
del os

You’ll need to change the path specified to wherever you placed utils.py containing the mf() function of course. You can download mine here if you’d like. The del statements are to clean up any trace of this operation once you get to the interactive interpreter. This way you get the added functions without mucking up your namespace.

This also loads in an obinfo(obj) function that I included in the same utils.py file. This is based on the interrogate() function written by Patrick O’Brien in the introspection article mentioned above. My version just adds a check for objects without docstrings and prints more of the docstring:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def obinfo(obj):
"""
Print useful information about object.

From http://www.ibm.com/developerworks/library/l-pyint.html
"""
if hasattr(obj, '__name__'):
print "NAME: ", obj.__name__
if hasattr(obj, '__class__'):
print "CLASS: ", obj.__class__.__name__
print "ID: ", id(obj)
print "TYPE: ", type(obj)
print "VALUE: ", repr(obj)
print "CALLABLE:",
if callable(obj):
print "Yes"
else:
print "No"
if hasattr(obj, '__doc__'):
doc = getattr(obj, '__doc__')
doc = doc.strip()
topfive = doc.split('\n')[0:4]
print "DOC: ", "\n".join(topfive)
else:
print "No docstring. Yell at the author."

It’s fairly useful:

1
2
3
4
5
6
7
8

>>> obinfo(["some", "list"].count)
NAME: count
CLASS: builtin_function_or_method
ID: 139922087930696
TYPE: <type 'builtin_function_or_method'>
VALUE: <built-in method count of list object at 0x7f422658ef80>
CALLABLE: Yes
DOC: L.count(value) -> integer -- return number of occurrences of value

Quick tips for NVIDIA and ATI graphics configuration repairs on Ubuntu

Samuel Huckins — Thu, 05 Nov 2009 02:15:10 +0000

While graphics card and display configuration on Ubuntu has come a long way from the days of always having to edit xorg.conf by hand, I still run into issues now and again. I almost never have problems any more when setting up a new system. But changing cards in an existing system is another story, especially if switching between NVIDIA and ATI.

Today I did indeed switch from an NVIDIA to an ATI graphics card on an Ubuntu 9.10 machine. I had the nvidia-glx kernel mod in place, and xorg.conf specified the NVIDIA related modules to load, so just switching out the card and rebooting resulted in a flickering text login, no more desktop. This post covers the commands needed to install the appropriate kernel module and reconfigure Xorg in such situations.

After putting in your new card if you aren’t able to get to your desktop/get sent to a text login when you boot/get an Xorg error, reboot and hit Escape when GRUB prompts you. Boot into the recovery mode for the latest kernel shown in the list. Drop into root prompt with networking when prompted.

NVIDIA to ATI

This assumes you have an ATI card in place. Once at the root prompt run:

1	apt-get install xorg-driver-fglrx

This will remove nvidia-glx if you have it installed and install the driver for ATI Radeon and FireGL graphics cards. After this is complete you need to reconfigure xorg.conf to use the new module. Fear not, no manual editing should be required to get a basic configuration working. Just run:

1	aticonfig --initial

This script comes with the xorg-driver-fglrx. Running this will add screen and device sections in /etc/X11/xorg.conf appropriate for an ATI card, backing up /etc/X11/xorg.conf beforehand. If you added any special configurations for your screen section before you might want to edit the file manually and copy the customizations into the appropriate new sections. The sections already in xorg.conf will still be there, just commented out or not actually called when loading X. Next run:

startx

If all goes well you will be back to a working desktop. You can also check from the command line by running “fglrxinfo”.

For more information on using the ATI drivers on Ubuntu see this wiki article.

ATI to NVIDIA

And in the reverse situation:

1
2
3

apt-cache search nvidia-glx
# Install the latest stable one. Currently:
apt-get install nvidia-glx-185

And the equivalent for aticonfig:

1	nvidia-xconfig

As above:

startx

For more information on using the NVIDIA drivers on Ubuntu see this wiki article.

No guarantees here, but these have worked for me.

Catching warnings from the MySQLdb module

Samuel Huckins — Mon, 26 Oct 2009 03:53:50 +0000

The MySQLdb Python module implements the Python DB API for MySQL. I’ve written about its use before. MySQL issues warning messages in a number of circumstances and PEP 249 (which specifies the Python DB API) describes a Warning error message to be included.

One issue I ran into recently was how to catch warnings thrown by this module when running queries. Oftentimes tutorials or forum discussions that cover warnings in the context of MySQLdb describe how to filter them (they can clog up script output). But in a recent case I wanted to grab and check the warning, logging a dependent result. I had hoped this clean implementation would work in a method used for all calls to the MySQL DB:

1
2
3
4
5
6
7
8
9
10
11

try:
cursor.execute(query)
except MySQLdb.Error, e:
raise e
except MySQLdb.Warning, e:
raise e
finally:
data = cursor.fetchall()
rows_returned = cursor.rowcount
cursor.close()
db.close()

But the warnings just went right through. Instead I needed the warnings module’s assistance:

1
2
3
4
5
6
7
8
9
10
11
12

import warnings
with warnings.catch_warnings():
warnings.simplefilter('error', MySQLdb.Warning)
try:
cursor.execute(query)
except MySQLdb.Error, e:
raise e
finally:
data = cursor.fetchall()
rows_returned = cursor.rowcount
cursor.close()
db.close()

This catches the warnings and raises them as errors, although their class is still correct, allowing a clean implementation to call the above code (wrapped into a method called do_query):

1
2
3
4
5

try:
self.do_query(make_cool_table)
logger.info("Created cool_table table.")
except MySQLdb.Warning:
logger.info("cool_table already exists.")

Viewing all users on a Linux system

Samuel Huckins — Sun, 04 Oct 2009 20:35:59 +0000

There are a number of widely-used and stable utilities on Linux systems that allow you to view information related to users. You can see who’s logged in with who, get info on a particular user with finger, see who you are with whoami and see who logged in last with last and lastlog. And there are commands to create, remove, and change users and their groups.

But what if you just want to see all the users defined on the entire system? Looking in /home won’t show you users that don’t have a home dir or one not located there. All users are indeed listed in /etc/passwd, but having to look through this file every time you just want a list of users is tedious. I haven’t found a utility that performs this role.

This was somewhat surprising, as it’s not an uncommon need. Perhaps I’ve missed some essential program along my Linux journey. If so, feel free to enlighten me in a comment In the meantime, let’s fill this gap:

1
2
3
4
5
6
7
8
9
10
11
12

# Prints all users, divided by login ability and homedir:
function userinfo {
echo "----- Users that can login -----"
awk -F":" '!/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- And have /home dir -----"
awk -F":" '!/bin\/false/ && /\/home/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- Users that can't login -----"
awk -F":" '/\/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd
echo ""
}

This function should work on any *nix system. Place it inside your ~/.bashrc, reload that (. ~/.bashrc), and try out “userinfo”:

Three sections are displayed showing usernames, user IDs, and homedirs for users in /etc/passwd who:

Don’t have /bin/false for a shell (i.e. users that can login)
Also have a homedir under /home (i.e. the users that aren’t for system processes)
Do have /bin/false for a shell (i.e. users that can’t login)

Note that the users that can login category only shows users to which one could switch with “sudo su – USERNAME” from a shell. This doesn’t mean anything about whether they can login via SSH or other access methods.

Great open source virtualization with VirtualBox

Samuel Huckins — Sun, 04 Oct 2009 17:49:26 +0000

My first forays into virtualization were at work and utilized VMware’s myriad offerings (Player, Workstation, Server, ESX, VI). I was impressed by the capabilities of VMware’s products, and of course excited by the possibilities of virtualization. But for home use I didn’t push past getting VMware Player installed. I was annoyed at not being about to create my own virtual machines with the free products, and the product not being open source didn’t generate much excitement for me either.

So when I finally decided to set up some VMs for my own use, my criteria were straightforward: I wanted a broadly supported, performant, free virtualization solution that was open source. There is quite a range of options out there, e.g. Xen and OpenVZ. I decided to give Sun’s VirtualBox a try and have been nothing but impressed so far:

I’ve found VirtualBox easy to install, quite fast, attractive and easy to use.

Installation

Installation on Ubuntu 9.04 was fairly simple following this guide. You add the VirtualBox repository, install the package, and follow the configuration wizard. I didn’t have to do anything special kernel-wise that the installation guide mentions. One thing they leave out: you need to add your user to the vboxusers group. Easy enough:

1	sudo usermod -a -G vboxusers YOURUSERNAME

The manual describes installing on a wide range of other platforms, and there are plenty of tutorials out there as well.

First steps

After VirtualBox is installed, you need to create a VM and then install an OS on it. Click “New” in the top left of the VirtualBox window and follow the wizard. You might run into an issue in this process (only annoyance I’ve encountered so far) when selecting the OS Type and Version. For me the OS options in the drop down show up as very light grey on white until you hover over them:

Not a critical issue, you can see the options by hovering down the list.

After you’ve followed the wizard to create the VM and provided it with storage space and other resources, you’ll end up at the VirtualBox main screen. At this point, select the VM you just made and click settings. Then go to CD/DVD-ROM on the left and check “Mount CD/DVD Drive”. If you have an actual install CD/DVD you want to use, select Host CD/DVD Drive. You can also just download the ISO for the OS you want to install and select ISO Image File:

After selecting your drive or ISO for installation, save the settings and start the VM. A new window will open displaying the output of the running VM. If all goes well, you should be shown the installation menu for the CD or ISO you’ve mounted.

Miscellaneous

You can take snapshots of your VMs in case you need to roll back to a known safe state. When the VM is turned off, go to the Snapshots tab to take new snapshots, review current ones, and roll back if needed.
Once you click on the VM window your output is sent to it alone. To get back out and interact with your own desktop, press the right CTRL key (configurable). Or install VirtualBox Guest Additions (see below) and not worry about this at all!
Your VMs (and assorted configuration info) are stored in ~/.VirtualBox:
Installing VirtualBox Guest Additions is quite helpful. Aside from graphical improvements, after it’s installed you don’t have to click inside a VM to get focus. You just move your pointer into the VM and it has focus. Move out and the host GUI has control again.

Resources

Homepage
Screenshots
Installation on Ubuntu 9.04
Manual (very well done)

Editing Trac comments at the DB layer

Samuel Huckins — Sat, 03 Oct 2009 00:58:34 +0000

Today I needed to alter the content of a comment made on a Trac ticket. I didn’t anticipate this would be a difficult task, but I wasn’t too familiar with Trac’s DB schema. So to save you a few consternated queries, here’s the spoiler. Comments on tickets aren’t kept within their own table (as you might expect) but are instead stored in the ticket_change table.

In my particular case the Trac DB was MySQL, so the examples I’ll provide will be MySQL queries. If you have an SQLite-backed Trac the operations should be the same, you’ll just need to use SQLite’s syntax.

Say the ticket with the naughty comment is #1234. To see all comments on this ticket:

1
2
3

SELECT * FROM trac.ticket_change
WHERE FIELD = "comment"
AND ticket = 1234;

And to update a comment:

1
2
3
4
5

UPDATE trac.ticket_change
SET newvalue = "New hotness"
WHERE FIELD = "comment"
AND ticket = 1234
AND oldvalue = "Old and busted";

Happy revisionizing.

Permanently remember password for gksudo

Samuel Huckins — Sun, 27 Sep 2009 03:44:49 +0000

I’m all about security practices on my remote server systems. On my firewalled desktop systems, however, I’d rather err on the side of convenience. I prefer to not have to type my password to use sudo or gksudo (graphical sudo). If I type my password to login to my account once, I want to have access to everything unfettered thereafter.

While there are plenty of posts on how to get sudo without password going, I never could seem to find information on how to do this for gksudo as well. So here’s how to do that.

First open the gconf-editor (gnome configuration):

1	gksu gconf-editor

Once that opens, go to /apps/gksu, and check “save-to-keyring”:

This will remember your password for gksudo, so you won’t get any more auth popups.

Get useful image information on the command line

Samuel Huckins — Sun, 27 Sep 2009 03:20:57 +0000

ImageMagick is an incredible set of programs that allow you to get all sorts of information on image files from the command line, as well as batch edit and generally alter images of many varieties. You can read all about these facilities on ImageMagick’s well-made documentation site.

What I often want, however, is simple: a command to get commonly desired information from any image file. Alias to the rescue!

1	alias imginfo="identify -format '-- %f -- \nType: %m\nSize: %b bytes\nResolution: %wpx x %hpx\nColors: %k'"

This produces a succinctly helpful output:

Often this is all I need for basic tasks. And if more is needed there’s always the full ImageMagick suite.

You can alter the format to whatever you prefer, using the wide variety of variables.

64-bit Ubuntu: Not a problem

Samuel Huckins — Sun, 27 Sep 2009 02:39:20 +0000

I recently had occasion to upgrade one of my home PCs. I went with a Core 2 Duo for the processor, and decided to give 64-bit Ubuntu (9.04, Jaunty Jackelope) a shot. I found it quite easy, with only a few adjustments needed to my usual system setup steps. General recommendations aside, I wanted to share a few particular application notes I found useful.

Adobe Air: I love Pandora One, TweetDeck, and a number of other Air applications, so getting this working on 64-bit was a requirement. While there aren’t any 64-bit packages for Air on Linux, the 32-bit version works just fine. I was a bit dismayed when I saw the lengthy instructions Adobe has listed for this process, but I must admit they did work perfectly. So grab the eyedrops and hit up this page. Be sure to follow the step at the end for 9.04, otherwise the Air apps won’t be able to connect to the internet on a 64-bit system.
Pandora One: You have to subscribe to get access to Pandora One, but rest assured it works just fine on 64-bit Ubuntu.
Java: This used to present some complications for Ubuntu users, but following this guide I had things running in no time.
Flash: All I needed to run was “sudo apt-get install flashplugin-nonfree”. Flash in Firefox worked perfectly.
TweetDeck: TweetDeck is a great application to keep better track of your Twitter feed. v0.30.5 works just fine on 64-bit Jaunty.

While the benefits of running a 64-bit OS aren’t as substantial for most desktop tasks, there is still a performance boost across the board. And at least for the applications I love and use everyday, the packages are definitely at a stable state for running under 64-bit.

An improved ruby debugger invocation

Samuel Huckins — Sun, 27 Sep 2009 01:17:41 +0000

I’d been wanting to write a post for some time on improvements I’ve found useful in using Ruby’s debugger. Friend Trevor Rosen beat me to the proverbial punch, however. Give his post a read first, I’ll wait.

Now on top of his suggestions I have one more refinement to add. I’ve defined a snippet in vim for whenever I need to use the debugger. You should be able to define the same in whatever editor you use (and if your editor can’t do snippets, get a real one). When in an .rb file, “debug” + Tab becomes:

1	require 'ruby-debug'; Debugger.settings[:autoeval] = true; debugger; rubys_debugger = "annoying"

In addition to the format Trevor suggested (which calls the necessary module, turns on auto evaluation, and calls the debugger itself) this adds a simple assignment after the debugger is called. I often find this necessary because I want to debug right at the very end of a given suite, a conditional, a method, etc. But if you invoke the debugger as the last statement in such a situation, it won’t actually be called. Here’s an example file:

1
2
3
4
5
6
7
8
9

#!/usr/bin/env ruby

def troubled_func(var1)
puts "Entering troubled function..."
secret_num = rand(10) * var1
require 'ruby-debug'; Debugger.settings[:autoeval] = true; debugger
end

troubled_func(5)

When you run this, you won’t get a debugger console. All the output you get is “Entering troubled function…” and the script exits. If instead you add the assignment I list above after debugger is called, you will get to the debugger console. Sad, but true. (You can select a less offensive assignment if you wish. I settled on it in disgust.)

Aside from this, the Ruby debugger is a useful and beautiful thing. So add the snippet and forget about it.