tail -f findings.out

Tune MySQL like a pro with MySQLTuner

Samuel Huckins — Sat, 13 Mar 2010 23:22:10 +0000

I don’t know why I didn’t know about this before (or why I forgot about it, more likely), but I came across MySQLTuner recently and was most pleasantly surprised. It’s a Perl script that only requires your MySQL user and password to provide detailed and useful checks of a running MySQL instance. First though, check out the awesome URL you download it at:

1	wget mysqltuner.pl

Yeah, that’s the actual URL that works. Pretty sweet.

Anyway, next you make it executable, run it, enter creds:

Then the goodies appear:

Oh, that query_cache_limit is tiny! At this point, no changes have been made to your setup. But at a glance you get helpful stats, validate that changes you’ve decided to make are in place, and get alerted to potential improvements.

If I knew Perl now and refreshed my MySQL tuning knowledge, I’d love to help as a maintainer of this lovely script. Maybe sometime soon…

Converting multiple images to one PDF on Linux

Samuel Huckins — Sun, 21 Feb 2010 18:12:26 +0000

There are a number of ways you can go about the process of converting image files into PDFs, most simply by opening a given image and printing it. You can select “Print to file” instead of a printer device, PDF format is the default for this type of output. But say you have a folder full of images and you want to make one big PDF of them. Here’s how you can do that:

1
2
3
4
5
6
7

cd FOLDER_WITH_IMAGES
FILES=$(ls *jpg)
mkdir temp && cd temp
for file in $FILES; do BASE=$(echo $file | sed 's/.jpg//g'); convert ../$BASE.jpg $BASE.pdf; done &&
pdftk *pdf cat output ../FINAL_NAME.pdf &&
cd ..
rm -rf temp

This loops through all the .jpg images in the directory and converts them to a PDF file of the same name. Once that is done they are all combined into FINAL_NAME.pdf by pdftk, a handy PDF utility. The temp dir business is there to make the temp PDF file removal easier.

git tip: Ignoring modifications to tracked files

Samuel Huckins — Mon, 08 Feb 2010 04:13:29 +0000

Problem

You have a file already tracked in your git repository, but you don’t want future modifications to it to be tracked.

A perfect example of this is the DB config file for Rails projects (config/database.yml). You’ll probably want to track this to keep the production and staging configuration stored and consistent. But it’s quite likely that individual development configurations will be different. Having a commit for each developer adding their own local configuration and thereby polluting it for others when they push is just silly.

Solution

1	git update-index --assume-unchanged FILENAME

The fun details

I had initially thought that git-ignore was the thing to use for this. That was wrong: git-ignore’s purpose is to allow ignoring of untracked files. In the case of the Rails DB conf example above, the file is already tracked. It has the instance information that needs to be shared and now we want to add local development details. Since it’s tracked, any modifications are going to be picked up by git-status. When you do a commit including changes to tracked files listed in .gitignore, the changes do indeed get pushed, which isn’t what we want.

When looking around for solutions to this, I saw “git rm –cached FILENAME” suggested as a way to stop tracking currently tracked files. And it does this, but it also deletes the file in the commit, which isn’t what we want either.

git update-index allows you to alter your staging area more manually than usual. With it you can perform a wide range of operations not otherwise possible through standard commands. Continuing with the Rails DB conf example, here’s what the workflow would look like:

config/database.yml with permanent instance info is added, committed, and pushed
Application code is pulled down by new developer
Local modifications made to database.yml for new developer’s local DB setup
git update-index –assume-unchanged config/database.yml

Thereafter, the changes to conf/database.yml won’t appear in status or get committed. You can continue making and committing other changes, going about business as usual. But any changes to config/database.yml won’t ever appear.

In case you decide you do want to start tracking changes on a file previously ignored, just run:

1	git update-index --no-assume-unchanged FILENAME

Thereafter changes to FILENAME will appear in git-status.

Gotcha

The annoying part about this solution is that it has to be run against every new checkout of the repo with the files whose modifications should not be tracked. The command sets a particular bit per checkout, and it’s not passed on to new checkouts.

Just as I was finishing up some fact checking for this write-up, I came across Jesper Rønn-Jensen’s post on this exact topic. If only it had come up in my initial searches! Here’s hoping the next searcher has better luck.

Fun with ASCII art

Samuel Huckins — Mon, 14 Dec 2009 06:06:02 +0000

No, in fact ASCII art isn’t dead, it’s all the rage. Now that that’s established, check out these handy programs for generating your own ASCII art messages:

1	figlet -f smslant Boxes rules. \| boxes -d parchment -a hcvc

Figure 1. boxes provides boundless entertainment.

1	figlet -f standard Also figlet. \| boxes -d mouse -a hcvc

Figure 2. figlet and its inherent awesomeness.

Spice up those bland Bash scripts with entertaining comment formats!

But seriously, these utilities can make for more interesting banners, alert messages, and so on:

boxes: Creates ASCII art boxes around entered text
figlet: Creates large ASCII art versions of entered characters

Just because it’s the command line doesn’t mean it has to be boring!

Hiding anchor tooltips on hover

Samuel Huckins — Sun, 29 Nov 2009 18:25:48 +0000

Navigation menus on websites most often consist of a list of links. When looking through a menu, it’s not unreasonable for a visitor to have their mouse cursor over some of the menu item links for a few moments as they read through the options. Using an unadorned anchor tag in this situation has a troublesome side effect:

The title attribute of the anchor tag will appear as a tooltip, potentially obscuring menu items and minimally being a useless distraction. But removing titles from navigation links isn’t a good option, since this attribute is helpful for SEO and accessibility purposes.

When discussing this problem with friend Google, I ran across many examples of how to change the display of anchor tooltips, as well as how to hide them altogether. But what I wanted was removal of the title attribute on navigation links only when the mouse cursor hovered over them, adding it back when the cursor left. The solution I settled on consisted of jQuery’s handy hover function and some basic JavaScript. hover() allows you to specify a function to call when the cursor hovers onto an object and another to call when the cursor hovers off. Perfect! Here’s a simple example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

<html>
<head>
<title>A test</title>

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js" type="text/javascript" charset="utf-8"></script>

<script type="text/javascript" charset="utf-8">
$(document).ready(function() {
// All links in main menu div
var menu_links = $('div[id=main-menu] a');
// On mouse hover
menu_links.hover(
// In: Store and remove title
function() {
old_title = $(this).attr('title');
$(this).attr('title','');
},
// Out: Replace title
function() {
$(this).attr('title', old_title);
}
);
});
</script>

</head>
<body>
<div id="main-menu">
<ul>
<li><a title="Titles are great for SEO and accessibility but don’t need to be seen by users normally!" href="zenoss.com">MenuItem1</a></li>
<li><a title="Titles are great for SEO and accessibility but don’t need to be seen by users normally!" href="zenoss.com">MenuItem2</a></li>
<li><a title="Titles are great for SEO and accessibility but don’t need to be seen by users normally!" href="zenoss.com">MenuItem3</a></li>
</ul>
</div>

<p>I am text! Hurray! <a title="bodylink1" target="_blank" href="zenoss.com">I am a normal link.</a></p>
</body>
</html>

You can verify the functionality by inspecting one of the MenuItem links with Firebug and watching the title attribute as you move the cursor on and off the same link.

After the page is finished loading, when a cursor hovers onto any links within the div of id “main-menu” the title attribute is stored and set to an empty string (so no tooltip). When the cursor leaves, the title attribute is set back to its original value. Simple but effective!

Better access to MySQL create view statements

Samuel Huckins — Mon, 16 Nov 2009 04:23:29 +0000

I’ve found that I often create several views that are variations on an initial one, providing a slightly different convenient slice of the data as the need arises. But when a view is created the syntax actually stored can be quite different than what you put in. The parser reorganizes and rewrites to make everything explicit. While better for performance in running the view, this has the unfortunate side effect of transmuting a once readable statement into a forest of parentheses and backticks that can and does induce tears.

First an example view that I created using the handy world database:

1
2
3
4
5
6
7
8

CREATE VIEW world.low_central_luzon AS
SELECT a.Name AS "City", b.Name AS "Country", a.Population, a.District
FROM world.City a
JOIN world.Country b ON (a.CountryCode = b.Code)
WHERE b.Continent = "Asia"
AND a.District = "Central Luzon"
ORDER BY a.Population
LIMIT 10;

This is a view of the 10 cites with the lowest population in the Central Luzon region of Asia. Now let’s see what actually got stored. To get a view’s create statement you can use the MySQL Query Browser and right click the view name shown in the right under Schemata, selecting “Copy SQL to Clipboard”. Or you can run a query like:

1
2
3

SELECT VIEW_DEFINITION FROM information_schema.VIEWS
WHERE TABLE_NAME = "low_central_luzon"
AND TABLE_SCHEMA = "world";

Here’s the result for the view above:

1
2
3
4
5

SELECT `a`.`Name` AS `City`,`b`.`Name`
AS `Country`,`a`.`Population` AS `Population`,`a`.`District`
AS `District` FROM (`world`.`City` `a` JOIN `world`.`Country` `b`
ON((`a`.`CountryCode` = `b`.`Code`))) WHERE ((`b`.`Continent` = 'Asia')
AND (`a`.`District` = 'Central Luzon')) ORDER BY `a`.`Population` LIMIT 10

I added the line breaks for a modicum of readablilty, the statement is stored without them. I sure don’t want to edit that version in making a variant! This example isn’t actually too bad, but for real world complicated views, it really is quite difficult to read views exported in this way. I often try to store the original create view query just to avoid this pain. But sometimes I only have the stored syntax.

I’ve created a Bash function to make this whole process less painful:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

function mysqldump_view {
# Grabs the create statement for the view passed in the db passed.
# Performs some text cleanup to make it more human readable, outputs
# to file.
USAGE="USAGE: mysqldump_view DATABASE_NAME VIEW_NAME"
opt_err="Must pass db and view."
if [ $# -ne 2 ]
then
echo $opt_err
echo $USAGE
return 1
fi
DB=$1
VIEW=$2
DEFAULT_USER="root"
OUTPUT_FILE="view-$VIEW.sql"

echo "Attempting to connect with mysql defaults, user $DEFAULT_USER..."
mysql -u $DEFAULT_USER -p --skip-column-names -e "SELECT VIEW_DEFINITION FROM \
information_schema.VIEWS where TABLE_NAME = \"$VIEW\" and TABLE_SCHEMA = \"$DB\";" \
| sed -e "1s/^$.*$$/CREATE VIEW \"$VIEW\" as\n\1/" \
-e 's/`,`/`,\n`/g' \
-e 's/ from /\nfrom /g' \
-e 's/ join /\n\tjoin /g' \
-e 's/ where /\nwhere /g' \
-e 's/ and /\n\tand /g' \
-e 's/ limit /\nlimit /g' \
-e 's/ order /\norder /g' \
-e '$s/$/;/' > $OUTPUT_FILE &&
echo "Done. View syntax written to $OUTPUT_FILE."
return 0
}

Now to get a cleaned up view you can call this function followed by the database and view names:

To get the create statements for multiple views you can use a loop. For example:

1	for VIEW in "high_central_luzon" "low_central_luzon"; do mysqldump_view world $VIEW; done

If you need some more flexibility than this I implemented it in Python after an initial fit of rage caused by trying to get various levels of nested strings properly escaped in Bash. That ended up being overly long and unnecessary, and a cooler head prevailed. But the script is here on GitHub if you want it. It creates the same file although without using sed and with some error handling.

A better way to search for methods of Python objects

Samuel Huckins — Sun, 08 Nov 2009 22:08:10 +0000

Python’s introspection abilities are quite extensive and useful. They are also well-documented, so I won’t go into the basics here. Check out this article if you need a good overview. N.B.: discussion and code below applies to both methods and attributes. I will simply refer to “methods” for simplicity.

Beyond simply listing the methods of an object, however, I often find that I want to search through them for something in particular. And eyeballing the output of dir(obj) is only efficient in the simplest of cases. hasattr(obj, “method”) won’t get you far either, as you need to match the “method” name exactly. What if you just have a good guess about the name of a method based on what you need to do? What if you want to know everything you can do with directories in the os module or ISO related methods in datetime.date? I haven’t found anything to help with this sort of problem yet. Approaches like writing a loop to do the search every time or perusing the pertinent API docs are too circuitous for such quick questions. So let’s make a tool to do this more easily!

We start with knowing some object and some string containing all or part of the method(s) we’re interested in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def mf(obj, term):
"""
Searches through the methods and attributes defined for obj,
looks for those containing the term passed.
Returns all matches or a warning if none found.
"""
meths = dir(obj)
match_meths = []
for meth in meths:
if meth.rfind(term) != -1:
match_meths.append(meth)
if match_meths:
print match_meths
else:
print "No matches!"

This simply iterates over the methods and attributes defined for the object passed and looks for the term passed within the name of each. rfind returns the highest index of the substring passed in the string it’s called on and returns -1 if no match is found. Once the matches are collected they are printed out. An example:

1
2
3
4

>>>from method_finder import mf
>>>import os
>>>mf(os, "dir")
['chdir', 'curdir', 'fchdir', 'listdir', 'makedirs', 'mkdir', 'pardir', 'removedirs', 'rmdir']

For convenience let’s make it available on the Python interactive prompt at every load without any extra effort. This is easy enough using a .pythonstartup file. This file can be used to load various useful items like tab completion for interactive Python sessions. If you haven’t used it before, you’ll need to add the following to your .bashrc:

1	export PYTHONSTARTUP="$HOME/.pythonstartup"

Then in your home directory create a .pythonstartup file containing something like the following:

1
2
3
4
5
6
7
8
9
10
11

import os
# Adds mf(obj, "str") allowing search for methods matching 'str' on obj
# as well as obinfo(obj) returning lots of info on obj
util_loc = "/home/shuckins/code/code_homerepo/python-programming/utilities"
if os.path.isdir(util_loc):
import sys
sys.path.append(util_loc)
from utils import mf, obinfo
sys.path.remove(util_loc)
del sys
del os

You’ll need to change the path specified to wherever you placed utils.py containing the mf() function of course. You can download mine here if you’d like. The del statements are to clean up any trace of this operation once you get to the interactive interpreter. This way you get the added functions without mucking up your namespace.

This also loads in an obinfo(obj) function that I included in the same utils.py file. This is based on the interrogate() function written by Patrick O’Brien in the introspection article mentioned above. My version just adds a check for objects without docstrings and prints more of the docstring:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def obinfo(obj):
"""
Print useful information about object.

From http://www.ibm.com/developerworks/library/l-pyint.html
"""
if hasattr(obj, '__name__'):
print "NAME: ", obj.__name__
if hasattr(obj, '__class__'):
print "CLASS: ", obj.__class__.__name__
print "ID: ", id(obj)
print "TYPE: ", type(obj)
print "VALUE: ", repr(obj)
print "CALLABLE:",
if callable(obj):
print "Yes"
else:
print "No"
if hasattr(obj, '__doc__'):
doc = getattr(obj, '__doc__')
doc = doc.strip()
topfive = doc.split('\n')[0:4]
print "DOC: ", "\n".join(topfive)
else:
print "No docstring. Yell at the author."

It’s fairly useful:

1
2
3
4
5
6
7
8

>>> obinfo(["some", "list"].count)
NAME: count
CLASS: builtin_function_or_method
ID: 139922087930696
TYPE: <type 'builtin_function_or_method'>
VALUE: <built-in method count of list object at 0x7f422658ef80>
CALLABLE: Yes
DOC: L.count(value) -> integer -- return number of occurrences of value

Quick tips for NVIDIA and ATI graphics configuration repairs on Ubuntu

Samuel Huckins — Thu, 05 Nov 2009 02:15:10 +0000

While graphics card and display configuration on Ubuntu has come a long way from the days of always having to edit xorg.conf by hand, I still run into issues now and again. I almost never have problems any more when setting up a new system. But changing cards in an existing system is another story, especially if switching between NVIDIA and ATI.

Today I did indeed switch from an NVIDIA to an ATI graphics card on an Ubuntu 9.10 machine. I had the nvidia-glx kernel mod in place, and xorg.conf specified the NVIDIA related modules to load, so just switching out the card and rebooting resulted in a flickering text login, no more desktop. This post covers the commands needed to install the appropriate kernel module and reconfigure Xorg in such situations.

After putting in your new card if you aren’t able to get to your desktop/get sent to a text login when you boot/get an Xorg error, reboot and hit Escape when GRUB prompts you. Boot into the recovery mode for the latest kernel shown in the list. Drop into root prompt with networking when prompted.

NVIDIA to ATI

This assumes you have an ATI card in place. Once at the root prompt run:

1	apt-get install xorg-driver-fglrx

This will remove nvidia-glx if you have it installed and install the driver for ATI Radeon and FireGL graphics cards. After this is complete you need to reconfigure xorg.conf to use the new module. Fear not, no manual editing should be required to get a basic configuration working. Just run:

1	aticonfig --initial

This script comes with the xorg-driver-fglrx. Running this will add screen and device sections in /etc/X11/xorg.conf appropriate for an ATI card, backing up /etc/X11/xorg.conf beforehand. If you added any special configurations for your screen section before you might want to edit the file manually and copy the customizations into the appropriate new sections. The sections already in xorg.conf will still be there, just commented out or not actually called when loading X. Next run:

startx

If all goes well you will be back to a working desktop. You can also check from the command line by running “fglrxinfo”.

For more information on using the ATI drivers on Ubuntu see this wiki article.

ATI to NVIDIA

And in the reverse situation:

1
2
3

apt-cache search nvidia-glx
# Install the latest stable one. Currently:
apt-get install nvidia-glx-185

And the equivalent for aticonfig:

1	nvidia-xconfig

As above:

startx

For more information on using the NVIDIA drivers on Ubuntu see this wiki article.

No guarantees here, but these have worked for me.

Catching warnings from the MySQLdb module

Samuel Huckins — Mon, 26 Oct 2009 03:53:50 +0000

The MySQLdb Python module implements the Python DB API for MySQL. I’ve written about its use before. MySQL issues warning messages in a number of circumstances and PEP 249 (which specifies the Python DB API) describes a Warning error message to be included.

One issue I ran into recently was how to catch warnings thrown by this module when running queries. Oftentimes tutorials or forum discussions that cover warnings in the context of MySQLdb describe how to filter them (they can clog up script output). But in a recent case I wanted to grab and check the warning, logging a dependent result. I had hoped this clean implementation would work in a method used for all calls to the MySQL DB:

1
2
3
4
5
6
7
8
9
10
11

try:
cursor.execute(query)
except MySQLdb.Error, e:
raise e
except MySQLdb.Warning, e:
raise e
finally:
data = cursor.fetchall()
rows_returned = cursor.rowcount
cursor.close()
db.close()

But the warnings just went right through. Instead I needed the warnings module’s assistance:

1
2
3
4
5
6
7
8
9
10
11
12

import warnings
with warnings.catch_warnings():
warnings.simplefilter('error', MySQLdb.Warning)
try:
cursor.execute(query)
except MySQLdb.Error, e:
raise e
finally:
data = cursor.fetchall()
rows_returned = cursor.rowcount
cursor.close()
db.close()

This catches the warnings and raises them as errors, although their class is still correct, allowing a clean implementation to call the above code (wrapped into a method called do_query):

1
2
3
4
5

try:
self.do_query(make_cool_table)
logger.info("Created cool_table table.")
except MySQLdb.Warning:
logger.info("cool_table already exists.")

Viewing all users on a Linux system

Samuel Huckins — Sun, 04 Oct 2009 20:35:59 +0000

There are a number of widely-used and stable utilities on Linux systems that allow you to view information related to users. You can see who’s logged in with who, get info on a particular user with finger, see who you are with whoami and see who logged in last with last and lastlog. And there are commands to create, remove, and change users and their groups.

But what if you just want to see all the users defined on the entire system? Looking in /home won’t show you users that don’t have a home dir or one not located there. All users are indeed listed in /etc/passwd, but having to look through this file every time you just want a list of users is tedious. I haven’t found a utility that performs this role.

This was somewhat surprising, as it’s not an uncommon need. Perhaps I’ve missed some essential program along my Linux journey. If so, feel free to enlighten me in a comment In the meantime, let’s fill this gap:

1
2
3
4
5
6
7
8
9
10
11
12

# Prints all users, divided by login ability and homedir:
function userinfo {
echo "----- Users that can login -----"
awk -F":" '!/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- And have /home dir -----"
awk -F":" '!/bin\/false/ && /\/home/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd

echo -e "\n----- Users that can't login -----"
awk -F":" '/\/bin\/false/ { print "username: " $1 ", uid: " $3 ", homedir: " $6 }' /etc/passwd
echo ""
}

This function should work on any *nix system. Place it inside your ~/.bashrc, reload that (. ~/.bashrc), and try out “userinfo”:

Three sections are displayed showing usernames, user IDs, and homedirs for users in /etc/passwd who:

Don’t have /bin/false for a shell (i.e. users that can login)
Also have a homedir under /home (i.e. the users that aren’t for system processes)
Do have /bin/false for a shell (i.e. users that can’t login)

Note that the users that can login category only shows users to which one could switch with “sudo su – USERNAME” from a shell. This doesn’t mean anything about whether they can login via SSH or other access methods.