tech.agilitynerd

Moved To Pelican

2013-03-17T12:00:00-05:00

Way back in 2004, when the content in this blog was a category in the main AglityNerd blog, I used the Perl Blosxom application to serve the blog. When my dog agility readers complained/were confused by the sprinkling of tech postings I split off tech.agilitynerd in another Blosxom instance.

Once modern web hosted blogs came in to existence I wanted to see how they worked and moved the tech content to posterous. That was only OK, the source code formatting was painful but it did support email and web based content creation. I was reasonably content, not bugged enough to move again.

Then posterous announced it is closing on April 30 so I finally had to do something. This site really only needs a few features:

easy format in which to write content
source code formatting
Atom/RSS feeds
comments

I use Disqus for comments on my other sites so that meant I could go with a statically generated site. Also since I primarily code in Python I wanted a platform to which I could contribute. I came across Pelican and it fit the bill nicely.

I exported the posterous posts, imported them through a temporary wordpress.com site using their posterous importer and finally re-exported them in wordpress format. After a quick bug fix to the pelican wordpress importer I had reStructured Text files for each post. After a few days of editing the embeded source code in the content files and fixing long broken links, I had the blog running and not looking bad at all.

I wanted a Twitter Bootstrap based responsive layout and I found azizmb's pelican-bootstrap-responsive-them which had a very pleasing layout. I tweaked it to have category and tag feeds and some other enhancements to get what you see today. I will see if Aziz is interested in pulling any of my changes back.

So far I like using Pelican and I've made the content of the blog and my edits to the theme available on github in case anyone is interested.

Google Analytics for JQuery Mobile With Internal and AJAX Pages

2013-01-21T02:00:00-06:00

I added google analytics to my agilitycourses.com website and was having problems getting the mobile version of the site to log analytics data after the first page. Google found this very helpful blog post: Using Google Analytics with jQuery Mobile and it got me very close to my final solution.

Since that time jQuery Mobile has changed to pageinit events and jqmData for page selectors, so I had to change the first line below to match the recomendations for the 1.x version of jQuery Mobile that I'm using:

$("div:jqmData(role='page')").live('pageinit',
  function (event, ui) {
    var url = '';
    var hash;
    try {
      /* urls begin with locale but track urls independent of locale: strip off leading locale */
      url = location.pathname.replace(/^/[-a-zA-Z_]{2,5}//, '');
      hash = location.hash;
      if (hash) {
        url = '/' + hash.substr(1); /* strip # from hash */
      }
      _gaq.push(['_setAccount', 'UA-XXXXXXX-X'],
                ['_setDomainName', 'example.com'],
                ['_setSiteSpeedSampleRate', 100],
                ['_trackPageLoadTime'],
                ['_trackPageview', url]);
    } catch(err) {}
});

My mobile site uses both internal pages and AJAX loaded pages. So unlike the blog post's solution, I needed to use URLs with and without hashtags. Per the Google docs on _trackPageview I also added a leading slash to the hashed URL.

Another change for my site is urls are localized, they begin with the locale for the user's language via django-localeurl. So I strip the leading "/es/", "/ca-FR/" or "/en_GB/" from the URL before sending it to Google.

There are a couple other variables I push on the _gaq array. I log my mobile site (m.agilitycourses.com) as a subdomain of my desktop site so I followed the settings from the Google analytics and set the domain name. I also enabled client side page load and site speed settings.

So I put the code in the head of the original post in the head of my pages and the code snippet above in the foot of the pages.

Hope this is helpful for others.

Create Posterous Posts from Google Forms

2012-05-12T23:57:00-05:00

I've been organizing dog agility bloggers for Dog Agility Blog Action Days and wanted to automate updating our group's blog when each blogger posted their article on the event/action days. For past events I had the bloggers email me with the information and manually created each entry in the blog.

Since posterous supports posting via email I knew I could create a form on one of my web servers and have the page format the data so posterous could turn it into a post. Here are the details for formatting emails for posterous: how to get the most out of posting by email.

Instead I stumbled upon an article on using Google Forms to send emails via the Google infrastructure: Get Form Data from Google Docs in an Email Message [Video Tutorial].

So all I had to do was create a form for my bloggers to fill out and then have the script run by their submission format and send an email to posterous to post.

Here's my modified version of the form script:

function sendFormByEmail(e) {
  // create a draft post - posterous will send you an email
  // when the draft is created.  var email = "draft@YOURBLOGNAMEHERE.posterous.com";
  // e.namedValues is a dictionary where the keys
  // are the names of your form's questions. So you can pick
  // them out and combine them as you wish.
  //
  // My form contained five questions:
  // 1. 'Title of your blog as you want it to appear'
  // 2. 'The URL of your blog as you type it in your browser'
  // 3. 'Title of your article'
  // 4. 'The URL of your article copied from your browser'
  // 5. 'Short description/teaser about your article'
  var v = e.namedValues;
  var subject = v['Title of your article'].toString();
  subject += "((tag: attitude))"; // add tags as you wish
  var message = '<a href="' + v['The URL of your blog as you type it in your browser'].toString() + '">';
  message += v['Title of your blog as you want it to appear'].toString();
  message += '</a > wrote:';
  message += "nn";
  message += v['Short description/teaser about your article'].toString();
  message += '< a href="' + v['The URL of your article copied from your browser'].toString();
  message += '"> Read the full article < /a >nn';
  message += "n#end"; // so your email signature won't cause a problem
  // This is the MailApp service of Google Apps Script
  // that sends the email. You can also use GmailApp here.
  MailApp.sendEmail(email, subject, message);
  // Watch the following video for details   // http://youtu.be/z6klwUxRwQI
  // By Amit Agarwal - www.labnol.org
}

In the script you can see how I've accessed the form fields by the full name and woven them into an email with links embedded - just have to watch the use of double quotes within single quotes. Which produces blog entries similar to this one I created manually.

So now I can just send the link to the Google form to participants and they'll create drafts that I can tweak and/or just submit. It took longer to write this post than it took to modify and test the script!

My Favorite ORM and Python Anti-Patterns

2012-05-12T23:57:00-05:00

At work I was looking at improving the performance of one of our slower web pages. It can be rewarding to find a little piece of code that can be easily optimized. This time there were several functions that were adding 10+ sec to the page in worst case. It wasn't a problem for most clients, but when clients with who are related to many other clients hit the page they'd experience terrible performance. Here's pseudo code for the combination of anti-patterns that caused the problem:

# Projects have users and users are in different organizations
# (project can contain multiple organization's users)
activeOrganizationProjectUsers = [x for x in project.users
        if x.active and x.organization == organization]
if activeOrganizationProjectUsers:
    # do something *NOT* using activeOrganizationProjectUsers

There are two main problems with this code:

It ignores the fact the project, users, and organization are backed by an ORM
The list comprehension is being used to find all matching elements when only a single element is needed.

Ignoring the ORM

The code above wouldn't be too bad if these were just lists of objects in memory. But being objects that are instantiated by an ORM a number of database queries will be issued. In this particular case (w/o eager loading across user to the organization table) the following queries where executed:

Join project to user and get all users for the project's id
For each user load their organization (one by one) if the user is active

So in the case where there were hundreds of users on a project there were hundreds of queries executed and hundreds of User and Organization instances were instantiated. Depending on the size of the objects (and the ORM's behavior) it can take "real time" to fetch and instantiate all these large objects.

This code base has this kind of code sprinkled through out it. At one time during it's development the developers were encouraged to treat ORM backed objects as though they were Plain Old Python Objects (POPOs). The developer wouldn't necessarily see the performance degradation using small data sets either. This is one of the reasons why I like to tail the database log (or use django-debug-toolbar if I''m using Django) to see the queries go by.

Using List Comprehensions When a Single Value is Needed

To make this situation worse, the activeOrganizationProjectUsers list wasn't actually used. This is a combination of a Python anti-pattern and the ORM anti-pattern. What was required was to determine if a single active organization user existed.

I believe the original developer(s) used the list comprehension solution in a combination of ignorance and syntactic sugar. They didn't want to write a new function to do the query and put it in the User class so they used the existing class's API. The syntactic sugar was using the list comprehension to get more values than the one that was needed. If this wasn't a (potentially) expensive ORM backed operation the original code could have been:

activeOrganizationProjectUsers = False
for x in project.users:
    if x.active and x.organization == organization:
        activeOrganizationProjectUsers = True
        break
if activeOrganizationProjectUsers:
    # do something

But this solution could still query all possible user/organization combinations. The other question would be: which set is larger the organization users or the project users? It is likely looping over the organization's users looking for active ones would be more efficient anyway.

Remember the Underlying Representation

When performance matters remembering the objects are ORM backed is important. So in this case a single query was all that was required (SqlObject pseudo syntax):

activeOrganizationProjectUsers = Users.selectBy(project=project,
                                                active=True,
                                                organization=organization).count() > 0

If abstracting out the ORM's methods is important this new function could be added to the appropriate class as a method. In my case making a change to use a query resulted in cutting the page load time by two orders of magnitude.

Reducing the Cost of Client Side Analytics

2012-01-12T22:10:00-06:00

I read Andy McKay's blog post on timing user experience on the Mozilla Webdev blog the other day and it reminded me of an idea I was thinking about for measuring client side timings at work. I had been toying with the idea of rolling our own library to capture JavaScript rendering time for our JS heavy pages (grids of hundreds of lines of data).

Andy's post mentions the boomerang JavaScript library and when I was reading it's docs they pointed out potential for abuse/load on the URL used to report the timings. For each instrumented page boomerang can hit the "beacon" URL to report the statistics it collects. So in the worst case you could double your page hits - although for specific pages/samples recording a few statistics shouldn't be too costly for low volume sites.

One solution is to only sample the pages/users of interest; selecting the sample could occur on the server and/or client. But another solution would be to collect statistics across multiple pages and periodically send batches of analytics to the beacon URL.

I've been playing with mobile web development for agilitycourses.com lately and will soon let users store the courses they create in localStorage on their browser. That got me thinking that sessionStorage could be used to store analytics across pages and then periodically send the stats to the server. This would reduce the number of hits on the beacon, allowing deployment to a larger sample of clients. It also gets flushed once the session is closed and (if kept small) doesn't prompt the user to approve storing the data.

A lot of modern browsers support session storage and for my purposes only ones with support would be relevant - due to our browser support policy at work.

The other problem the boomerang docs discuss isabuse of the beacon (accidental or malicious). A solution would be to piggyback reporting of analytics into application form post payloads. This is trickier to implement and it suffers from coupling analytic reporting into the application itself.

To try to solve it some what generally... The client side JS library could add a hidden field to any/some/specific forms into which it writes the analytics data collected thus far. If it hooked the form submit callback it could know if the form was successfully submitted and clear the session storage.

Server side middleware could detect the hidden analytics field in the form and extract/store the data. It could also remove the field before passing the request data along to the app server.

All in all a fair amount of twiddling to keep from exposing a recording URL to the outside world.

Of course if an authenticated session was being used then abusers would have to have a valid session to post to the beacon URL.

I don't know if I will have time to play with the sessionStorage idea but I think it might be a worthwhile extension to boomerang or other client side analytics capture libraries.

YellowGrass - Web Based Issue Tracking

2011-06-07T12:24:00-05:00

I was doing some reading on mobl and saw that they are using a free web based service called YellowGrass for issue tracking. It has some nice features and seems easy to use. Everything is tag based. I think I'll try to use it for tracking enhancements to agilitycourses.com.

Django Shrink The Web django-stw 0.2.0 Released

2011-04-23T13:32:00-05:00

Shrink The Web has announced a new API for free users using their new preview verification feature. This change required changes to my django-stw package.

The changes (lifted from the CHANGELOG.txt):

Changes to the shrinkthewebimage template tag:

The shrinkthewebimage template tag is NOT backward compatible with version 0.0.1. The alt argument is no longer accepted.
The shrinkthewebimage template tag is now intended for use by free accounts, it adds the required preview feature. It can also be used by PRO account users wanting the preview functionality.
The shrinkthewebimage template tag now accepts PRO key-value arguments in the same manner as the stwimage tag. This functionality is shown in theexample template but may not yet be fully implemented by the STW web service.

Changes to the stwimage template tag:

The stwimage can now only be used for PRO features.

Common changes:

Template tags now throw exceptions in their constructors instead of in the render function so configuration errors are visible during development.
django-stw defines a key 'lang' for the SHRINK_THE_WEB dictionary that can be passed along as a default to the preview tag. Alternately a 'lang' keyword can be supplied in each template tag invocation. django-stw defaults it to 'en'. This functionality is not yet implemented by the STW web service.

The v 0.2.0 package is available on PyPi, as a source download on github, or via git clone.

Python dict.get's Default Value is Always Evaluated

2011-04-05T17:16:00-05:00

This is a gotcha I ran across in some production code that is obvious in retrospect. I was profiling the code to find places where we were calling an_expensive_database_function and came across code like this:

def doit(*args, **kwargs):
     value = kwargs.get('key', an_expensive_database_function())

The original author probably assumed that if 'key' was present in the kwargs dictionary an_expensive_database_function wouldn't be called; that it would be short circuited in the same manner as Boolean expressions. But since get is a function the arguments are always evaluated on the way into the function. So in this case even if the value of an_expensive_database_function was already present in the kwargs dictionary the database function would be called again.

Here is a "look before you leap" solution:

def doit(*args, **kwargs):
    value = kwargs.get('key')
    if value is None:
         # assuming default value None isn't a valid value
         value = an_expensive_database_function()``

Here is the "easier to ask forgiveness than permission" solution:

def doit(*args, **kwargs):
    try:
        value = kwargs['key']
    except KeyError:
        value = an_expensive_database_function()``

Blosxom Plugin for Generating Facebook Comment xids

2010-11-23T05:00:00-06:00

I've been using Blosxom to power my dog agility blog for over 6 years. In the past year or so I've enabled Facebook comments in addition to my site's own comment plugin. I ran into a problem using Facebook's comments: if the user enters a comment on a page and the URL has any additional URL parameters then the comment is only associated with the page when accessed with those parameters, others hitting the page w/o parameters won't see the comments.

This behavior is documented by Facebook when the xid attribute isn't set in fb:comments HTML element. I didn't think I'd encounter this situation since my blog post URLs don't contain any parameters. However, when people link to one of my articles within Facebook, Facebook appends various parameters to the base URL.

The solution is to specify an xid attribute in the fb:comments element containing the URL encoded URL of the page (Facebook's default xid). This causes existing comments to show up and causes comments created when the page is loaded with URL parameters to use the same encoded URL.

So I created a simple Blosxom plugin to perform the encoding so the encoded URL can be placed in the story.html template:

# Blosxom Plugin: urlencode -*- perl -*-
# Author: Steve Schwarz <http://agilitynerd.com/>
# 2010-NOV-22    0.1 initial version.package urlencode;
# puts the urlencoded string of the URL for this page
# into $urlencode::url without any params# use this in fb.comments xid to give the same xid
# even when query params are provided
# -------------------
use CGI qw/:standard/;
use URI::Escape;

sub start {
    return 1;
}
$url = '';

sub story {
    my ($pkg, $path, $filename, $story_ref, $title_ref, $body_ref) = @_;
    $urlencode::url = uri_escape("$blosxom::url/${blosxom::path_info}");
    return 1;
}
1;

Then it is used in the story template:

<fb:comments width="550" numposts="5" xid="$urlencode::url"></fb:comments>

Now my readers won't have to worry that their comments won't show up.

Obtain Short URLs and QR-Codes for Django Apps

2010-10-22T04:00:00-05:00

Lately I've been interested in improving the interaction of my agilitycourses website for mobile users. One such improvement is to add QR Codes (aka 2D barcodes) representing the page URLs to the printed representations of pages served as PDFs.

I found that developers have reverse engineered the "api" of the goo.gl URL shortening web site. In my brief testing it is very fast. What makes that service extra useful is by adding ".qr" to a shortened URL it returns a PNG image of the QR Code for the shortened URL. That made it perfect for providing both short text and QR Code URL representations for my printed documents.

I threw together a few functions and put them in a module to make it easy to shorten a long URL, obtain the QR Code PNG and store it using Django's Storage functionality:

import osimport urllib
from django.utils import simplejson

def googl_shorten_url(long_url):
    """
    Returns goo.gl shortened url for the provided long_url.
    Code taken from: http://djangosnippets.org/snippets/2220/
    Parameters:
    - `long_url`: the url to supply to goo.gl to be shortened.
    """
    params = urllib.urlencode({'security_token': None, 'url': long_url})
    f = urllib.urlopen('http://goo.gl/api/shorten', params)
    return simplejson.loads(f.read())['short_url']

def googl_qrcode(googl_url):
    """
    Return file containing qr code image file for the given goo.gl url.
    Parameters:
    - `googl_url`: url from which to obtain the qr code.
    """
    return urllib.urlopen(googl_url + '.qr')

def get_url_qr_code_image(long_url, storage, storage_image_file_path=''):
    """
    Return goo.gl shortened url and storage name of qr code corresponding to
    the shortened url for the supplied full url. Contacts goo.gl to shorten
    the supplied long url then downloads and stores the qr code image file
    in the storage instance using the file path and the shortened url name
    as the storage name.
    Parameters:
    - `long_url`: the url to shorten.
    - `storage': a Django storage instance into which to store the qr code
    image.
    - `storage_image_file_path`: file system path to prepend to shortened
    url. This path must exist prior to calling this function.
    """
    try:
        googl_url = googl_shorten_url(long_url)
        qr_file_name = googl_url.split('/')[-1] + '.qr'
        qr_code_name = os.path.join(storage_image_file_path, qr_file_name)
        if not storage.exists(qr_code_name):
            qr_buffer = storage.open(qr_code_name, 'wb')
            qr_buffer.write(googl_qrcode(googl_url).read())
            qr_buffer.close()
    except:
        googl_url = None
        qr_code_name = None
    return googl_url, qr_code_name

Yes, it has a nasty bare try/except. For my uses this is optional functionality so I never want a failure to stop the main functionality of the views that use it. Add exception handling appropriate for your needs.

The main entry point is get_url_qr_code_image(). Here is an example of its use (assuming you save the code in googl.py):

>>> import googl
>>> from django.core.files.storage import default_storage
>>> short_url, qr_code_storage_name = googl.get_url_qr_code_image('http://google.com', default_storage)>>> short_urlu'http://goo.gl/mR2d'
>>> qr_code_storage_nameu'mR2d.qr'
>>> default_storage.path(qr_code_storage_name)u'/home/dev/agilitycourses/static/mR2d.qr'
>>> default_storage.url(qr_code_storage_name)u'mR2d.qr'
>>>

Hope you find this useful.

Adding pyrsvg to a virtualenv created with --no-site-packages

2010-10-12T03:02:00-05:00

I set up my development and deployment environments on Ubuntu with virtualenvwith the --no-site-packages option to isolate them from packages in the system installation. My application uses pyrsvgand it is installed by default as a system package. Consequently I had to link the shared libraries it installs (w/in gtk) into my virtualenv.

Here are the links I created (workon and cdsitepackages are virtualenvwrappershell aliases):

$ workon project
$ cdsitepackages
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/rsvg.so .
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/gobject .
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/glib .

Mobile Web Site Redirects in Django

2010-10-05T14:34:00-05:00

For the mobile version of agilitycourses.com I wanted to follow the approach Google appears to be using on some of its sites:

If the user views agilitycourses.com from a desktop browser they should see the standard/desktop version of the site.
If the user views agilitycourses.com from a mobile browser they should be redirected to a mobile domain (m.agilitycourses.com).
The mobile version of the website includes a link to the standard version.
If the mobile user chooses the standard website they should "stick" on that site and not be redirected to the mobile site.

I wanted to run two different websites but share templates and have the templates and css change for the mobile site. That meant that I'd need to set a variable(s) in the request to use to generate the appropriate HTML. So I found the simplest mobile device detector minidetector and initially used that. I later found Chris Drackett's fork has a number of useful enhancements and switched to it.

But minidetector didn't provide the ability to redirect to another site. I found Scott Newman's article on using multiple templates which had a section on performing the redirect and storing the user's selection in the session. So I forked Chris' minidetector and modified it to include the redirect and session storage. At the same time I decided to store all the minidetector variables into the session and add them, via middleware, to the request so the raw request wouldn't have to be parsed each time. My fork is available here with details on the new configuration options.

I'm using two domains so I can track analytics for the mobile and non-mobile sites separately and allow users to bookmark the desired site's pages. I use Google Analytics (via django-google-analytics) and Awstats for analytics.

Since I'm using two separate domain and sharing everything else I'm using a setup similar to the one described by Dustin Davis. I have a settings.py file and a mobile_settings.py that only overrides the features I need:

from settings import *
SITE_ID = 2
CACHE_MIDDLEWARE_KEY_PREFIX = "m.ac-"

I use a different memcached key prefix so the cached pages for the mobile site don't clash with those for the desktop site.

I setup m.agilitycourses on my server using the same Gunicorn setup I used for agilitycourses.com with the only changes being specifying the --bind address/port and the name of the mobile settings file:

#!/bin/sh
GUNICORN=/home/user/virtualenvs/myapp/bin/gunicorn_django
ROOT=/home/user/source/myapp
PID=/var/run/myapp.pid
if [ -f $PID ]
    then rm $PID fi
cd $ROOT
exec $GUNICORN --bind 127.0.0.1:8001 -c $ROOT/gunicorn.conf.py --pid=$PID $ROOT/mobile_settings.py

If my templates/content start to diverge more significantly between the mobile and desktop sites I may set the TEMPLATE_DIRS differently in the mobile_settings file. Or I can move to Dustin's approach and create a new application containing the urls.py and views.py specific to my mobile deployment. I would think diverging further would call for a refactoring of the common functionality to its own application which could be imported into separate code branches for each domain.

Debug Site for Website Redirects By Referer String

2010-09-29T18:42:00-05:00

I'm adding an "m" subdomain to agilitycourses.com to provide a better mobile browsing experience. I'm using the referrer string in Django middleware (currently using minidetector) to detect whether the client is mobile and redirect them to the mobile site. Since it is likely that some folks will/won't get appropriately redirected I was looking for an easy way for them to tell me when they were incorrectly redirected. I'd need to know their referer string.

A little googling turned up a nice one purpose website: www.whatismyreferrer.com/

Configuring Runit for Gunicorn and Django Installed in a Virtualenv on Ubuntu

2010-09-08T03:08:00-05:00

I couldn't find any documentation that covered all the pieces for configuring my latest Django site so I hope this helps someone else out.

I had used mod_wsgi under Apache for my other Django sites. But now I'm using different python versions for the sites (until if/when I update the older sites) and I wasn't getting the correct versions of some python libraries (even though virtualenv apeared to be putting the appropriate python packages at the start of the sys.path). So I decided to configure Apache to ProxyPass to Gunicornso I could run my Django app in its virtualenv without it getting any other python modules.

Installing Gunicorn

I installed Gunicorn into the virtualenv for my application, which simplifies using gunicorn from the command line. Assuming /home/user/virtualenvs/myapp is the location of the virtualenv:

$ source /home/user/virtualenvs/myapp/bin/activate
$ pip install gunicorn

# or
$ easy_install gunicorn

This copies gunicorn_django to the /home/user/virtualenvs/myapp/bin directory. Test gunicorn with your app, assuming your Django app is located at /home/user/source/myapp, as follows:

$ source /home/user/virtualenvs/myapp/bin/activate
(myapp)$ cd /home/user/source/myapp
(myapp)$ gunicorn_django

Gunicorn starts myapp using the settings.py file in the current directory on 127.0.0.1:8000. Ctrl-C to stop the process.

Installing Runit on Ubuntu

There are two runit packages. You want the one that only runs services you add to it:

$ sudo apt-get install runit
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  runit-run socklog-run
The following NEW packages will be installed:
  runit0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/113kB of archives.After this operation, 537kB of additional disk space will be used.
Selecting previously deselected package runit.
(Reading database ... 209845 files and directories currently installed.)
Unpacking runit (from .../runit_2.0.0-1ubuntu2_i386.deb) ...
Processing triggers for man-db ...
Setting up runit (2.0.0-1ubuntu2) ...
runsvdir (start) waiting
runsvdir (start) startingrunsvdir (start) pre-start
runsvdir (start) spawned, process 9575
runsvdir (start) post-start, (main) process 9575
runsvdir (start) running, process 9575

You'll want to create a directory for the application and a run script in /etc/service:

$ sudo mkdir /etc/service/myapp
$ sudo vi /etc/service/myapp/run
# enter the run script I'll show below
$ sudo chmod +x /etc/service/myapp/run
# stop runit from trying to run gunicorn until we are ready
$ sudo sv stop myappok: down: myapp: 0s, normally up

The example run script checked into Gunicorn had some syntax errors and wasn't quite what I wanted. Here's my version:

#!/bin/sh
GUNICORN=/home/user/virtualenvs/myapp/bin/gunicorn_django
ROOT=/home/user/source/myapp
PID=/var/run/myapp.pid

if [ -f $PID ]
    then rm $PID
fi

cd $ROOT
exec $GUNICORN -c $ROOT/gunicorn.conf.py --pid=$PID

You can create a configuration file for gunicornto use or just create an empty file for now:

$ touch /home/user/source/myapp/gunicorn.conf.py

If you have multiple appserver you'll need to run gunicorn on different ports, you can put the configuration in the gunicorn.conf.py file:

bind = "127.0.0.1:8111"

Putting it Together

Now you can test that the run script works when run as root:

$ sudo /etc/service/myapp/run

Gunicorn should start and start the appserver. If it fails you can debug the script via:

$ sudo bash -x /etc/service/myapp/run

Tell runit to start and keep gunicorn running:

$ sudo sv start myapp
ok: run: myapp: (pid 7540) 0s
$ sudo sv status myapp
run: myapp: (pid 7543) 1s

MOTD

2010-08-31T20:21:00-05:00

I've always preferred maintainable code over clever code:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Brian W. Kernighan

Kernighan had it right all those years ago.

Confidently Refactoring Django URLs, Views, and Templates

2010-08-22T05:12:00-05:00

Googility.comis my first Django website and under the covers the oldest code looked like it. I had originally written it with the sole intent of allowing people to enter dog agility businesses and websites into a database that I could use to create a Dog Agility Google Custom Search Engine. The primary mistake I made was making the "project" (in Django speak) effectively equivalent to the primary application. In other words I didn't divide the major features of the site into standalone applications (which would allow them to be more easily reused, extended and tested).

As I continued to work on it I learned more about organizing Django projects. When I added the periodical search to the website I created it as a standalone application. I recently split out my django-shrinktheweb application from the main code base.

The Custom Search Engine (CSE) functionality is a worthwhile application that I'm planning on releasing as its own reusable application. I had already created an application directory called "cse" into which I had placed my models, views, urls, and tests specific to the CSE functionality. But I wanted to make the following changes:

Move CSE templates into a cse template subdirectory
Name the templates to match the views that use them
Name the urls in the urls.py prefixed with the application name ("cse_")
Covert all reverse() calls in the views and url template tags to use the named urls

Those are enough changes that I was concerned that I might miss something that would fail either in the view code or in rendering of the templates.

The Django test client makes it easy to test the forward and reverse url matching, calling the view and rendering the template. It is kind of a coarse grained test but the changes I was making were perfect for this tool. Given a urls.py:

urlpatterns = patterns('cse.views',
                    url(r'^site/view/(?P<id>d+)/$', 'view', name='cse_view'),)``

and a view:

def view(request, id, template='cse/view.html'):
    """Display an end user read only view of the site information"""
    site = get_object_or_404(Annotation, pk=id)
    return render_to_response(template,
                          {'site': site,
                           'labels': get_labels_for(site, cap=None),
                           },
                          context_instance=RequestContext(request))``

I then wrote a test class to create the required test instances and tests for each url to verify that the url can be found by name (via reverse()), the url maps to a view, the view invokes the desired template(s), and the {% url %} calls within the template can all be resolved:

from django.test import TestCase
from django.test.client import Client
from django.conf import settings
from django.core.urlresolvers import reverse
from cse.models import Label, Annotation

class ViewsTestCase(TestCase):
    def setUp(self):
        self.client = Client()
        self.ROOT_URLCONF = settings.ROOT_URLCONF
        # can provide a custom urls.py for testing so the tests can be run when
        # the application is incorporated into another project
        # settings.ROOT_URLCONF = 'cse.tests.cse_test_urls'
        # override the template context processors if there are special ones in place
        # that either you want to test or want to avoid
        self.TEMPLATE_CONTEXT_PROCESSORS = settings.TEMPLATE_CONTEXT_PROCESSORS
        settings.TEMPLATE_CONTEXT_PROCESSORS = ()
        # Create some instances on which we can invoke views
        self.label = Label(name='name', description='description')
        self.label.save()
        self.annotation = Annotation(comment='Site Name', original_url='http://example.com/')
        self.annotation.save()
        self.annotation.labels.add(self.label)
        self.annotation.save()

    def tearDown(self):
        # put settings back so the next tests aren't effected
        settings.ROOT_URLCONF = self.ROOT_URLCONF
        settings.TEMPLATE_CONTEXT_PROCESSORS = self.TEMPLATE_CONTEXT_PROCESSORS

    def test_view(self):
        response = self.client.get(reverse('cse_view', kwargs={"id":self.annotation.id}))
        self.assertEquals(200, response.status_code)
        self.assertTemplateUsed(response, 'cse/view.html')

The normal unittest asserts are available in the tests. I'm using one of the special asserts provided by the Django test Client to verify that the template I expected was used. All the templates used (due to template inheritance) are collected by the client and can also be verified.

I used these tests in a TDD-ish manner, I wrote the test for a view, ran the tests and kept resolving errors in the templates as I made the changes in my bullet list. It made a tedious job simple and gave me good confidence that I'd found all the renamed urls, views, and templates.

Haystack Search Result Ordering and Pre-Rendering Results

2010-08-10T03:41:00-05:00

I use Haystack and the Python Whoosh project to provide search over ~3400 articles in my Googility.com database. I had originally implemented the search in the "simplest way that works". I was making some other enhancement to Googility and noticed the search result page had two undesirable behaviors:

The ordering of results was basically random for all matching articles. For the domain of magazine article search having a bias toward the most recent publications would be more desirable.
Looking at the django-debug-toolbar output each element in the search results was hitting the database twice (once for the Article instance and again for its corresponding Periodical). So a single result page was making as many as 60 database selects.

Haystack provides mechanisms to help with both of these issues.

Imposing an Order on the SearchQuerySet

Haystack models search using an API based on Django's QuerySet. The only thing to remember is it performs its queries over the Haystack SearchIndex subclass(es) you create instead of over the Django ORM. So you define a SearchIndex subclass that contains the data from the application's model overwhich you'd like to search. You can also define additional fields that can be used to modify the results of the query. Here is my magazine Article search index:

from haystack.sites import site
from haystack import indexes
from periodicals.models import Article

class ArticleIndex(indexes.SearchIndex):
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='issue__pub_date')

site.register(Article, ArticleIndex)``

The text field contains the "document" over which the search engine (Whoosh) will actually perform the search. I'm using the template feature that allows me to use Django templates to format the data presented to the search engine.

I added the pub_date field to the index to allow the matching search results to be ordered by the pub_date field. The 'issue__pub_date' syntax mirrors the Django QuerySet syntax and means extract the "pub_date" attribute of the Article's "issue" attribute (it joins Article to Publication and get's the Publication's published date).

Then the urls.py is modified to change the SearchQuerySet passed to the default haystacksearch view to order by the ArticleIndex's pub_date attribute:

from haystack.views import SearchView
from haystack.query import SearchQuerySet

# query results with most recent publication date first
sqs = SearchQuerySet().order_by('-pub_date')
urlpatterns = patterns('',
                       url(r'^search/',
                           SearchView(
                               load_all=False,
                               searchqueryset=sqs,
                               ),
                           name='haystack_search',
                           ),<snip>``

Pre-Rendering Result HTML

Since I have only a few thousand records I decided to follow the Haystack Best Practices for Not Hitting the Database. This solution trades space in the Whoosh index files by generating the HTML that will be displayed when each article matches along with the data used by Whoosh to match articles to search keywords. The changes were pretty simple. In the ArticleIndex:

from haystack.sites import site
from haystack import indexes
from periodicals.models import Article

class ArticleIndex(indexes.SearchIndex):
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='issue__pub_date')
    # pregenerate the search result HTML for an Article
    # this avoids any database hits when results are processed
    # at the cost of storing all the data in the Haystack index
    result_text = indexes.CharField(indexed=False, use_template=True)site.register(Article, ArticleIndex)``

The use_template keyword requires you to create a Django template file that is used during index creation to build the HTML that will be displayed. The only peculiarity I found was figuring out where the template should live. On my system it was at templates/search/indexes/periodicals/article_result_text.txt. I understand the periodicals/article_result_text part but I haven't looked into where the search/indexes is generated from. I imagine a reverse() to find the url for the view and "indexes" is appended to that...

The final change is the template used to display the search results. In order to not hit the database the object list generated by the haystack SearchView is placed into the context used by the template and only the result_text attribute should be accessed:

{% if page.object_list %}
<div class="search-results-title">Results <b>{{page.start_index}}</b>  - <b>{{page.end_index}}</b> for <b>{{query}}</b></div>
<div class="search-results-list">
{% for result in page.object_list %}
  {{result.result_text|safe}}
{% endfor %}
<div class="pagination">
    <span class="step-links">
      {% if page.has_previous %}
          previous
      {% endif %}
      <span class="current">
          Page {{ page.number }} of {{ page.paginator.num_pages }}
      </span>
      {% if page.has_next %}
          next
      {% endif %}
    </span>
  </div></div>{% else %}<h2>No matching articles found.</h2>
{% endif %}

The actual result is placed in the template via {{result.result_text|safe}} the safe filter is required since the HTML doesn't need to be escaped again - it was escaped by Django when it was placed into the SearchIndex.

So now my search results are in reverse chronological order and they render using only 3 database queries and at least 10x faster than before.

Improving Google Ads and Google Search Descriptions

2010-08-03T01:46:00-05:00

I was looking at the google search results for my Googility web site and noticed that the descriptions shown underneath the title often contained text from my navigation links instead of content from the body of the page:

I did some searching and found the Google Webmaster blog post about description meta tags. Since almost all of the pages on Googility are generated by fewer than a dozen Django templates I edited the templates and inserted meta tags and filled the description in with data from each database entry. This avoids boilerplate information that would be ignored by Google and improves the descriptions shown to Google searchers. Some of my pages have already been reindexed:

Yahoo and some other search sites use a class robots-nocontent on any page elements it should ignore for it's index, Unfortunately, Google doesn't follow this standard. So I might end up making that edit to the templates also. Looking at my site's log files it appears the Yahoo spider is hitting my site more frequently than Google's and the Yahoo index is more up to date. Looking at my analytics reports though Google refers far more readers to my site than Yahoo...

I also noticed that the ads served on pages containing mostly links appeared to be using words in my navigation or other boilerplate instead of the few lines of valuable content. More searching to the rescue and I found this Google Adsense article on section targeting. Once again editing the dozen or so templates I used were easy to edit to add in these HTML comment tags. Checking back a couple days later showed improvements in the ads being generated for those pages. I keep an eye on my Adsense click rate and see if there is any increase in ad clicks.

So a couple simple edits made noticeable improvements not bad for a couple hours investigation and implementation.

Firediff: track and save CSS edits in Firebug

2010-07-25T16:36:00-05:00

When I'm making fiddly changes to a web page I like to tweak the CSS using the Firefox Firebug plugin. It has the advantage of letting you try changes quickly and see the effect. The downside has always been that you had to then change the source CSS file to include your changes. Which increases the risk that you forget to include a change.

I was thinking about that recently when I changed the color scheme on my AgilityNerd blog and went searching for a way to at least identify the changes. Turns out there is a nice plugin that provides exactly that functionality. The Firediff plugin not only tracks changes to the CSS it also allows you to save those changes for overwriting or diff'ing into you CSS file(s).

Like other firebug related extensions you need to enable this feature per page by clicking on the arrow next to the new Changes tab.

Here is a screenshot of the plugin in action:

You can see I've changed the line-height attribute - it shows the previous and final values. A right mouse menu lets you save those changes to a file or revert them.

Another feature that can be configured in the Changes menu is whether to track changes made by other JavaScript to the page. This is an interesting feature if you were wondering how a JQuery or other JavaScript modified the page for a given effect.

This is a great plugin that I'll continue to use regularly.

Initial Release of django-stw

2010-07-11T15:47:00-05:00

I have been using the free website thumbnail service from Shrink The Web on my dog agility search website Googility since I launched it. It is quick and easy to use and it adds a lot to the look of the pages.

I had created a simple Django template tag for inserting the little snippet of HTML needed by their service.

Recently they asked me to add support for their advanced features to my template tag. I used this opportunity to convert my templatetag to a Django application. This mostly makes it a lot easier to install but it also let me to bundle tests and an example template with the template tag.

I kept the existing shrinkthewebimage template tag and added a new tag called stwimage to enable the new features.

I'm hosting the example page included in the package here so you can see how the template tags work.

I've hosted the project source on github and uploaded the initial release to the CheeseShop for easy installation.

Embedding JSON Within Generated HTML

2010-07-08T21:03:00-05:00

Ran into an interesting problem at work this past week that had a simple and pleasing resolution. We have an in house developed JavaScript grid on some of our pages and when users entered some text strings we'd generate invalid JSON payloads that would give the user an error page. If they entered strings that looked like an HTML Entity i.e. &#13 which (with the addition of a trailing ; ) is a non-visible HTML character (carriage return) the text wasn't displayed in the widget. To further complicate things some of the content displayed in the grid is HTML which is inserted into the grid as is and can contain escaped HTML characters.

The grid gets its content as a JSON payload from within a hidden div in the HTML which is generated via a template mechanism. Heres a portion of the template where <%= and %> stringifying of the value of the Python variable(s)/code they surround:

<div style="display:none;" id="grid-init-args-<%= count %>">
  <textarea>
  <!-- this is the JSON payload loaded via the grid JavaScript -->
  <%= [ columnsIndex, indexColumns, columns, rowBuffer, footerRows, formulas] %>
  </textarea>
</div>

This approach has a number of problems:

By using the template mechanism to create the JSON payload this template was relying on the similarity of the string representation of Python objects to JSON. After some testing I found the following scenarios: If a string contained a single quote character the string representation was a double quoted string around the text and the single quote; a valid JSON string. If the string contained a double quote character the string representation was a single quoted string around the text and the double quote; an invalid JSON string. If the string contained both a single and a double quote the string representation would be a single quoted string containing a slash escaped single quote and the double quote; an invalid JSON string. Depending on the browser (of course) the JSON string would fail to parse correctly when the double quote was encountered within the single quoted string.
The JSON payload had to be HTML encoded (converting <, >, ", and &) since it was parsed by the browser as HTML.
The HTML encoding would encode or double encode HTML to be inserted directly into the grid's DOM.

The variation in single/double quoting was an easy fix, I changed to simplejson.dumps() which correctly double quotes key/values in dicts and escapes embedded double quotes (single quotes don't need to be escaped). I didn't time it but with the C extension it may be faster than the template engine for our larger datasets.

I played around with (not) encoding various portions of the payload and then it hit me that I should change the grid to get its payload from a non HTML element so that only HTML destined for insertion into the DOM would be HTML encoded (which is as you'd expect for normal HTML handling). I started changing the payload to be stored in JavaScript generated in the template but didn't like the impact the change would have on all the existing templates. So I started Googling and found Ben Nadel's blog post on using script tags as data containers.

So here's my solution:

<div style="display:none;" id="grid-init-args-<%= count %>">
<script type="application/json">
<%= simplejson.dumps([ columnsIndex, indexColumns, columns, rowBuffer, footerRows, formulas]) %>
</script>
</div>

There were two changes:

Used simplejson.dumps to correctly double quote and escape double quotes within the variables in the payload.
Change the textarea to a script element.

By converting to a script tag within the hidden div the HTML parser no longer parsed the content of the JSON payload. so the JSON payload only needed to HTML encode HTML elements that were being inserted into the DOM created by the grid.

This change also meant I was able to delete the unnecessary HTML encoding of non-HTML JSON payload data. Got to love solutions that involve deleting code.

Ultimately, we'll convert to loading the JSON payload as a separate AJAX request from the page to the server, but for now this simplifies the markup and handles all types of user input and HTML encoded characters correctly.

Django Shrink The Web Template Tag Updated

2010-07-01T23:00:00-05:00

I recently updated my Django template tag for simplifying the use of Shrink The Web images. They recently announced a CDN based distribution of images and they took the opportunity to modify their API.

The updated template tag is on django snippets.

The STW folks have asked be to extend my template tag with support for their PRO features. With luck I'll make that available sometime this weekend.

Multiple YouTube Videos per page using VideoLightBox

2010-06-04T22:58:00-05:00

I decided to stop displaying the default YouTube video players within posts on my AgilityNerd blog and I started looking for a light boxed player. Their were two main reasons. The smallest video playback window provided by YouTube for HD videos is too wide for my two column layout and now that I'm posting more videos the load time of the page is delayed by the communication with all the off site webservers; serving the YouTube static image of the video will be much faster/lighter weight.

I looked around and really liked the lightbox containing the default YouTube player provided by VideoLightBox and started playing around with their demo. VideoLightBox (VLB) has an interesting approach. You download an application (PC or Mac), configure how you want your video(s) to look and it generates a directory of files on your local disk (or uploads files to your website via FTP) along with an index.html file from which you copy the code to put in the <head> and <body> of your web page. For YouTube it also downloads a static image for each selected video which is used as the image link within the HTML page. Straight forward and works well.

For my purposes there was a problem with their approach, its locates the image used to launch the light box using an element id. This assumes a single video or gallery of videos per web page. On my blog's main page or the category pages there will be multiple videos (possibly multiple videos within a single post). I figured a little bit of CSS and JQuery hacking would solve the problem and it did.

I decided to modify their HTML/CSS/JS to use a CSS class instead of an element id to allow for multiple videos per page. At first I just modified the generated files. Then I saw that VLB has template files in their installation. So I started modifying the templates to output the new code. Two hours later I bailed. Using procmon it looks like the client app reads the template files but then doesn't actually use the files to generate the output files(?). I was only able to modify one of the three template files that needed to change and have it effect the generated files.

I'm going to provide my edits to the VLB developers in case they are interested.

So the solution is to edit one of the template files and then edit two of the generated files; not ideal but once you put the generated files on your webserver you'll probably not touch them unless you are changing CSS styles. The modifications aren't hard but you need to be careful and typos will definitely break things. You should backup the VLB directory before your start or be prepared to uninstall and reinstall from their installation program.

Navigate to the VideoLightBox directory (i.e. C:Program Files (x86)VideoLightBox)
Change the permissions on the templates subdirectory to give your user full access to overwrite the files
For each directory in the templates subdirectory open the videolightbox.js file in an programming editor (a keyboard macro makes this trivial):
1. Globally replace $("#videogallery a[rel]") with $(".videogallery a[rel]").each(function(idx){$(this)
2. Go to the end of the line and add });
3. Save the file

Then generate the output files using the VLB executable for one or more videos, saving the results to your local file system

Navigate to the output directory
Open the index.html file in an programming editor
1. Globally replace #videogallery with .videogallery
2. Globally replace id="videogallery" with class="videogallery"
3. Save the file
In the engine/css subdirectory open the videolightbox.css file in an programming editor
1. Globally replace #videogallery with .videogallery
2. Save the file

Then you can copy the files just as specified by the VLB installation instructions.

The other change I'll be making for my deployment is to rename the video images. They are named 0.png, 1.png, etc. I'm going to put them all in a directory on my resource webserver so I'll rename the files and their references in the code copied from the index.html to use the YouTube video id.

I'll be changing my existing web posts over to this new scheme over time...

Blosxom hitcounter, favorites and lastread plugins for MongoDB

2010-05-17T10:48:00-05:00

When I wrote my plugins for tracking the visits to each page (hitcounter), presenting the counts by popularity (favorites), and displaying the most recently read posts (lastread) I used Perl's Storable module as a simple way to serialize the data structure to and from disk. At the time I wasn't too concerned with performance, my blog agilitynerd.com didn't suffer under the load of the disk reads/writes.

But periodically I get DOS'd and my website becomes unresponsive. Not too suprising since Blosxom has to run as a plain CGI script; but I'm certain the disk writes of these plugins weren't helping. Occasionally the Storable on disk would become corrupted which would take down the site (not enough exception handling in the plugins).

I decided to go with Mongo as the backing data store to remove the disk reads/write from the cgi script. It supports an increment operation (ala memcached) so multiple threads can update the hit count and MongoDB will "do the right thing". It also has a query API that made rewriting favorites and lastread trivial.

I haven't tested the performance but my website, especially the favorites page, seems faster. Once I write a JavaScript client to update the hitcounts I'll be able to move the website to a fully cached deployment while still tracking hits which should make the load of running the agilitynerd blog very low.

I've hosted the plugins on github: http://github.com/saschwarz/blosxom-mongodb-plugins The mongohitcounter plugin can be configured to automatically import existing count data from the hitcounter plugin's Storable. So conversion to the new plugins just requires updating the template's variables to the new plugin names.

Using django-sitemap with django-tagging

2009-11-27T17:33:00-06:00

I was adding django-sitemap to googility.com yesterday and found that Tags don't implement get\_absolute\_url(). Which makes sense since the site developer would want to decide how to expose them in the URL space.

It is also arguable that links to pages displaying the tag view already exist in the page for models that are already in the sitemap so they don't need to be put in the sitemap explicitly. For example, a page for an Article might be at /article/django-11-release and that page would contain the links to pages linked with the tags for that article e.g. /tag/django/ and /tag/python/

But I figured having the tag pages indexed by Google would be useful. It also allows a different priority to be specified for the pages. So I made a little class that derives from GenericSitemap that allows the url and suffix for the Tag name to be specified:

class SlugSitemap(GenericSitemap):
"""Use for objects that don't implement get_absolute_url
   but have a slug field used in creating their url"""

def __init__(self, info_dict, priority=None, changefreq=None):
    GenericSitemap.__init__(self, info_dict,
                            priority=priority,
                            changefreq=changefreq)
    self.url = info_dict.get('url', '/')
    self.slugfield = info_dict['slugfield']
    self.suffix = info_dict.get('suffix', '')

def location(self, obj):
    return "%s%s%s" % (self.url,
                       getattr(obj, self.slugfield),
                       self.suffix)

Here's how I use it:

sitemaps = {'tag_detail': SlugSitemap({'queryset':Tag.objects,
                                       'url':'/tag/',
                                       'slugfield':'name',
                                       'suffix':'/'},
                                       changefreq='monthly',
                                       priority='0.5'),
}

The urls for tags are at /tag/slugname/ where /tag/ is prepended to tag.name and / is appended to the end

This class can be used to create sitemap entries for any url parameterized on a single field of an instance returned by the QuerySet.

Firefox Web Developer Add-on AJAX NON-Caching Problem

2008-12-30T04:08:00-06:00

I ran across some "interesting" behavior of the Firefox Web Developer Add-on today. When editing JavaScript code I normally leave Web Developer set to "disable cache". This causes all images and, significantly for my purposes, the JavaScript files to be downloaded from the server on every page request. But I ran into a unexpected problem with this setting that cost me a couple hours today.

I was working on a legacy page today that uses an AJAX POST and the response is sent to an IFRAME. The POST was successful, the results were valid and present in the IFRAME. The problem was as soon as the content of the IFRAME was eval'd() a GET was being sent to the server for the URL of the page. This GET was a problem because it had a side effect of clearing the session data (which contained the POSTed AJAX data) for the current page. So when the page's FORM was finally POSTed the server couldn't find the data in the session.

After trying numerous code changes including disabling event handlers and events, it finally occurred to me that the cache disable feature of the add-on might be causing the unexpected GET. Sure enough, once I re-enabled caching the GET stopped being sent.

It was strange that the eval() caused the GET to be requested. All the other code around processing the IFRAME's data didn't trigger it, only the eval. Strange but true.

So just something to keep in mind when strange behavior occurs on AJAX pages where session data is being used.

reCAPTCHA in Django

2008-12-23T22:04:00-06:00

I first read about ReCAPTCHA in this article in Wired magazine last year.

reCAPTCHA provides a free CAPTCHA web service that pairs together two words from OCR scanned books. One of the words is known and the other couldn't be recognized. The user types in both words not knowing which is unknown to the system. As reCAPTCHA collects the responses for the unknown word they get human verified character recognition. So the millions of users of the system are clearing up millions of unrecognized words. It is a very clever human "cloud computing" system using only seconds of human effort for each use of the system.

I'm using a FIGLet based ASCII CAPTCHAon my websites since it was easy to integrate into the Blosxom writeback plugin. But I wanted to give reCAPTCHA a try while converting my Googility site to Django. John DeRosa made my job trivial by writing up the steps with a clear example.

So I followed his directions which involved installing the recaptcha-client python library on my dev and production systems and obtaining a free public/private license key from the reCAPTCHA site. Then I updated my Django view and template files for the one form that needed CAPTCHA protection. It was dead simple and working within minutes. The only minor addition I'd make to John's article is of course you need to pass the captcha_error variable from your view to the template:

return render_to_response('edit.html', {'form': form, 'captcha_error':captcha_error})

So give reCAPTCHA a try for your next project. It was so easy to do I might even convert my Blosxom blogs to use it via Lars Engel's recaptcha plugin.

Improving Site Navigation - Add Titles to Blosxom Pages With storytitle

2008-06-20T22:03:00-05:00

I've always wanted my AgilityNerd blog to serve as a reference for interesting dog agility subjects. Consequently, I'm interested in making it as easy as possible for readers to locate, explore, and learn more about the sport. So improving navigation is a strong interest to me.

I have been reading Designing Web Navigation by James Kalbach. In the section discussing Browser Mechanisms Kalbach discusses the back and forward buttons and the Session History drop down. The session history is usually a menu that drops down from the back button (or near it). It lists the pages the user has visited in reverse order showing their page titles. Kalbach states: "Session history is a good reason to supply meaningful browser titles". That reminded me that for my blog the session history has always just displayed "AgilityNerd" for each page visited on my site. I was leaving out a valuable aid to my readers who use the session history to find where they had been previously.

Another benefit of setting the title in pages is in tracking site statistics. It had always bothered me that GoogleAnalytics never showed statistics for my pages by page title.

So these two things finally moved me to take some action. I had been thinking about writing a Blosxom plugin to set the title element in the HTML head section but I've always been too busy. Luckily I didn't have to.

I found out about the storytitle plugin from Nick Leverton's Serendipity blog. He had made some changes and released version 0.7 of the plugin. It was trivial to setup, I just followed the directions in the plugin and had this site and agilitynerd.com/blog working within minutes.

So if you are a Blosxom user just grab this plugin. Otherwise, take the time to set your page title sections so you can help your readers and help analyzing your own site's statistics

Blosxom Plugin lastcommented Enhancement

2008-02-29T18:50:00-06:00

I had a comment entered on my dog agility blog that I subsequently deleted and of course my lastcommented plugin recorded the comment as the most recent comment. Since the plugin uses Storable to store the data I couldn't manually edit the file to remove the entry.

So I modified my lastcommented plugin to allow you to specify the, base zero, index of the entry on the URL and delete the entry. For example to delete the second entry in the list (index 1):

http://example.com/blog/?lastcommentremove=1

To keep mischief makers from deleting all your comments the plugin has a variable in its configuration section called lastcommentremove that you set to a non zero value to enable this feature. Since this is probably a relatively infrequent activity (if you have blacklisting or wbcaptcha comment spam protection enabled) I figured this was a sufficient mechanism.

You can download the latest version of this plugin here.

Minor Additional Mods to Blosxom moreenties Plugin

2008-02-24T16:24:00-06:00

I had previously extended Jason Clark's moreentries plugin for Blosxom to allow adding images to links to the previous and next group of articles/entries in the head or foot of a Blosxom weblog. While attempting to have valid HTML on my blogs I found that I had left an img tag unclosed. I also found that it wasn't as easy to change the styling of the links as I had originally planned.

So I've closed the open img tags and added the configuration variable $selfstyle into which you can place any CSS style information you'd like added to the td containing each page link. For example:

# Set to the CSS style for $moreentries::links
$selfstyle = 'style="padding: 0 3px;"'; # set to '' for no style

New Scheme Using YouTube to Get "Google Juice"?

2008-02-15T04:32:00-06:00

In the last couple days my YouTube account has fallen victim to a new scheme by porn websites attempting to garner Google Juice. It is pretty straightforward. A generic YouTube account is created with a user name like "babiegirl123xyz" and the "user" has a single line profile with the line "all my pics are at the link below". The link is to an adult sex/dating site.

All by itself creating a bogus YouTube account with a link to your site won't get a site listed any higher in a Google search. But the clever part of this approach is the "user" subscribes to many YouTube channels. Since YouTube users often have channel pages with links to all of their subscribers, the more subscriptions the bogus user creates; the more incoming links they have to their profile and consequently to their adult website. But I'd think the Google algorithm wouldn't weigh links from YouTube too heavily anyway...

I came across this scheme because I got several subscription emails from variants of the babiegirl accounts all with the same template of a profile page and all linking to the same website. It doesn't really matter much to me that they go to this trouble, but it is strange since I don't display a list of my subscribers on my channel so they don't get any real benefit.

I guess another way this scheme could help increase traffic would be through the subscription emails. Heck I clicked on the links to see what the site was about. It could be there are enough folks who do the same in response to the subscription email. If only a few end up becoming members of the adult sites it could make this a worthwhile means for these sites to get more page hits/subscribers.

Sigh...

Web Site Themes for Techies

2008-02-04T05:34:00-06:00

I'm no graphic artist so when it comes time to create or change the look of my website I always have a difficult time. So I've discovered some tools and tips that might help other non artists create reasonable looking websites.

Here's the general approach:

Select an existing template or layout to use as a framework
Select a photo or logo that represents the theme/color scheme of the website
Create a background or background image using a sample of the photo/logos color
Sample colors from the background, photo or logo for highlighting other elements of the website

I'll present how I applied this approach to my latest website makeover and I'll discuss a some websites/tools I found helpful in these areas.

Selecting a Layout

Unfortunately I don't have any suggestions for templates for specific blog engines (apart from searching with Google). But I'll try to make up for that with my next discovery.

If you are familiar with using CSS for website customization Alessandro Fulciniti has created a set of 40 one, two, and three column layouts on his website. They are straight forward to follow if you've done any CSS coding. If you haven't please skip ahead to the next section for some other tips that might be useful.

I have to admit to having some problem retrofitting my old CSS design. My issue was with the rendering of the right hand column. When the negative offset for the column was larger than the right margin for the containing div the right column would be rendered at the bottom of the page. It would look OK in Firefox but not in IE.

Of course it is important to test any new layout in at least Internet Explorer and Firefox. Now that Safari is available for Windows it is easy to test on that browser as well.

Select a Photo or Logo

If you already have a color logo that you use for your business then you can help reinforce that logo by using those colors in your site design. Since my logo is currently black and white that didn't help me much.

I was looking to change my site to reflect the winter season so I went to StockXCHNG and found a nice royalty free snow scene by photographer ijsendoorn. There are thousands of interesting photos suitable for use in blogs hosted on Stock.XCHNG, it is a great resource and I heartily thank the photographers who post their work.

I then used the free Gimpimage editing software to select a section of the appropriate size for the header of my site. The Gimp is easy to use and there is a fair amount of online help available if you are doing something tricky. Of course you can use Photoshop or a similar program if you have it.

Create a Background

This is an area where I was really at a loss. I've never been good at creating graphics, and a tileable image is even harder to create. There are some tutorials you can find via Google but I just wasn't up for the challenge. Of course you can use a complementary solid color background by sampling your photo or logo (see the next section).

Fortunately I came across a software program called Gliftex - An Infinity of Tileable Graphics Designs. This is an interesting program that can create a multitude of different images suitable for tiling across the background of a web page. Gliftex has a number of algorithms for choosing the form, color scheme, and interpretation and generating images. Here is a good overview of the product along with some examples.

Gliftex has a feature that makes it perfect for creating a tileable background image for a website, it can sample an image and use the colors in the image as its palette of colors. So I took my snow scene section and had Gliftex sample it. I pressed the Form and Interpretation buttons a number of times and after a while came up with these understated and simple tileable images:

As far as I'm concerned the $ 50 US for the Gliftex Home download was a great investment. They have a free demo version available for download if you want to play around before you buy.

Sample Colors for Highlights

So now I had an image for the header and a repeating image for the background. I just needed to add some matching highlights for some other areas of my blog to tie the whole theme together. I decided to color the headings of my right and left sidebars to match one of the darker colors of the header image. But how to pick out colors from the image?

Gimp to the rescue again. I loaded my image again and selected Tools > Color Picker. You are then presented with an eye dropper pointer. I just clicked on interesting colors in my image to obtain a color value that I could type into my theme's style sheet for the background colors. In the picture below you can see the color value is "bec7d0" which you would write as #bec7d0 in the style sheet.

Results

While there are an many additional changes to my theme I could make I am pretty happy with the cool, blue-ish, wintry theme I've created:

Take a look at AgilityNerdto see it in action or, depending on when you read this, to see my next seasonal theme. I hope some of these tips might be helpful for helping you create your next theme.

Stock.XCHNG - Free Stock Photos

2008-01-19T19:41:00-06:00

I was looking at templates for ZenCart and came across some templates that use images from Stock.XCHNG. Over an hour ago I started browsing the images and am really impressed with the quality and breadth of photos that artists are making available for free use.

I'll definitely use some of these images to spice up my sites.

Feed2JS - Adding AgilityNerd Headlines to Your Website

2008-01-11T22:24:00-06:00

Updated 16-Aug-2006 to reflect Feed2JS's move to http://feed2js.org

I never thought anyone would want me to syndicate my blog and put links to my latest articles on their website. But Marj Kibby in Australia with her very nice dog raising and training blog Choose and Raise a Puppy did just that. So I started thinking about how I might provide an HTML feed that could be included into blogs using client-side JavaScript as not all web/blog writers are developers who can integrate this data on the server side.

Well Google to the rescue: the Feed To JavaScript Open Source project provides a really great service that exactly fits the bill. If you have a website or blog and wish to list the latest articles from another website you can just fill out a form on the Feed2JS site and get the HTML to paste into your website. All you do is supply the RSS feed from the source site to Feed2JS, select the options and click Preview or Generate JavaScript to get the code you need. Feed2JS does all the "heavy lifting".

Here is an example of the generated HTML as it would appear in your website using the current data from this site:

The code for the above is a single (long) line of code:

<script src="http://feed2js.org/feed2js.php?src=http%3A%2F%2Fagilitynerd.com%2Fblog%2Findex.rss10&chan=y&desc=0&date=n" type="text/javascript"></script>

There are many other ways to configure/display the information from AgilityNerd on your site. Just go to: http://feed2js.org/index.php?s=build and enter: http://agilitynerd.com/blog/index.rss10 in the URL text entry box. Then answer the other questions.

Feed2JS kindly caches the latest RSS feed from the source site you specify and replies with the above HTML each time someone visits a page on your site. This has the added advantage of not taxing the webserver of the source site to feed your website; which is nice for sites like this one with a small monthly bandwidth quota.

Feed2JS also has a number of ways to style the HTML that is delivered so you can change the look of the links to match your site. Of course, web developers fluent in CSS can just style the HTML directly.

For Blosxom bloggers I've been playing around developing a new plugin based on the headlines plugin to provide this same functionality. I hope to post this new plugin when/if I get it working. Either way I strongly recommend Feed2JS, it is a great project developed in the true spirit of the Open Source movement.

New hitcounter, favorites Blosxom Plugin Features

2008-01-09T13:25:00-06:00

After fixing a bug in these plugins I decided to add some new features:

The links generated by the favorites plugin used to display the path and filename of each popular article. It now displays the article title and the filename is displayed when hovering over the link. This makes for a much nicer looking and more useful display. Thanks to Doug for the suggestion.
The hitcounter plugin now tests for existence of articles before adding them to its hash of counts.
Requesting a page which no longer exists deletes the page and its counts from the hash of counts. Since previous versions of this plugin would record non-existent pages they could appear in the favorites listing. So now you can delete them by clicking on the link in the favorites page.
As pages previously stored in the hitcounter hash of counts are visited their titles will be added so they will appear in the favorites' generated links.

Both plugins must be upgraded together and will upgrade the existing hitcounter database as pages are accessed. The new versions can be downloaded here:

Notes on Configuring Postfix on Ubuntu Gutsy to Send Email via Google Apps

2008-01-05T23:53:00-06:00

Here are some notes I took on configuring my Slicehost Ubuntu Gutsy installation to use Postfix to send emails via Google Apps. I am far from an expert on postfix configuration but maybe these notes will be helpful to others needing this configuration.

These sites contain the key information:

I basically followed the Behind My Screen tutorial (read the comments too) with the updates from The Prancing Tarantula and the following changes.

My Ubuntu server install didn't have the Thawte certificates installed by default so I installed them:

sudo aptitude install ca-certificates

Then you can append that file to your /etc/postfix/cacert.pem If you "sudo su" before doing the append to the file you won't get messed up by the shell:

$ sudo su
# cat /etc/ssl/certs/Thawte\_Premium\_Server\_CA.pem >>
/etc/postfix/cacert.pem

Since I have Google Apps setup for my domain I don't just want to relay email as "user@gmail.com", I want the email to be sent as though it came from my domain ("me@agilitynerd.com"). This requires some simple changes to the config files.

In my transport file I have:

agilitynerd.com smtp:[smtp.gmail.com]:587

In my generic file I have:

demo@myservername.agilitynerd.com me@agilitynerd.com

Where "demo" is the login name and "myservername" is my slicename. In my sasl_passwd file I have:

[smtp.gmail.com]:587 me@agilitynerd.com:me\_gmail\_account\_password

After restarting postix you can test sending email from your server:

$ sudo aptitude install mailx
$ mailx -s "test email" someotheraccount@gmail.com <
~/sometestfile\_to\_send

Check your logfiles for errors/warnings:

sudo tail /var/log/mail.\*

I hope these notes might help folks "get over the hump" if they are setting up the same configuration.

hitcounter, favorites, lastcommented and lastread Blosxom Plugin Updates

2008-01-01T18:44:00-06:00

I found a bug in my hitcounter, favorites, lastread and lastcommented plugins due to my not using the correct Perl Storable functions. For some reason I didn't use the locking versions of retrieve() and nstore(). So when my old webhost had some problem causing long page load times (and many simultaneous requests) I ended up having my favorites data file written as zero sized.

So I changed to Storable::lock_retrieve() and Storable::lock_nstore() for all four plugins. The new versions are here:

I never heard any bug reports about these plugins, but I'm sorry if anyone ran into any difficulties due to these bugs.

Taking Better Photos - Building a Light Box

2007-12-31T19:15:00-06:00

I have often been disappointed with the quality of photos of objects I've posted on my site. My biggest problems have been providing good lighting and a nice background (see this for example). So I was Googling to find a photographer's light box to build when I found Bill Huber's great directions for building your own Light Box. So I built my own light box based on Bill's design.

I got two 10 foot lengths of 1/2 PVC and the fittings Bill outlines. Nancy picked up a couple yards of white cloth from the fabric store. I went to a craft store and bought 10 different colored sheets of poster board about 20" by 26". I already had some goose neck desk type lamps into which I put some compact fluorescent bulbs.

I figured I'd size the box so that I could use the poster board oriented in either direction. That made my pipes for the width and depth 28" by 22" (to leave room for the pipe going into the fittings). I made two sets of legs at 12" and 16" long. I got to use my PVC Cutter to quickly cut the pipe. I didn't bother gluing any of the pipe together since the friction fit is plenty tight enough. I also wanted to be able to easily make the box smaller if I over estimated the size I would need. It is a pretty sizeable box.

Light Box with Backgrounds and Cover Sheet Pulled Back

So if you are looking to take photos for your blog or for selling items on eBay this is a simple and inexpensive project. We'll see if it helps my photos look nicer.

Blog Spammers Using URL Encoding

2007-12-29T23:16:00-06:00

I was getting hit by comment spammers in the last week who were using URL-encoding of their addresses to get around the comment blacklisting filter I use. By replacing regular characters with the multi-character encoded representation of those characters within the URL the spammers were able to post comment spam with links to casino and porn websites to my blog.

This type of spam is a reversal of the same method for hiding your own email address in a web page so it won't be harvested by email spammers (see for example Chip Rosenthal's Blog entry).

I just added a regular expressions in my blacklist file to block the use of URL encoded characters in all links. If your blog comment software supports this approach you might want to do the same thing.

Blosxom - Default Flavour Plugin - Fixes Unknown Flavour Error

2007-12-29T23:16:00-06:00

I previously blogged a solution for the Unknown Flavour error that involved modifying the blosxom.cgi script itself. I recently found my modification caused requests for the RSS 0.91 feed and atom feeds to return the default flavour instead of the desired feed. I suspect any plugin using a flavour that inserted itself into the Blosxom template hash table without corresponding flavour file(s) probably didn't work correctly after my modification.

While investigating a fix for that bug I noticed that a better solution would be to override the Blosxom template subroutine. So I created the defaultflavour plugin. This plugin fixes the bug by first applying any registered flavour templates if the requested template files are not found. If that registered flavour templates don't exist the plugin finally applies the default flavour template files.

Download version 0.1 of the defaultflavour plugin here.

No configuration of the plugin is required. Just copy it into the plugin directory.

Blosxom - Hit Counter and Favorites Plugins

2007-12-29T23:16:00-06:00

I added a Favorites page to the side menu of my AgilityNerd site. Since then I've been updating it manually about once a month based on the AWStats reports from my web hosting provider (HostMagix). This task is unnecessarily complicated because AWStats keeps page counts independently for each URL, which includes the Blosxom flavour (filename extension). For my purposes I don't want to distinguish between say HitCounterFavorites.html and HitCounterFavorites.htm

So I decided to write a plugin to track the hits per page and "lump" together counts independently from the file extension. This plugin allows me to display the hit count of visitors for each page on each page. I wrote a second plugin to use the hit count data to automatically generate my Favorites page.

Hitcounter Plugin

I based the hitcounter plugin on the categories plugin written by Todd Larason. The plugin has a couple features of interest:

The $reset\_count flag within the plugin lets you provide a starting count value for any page. For example to set a page's count to 10 append "?count=10" to the page's URL. Disable this flag once you've set your counts to avoid mischevious count setting by outsiders.
The $retrieve\_only\_flavour flag within the plugin can be set to a flavour you want to use for retrieving counts without incrementing the count. Use this flavour to view the counts for URLs of interest.
You can add filters to the start() subroutine to exclude certain requests from updating your counters. I exclude RSS and Atom feed requests from my counts.
As of version 0.5 you can filter out loading and incrementing page counts for specific user agents via the ignore\_agents array.

The hitcounter plugin stores the $blosxom::path as the key in a hash whose value is the count of hits. The hash is stored in a file in the data directory. The same hash is used for the entire site. As each page is "hit" the value is incremented and stored in the $hitcounter::count variable. I put this variable in the footer of my pages.

Favorites Plugin

My favorites plugin uses the data file containing the hash of paths and counts as its input. This plugin generates an HTML unordered list of the most visited URLs with the following configuration options:

The $num\_entries variable controls how many URLs are listed.
The $include\_counts variable controls if directories will be included in the list of entries.
The $include\_counts variable controls whether or not the number of counts is appended to each list entry.
The $anchor\_format variable controls if the full path or only the filename is displayed in the list entry.
The $anchor\_link variable controls whether or not the entry is wrapped in an anchor <a>.
The $excludes variable controls which pages are excluded from consideration. I exclude the main URL for my site from showing up in the list.
The $groups array holds regexps that will generate separate unordered lists grouping together pages matching each regexp of $num\_entries list elements. See the configuration section for an example. I use this feature for my Favorites page to group pages for agility and tech categories into separate lists.
The $url\_flavour variable lets you specify the flavour to use for anchors to non-category URLs. It defaults to to $$blosxom::default\_flavour.
You can add filters to the start() subroutine to exclude certain requests from doing the work to generate the HTML. For example, I only run this plugin for my Favorites page.

When this plugin runs the $favorites::count variable is populated with the HTML. I then put this variable in my Favorites.txt page.

Download version 0.5 of the hitcounter plugin here.

Download version 0.1 of the favorites plugin here.

Comments/bug reports are welcome.

Blosxom Plugins: lastcommented and lastread

2007-12-29T23:16:00-06:00

In preparation for an overhaul of my website I wrote two new Blosxom plugins. The lastread plugin allows me to create a list of links to the last n articles that were viewed by readers. The lastcommented plugin allows me to create a list of links to the last n articles that had comments added by readers.

I took pains with these plugins so they only write their state files to disk when a single article is read (not when the index pages are hit) or when a comment is actually posted. I've also made it optional to not rewrite the state file if an already existing entry in the list is visited/commented again. There are a lot of Blosxom plugins that do a lot of work on every page access and I didn't want to add to the server load.

The configuration of each plugin is pretty straightforward. Both plugins require the Storable plugin for storing the state files. Only the lastcommented plugin requires any coding; whatever comment plugin you use needs to set a variable in order for lastcommented to do any work.

Download the lastread plugin

Updated 2007-06-23 to version 0.3 to fix unshift() error and ignore unknown file requests and filter out user defined HTTP Agents.

`Download the lastcommented plugin`_ Updated 2007-06-23 to version 0.2 to fix unshift() error.

Hopefully I'll have my new site design using these plugins available in a few weeks.

Blosxom - Removing the Unknown Flavour Error

2007-12-29T23:16:00-06:00

After my recent Referer Spam attacks I've been checking my logs and AWStats reports daily. I noticed that occasionally I'd get people requesting pages with incorrect suffixes. i.e. foo.ht or bar.|id| (some of these are probably spammers too). A "feature" of Blosxom is that it will still serve the page with the default layout or flavour. This feature causes a not too helpful and ugly error message to be displayed on the top of the page:

Error: I'm afraid this is the first I've heard of an "ht" flavoured
Blosxom. Try dropping the "/+ht" bit from the end of the URL.

Of course removing the suffix from the URL doesn't usually work; the client is served with an empty (but flavoured) page. So I though a better solution would be to have Blosxom serve my default_flavour whenever the flavour couldn't be identified. A little Googling later turned up this Blosxom mod by James Vasille. James explicitly loads a flavour file named "default" whenever the requested flavour can't be found.

For my site it is more appropriate to serve the $default_flavour when the requested flavour can't be found. That way I didn't have copy/create head.default, foot.default, etc files. So my code change is almost the same as James':

$template =   sub {
  my ($path, $chunk, $flavour) = @_;
  do {
    return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$flavour");
  } while ($path =~ s/(/*[^/]*)$// and $1);
  # Begin added code
  do {
    return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$default_flavour");
  } while ($path =~ s/(/*[^/]*)$// and $1);
  # End added code
  return join '', ($template{$flavour}{$chunk} || $template{error}{$chunk} || '');
};

So now a less ugly page will be provided when an unknown suffix/flavour is requested.

Update 19-Apr-2005 - this algorithm has a bug that has been fixed by the defaultflavour plugin.

Comment Spam and wbcaptcha Plugin Enhancements

2007-12-29T23:16:00-06:00

I have been fighting automated comment spam robots (see also refererblock update, and hit counter changes) with my modified writeback plugin that uses a blacklist of spam words and URLs. However, I'm getting tired of updating the blacklist and removing spam comments that get through the filter. So I decided to install the wbcaptcha plugin written by Pasi Savolainen. His plugin uses the FIGlet program to generate an ASCII "image" of random letters:

              _
  ___  _ __  | |__    __ _
 / _ \| '_ \ | '_ \  / _` |
|  __/| | | || | | || (_| |
 \___||_| |_||_| |_| \__,_|

A person leaving a comment then enters the letters (enha for the image above) as a "key" along with their comments. The plugin then validates that the key matches the generated image before it allows the comment to be posted. This is similar to the approach used by other blog software to distinguish humans from automated spam programs.

Pasi's plugin worked by executing figlet in a subshell whenever someone wanted to post a comment. I wanted to change this approach since I don't have the ability to install figlet on my web host and I didn't want to pay the performance cost of calling a program through the shell.

So I found the Text::FIGlet Perl package on CPAN and made a few small edits to Pasi's plugin to allow it to invoke the FIGlet Perl code without the shell. This simplified the plugin and sped up it's performance. I've sent my changes to Pasi in case he wants to incorporate them in a future release of his plugin.

I've decided to leave my blacklist in place, since I'm sure I'll still be facing human comment spammers looking to promote their websites. The last change I can think to make to further discourage spammers will be to change all <a> elements in comments to include the rel="nofollow" attribute supported by most portal/search websites.

I also took this opportunity to add some name anchors to my foot.htm and foot.html files so clicking on the "Comments" link on my index pages now takes you directly to the comments section of the individual article. Similarly, selecting "Add Your Comment" and Posting a comment also show the writeback for and the last added comment respectively. This makes for less scrolling on long articles and multiple comments.

Unfortunately, I don't think this will actually reduce the spammer hits on my site since these are due to automated "bots". I doubt the spammers actually check if their comments get posted, they are just hoping for a comment to "get through" occaisionally so their sites can can get some "Google Juice". So they'll keep pounding on my site although hopefully fewer spam comments will get through.

Please email me at the address shown in the right hand menu bar if you have any problems posting comments.

Directing Search Engines to Blosxom Content Pages

2007-12-29T23:16:00-06:00

I've noticed that folks who come to my site by querying through a search engine often end up on my main page and the article they are searching for has already rolled off the front page. This is because the search engine robots tend to choose the index pages as more relevant than the content pages linked to by the index pages. This is especially frustrating to me since I want my blog to serve as a reference for other Agility enthusiasts - so making my articles easily retrievable from search engines is important to me. It turns out this is a pretty common problem faced by many blogs.

I found that Jason Clark had run into this problem with his Blosxom blog and resolved it by using the following meta element on his index pages: <meta name="robots" content="noindex,follow" >. His post describes how he conditionally includes these robot meta tags through his head flavour files (using the interpolate_fancy plugin and his storystate plugin) only on his index pages and not on other pages.

For my site it is even easier. I use separate head.index pages to style my index pages differently from my htm/html pages. So I just directly inserted the <meta name="robots" content="noindex,follow" > into my head.index pages.

There are a number of other robots meta tags that I might consider adding in the future. Websnob has a thorough explanation of the tags and the robots that use them.

Hopefully over the next few months these changes will cause search engines to give their users links directly to the articles in my site.

Image Theft Ranting And Blocking

2007-12-29T23:16:00-06:00

I was really disappointed yesterday when I checked my blog's statistics and found that someone from a Hungarian Agility discussion board had directly linked course images from my website. Direct linking of an image is placing the URL of the image directly in a webpage hosted on another server. Each time a browser loads that page the direct link causes the other server to send the image to the client's browser. So each time the forum is viewed my server has to send the images to the forum's viewer.

I purposely try to make my image sizes small to make it possible for dial up visitors to view my site without too much delay. So direct linking doesn't impact my bandwidth costs too much; so far it is about 1Mb a day. The additional burden it puts on the webserver should be small too, but since I don't own the shared server I don't really know.

Ranting

The aspect of this that most irritates me is that someone would copy my work without attribution. That is the only thing I ask of visitors to my site who wish to reuse my content. The license link at the bottom of each page should make this clear. I really don't think this is it too much to ask.

I take copyright infringement very seriously. To me it isn't just the legal requirements of using other's material in accordance with their wishes that is important, taking credit, explicitly or implicitly, for another's work is just wrong. This is one of those All I Really Need To Know I Learned In Kindergarten concepts: Don't take things that aren't yours. As a friend of Nancy's says "Some people don't have good home training".

Lastly, the Agility community is still a small community and the online Agility community even more so. I guess I am naive, but I hoped that the members of our community wouldn't do things like this.

Blocking

My first step was emailing the webmaster of the site. The site was entirely in Hungarian so it is possible the webmaster may not have understood my English request. In any event, after 24 hours they hadn't removed the links.

So I went to look for a technical solution. There are a few well known technical solutions for this problem. A search of Google for blocking direct linking or blocking hotlinking will turn them all up. The most useful solutions include:

Rename the direct linked images

This mean updating all posts one your site to match the new name. But if you only have a few images and/or posts to them you can do this on a per direct link basis.

Randomly generate image file names that change over time

This is usually used for photo galleries where there is no text referring to each image.

Serve images through a script

This script would reject requests for images based on information in the request. This can be a compute intensive approach since it causes requests even from your own site to go through the script. For sites where you don't have control over the webserver this may be required.

Use a Rewrite rule to serve a different image to non-local referers

This is the technique I used.

Based on this altlab.com article I originally added these rules to my .htaccess file on my server:

RewriteEngine on
RewriteCond %{HTTP\_REFERER} !^$
RewriteCond %{HTTP\_REFERER} !^http://(www.)?agilitynerd.com/.\* [NC]
RewriteRule .\*.(jpg\|jpeg\|gif\|png\|bmp)$ /images/nodirectlink.g [L,NC]

I then created an image called nodirectlink.g shown below.

Redirected Image

Don't use the same filename suffix as one of the real image filenames you use or you'll loop the rewrite engine.

I had forgotten about images direct linked by RSS feed readers that access the root. Rather than rewrite those requests I moved my Rewrite rules into the .htaccess file in the images directory. I also decided to not send the image after all, I'll just fail the request. There is no sense in even wasting the bandwidth, the clients will now get the broken image icon from their browser. The official Apache URL Rewriting Guide describes this in the Blocked Inline-Images section. So here is my final solution:

RewriteEngine on
RewriteCond %{HTTP\_REFERER} !^$
RewriteCond %{HTTP\_REFERER} !^http://(www.)?agilitynerd.com/.\*$ [NC]
RewriteRule .\*.(jpg\|jpeg\|gif\|png\|bmp)$ - [F]

I took this opportunity to modify some of my very first articles and move their images into the /images directory. Those images were being served by the Blosxom binary plugin back when I thought it was a good idea. Now I won't pay any extra processing cost for those images.

Conclusion

So now I have a technical solution to a human problem. There is a chance that this change may cause some viewers of this site to not see images on this site. But hopefully that should be a very, very small number of people. But if you think you are getting my blocking image incorrectly Please email me: steve at agilitynerd.com.

I'm still disappointed I was forced to resort to this change.

Modifications to Blosxom entriescache PluginHandles Articles With Future meta-creation_date

2007-12-29T23:16:00-06:00

Fletcher Penney created the original entriescache Blosxom plugin. It is designed to cache all the articles and their creation dates in a Blosxom blog to avoid the overhead of scanning the file system for new/modified files. He merged the meta plugin functionality into the plugin to allow authors to date their entries without relying on the file creation date (this makes it easy to move from one web hosting provider to another without having your article dates get reset).

Unfortunately, articles posted to a Blosxom blog with meta-creation_date's in the future would be displayed immediately. I had wanted to modify Blosxom to allow me post articles with future creation dates and have them hidden until that date. After a request for this functionality on the Blosxom email list, I looked into the required modifications and found it wasn't too difficult to make them.

You can download the modified plugin here. Let me know if you have any questions or comments.

Refererblock Version 0.2

2007-12-29T23:16:00-06:00

I came up with two improvements to the first release of my refererblock plugin:

If the referer string matches the site's URL it passed immediately and isn't checked against the blacklist.
The blacklist.txt file was being read even if the referer string was empty. Now it is only read if the referer string is not empty and it isn't for the site's URL.

These optimizations do improve the performance of the plugin. My testing on a PIII 800MHz running Fedora Core 3 Linux with Apache 2.0 showed the following average latencies:

1.5 ms - Empty referer string or current domain.
2.0 ms - Referer string matching the first regex of the example blacklist file.
3.0 ms - Referer string matching the final regex of the example blacklist file.

I was kind of surprised at how little additional time was required to load the blacklist file and process the regular expressions. This is probably due to the file remaining in the disk cache for subsequent requests. Of course your mileage may vary.

Download version 0.2 of the plugin here.

See my original plugin description for installation, configuration, and testing information. Please let me know if you use this plugin or if you have comments or suggestions for improving it.

Those Darn Spammers Blosxom - Hit Counter and Writeback Changes

2006-08-11T19:00:00-05:00

Over the past year I'd noticed that comment and trackback spammers had been hitting the same dozen or so pages of my blog multiple times daily. (It is probably the same person/group who took a snapshot of the articles on my front page at that time and just reuses those URLs for all of their different domains). Last I calculated, about 50% of my overall website traffic is due to spammers. This constant barrage of hits skews my AWStats statistics and, more importantly, skews the results on my Favorites page. So I took a little time to work on this problem.

FWIW I'm also starting to see a lot of spam coming from "blogs" being setup on BlogSpot that are a single page all of whose links point to the real site. Some of these bogus blogs use the BlogSpot temmplate which contains a flag used to alert BlogSpot admins to content in violation of their Terms of Service. This allows anyone to report the bogus blog by just clicking on the flag. Other "blogs" just use their own HTML and BlogSpot support would have to be sent an email with the offending site's URL.

So anyway, I've taken the following steps:

Disabled trackback comments entirely using the configuration variable in the writeback plugin. I've never received a legitimate trackback ping.
Modified my modified version of the writeback plugin to set a variable $rejected with a 1 if the comment was rejected or if trackback was attempted.
Modified my HitCounter plugin to read the $writeback::rejected variable and then not increase the counter for the spammed page.
I had to change the ordering of the hitcounter plugin to run after writeback so the variable would be set correctly when hitcounter ran.
Set the $hitcounter::reset_count variable and reset the counts of the spammed pages back to "reasonable" counts.

Another couple hours wasted messing around against spammers.

I've previously written how I've been using comment content blacklisting to reject comment, trackback, and referer (sic) spam. My current blacklist file has over 40 regular expressions containing over 250 words and patterns. I update it whenever a spam comment slips through. So that is an ongoing almost daily effort. I might just have to go to the trouble of getting a CAPTCHA plugin to work.

How Safe is Your Personal Information in the Hands of Website Developers?

2005-06-23T19:00:00-05:00

I was going through the webserver statistics for this site to see if any new sites had linked to any of my articles (it is always nice to see that what I have to say is useful to someone). Anyway, I ran across someone who had come to my site through a Google query (I won't mention what the query was for reasons you'll soon see). I ran the same query on Google to see what else came up since it was a rather unique query. Another Google link was for a site that looked like it had raw data - not your usual HTML pages.

When I went to the site I found what looked like a website developer's development directory wide open to the internet. There were at least three company's websites sitting in subdirectories. The file referred to in the Google result page was a backup of an SQL database dump file. Not just any database file - a backup of all the customer information for running one site's shopping cart database. It included names, addresses, email addresses, and phone numbers! (I didn't poke around to see if it had any more sensitive data).

I was able to figure out the original data owner's domain name from some info in the header of the file. So I just sent them an email letting them know that their customer information is posted for all to see on someone else's website. It will be interesting to see if they respond. I hope it is just their website developer who has a test server running and accidentally left this SQL dump in a publicly accessible area of their webserver. I'd hate to think this data was stolen from the real website and being used for spamming purposes.

As a software developer I've read numerous cautionary tales of accidental (and malicious) data theft occurring when real customer data is used in test systems. I just never imagined I'd stumble across such an egregious privacy violation. So this experience makes me wonder about all the online systems into which we type our personal information. All it takes is one careless developer (not even a malicious one) to expose our private information to a much wider audience...

Blosxom Plugin to Block Referer Spam

2005-02-12T20:00:00-06:00

Like so many other bloggers who allow comments on their websites and blog articles, I was facing increasing comment spam as my blog got noticed by more spammers. The size of this problem is illustrated by this Google query for "comment spam" that returned 1.5 million hits. For the uninitiated comment spam is like email spam for blogs; the spammer inserts fake comments in a blog where either the comment text contains from one to dozens of links to the spammer's websites. When web search sites "spider" the blog the links to the spammer's site are treated as "endorsements" of the spammer's sites and the spammer's sites are raised to the top of the search site's result lists.

There is another growing type of blog spam called referer spam (yes it is officially misspelled). When a web surfer clicks on a link in a web page that sends them to another web page most web browsers fill in the URL of the referring page into the request called the HTTP_REFERER. Some websites and blogs capture that page link information when they are on the receiving end of a web request. These sites might have a section on each page indicating the sites that link to that page. These links are referer links.

Referer spam uses the same mechanism as comment spam to raise the search sites ranking of the spammer's websites. But referer spammer's don't post comments; they post fake referrals to a website. The are hoping that the website or blog displays links of the sites that refer to them. So when the website is spidered the search ranking is raised.

Blosxom Plugins Addressing Spam

Like so many bloggers once I started getting comment spam I was able to manually delete them as they occurred. But that got old fast. After some Googling I discovered Doug Alcorn's Blosxom writeback blacklist plugin. I had been using the original writeback plugin. Doug's improvements provided enough protection (so far) with less than a dozen regular expressions removing all my comment spam.

Referer spam started hitting me three weeks ago. What was most infuriating was that I don't display any links of referring sites on my site at all. So all these spammers were succeeding in doing was skewing my site statistics and using my bandwidth with their fake referer attacks several times a day.

Of course all the referer spam site's addresses contained one or more of the same dozen blacklisted words I had already configured for comment spam. About the same time I saw Jason Clark post his deferer plugin to return a 301 permanent redirect for the IP address of one particular referer spammer who was attacking his site. I thought that by combining Doug's blacklisting plugin with Jason's immediate redirect plugin I could reduce the referer spam from my log files.

This plugin hasn't removed the entries entirely from my logs, since the initial request is still logged with a 301 status. But it has stopped the subsequent downloading of images for the pages whose content is now not served. Now that the log contains 301 status messages for these requests they are ignored by my host's statistics program (Advanced Web Statistics - AWStats).

On a re-reading of Jason's blog entry for deferer he also mentions the idea of white and black lists for referer filtering. So I might have subliminally remembered his idea and implemented my refererblock plugin based on his idea. In any event, I fully credit Jason and Doug for giving me the ideas and code with which to put together my plugin.

Referer Block Plugin

The refererblock plugin's tar file can be downloaded here. It contains the refererblock plugin already named 000refererblock so that it runs before all other plugins (you want to discard the blacklisted requests before all legitimate requests). A sample blacklist.txt file is provided and contains some example regexs. It uses the same blacklist file format and file name as Doug's writeback modification (I took the code from his plugin with only cosmetic changes). See Doug's website for links to the Movable Type and other blacklists.

The only configuration variable you can set is $log_blacklisted. If set to a full path file name the script logs the UTC date/time, referer string, and the page to which they were referring. You could use the frequency of words in the rejected referer strings to fine tune the content and ordering of your blacklist.txt file to match the spammers hitting your site. Be aware that this file isn't trimmed so you might want to keep an eye on its size.

Lastly, the zip file contains a simple Perl script you can use to test the plugin. Execute it as:

referer_test.pl http://example.com http://referer-spam.com
where the first URL is your website and the second URL is the referer

to be sent in the request. This script uses HTTP::Request to send the request. The script returns the status "200 OK" for the requested page if it isn't blacklisted and "301 Moved Permanently" if the referer is blacklisted.

Please let me know if you use this plugin or if you have comments or suggestions for improving it.

Online Button Creation Tool

2004-12-08T20:00:00-06:00

I was reading the Blosxom email list tonight and came across an interesting photoblog by Rob Park. After enjoying some of his photos, I noticed his post about an online form for creating the XML, RSS, etc. buttons popular on so many websites. Adam Kalsey has created a simple (dare I say elegant?) form that gives you just enough inputs to completely control the look of the buttons. The image is generated in PNG format.

Had I known of this site before it would have saved me a couple hours messing around with Fireworks creating the buttons on my AgilityNerd Feeds page...

AgilityNerd Site Enhancements

2004-10-29T23:16:00-05:00

I've spent some time over the past couple nights trying to improve the access to information on my site. As a software developer it is often more fun to play around with the technology than it is to do "real" work.

I've made the following enhancements:

Added Quick Links to the right hand navigation to allow folks to get to the three main Agility sections of the website without going to the Navigate page. Used the [[Blosxom]] file pluginto include the new HTML in all pages.
Changed my AgilityGlossary page to have a preface with an alphabetical listing ofdefinitions. I ended up using three new Blosxom plugins to make thiswork: config,postheadprefootand sort_order.

For other Blosxom users I've updated my site pluginspage with the details.

Hopefully these changes along with the page links and previous/nextpage arrows at the bottom of the page will give everyone enough ways to move easily around the sight.

Modifications to Blosxom moreenties PluginLinks to a Configurable Number of Index Pages

2004-10-21T19:00:00-05:00

Jason Clark created the moreentries plugin for Blosxom to allow adding links to the previous and next group of articles/entries in the head or foot of a Blosxom weblog. He solved a problem that most weblogs have; it can be difficult for visitors to browse older articles. Although he calls the solution a hack, he nicely solved an important problem.

I had been using the morentries plugin but found the text links didn't make it obvious to some visitors that there were more pages of entries. So I added images to the links to make them stand out. But I decided it might be more intuitive to use the same model used by search sites; create a graphical and textual link for each page of articles along with previous and next links.

So I decided to modify Jason's plugin to provide this feature. If you take a look at the bottom of my AgilityNerd home page page with more than $blosxom::num_entries entries and scroll to the bottom) you'll see grey dots, arrows, and page numbers each linking to a page of articles.

You can download the modified plugin here. The prolog of the file describes the configuration options. I've included the public domain icons I use on my site in the zip file for your experimentation. Since there are a few configuration options I've put some examples below so you can see how the new parameters control the html stored in $moreentries::pagelinks.

With these settings:

$numpagelinks = 10;$textlinks = 1;                # set to 1 to enable text links 0 to disable
$imagelinks = 0;        # set to 1 to enable image links 0 to disable

A screenshot of the HTML generated when the fifth page is selected:

With these settings:

$numpagelinks = 10;
$textlinks = 0;                # set to 1 to enable text links 0 to disable
$imagelinks = 1;        # set to 1 to enable image links 0 to disable
$prevlinkimage = "/images/left.gif";
$nextlinkimage = "/images/right.gif";
$currentpageimage = "/images/ball.red.gif";@pageimages = qw( /images/ball.gray.gif );

A screenshot of the HTML generated when the fifth page is selected:

With these settings:

$numpagelinks = 10;
$textlinks = 1;                # set to 1 to enable text links 0 to disable
$imagelinks = 1;        # set to 1 to enable image links 0 to disable
$prevlinkimage = "/images/left.gif";
$nextlinkimage = "/images/right.gif";
$currentpageimage = "/images/text.gif";@pageimages = qw( /images/ball.gray.gif /images/ball.red.gif );

A screenshot of the HTML generated when the fifth page is selected:

Thanks again to Jason for creating this plugin. It was fun to modify and I hope it is useful for other Blosxom webloggers.

See More... Added to Article Display on Index Pages

2004-09-26T19:00:00-05:00

I know my Agility readers will give me a hard time about this non-Agility post, but hey I'm "thinking about technology" too. I was limiting to five the number of articles/posts on my main and index pages to minimize the download time for non cable/DSL visitors. But I've noticed some searches hit my site for which an article was moved off my main page and I'm sure they didn't hunt around for the missing post. So I wanted to show more articles without increasing the download time.

The Blosxom plugin seemore automatically adds a link labeled "See More..." in place of a special label in each article. Selecting the link takes you to the full text of the article. This useful plugin was developed by Todd Larason, a blogger who's developed a number of powerful Blosxom plugins.

I made a minor modification to seemore, which I've forwarded to Todd, to allow showing the entire article for a configurable number of articles on a page. This lets you see the full text and diagrams for the latest articles and only seeing the first couple paragraphs of the older articles.

If you are a Blosxom blogger and would like my modifications you can download my modified version here. Unzip it and put it in your plugin directory. By default it should work the same as Todd's original version. A new configuration variable $show_all_of_stories_until can be set to the number of stories on each index page that you'd like to show in full.

So I've modified most of my posts to include the seemore label and increased the number of articles shown on each index page to $blosxom::num_entries. I like this new site configuration; let me know if you do too.

Backup Your Data Lately?

2004-08-20T19:00:00-05:00

A few weeks ago I bought a Ximeta NetDisk 120 GB external hard drive to back up the data on my home computers. There are a number of vendors making hard drives that support USB connections; this model is unique in that it also supports direct ethernet connections. Unlike more expensive Network Attached Storage devices (more than $ 1000) this unit only cost a little more than the hard drive itself; about $ 150. The only downside is that it uses it's own proprietary network protocol which requires installation of a driver on any computers using the drive.

My goal was to put this drive on my network switch and back up data from my XP, WinME, RedHat 8.0, and Win95 machines. It turns out this was a little trickier than I expected. So I spent the better part of the morning on a cool, grey Chicago day making this all work.

The disk comes formatted for NTFS but in order for it to be shared on the older Windows platforms and my Linux machine it needs to be reformatted to use a FAT32 file system. Windows XP doesn't support formatting drives for FAT32 so I had to install the Ximeta driver software on my WinME machine and install the NetDisk on my network.

Running fdisk and reformatting the hard drive is the only "scary" aspect of the installation. Choosing the wrong drive or partition would be a bad thing. This document gives a good step by step description.

I was skeptical that running fdisk over the network would work correctly, but it did. At this point I was able to view the drive on both my WinME and XP systems and copy data to the drive as if it was locally connected.

I was pretty sure making the drive work for Linux would be difficult. Unfortunately, the PDF documents from the Ximeta website are unreadable as they require installing the Korean Acrobat extensions... thankfully Google has its "View has HTML" facility which let me read the RedHat instructions. The docs on the install CDROM are viewable (but don't include the RedHat docs).

I'll spare you all the trial and error but after downloading the driver RPM from the website and installing it I couldn't configure and connect to the drive on the network. It could be that ports required for their protocol aren't opened on my Linux machine, but Ximeta doesn't give any information on what ports are used by their driver. The admin tool gives some cryptic error messages that Googling and the docs didn't explain (the docs recommend reinstalling the drivers for any errors...). I ended up connecting the drive directly via USB and was able to mount the drive and backup my user and system accounts to the disk.

So it looks like my goal of leaving the drive on the network and copying to it from any computer will only work for Windows machines. But at least I have a mechanism for backing up all my machines, that is easy enough that I'll use it all the time. My next step is looking into configuring rsync or a similar mechanism to only backup the changed files to the NetDisk.

In summary, I'd recommend this drive for anyone who is using Windows XP; it is plug and play for that operating system. If you are computer savvy you can make this hard drive play with other systems too.

However you do it, backup your computer!

Centering DIVs in CSS

2004-08-15T00:00:00-05:00

I took a look at this site on an 800x600 resolution monitor and didn't like how the original layout looked. Having the menu box on the left hand side with a 20px margin to the left used up too much screen space. So I decided to move the menu box to the right of the main body of the page.

I could have hard-coded the box position; but decided that I wanted to have the main body remain centered regardless of browser size. That would complicate the positioning of the menu box (which is it's own div not contained with the main page). Once again A List Apart had a good articledescribing how they got their CSS centering to work.

The ALA article had links to these two pages describing the methods I used: Auto Width Marginsand NegativeMargins.

The AgilityNerd site has two main divs: #page for the main body and#sidebar for the box containing the menu. I used the Auto Width Margins method to center the main body:

#page { font-size: 13px; background: white; border: 1px solid black;
top: 0px; width: 640px; margin: 0px auto; margin-bottom: 50px;
text-align: left;}

The two lines of interest are margin: 0px auto;` and ``text-align: left;. Thelater is required because I used the IE 5.x workaround of specifying text-align: center; in the body element.

So now that the main body of the site was centered, I wanted the menu to be floating off to the right of the main body. The Negative Margins method let me horizontally center the divand also offset it horizontally by any amount. I offset it by half the(fixed) width of the main body plus a little space. The additionalspace keeps IE 5.x browser's miscalculation of sizes from causing theborders to not meet correctly:

#sidebar { background: white; font-size: 12px; position: fixed;
width: 140px; height: auto; top: 70px; left: 50%; margin-left:
330px; text-align: center; border: 1px solid black; }

Unlike the article, I used position: fixed instead of absolute. This allows the menubox to float in place during scrolling (but only for CSS compliant browsers like: Mozilla/Netscape).

CSS for Print Media

2004-08-14T23:16:00-05:00

In keeping with my desire to have this site's layout be easy to read,I wanted it to look good in print too. The layout uses CSS instead of HTML tables. This lets me position the divscontaining various text for the screen and it allows me to positionand hide text differently for the printed media.

Any CSS book covers the basics of changing the display, position, margins, and padding along with the use of separate style sheets forprint and screen media. But I ran into problems with printing mysite's pages with the Mozilla/Netscape browsers. Only the first page of my main div would print. Alittle Googling turned up this excellent article on A ListApart. ALA is a great resource for web developers.

The only clarification I have is that since both the screen and the print CSS files are included, the print CSS file's elements may needto set additional properties as well as reseting properties that existin the screen media file. Otherwise the definitions for the screen will be used. I had to set some div's position properties to static to get them back into the document flow.

AgilityNerd Blosxom Plugins

2004-08-10T19:00:00-05:00

Updated 20-Jun-08 for addition of storytitle

I'm really impressed with the Blosxom plugin mechanism. Rael Dornfest, author of Blosxom, put hooks at strategic points in the page generation process. At each hook point the main script iterates all the plugins present in the plugin directory. Plugins that implement the hook function can then modify the state or add additional data for use in the page presentation.

Consequently, a plugin only implements the hook functions it needs invoked to provide its features. Some powerful plugins need only implement one or two hook functions and are implemented with a couple dozen lines of Perl code.

When I started building this site I found it a little confusing to sort through all the plugins and understand their functionality. The ease of creating and modifying plugins has allowed a lot of developers to contribute to the Blosxom plugin registry. I found it helpful when Blosxom sites listed the plugins that implemented their site's features. So here is the list of plugins I use.

atomfeed - Provides the AgilityNerd Atom feed. Point your browser to http://agilitynerd.com/blog/index.atom

breadcrumbs - Creates the click-able trail to the current position in the weblog's path shown at the top of all .index pages and at the bottom of each article.

binary - Used to supply images from the directory in which an article exists. Modified to use LWP::MediaTypes since MIME::Types isn't installed on this web server. I also commented out the interpolate() function and added it to the one in interpolate_fancy to make this plugin coexist with interpolate fancy.

I no longer use binary because it is faster to just have Apache serve binary files directly from one or more directories outside of the Blosxom src tree.

I've stopped using this plugin since it requires Blosxom to look at each file to determine whether or not it should be skipped. So I'm starting to put all binary files in my images directory outside of Blosxom. This has the added advantage that I can just put "/images" and "/video" in by robots.txt file so my images and videos don't get indexed by search engine robots.

categories - Displays the tree of paths/categories of articles on the navigate page. Slightly modified to remove the root directory from the display. Added a hide feature to hide specified directories entirely - used this to prune subdirectories for my side bar menu. Removed flavour ending from all generated links since all paths are valid links, are shorter for people emailing links around, and would always use the default flavour. Modified to not calculate breadcrumbs since I use the breadcrumbs plugin for that feature.

config - Allows any Perl variable to be set differently in any category/directory. I use this to change the sort order for my glossary page to be alphabetical order by changing the sort_order plugin's sorder variable.

entries_cache - Speeds up processing of the site by caching the article index to avoid scanning directories until a configured amount of time has elapsed. Also caches article creation time to allow for editing of articles without changing their timestamps.

favorites - Used to automatically generate my Favorites page based on the visitor hit counts recorded by my hitcounter plugin.

file - Used to include the navigational side div into all pages without copy/paste of the same HTML.

find - Provides search capabilities for all articles, comments, and linkbacks on the web site. HTML slightly modified to remove the advanced search link. Also set the hidden "path" input to the empty string so the entire site is searched regardless of where in the hierarchy the search was initiated.

foreshortened - Ends articles at the end of the first sentence. Used to provide the shortened version of articles for use in RSS and Atom feeds.

headlines - Used to determine all the entries in my glossary category and create links in alphabetical order. Requires interpolate_fancy.

hide - Excludes directories of articles from the index views (like the agilitynerd.com main page) but still allows them to be searched by find.

hitcounter - Used to automatically generate the number of visitors displayed in the footer for each page on my site. These hit counts per page are used by my favorites plugin to generate my Favorites page.

interpolate_fancy - Used to remove the space allocated for the results of find and the breadcrumbs plugin when their variables aren't defined. I think I'll use this more in the future to dynamically change the page format.

lastcommented - Used to automatically generate the list of the last 10 articles that were commented shown in the side bar on each page of the site.

lastread - Used to automatically generate the list of the last 10 articles that were read shown in the side bar on each page of the site.

moreentries - Creates the "Previous" and "Next" links at the bottom of each index or search page when there are more than $blosxom::num\_entries entries. I've modified this plugin as I described here.

postheadprefoot - Allows each category/directory to have unique text inserted after the header and/or before the footer. I use this on my glossary page to add a category specific subsection. I had to modify the script to have paths resolve correctly.

redirect - Unmodified version of this plugin. I had to use it when I moved a directory within my tree so search engine and email list links would redirect correctly.

refererblock - A blacklist driven plugin to permanently redirect referer spam. The redirect probably has little effect on the receipt of referer spam but the 301 redirect code keeps the spammer's sites out of my site statistics.

rss10 - Provides the extra information to provide a valid RSS 1.0 feed for syndication. Slightly modified to work with the writeback plugin.

seemore - Adds "See more..." link within the text of a post when the post is shown in an index page. The location of the link is controlled by placing a special tag in the posting source file. I have modified this plugin to show the first "n" entries of an index page in full. Furthe

r modified to always create a link to my html flavoured pages.

sort_order - Allows sorting by date, directory names, or file name. I modified it to not read the URI parameters and just use a configuration variable. Then I used the config plugin to change the sort order for my glossary page to sort by file name instead of in reverse chronological order.

storytitle - sets the <title> element in the HTML <head> with the title of the article or the name of the category. I use it to aid in navigation and site statistics]].

timezone - Adjusts timestamps automatically created or entered using meta-creation_date: to be offset correctly to my timezone; which is different from the web server's timezone.

I no longer use timezone since I can set the timezone on my Slicehost server.

defaultflavour - Provides the user with the default Blosxom flavoured feed when an unknown flavour is requested. This keeps the user from getting the Unknown Flavour error across the top of the page.

wbcaptcha - Provides an ASCII image via FIGlet when visitors enter comments to stop spam bots from saturating my blog comments. See this article for my modifications to this plugin.

wikiwordish - Provides WikiWord-like linking to Wiki articles, local article file names and modified to replace WikiWords with HTML <a> links. This plugin saves me from having to enter links to common sites to which I always refer. I had to name this file 00wikiwordish so that WikiWords would be correctly replaced in RSS and Atom feeds.

writeback blacklist plugin - Provides comment and TrackBack capability. This is Doug Alcorn's blacklist modified version to help fight comment spam. I've slightly modified mine to try to protect comment poster's from spam by obfuscating their email addresses.

In case anyone is trying to get these plugins to "play" together I have them named as follows:

000refererblock
001redirect
002defaultflavour
005wikiwordish
007google_highlight
008wbcaptcha
01atomfeed
01breadcrumbs
01categories
01config
01entriescache
01favorites
01file
01find
01foreshortened
01fullcategories
01headlines
01hide
01moreentries
01postheadprefoot
01rss10
01seemore
01sort_order
01storytitle
01writeback
02hitcounter
02lastcommented
02lastread
02recentwritebacks
50interpolate_fancy