tech.agilitynerd

Running Django Management Commands Periodically in Docker

2023-01-01T16:00:00-06:00

This past week I updated my Agility Course Master API to Django version 4.1.4 and Python version 3.11. I noticed that I hadn't run some of the database cleanup management commands (i.e. clearsessions ) that I should have been running periodically. So I started investigating how to run them automatically.

Overview

First, I started investigating how to run cron-like jobs in Django. There are a number of django packages that support importing/running Django application code in cron jobs. In all my Django projects I used django-extensions which contains a simple Job model for defining tasks which are then run by its runjobs management command. So that was my choice.

You don't strictly need a separate package, you could just invoke python in your virtualenv with a script that imports the necessary packages, connects to the db, etc. But, these packages/apps take care of that boilerplate for you and you can use familiar Django idioms.

But, none of these apps/packages can run simultaneously in the same Docker container that serves the Django webapp/API. So you need to run some sort of cron process to control running the scripts.

After much searching, I figured out that I wanted to create and run a new container; based on my Django API image that runs cron in the foreground. That way, it has the same configuration and code as my API container and can run any Python code/Django management commands I want.

Here are some articles I found helpful:

Details

Some of my cron configuration is specific to the Debian slim-buster image upon which I base my Django API container. But, it should be similar enough to other Linux containers to get you started.

1. Install cron in your Dockerfile if it isn't already part of your Docker image:

RUN apt-get update && apt-get -y upgrade && apt-get install --no-install-recommends -y <your packages here> cron

2. Install django-extensions and follow it's instructions for creating your job file(s) that will be run by cron. Unlike, their examples of crontab files, Debian uses a different format:

# run daily at 7:07 UTC
# has no environment and needs to redirect to stdout so docker can see results
7 7 * * * root source /etc/environment; /app/manage.py runjobs daily >/proc/1/fd/1 2>/proc/1/fd/2

Notes:

I'm running the runjobs daily management command at 7:07 UTC everyday as root. Create as many entries/crontab files as you need.
I put the above file named djangocrondaily in the root of my repo in a crontab directory.
I found out the hard way that cron jobs don't inherit any environment. In my containers I don't hard code any values into the settings.py file, they are all read from the container's environment variables. That's why I had to source /etc/environment which is populated in the entrypoint.sh below.
You can test running your script with no environment within your Docker container by running env -i /bin/sh scriptname or env -i /bin/bash scriptname as appropriate.
In order to get the output of the cron jobs in docker logs you need to redirect their output to stdout.

3. entrypoint.sh shared by my Django and cron containers

#!/bin/sh

# store environment variables in a file so cron can source them
env >> /etc/environment

echo "Waiting for $SQL_DATABASE database..."

while ! nc -z $SQL_HOST $SQL_PORT; do
  sleep 1
done

echo "$SQL_DATABASE ready!"

# production/staging environments
if [ "$1" = "gunicorn" ]
then
  mkdir -p staticfiles
  python manage.py collectstatic --noinput
  python manage.py migrate
fi

exec "$@"

Notes:

This writes the environment variables into /etc/environment at runtime so that file can be source'd by each crontab entry.
The final exec "$@" executes the CMD for each of the docker-compose.yml services.

4. Install crontab used by cron in the final build stage of the Dockerfile

FROM python:3.11-slim-bullseye as base

ENV PYTHONFAULTHANDLER=1 \
  PYTHONHASHSEED=random \
  PYTHONUNBUFFERED=1 \
  PYTHONDONTWRITEBYTECODE=1

ENV VIRTUAL_ENV=/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

WORKDIR /app

######################
FROM base as builder

ENV VIRTUAL_ENV=/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
ENV PIP_DEFAULT_TIMEOUT=100 \
  PIP_DISABLE_PIP_VERSION_CHECK=1 \
  PIP_NO_CACHE_DIR=1

RUN apt-get update && apt-get install --no-install-recommends -y git curl netcat build-essential libpq-dev libjpeg-dev zlib1g-dev libffi-dev

RUN curl -sSL https://install.python-poetry.org | python3 -

COPY pyproject.toml poetry.lock ./

# Only non dev dependencies
RUN $HOME/.local/bin/poetry install --no-dev

#####################
FROM base as final

COPY --from=builder /venv /venv
COPY --from=builder /root/.local /root/.local

# install runtime dependencies
RUN apt-get update && apt-get -y upgrade && apt-get install --no-install-recommends -y libffi-dev libpq-dev libjpeg-dev memcached libmemcached-tools netcat cron

# copy project
COPY . /app/

# install crontab to run django_extensions runscript - it is only run in the cron container
COPY crontab/djangocrondaily /etc/cron.d/djangocrondaily
RUN chmod 0644 /etc/cron.d/djangocrondaily

EXPOSE 8000

# run entrypoint.sh
ENTRYPOINT ["/app/entrypoint.sh"]
# entrypoint.sh gets CMD as arguments
CMD ["gunicorn", "api.wsgi:app", "--bind", "0.0.0.0:8000", "--log-level=info", "--access-logfile=-", "--error-logfile=-", "--workers=2", "--threads=4", "--worker-class=gthread", "--worker-tmp-dir=/dev/shm"]

Notes:

the default CMD starts gunicorn for my production and staging environments

5. Simplified docker-compose file that starts my django and cron docker services:

version: '3.7'

services:
  django:
    image: registry.xxx.com/yyy/zzz/api-django
    build:
      context: .
      dockerfile: Dockerfile.prod
    container_name: api-django
    ports:
      - 8000
    env_file:
      - ./env.prod
    depends_on:
      - db

  cron:
    image: registry.xxx.com/yyy/zzz/api-django
    build:
      context: .
      dockerfile: Dockerfile.prod
    container_name: api-cron
    env_file:
      - ./env.prod
    command: >
      cron -f -L 15

Notes:

both services use the same dockerfile, image, context, and env_files settings.
the cron service overrides the CMD/command to run cron in the foreground instead of the default gunicorn process.

I hope you find this helpful if you need to run cron to execute periodic Django tasks.

Building/Uploading Source Maps and Triggering a Deploy in Angular 11 to Rollbar

2021-02-07T13:00:00-06:00

I recently updated my Agility Course Master Ionic PWA to Angular 11 and wanted to fully automate generating and uploading of source map files to Rollbar. I found an existing project that almost did everything I needed: angular-rollbar-source-maps. It was missing triggering Rollbar's deploy endpoint to indicate that a new release had been deployed.

So I took a little time to add that feature and I also changed the scripts so only node was needed to generate the Git SHA revision and to perform the upload/deploy HTTP POSTs to Rollbar.

Overview

During the build the file src/environments/version.ts is automatically generated containing the definition of the current Git revision:

// this file is automatically generated by git-version.js script
export const version = {revision: '12f54f9'};

Import that version.ts file into the file where you create your Rollbar.Configuration and use it to set your code_version Rollbar parameter.

The same version is used when uploading the source maps and informing Rollbar of the deploy.

The entire process is initiated by a single command:

npm run prod

Installation

Edit package.json and add the prod target line to the scripts section. This generates and uploads the source maps and then deletes them from the www directory so they aren't uploaded to your site.
Create a .env file with values populated for your site (you probably want to add .env to your .gitignore file also):
npm install dotenv
Copy git-version.js and rollbar-deploy.js (below) into the same directory as your package.json
Edit rollbar.ts to import the environment/version.ts file and set the version in the Rollbar configuration.

Building and Deploying

When you are ready to build/deploy:

npm run prod

Once the script is done the www directory contains only the artifacts you need to deploy your site (no JS source map files). In my case I deploy to firebase via:

firebase deploy

Lightning Talk: StorybookJS Streamlines UI Component Development

2020-10-24T20:00:00-05:00

I became interested in Storybook.js when I was creating a new UI component and wanted a way to quickly iterate on the design and be able to demo the different ways it could be used to my team members.

In the past I would temporarily add a page to our application, add multiple versions of the component, configure them for each variation, and then demo it. Then repeat as we refined the design. The temporary page would never be committed to the product's main branch.

Storybook had recently released version 6 so I thought I'd give it a try. I was really impressed with how easy it was to use for my scenario.

You define your component as you normally would (in VueJS in my case) and import it into a "story file". A story file is just a JS (or Typescript file) where you define the properties for your component for any variations (use cases) in which you are interested. Storybook then reads that file and renders pages containing the component for each story.

I was able to quickly modify my component, interact with it, style it, create new stories for different property configurations, and Storybook re-displayed them on each save. It made for a fast and easy development cycle!

TL;DR: These are the key takeaways for using Storybook when developing web components:

Installs along side your components/project.
Watches your component and story files for changes and automatically hot reloads.
Story files define your components' expected use cases.
The Storybook UI also allows for interactive modification of properties in the browser.
The Storybook UI provides basic documentation for users of your components.
Individual stories can be directly imported into spec files unit and visual regression tests. This is a valuable Storybook feature that is often overlooked!

I was so impressed by Storybook I put together a "lightning talk" on using it with VueJS for the Northwest Chicago JavaScript meetup September 2020 virtual meeting. Check it out to see how easy and powerful Storybook is to use. Here's the video of my presentation (unfortunately, the first few minutes didn't get recorded):

If you'd like to see my slides they are online. Use arrow keys/spacebar to navigate them.

My example repo is on GitHub.

If you develop web components/controls give Storybook.js a try to simplify component development and test, provide an interactive component browser, and component documentation.

Moving the AgilityNerd.com Blog to GatsbyJS

2019-11-28T08:00:00-06:00

As a software developer, one of the ways I learn about new web technology is by attending the monthly Northwest Chicago JavaScript meetup. It's a supportive and enthusiastic group of folks interested in continuing to learn more about software development in JavaScript. Every month there are one or more informal presentations by members of the group with questions and discussions.

This past week I gave a talk titled: "Moving the AgilityNerd Blog to GatsbyJS". I started my dog agility blog AgilityNerd in 2004 using the now defunct, Perl-based, flat file CGI site generator, blosxom. Over the years I wrote, enhanced, and fixed a number of plugins to the Blosxom project. Though it worked well, technology has long since moved passed Blosxom's humble beginnings.

The costly hosting requirements for Blosxom at the AgilityNerd's site size (with over 800 source pages generating 1,500 pages dynamically), authoring in HTML, lack of preview, and enhancing a nearly 20 year old code base became so onerous that I decided to extract the content and move to a more modern static site generator.

This site, tech.agilitynerd.com, uses the Python powered Pelican static generator and deploys to GitHub pages. It is simple to author, deploy, and free to host. But I was looking for a static site generator that would make it easy to integrate custom web components, provide image processing, dynamic image/content loading, and integrate 3rd party plugins like: Google Analytics, Rollbar error tracking, and advanced search (Algolia).

My search boiled down to React-based GatsbyJS and Vue-based Gridsome. I've been using Vue daily at work for almost two years, but Gridssome isn't as far along in its development as Gatsby. It has been a while since I did any serious React development so I decided to go with the more popular GatsbyJS as a reason to get my skills on React more up to date.

Gatsby's offering an easy "GitOps" continuous deployment model hosted (for free) on Netlify was just icing on the cake.

Here's a link to my slides. Interestingly, the slides are written using MDX Deck which is itself built on top of Gatsby. Of course I used that functionality to deploy the slide deck to Netlify too.

Here's the video of my presentation which goes into a lot more detail on my reasoning for choosing Gatsby, setting it up, how it works, how plugins work, and deploying a Gatsby site:

Since this presentation, I've started using Gatsby Cloud's build service to get faster builds and then have it upload the assets to Netlify for hosting.

Disabling Rollbar and Enabling Logging During Vue.js Development

2019-03-30T08:00:00-05:00

I was adding Rollbar support to a Vue.js application and ran into an issue that made it inconvenient for use in development environments:

Captured exceptions and rollbar.error|warning|log|... calls are always sent and logged in the Rollbar dashboard.

You could configure different a Rollbar environment for development and filter the dashboard; but I'd rather not even send "developer-induced" errors to Rollbar at all. After speaking with support they don't yet have a "log locally and don't send to Rollbar" configuration.

So I came up with this simple Vue plugin that is only installed during development builds:

if (process.env.NODE_ENV === 'development') {
  const Rollbar = function () {
    // I only call $rollbar.error in the app
    this.error = console.error
    // define other rollbar API mocks here
  }
  const RollbarMockPlugin = {
    install: function (Vue, options) {
      Vue.rollbar = new Rollbar(options);
      Vue.prototype.$rollbar = Vue.rollbar;
    }
  }
  Vue.use(RollbarMockPlugin)
} else {
  // Your normal Rollbar configuration
  // I set some env vars in the build script
  Vue.use(Rollbar, {
    accessToken: process.env.VUE_APP_ROLLBAR_POST_CLIENT_ITEM_KEY,
    captureUncaught: true,
    captureUnhandledRejections: true,
    enabled: process.env.NODE_ENV === 'production',
    environment: process.env.DEPLOY_ENV, // dashboard filtering
    verbose: true, // also log to console
    payload: {
      client: {
        javascript: {
          guess_uncaught_frames: true,
          source_map_enabled: true,
          code_version: process.env.VUE_APP_VERSION
        }
      }
    }
  });

So when I have code like this in the app:

async save(value) {
  this.loading = true
  try {
    const payload = { todo: value }
    await this.saveTodo(payload)
  } catch (e) {
    this.$rollbar.error(e)
    this.notifySaveFailed()
  }
  this.loading = false

The call to this.$rollbar.error(e) now logs only to the dev console when running in development.

So I hope this little shim helps you keep your Rollbar dashboard free from development environment clutter.

Build In Customer Support to Your Mobile/Web App/PWA

2018-10-28T08:00:00-05:00

Here are my slides on Build in Customer Support: Simplify support of your mobile/web applications for a "lightning talk" that I gave at the October Northwest Chicago JavaScript meetup.

I've been a developer for a while and have worked in organizations who have been focused on the customer (really!). So I've always been interested in helping support folk and developers build and improve the product. There is a lot of discussion on design to help products meet customer needs and improve usability/interaction. But I hadn't found any discussions on how to build your product so you can provide your customers with better support.

So I put together this talk on what I've learned. I focus on tools/techniques for web/mobile apps and support for small groups/single developers. Here's the outline:

Overview support options and choose channels that fit your customers.
Demonstrate some tools for those channels that are helpful for startups/side projects.
Can you avoid support?
Find/fix bugs before they are detected by customers.
Acting as the customer
Reproducing bugs
Acting on and Documenting issues

There is also video of my talk:

Cypress E2E Testing Overview Presentation

2018-06-21T08:00:00-05:00

Here are my slides on Cypress E2E Testing for a talk that I gave at the June Northwest Chicago JavaScript meetup.

I first tried Cypress when testing my Agility Course Master hybrid mobile (Ionic) application/website. I was really impressed with how easy it was to install and use. The thorough and well written documentation together with their online support further "sold" me on it for testing.

So I put together this talk on what I've learned. Here is video of my talk:

PouchDB and Couchbase Replication for the Offline-First Web

2017-09-21T08:00:00-05:00

I gave a short talk on 'PouchDB and Couchbase Replication for the Offline-First Web' at the September Northwest Chicago JavaScript meetup.

It builds on Peter Mbanugo's blog post: Using PouchDB and Couchbase in an Offline-First Application by using the PouchDB API in place of the Hoodie API Peter used.

My source code is in this GitHub repository.

Here is a video of my talk:

Using the Ionic Framework for Mobile and Web Development

2017-07-20T08:00:00-05:00

Here are my slides on Using the Ionic Framework for Mobile and Web Development for a talk that I gave at the July Northwest Chicago JavaScript meetup.

I've been using Ionic since early 2017. I was drawn to it because it allowed a single code base to create native applications for both iOS and Android (with Cordova providing the abstraction on top of the mobile platforms). Ionic provides a set of platform styled widgets, integration and tooling to make it work together easily. Their use of Typescript/Angular also meant I could create a purely web version of the application also using the same code.

Here is a video of my talk:

Django Shrink The Web django-stw 1.0 Released

2017-01-10T13:00:00-06:00

It has been several years since I've worked on my Django driven website that uses Shrink The Web (STW) screen shots. During that time STW has simplified their feature set and their API.

As I upgraded my site (Googility.com) I took some time to modernize the django-stw template tag and support API changes for the latest Django and Python versions.

I also tested the package when deployed using HTTP and HTTPS.

Here are the changes (from the CHANGELOG.txt):

1.0.1 Use https for all requests to shrinktheweb.com
- Stops browser warnings when used on https websites while not interferring with sites running on http.

1.0.0 Updates for Django 1.9/1.10 and Python 2.7/3.5
- Old versions of Django/Python are no longer supported.
- Removed ``shrinkthewebimage`` template tag since there is no longer an API distinction between free and Pro accounts.

You can the template tag in action on this demo page in a nicer "Bootstrap-ified" UI:

The latest package is available on PyPi:

pip install django-stw

Or as a source download on github, or via git clone.

Using Gestures in Angular 2 with TypeScript Lightning Talk

2016-12-14T14:00:00-06:00

Here are my slides on Using Gestures in Angular 2 with TypeScript for a lightning talk that I gave at the December Northwest Chicago JavaScript meetup.

I converted the Reveal.js slides checkout to use the Angular 2 CLI. That let me include two small Angular 2 components within the slides themselves. Which also allowed me to demonstrate using the pan gesture without leaving the presentation. The Angular CLI also made it easy to generate and publish the presentation to the GitHub pages associated with the slides via ng github-pages:deploy. Here's the source code for the slides/components.

I have more examples of using gestures in the larger demo app I showed at the beginning of the talk: Angular 2 TypeScript Virtual Image Cropping repository.

Here is the video of my presentation:

Maximize and Minimize Code Blocks in Reveal.js Slide Shows

2016-12-04T12:00:00-06:00

I was working on a slideshow about Using Gestures in Angular 2 Components for a lightning talk at the December Northwest Chicago JavaScript meetup and I found the code sections just weren't large enough. So I threw together a little JavaScript to add "+" and "-" buttons next to each code section that maximizes/restores the code blocks for easier viewing during the presentation:

Just copy/paste the following into your index.html file:

Pelican For Fast Site/Project Development

2016-11-08T13:00:00-06:00

I was working on a simple single page website for calculating dog agility jump heights and was really missing the tool chain I normally use in Flask and Django web sites for bundling, compressing, and versioning CSS and JS files and a mechanism for putting the bundled/versioned file names in the HTML files. I was searching for what I needed and was about to write a little script to do it and then it occurred to me that Pelican already has just what I needed!

Pelican is known for making static generation of blogs easy. But it also has a lot of powerful features that can be easily leveraged to create small web sites and web sites for projects:

webassets integration to SASS/LESS, minify, bundle, and version CSS and JavaScript. It automatically inserts versioned bundled names in the HTML.
Dozens of themes.
Themes are implemented using Jinja2 templates and allow sharing page layouts across your project's pages. You can also have custom templates per page.
Theme templates already contain useful integrations which can be used in you templates:
- Navigation
- Analytics
- Disqus
Dozens of Pelican plugins can be installed to add new features.
During development regeneration of files is automatic when you save files.
Many deployment options are also available:
- GitHub pages
- FTP/SSH to your own server
- Dropbox
- S3
- Rackspace Cloud Files

Overview

I've written a lot of detail on every step but it is actually very easy to use Pelican for non-blog web sites. Here's how it works:

Create a virtualenv and install Pelican and webassets Python packages (I use the same venv for all my Pelican projects).
Checkout pelican-plugins.
Edit the Pelican theme's base.html Jinja template to include the CSS and JavaScript files you need.
Create your pages' body content in files in content/pages in HTML, Markdown or reStructuredText.
Run make devserver and refresh your browser to see your changes or use this LiveReload script to automatically reload your browser. Repeat steps 3 and 4 until you are done.
Deploy to GitHub Pages: make github. Or deploy to your own server.

Setup the Environment

Install and configure Pelican for creating non-blog web sites:

Create a virtualenv and install pelican, webassets, cssmin, and jsmin (or any other CSS/JS filters supported by webassets):
```
virtualenv ~/virtualenvs/pelican
source ~/virtualenvs/pelican/bin/activate
pip install pelican webassets cssmin jsmin
```
Check out the pelican plugins repository outside of your project (plugins are only used during the build process):
```
git checkout https://github.com/getpelican/pelican-plugins.git
```

Create your project directory, run pelican-quickstart and answer the questions just like for a blog site:

mkdir myproject && cd $_
pelican-quickstart

Welcome to pelican-quickstart v3.6.3.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.

> Where do you want to create your new web site? [.]
> What will be the title of this web site? Dog Agility Jump Height Calculator
> Who will be the author of this web site? Steve Schwarz
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) y
> /agility-jump-heights           <-- enter the name of your GitHub repository
> Do you want to enable article pagination? (Y/n) n
> What is your time zone? [Europe/Paris] America/Chicago
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n) y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) y
> Do you want to upload your website using FTP? (y/N) n
> Do you want to upload your website using SSH? (y/N) n
> Do you want to upload your website using Dropbox? (y/N) n
> Do you want to upload your website using S3? (y/N) n
> Do you want to upload your website using Rackspace Cloud Files? (y/N) n
> Do you want to upload your website using GitHub Pages? (y/N) y
> Is this your personal page (username.github.io)? (y/N) n
Done. Your new project is available at /home/dev/agility-jump-heights

Paste the following at the end of your pelican.conf:

STATIC_PATHS = ['images']  # put page specific assets here
PLUGIN_PATHS = ['../pelican-plugins']  # set this to the location of your plugins checkout
PLUGINS = ['assets']
THEME = './theme'          # All CSS/JS files go in directories under here
# I only want to generate Pages so I disable all "blog-like" pages see Note in:
# http://docs.getpelican.com/en/stable/settings.html?highlight=url_for#url-settings
TAGS_SAVE_AS = ''          # Don't generate Tags pages
TAG_SAVE_AS = ''
CATEGORY_SAVE_AS = ''      # Don't generate Category pages
AUTHOR_SAVE_AS = ''        # Don't generate Author pages
DIRECT_TEMPLATES = ['index']  # Don't generate tag, category, or author output for some themes
# In the generated output directory move files to the root and adjust their URLs to match:
PAGE_URL = '{slug}.html'
PAGE_SAVE_AS = '{slug}.html'
INDEX_SAVE_AS = "/ignore/index.html"  # don't create normal index.html which lists all articles and pages

Copy any theme from pelican-themes into .theme or I just copy the notmyidea theme installed with Pelican from the virtualenv:
```
cp -pR $VIRTUAL_ENV/lib/python*/site-packages/pelican/themes/notmyidea/ theme
```
Have Git ignore the output directory:
```
echo "/output" >> .gitignore
```

Create Your Project Web Site

Setup Templates

Edit ./templates/base.html and delete/add any sections, stylesheets and javascript you like. Your pages only need to define content that goes in the content block of the Jinja templates. Of course you can define your own templates and use the full power of Jinja templating even for individual pages.

For small projects it is easiest to serve the same JS/CSS on all pages so I put them in the base.html file. Using Jinja template inheritance you can also create and serve separate bundles for individual pages.

I use webassets right in the template to define how to combine JS/CSS files into bundles, minify and version them. For CSS files in the head of my base.html:

 {% assets filters="cssmin", output="css/style.%(version)s.min.css", "css/normalize.css", "css/skeleton.css", "css/style.css" %}
<link rel="stylesheet" href="{{ SITEURL }}/{{ ASSET_URL }}">
 {% endassets %}

For JavaScript the bundled, versioned, compressed script tag(s) is defined similarly just before the end of the HTML body tag:

{% assets filters="jsmin", output="js/main.%(version)s.min.js", "js/main.js" %}
<script src="{{ SITEURL }}/{{ ASSET_URL }}"></script>
{% endassets %}

For more options see the webassets README.

Edit theme/templates/page.html to suite your needs. I just put in a wrapper div around the content:

{% extends "base.html" %}
{% block title %}{{ page.title }}{% endblock %}

{% block content %}
<div class="container">
{{ page.content }}
</div>
{% endblock %}

You can also delete any CSS, JS, images, and unused Jinja templates from your copied theme.

Write the Pages

Create the pages directory:

mkdir content/pages

Lastly put each page's body content in a file in the content/pages directory. I like to write the body content in HTML. You put the Pelican metadata in meta elements in the head element as shown the Pelican docs. Here's index.html and I recommend specifying the title and save_as:

<html>
    <head>
        <!-- By default used to create the URL slug -->
        <title>Dog Agility Jump Height Calculator</title>
        <!-- Override the default URL made up of the slug; needed for the index.html -->
        <meta name="save_as" content="index.html"/>
        <!-- any other metadata attributes as meta tags; none normally needed -->
    </head>
    <body>
        <!-- all  markup goes here. e.g. -->
        <h1>Hello World!</h1>
    </body>
</html>

You can use any input syntax supported by Pelican e.g. ReStructuredText, Markdown, or even write a Reader class for your own custom input file format.

Start up the Pelican development server to watch for file changes and regenerate changed files:

make devserver

Point your browser to http://localhost:8000/.

I recommend using this LiveReload script as it also watches for changes to the themes directory and automatically reloads your browser on http://localhost:5500.

Once you are setup just edit your templates, JS, and CSS under the theme directory and add/edit pages in your content/pages directory.

Deploy

I like to deploy small projects to GitHub Pages and it's this easy:

make github

Then on GitHub enable GitHub Pages in your project's settings.

To see this whole setup in action take a look at this single page calculator application with one JS and HTML file.

The next step to make this even easier would be to use Cookiecutter to make setting this up via one command.

LiveReload with Pelican

2016-11-06T00:00:00-05:00

I was looking to use LiveReload while developing using Pelican and I came across this nice simple solution by Merlijn van Deen.

In my use case I also wanted to watch the pelicanconf.py file and themes directory for changes and then regenerate the output and reload the browser. Lastly I wanted to use the host/port defined in my pelicanconf.py. So I made some small edits to his script named pelican-livereload.py:

Just copy it into your Pelican top level directory and execute it:

python ./pelican-livereload.py

The LiveReload server automatically injects the livereload JavaScript script tag into the HTML so you don't need to install the LiveReload browser extension.

So all you need to do is visit the SITEURL you've specified in your pelicanconf.py otherwise it defaults to http://localhost:5500. Then any edit you make causes Pelican to regenerate the files and the browser immediately refreshes. The only downside to the regular Pelican watcher feature is all files are regenerated instead of just the modified file. But for me having the browser automatically reload is is worth the extra brief delay.

HTTP/2: A Quick and Easy Website Speed Up

2016-09-23T18:00:00-05:00

I always want my websites to be secure and fast. HTTP/2 is the latest enhancement to the HTTP protocol that can provide significant performance improvements:

The primary goals for HTTP/2 are to reduce latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, and add support for request prioritization and server push.

—High Performance Browser Networking - O'Reilly:

It turns out all browsers that support HTTP/2 also require TLS (HTTPS). So my first step was adding HTTPS support to agilitycourses.com which also provides a backbone on which I'll implement secure user profiles. Then I wanted to enable HTTP/2 to see if it improved the speed of pages served to end users.

But First an OS Upgrade

After a little research I found that recent releases of NGINX support HTTP/2 and those versions are packaged with Ubuntu 16.04 LTS. That made upgrading to this latest long term support OS version a better approach than just upgrading NGINX on my older OS. I basically went through the standard upgrade process.

While I had to work through a number of small issues upgrading my older websites; the biggest change was converting my upstart scripts to systemd scripts. But the beauty of hosting with virtual servers is it was trivial to create an image of my existing server, spin up a temporary server using that image, and then practice the ugprade/migration on the temporary server. I was able to move some configuration changes back to my live server and test out the changes to my sites' Fabric deployment scripts.

Once everything was working I configured my websites that contain user modifiable data into maintenance mode on my temporary server. Then I pointed the DNS for all my domains to the temporary server. After DNS propagated I shutdown Postgres and the webserver on my original server and created a backup image. Then I went through the upgrade process. Once I validated the migration was successful I switched the DNS back to my original server and shutdown the temporary server. After a couple days I deleted the temporary server and the pre-migration backup image.

Enabling HTTP/2

Once I was on NGINX version 1.10 the change to the website to enable HTTP/2 was as simple as changing the virtual server from:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
...
}

to:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
...
}

Then I just reloaded the Nginx configuration: sudo systemctl reload nginx. That was it!

Testing

Here's the network view of the timing of requests for a page using HTTP/1.1 over HTTPS in Chrome:

Here's the same page using HTTP/2 over HTTPS:

The obvious difference is in the Timeline - Start Time column. In the first diagram you can see the "waterfall" queueing up of requests for images as all the open sockets to the domain were in use. In the second diagram you can see them all interleaved as they are multiplexed across the same connection.

Also the bottom line is the page is fully loaded in 1.73 sec using HTTP/1.1 and in 1.16 sec using HTTP/2 for a 33% end user page load improvement!

Next Steps

So that was a nice free speed up and there are plenty of areas I can still investigate to further improve the performance of this web site:

Bring 3rd party resources back to my domain to save DNS lookups and see how that affects overall performance. This may hurt performance for HTTP/1.1 clients who are limited to the number of connections per domain.
Change multiple SVG files (which are all steps in a progression) into a single file and use CSS to hide/show appropriate steps using a single image. This will greatly reduce the number of individual files downloaded and save some duplicated data within those files.
Move Google Analytics locally. GA has a number of problems (short cache times and multiple files downloaded) so at the cost of periodically updating the files I could host them all locally and save some time.
Disqus is even worse than GA when it comes to many file downloads, redirects, short cache times etc. I might change to have Disqus data loaded on user demand.
Experiment with other HTTP/2 features:
- preconnect might help reduce the cost of DNS look ups that result from redirects like those used by Google Analytics, Google Fonts, and Disqus resources.
- preload to load resources before they are accessed by JS during onload or via user actions.
- prefetch to load JS/images for future pages. There are some common workflows on this site that could benefit from prefetching the JS for subsequent pages.

NGINX, HTTPS, Let's Encrypt, and Django

2016-09-04T12:02:00-05:00

My agilitycourses.com website is served by nginx proxying to gunicorn running my Django application. I'll be adding user accounts soon so I wanted to convert the site to be more secure by using HTTPS encryption. Also Google has announced it will likely prefer sites using HTTPS.

The site is running on Ubuntu 14.04 LTS. I won't recount the whole process, I followed some great resources and I'll discuss a couple adjustments that might be helpful to others.

I basically followed the instructions in this excellent Digital Ocean tutorial: How To Secure Nginx with Let's Encrypt on Ubuntu 14.04.
I confirmed via the SSL Labs SSL Server Test that my IPv4 and IPv6 server configurations had "A+" ratings.
While looking for other SSL testing sites I came across securityheaders.io developed by Scott Helme. My initial score was a sad "D". The site has snippets for NGINX and Apache configuration changes and in depth articles describing the how and the why.
While investigating the changes to the HTTP Headers to improve my test score I came across this nginx-conf GitHub repository. Specifically the idea of putting the header settings into an NGINX include file. I have several other domains on the same server and will also be converting them. I used that idea to include ssl and header configurations into any virtual host.

Here's my /etc/nginx/ssl.conf file:

# From https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-14-04
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;

And my /etc/nginx/security_headers.conf file:

# See https://github.com/jonnybarnes/nginx-conf/blob/master/conf/includes/security-headers.conf
add_header X-Xss-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";
add_header Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'";
add_header Strict-Transport-Security max-age=15768000;

So my server blocks with all these edits are now:

# redirect http://www.tld and http://tld to https://www.tld
server {
    listen 80;
    listen [::]:80;
    server_name www.agilitycourses.com agilitycourses.com;

    # letsencrypt location
    location ^~ /.well-known/ {
        allow all;
        root /usr/share/nginx/html/;
    }
    location / {
        return 301 https://www.agilitycourses.com$request_uri;
    }
}

# redirect https://tld to https://www.tld
server {
    listen 443 ssl;
    listen [::]:443 ipv6only=on ssl;

    server_name agilitycourses.com;
    # certificates are needed here too
    ssl_certificate /etc/letsencrypt/live/agilitycourses.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/agilitycourses.com/privkey.pem;
    return 301 https://www.agilitycourses.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name www.agilitycourses.com;
    root /home/agilitycourses/production/current/;

    ssl_certificate /etc/letsencrypt/live/agilitycourses.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/agilitycourses.com/privkey.pem;

    include /etc/nginx/ssl.conf;
    include /etc/nginx/security_headers.conf;
    ...
}

So now I have an "A" score from securityheaders.io

The Digital Ocean tutorial sets up a root crontab entry to automatically update the SSL Certificate. I decided to also update the letsencrypt client software automatically:

# m h  dom mon dow   command
20 2 * * 1 cd /opt/letsencrypt && git pull
30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /etc/init.d/nginx reload

The last change I made was to pass along the presence/absence of HTTPS from NGINX to Gunicorn/Django via the X-Forwarded-Proto header as described in the Django SSL/HTTPS docs

location @proxy-to-app {
    proxy_pass http://agilitycourses-production-gunicorn;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Accept-Encoding "";
    proxy_read_timeout 120;
    proxy_send_timeout 120;
    ...
}

Based on the Django recommendations I also made these changes in my settings.py:

# SSL settings
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

Even with a lot of web browsing to learn about these settings the whole process only took a couple hours. Now that I've done it once (and updated my Fabric fabfile.py) it will be easier to convert my other domains.

NGINX CGI Parameter Gotcha

2016-07-31T12:02:00-05:00

When I first started the agilitynerd blog in 2004 I had my Blosxom blogging CGI script running via Apache. Later on I moved all my sites to nginx or took advantage of nginx's caching features to have it act as a proxy in front of Apache. I finally decided to remove Apache entirely and that meant solving running CGI scripts using nginx.

After some googling I found FastCGI and fcgiwrap

I'm on Ubuntu so installation was as easy as:

sudo apt-get install fcgiwrap

That setup the init script that starts the fcgi daemon. To run the cgi script(s) nginx has to be configured to parse apart the incoming URL, execute the appropriate script and pass along any arguments needed by the CGI script. Sounds easy.

I only want to support running a single CGI script: index.cgi and pass along the path after the root of URL as the argument to the script. Most examples are more generic and parse out any cgi script and any arguments.

The key built-in to nginx to do the splitting is fastcgi_split_path_info which takes a regex with two captured groups to parse out the script name and the arguments. These are stored in the $fastcgi_script_name and the $fastcgi_path_info variables respectively. This Digital Ocean article discusses FastCGI and has an excellent discussion of the variables available and also used by fcgiwrap.

So I created this configuration file that matches http://agilitynerd.com/blog/foo.html and invokes /home/agilitynerd/cgi-bin/index.cgi foo.html:

location /blog/ {
    root /home/agilitynerd/cgi-bin/;

    fastcgi_split_path_info ^(/blog)(.*)$;
    include /etc/nginx/fastcgi_params;
    fastcgi_param DOCUMENT_ROOT /home/agilitynerd/cgi-bin/;
    fastcgi_param SCRIPT_NAME index.cgi$fastcgi_path_info;

    # Fastcgi socket
    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
}

You'll notice: include /etc/nginx/fastcgi_params is used to get default values for the fastcgi_param variables.

And it didn't work. I kept getting errors:

Cannot get script name, are DOCUMENT_ROOT and SCRIPT_NAME (or SCRIPT_FILENAME) set and is the script executable?"

Clearly I'm setting DOCUMENT_ROOT and SCRIPT_NAME. After almost a day of googling and testing (during which I found this helpful article on nginx debugging) I temporarily commented out fastcgi_pass, and returned the variables.

I found that they were being set as I expected... Strange!

Then I came across this Digital Ocean article where they have a critical discussion on overriding variables in which they state:

This inconsistency and unpredictability means that you cannot and should not rely on the backend to correctly interpret your intentions when setting the same parameter more than one time. The only safe solution is to only declare each parameter once. This also means that there is no such thing as safely overriding a default value with the fastcgi_param directive.

So in my case I commented out DOCUMENT_ROOT and SCRIPT_FILENAME in the /etc/nginx/fastcgi_params file, reloaded nginx, and voila! Everything worked. Hope this helps you if you run in to the same problem.

ReactJS SVG Path Player Component

2016-01-24T12:00:00-06:00

I've been a big fan of SVG images for many years for their light weight and resolution independence. I started playing with them back when most browsers needed a plugin to render them; which kept me from using them in web sites. Within the past few years SVG has become natively supported by almost all browsers and mobile devices so I could finally use them on my agilitycourses.com website to display dog agility obstacles and the sequences through them.

My next enhancement to the site was to animate the shortest/fastest paths dogs could take through the obstacles. I had found the Snap.svg JavaScript library which is light weight, resonably well supported and, to make it even easier, I found a great example of animating a path and a marker along the path using it.

I decided to make a "media player component" that I could instantiate in multiple places in the site and realized if I generalized the component a little it might be useful for others.

I also decided to use React to create it. React has a well defined component approach and I thought this would be a good learning experience with the library and the tooling/packaging required to share it.

Here's what the SVG Path Player component looks like in action:

Here's an article from my dog agility blog explaining how dog agility handlers can use this player.

Some Things I Learned

I won't walk through the code, it's all on GitHub, but I'll give you some pointers and links to articles/videos I found useful.

The SVGPathPlayer component itself renders the UI and provides the button's callback methods that call to a Snap.svg element that "owns" the SVG image it controls. The componentDidMount method uses Snap.svg to load the SVG image and select the path(s), and optional marker within it. Within the render method the element into which the SVG image is rendered is a React ref child element:

<div className="svg-container svg-container-box" ref={(ref) => this.svgImage = ref}></div>

I chose to store the ref on this and not in the state of the component since it doesn't impact the component state; it is a data member used only by Snap.

Once loaded the render method uses a Controls component to show the animation status and allow the user to start, stop and step forward/backward within the animation. The Controls component is a "stateless" or "dumb" compone t; it's buttons invoke callbacks provided via it's props by it's parent "smart component": SVGPathPlayer.

Making Controls and Spinner dumb components made writing tests for them really easy. I might never reuse these components, but they made reasoning about responsibilities easier and helped me simplify the interfaces (props) passed in to each component. Going through the refactoring into components I also deleted some internal state I didn't really need in the original monolithic component.

Writing the React code was straight forward and I found using ES6 syntax made it even easier. The big challenges with this project were packaging it as a reusable component that could be used in both "script" and "npm" installations, generating GitHub hosted pages, and automating the testing/packaging/deployment as part of the Travis-CI automation.

There are a lot of boilerplate React application projects out there but not too many for reusable React components. I found survivejs/react-component-boilerplate to be very well supported and included most of the functionality I wanted. I also bought the ebook SurviveJS - Webpack and React - From apprentice to master which has been continuously updated to incorporate all the recent changes in the Webpack/Babel tools (I still need to migrate my tooling to the latest Babel release).

So checkout my project's package.json npm scripts and the webpack.config.babel.js distribution configurations for the scripts/configurations to create all the pieces. Getting all of this to work is still a little complex and I should document how I got it working as well as creating my own boilerplate project. I know I would have benefitted from an annotated versions of those files.

Another very helpful resource was Kent C. Dodds egghead.io videos on How to Write an Open Source JavaScript Library particularly for Travis-CI integration and using semantic-release.

If you are looking to make your own redistributable React components I strongly recommend reviewing both of those resources and looking through the configuration of my component. I hope to write up the details after I find some time to migrate the webpack tooling to the latest versions.

Upgrading Pelican and Migrating to GitHub Pages

2015-11-29T15:00:00-06:00

I've been using Pelican for this blog for almost three years with source and output stored in a GitHub repository. The output files were then checked out and hosted as static content behind an NGINX web server on my VPS. Since I set that up GitHub introduced GitHub Pages with support for custom domains and all the "cool kids" started hosting their static web sites right on GitHub.

I had some free time this weekend and decided to see what it would take to upgrade my Pelican version to the latest (3.6.3) and host my files on GitHub Pages. I had four steps to perform:

Create a new environment with the latest Pelican
Update my content files for the changes in Pelican versions
Put output files into GitHub and verify them on GitHub Pages
Move my subdomain to point to my GitHub Pages

Create New Environment

I didn't want to screw up my existing/working virtual environment so I created a new one containing Pelican and ghp-import which does all the work of updating the gh-pages branch with the output:

# Create a new virtualenv
mkvirtualenv pelican-new
# Install pelican and ghp-import:
pip install pelican ghp-import

Update Content

This was arguably the most painful part as I wasn't using appropriate reStructuredText markup for my images and the location of images required removing /static from the path. So my markup went from:

.. raw:: html

   <div class="thumbnail">

<img src="/static/images/myimage.png" />

.. raw:: html

   </div>

to this (which includes adding a missing alt tag):

.. class:: thumbnail
.. figure:: {static}/images/myimage.png
   :alt: Clever alt image text goes here

Those changes were mostly mechanical and using figure:: in place of raw:: also cleaned up the mark up. I tested the changes locally and confirmed all modified pages where displaying correctly.

Convert to GitHub Pages

This setup is now documented in the Pelican docs on Publish to GitHub and is easy.

It looks like the clever setup is to put the source for the blog in the master branch and then check the output of running pelican into a branch called gh-pages. The ghp-import python package does all the work of creating and updating the gh-pages branch from the output directory for you!

The first thing I did was to switch to my master branch and then remove the content directory and all of it's files:

git checkout master
rm -rf output

Then I edited the .gitignore file to exclude the output directory.

I wanted to keep my existing blog working until I worked out all the kinks in the migration. So I delayed pointing DNS to the GitHub pages. That meant I needed to temporarily change the URL of the blog to match where it will be hosted on GitHub pages. So I edited the publishconf.py configuration file and changed the SITEURL temporarily from:

SITEURL = 'http://steve.agilitynerd.com'

to it's location on GitHub Pages:

SITEURL = 'http://saschwarz.github.io/steve-agilitynerd'

Get the URL by clicking on the Settings tab for the GitHub repository:

Now that the master branch is set up I checked in and commited the changes:

git commit -a -m"Migration to GitHub Pages"

Now I followed the instructions in the Pelican docs to generate the output and add it to the gh-pages branch via ghp-import (except they show using pelicanconf.py which I use for local development)

pelican content -o output -s publishconf.py
ghp-import output

or, since I opted to have Pelican automation setup, I did:

make github

ghp-import committed and pushed the output to GitHub and I tested that files/images were correctly being served by going to the GitHub Pages URL in my browser.

Move Subdomain to GitHub Pages

This step is well documented in the GitHub help page: About custom domains for GitHub Pages sites. In my case I was already using a subdomain for my Pelican blogs so I just followed their instructions.

On my VPS's DNS configuration screen I deleted my subdomain's A record pointing to my VPS and added a CNAME record pointing to my GitHub .io account.

Then don't followed these instructions: Adding a CNAME file to your repository to setup a CNAME file in the gh-pages branch. The instructions work but ghp-import deletes the content of the gh-pages branch before re-adding files and that deletes the CNAME file you just added!

After some googling I found Tip #2 in the Pelican Tips and followed their instructions. I added the following to my publishconf.py:

STATIC_PATHS = ['images', 'extra/CNAME']
EXTRA_PATH_METADATA = {'extra/CNAME': {'path': 'CNAME'},}

Then I created the CNAME file in the new content/extra directory with the name of my subdomain in it:

steve.agilitynerd.com.

Undo the edit to publishconf.py so it uses the subdomain name:

SITEURL = 'http://steve.agilitynerd.com'

Commit that edit to the master branch and then regenerate the output and commit it to gh-pages branches:

git commit -a -m"Done with migration to sub domain"
git push
make github

Opened the browser to my subdomain and verified that images and links within the site were working correctly. I went back to my VPS and disabled the subdomains from NGINX and deleted the blog check outs to free some resources. Two fewer websites to maintain on my VPS!

Django Migrating Models from an Abstract Base Class to a Concrete Base Class

2015-11-15T05:00:00-06:00

On my agilitycourses.com I had been modeling three types of dog agility courses using an abstract base class Course with three child classes: Box, StarBox, and DoubleBox. This created three tables in the database prepended with the Django application name box: box_box, box_starbox, and box_doublebox. I needed to add a relationship to all three classes from a new table and, rather than creating three separate tables relating to each child table, I decided to convert the Course class to a concrete class/table and relate the new class/table to it instead of each child class. For my purposes the extra join to the child class won't impact performance significantly (if it does I'd move the identity of the type of subclass into a column in the parent Course table and delete the child tables/models).

I didn't find any examples of this type of migration online so I thought I write down my notes in case they are useful to others.

This ends up being a schema migration to put columns in place for inserting data into the parent table, a data migration to populate that table and the new many-to-many table(s), and then another schema migration to remove the temporary columns.

After playing around with a few approaches I found it was easiest to put temporary join ids on the parent class that I could use during the migration and then remove them when I was done. I added these fields to the parent in the models.py:

subclass = models.IntegerField(default=-1)
subclassid = models.IntegerField(default=-1)

and removed the abstract = True Meta class attribute from Course.

I struggled for a while when I found I couldn't control the OneToOneField Django automatically creates from the child classes to the parent. I then saw this StackOverflow answer on a table inheritance question which gave the null/blank field attribute that you'll see I use below.

Backup Your Database

It took me a few attempts to get this right so backups are wise...

First Schema Migration

Now that the models are prepared I created the first database migration adding a default of -1 for the foreign key from the existing child tables to their new concrete parent (which I'll remove manually):

$ python ./manage.py makemigrations box --settings=dev_settings
You are trying to add a non-nullable field 'course_ptr' to box without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
>>> -1
You are trying to add a non-nullable field 'course_ptr' to doublebox without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
>>> -1
You are trying to add a non-nullable field 'course_ptr' to starbox without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows)
 2) Quit, and let me add a default in models.py
Select an option: 1
Please enter the default value now, as valid Python
The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
>>> -1
Migrations for 'box':
  0004_auto_20151114_1255.py:
    - Create model Course
    - Remove field created from box
    - Remove field generator from box
    - Remove field id from box
    - Remove field sequence from box
    - Remove field short_url from box
    - Remove field skills from box
    - Remove field created from doublebox
    - Remove field generator from doublebox
    - Remove field id from doublebox
    - Remove field sequence from doublebox
    - Remove field short_url from doublebox
    - Remove field skills from doublebox
    - Remove field created from starbox
    - Remove field generator from starbox
    - Remove field id from starbox
    - Remove field sequence from starbox
    - Remove field short_url from starbox
    - Remove field skills from starbox
    - Add field course_id to box
    - Add field course_ptr to box
    - Add field course_id to doublebox
    - Add field course_ptr to doublebox
    - Add field course_id to starbox
    - Add field course_ptr to starbox

This automatic migration drops the columns in the subclass tables and with them all the existing data (including keys used in foreign key tables) is lost. But at least I can modify the migration to do what I need for the first migration. The steps will be:

Keep the CreateModel of the parent class, Course, table.
Manually edit the AddField of the OneToOneField on the child classes to keep the existing primary key on the table during the migration. Change them from this:

migrations.AddField(
    model_name='box',
    name='course_ptr',
    field=models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, default=-1, serialize=False, to='box.Course'),
    preserve_default=False,
),

to remove the primary_key, default and add null/blank parameters:

migrations.AddField(
    model_name='box',
    name='course_ptr',
    field=models.OneToOneField(parent_link=True, auto_created=True, null=True, blank=True, serialize=False, to='box.Course'),
    preserve_default=False,
),

Delete all the RemoveField entries in the migration. They'll be added in our final migration.

If you want to see/validate/test the SQL that will be run you can use the sqlmigrate management command (just give it your app name and the number of the migration):

$ python ./manage.py sqlmigrate box 0004
BEGIN;
CREATE TABLE "box_course" ("id" serial NOT NULL PRIMARY KEY, "sequence" varchar(64) NOT NULL, "short_url" varchar(64) NOT NULL, "created" timestamp with time zone NOT NULL, "generator" varchar(2) NOT NULL, "subclass" integer NOT NULL, "subclassid" integer NOT NULL);
CREATE TABLE "box_course_skills" ("id" serial NOT NULL PRIMARY KEY, "course_id" integer NOT NULL, "skill_id" integer NOT NULL, UNIQUE ("course_id", "skill_id"));
ALTER TABLE "box_box" ADD COLUMN "course_ptr_id" integer NULL UNIQUE;
ALTER TABLE "box_box" ALTER COLUMN "course_ptr_id" DROP DEFAULT;
ALTER TABLE "box_doublebox" ADD COLUMN "course_ptr_id" integer NULL UNIQUE;
ALTER TABLE "box_doublebox" ALTER COLUMN "course_ptr_id" DROP DEFAULT;
ALTER TABLE "box_starbox" ADD COLUMN "course_ptr_id" integer NULL UNIQUE;
ALTER TABLE "box_starbox" ALTER COLUMN "course_ptr_id" DROP DEFAULT;
ALTER TABLE "box_course_skills" ADD CONSTRAINT "box_course_skills_course_id_4bbae33e06b494d4_fk_box_course_id" FOREIGN KEY ("course_id") REFERENCES "box_course" ("id") DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE "box_course_skills" ADD CONSTRAINT "box_course_skills_skill_id_35b3dcfd6d387281_fk_box_skill_id" FOREIGN KEY ("skill_id") REFERENCES "box_skill" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "box_course_skills_ea134da7" ON "box_course_skills" ("course_id");
CREATE INDEX "box_course_skills_d38d4c39" ON "box_course_skills" ("skill_id");
ALTER TABLE "box_box" ADD CONSTRAINT "box_box_course_ptr_id_9f73cfe60a5d542_fk_box_course_id" FOREIGN KEY ("course_ptr_id") REFERENCES "box_course" ("id") DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE "box_doublebox" ADD CONSTRAINT "box_doublebox_course_ptr_id_6b112382d489a445_fk_box_course_id" FOREIGN KEY ("course_ptr_id") REFERENCES "box_course" ("id") DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE "box_starbox" ADD CONSTRAINT "box_starbox_course_ptr_id_25fd8909f85eb93a_fk_box_course_id" FOREIGN KEY ("course_ptr_id") REFERENCES "box_course" ("id") DEFERRABLE INITIALLY DEFERRED;

COMMIT;

If you are happy then save and run the migration:

$ python ./manage.py python migrate box

Data Migration

I decided to use SQL (via RunSQL ) for the data migration since it was easier/faster than instantiating each Django model instance as part of the migration. I didn't write reverse migrations since I won't be needing them.

Here's my approach:

Copy subclass rows into parent course table with the subclass column set to a unique value for the subclass (just used a number for each subclass: 1, 2 & 3) and subclassid set to each the child table's id (primary key) value. Together they are a composite key that will be used to tie the parent records back to the child records and their many-to-many relationships.
Update the subclass course_ptr foreign key column with the primary key id of the course table rows having the subclass's id and subclass number value.
Insert subclass's many-to-many table data into the corresponding many-to-many parent table.

Create an empty migration:

$ python manage.py makemigrations --empty box

Then add the migration queries to it (repeat the following for each of the subclasses giving each a different number):

operations = [
    # insert data from subclass into parent class with subclass 'number' and primary key/id
    migrations.RunSQL("""INSERT INTO box_course (sequence, short_url, created, generator, subclass, subclassid)
                      SELECT sequence, short_url, created, generator, 1, id
                      FROM box_box;"""

    ),
    # update subclass primary key to point to parent class (notice composite key values):
    migrations.RunSQL("UPDATE box_box box SET course_ptr_id=course.id FROM box_course course WHERE course.subclassid=box.id AND course.subclass=1;"
    ),
    # insert child's many-to-many foreign key references into it's parent's many-to-many table
    migrations.RunSQL("""INSERT INTO box_course_skills (course_id, skill_id)
                      SELECT box.course_ptr_id, skills.id
                      FROM box_box box JOIN box_box_skills skills
                      ON box.id = skills.box_id"""
    ),
]

Final Schema Migration

Then it is time to edit the models.py file and remove the temporary members/fields in the parent class: subclass and subclassid. Then create the schema migration which will drop those columns and the migrated columns from the child tables:

$ python manage.py makemigrations box
  You are trying to add a non-nullable field 'course_ptr' to doublebox without a default; we can't do that (the database needs something to populate existing rows).
  Please select a fix:
   1) Provide a one-off default now (will be set on all existing rows)
   2) Quit, and let me add a default in models.py
  Select an option: 1
  Please enter the default value now, as valid Python
  The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
  >>> -1
  You are trying to add a non-nullable field 'course_ptr' to starbox without a default; we can't do that (the database needs something to populate existing rows).
  Please select a fix:
   1) Provide a one-off default now (will be set on all existing rows)
   2) Quit, and let me add a default in models.py
  Select an option: 1
  Please enter the default value now, as valid Python
  The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
  >>> -1
  You are trying to change the nullable field 'course_ptr' on box to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
  Please select a fix:
   1) Provide a one-off default now (will be set on all existing rows)
   2) Ignore for now, and let me handle existing rows with NULL myself (e.g. adding a RunPython or RunSQL operation in the new migration file before the AlterField operation)
   3) Quit, and let me add a default in models.py
  Select an option: 1
  Please enter the default value now, as valid Python
  The datetime and django.utils.timezone modules are available, so you can do e.g. timezone.now()
  >>> -1
  Migrations for 'box':
    0006_auto_20151114_1708.py:
      - Remove field created from box
      - Remove field generator from box
      - Remove field id from box
      - Remove field sequence from box
      - Remove field short_url from box
      - Remove field skills from box
      - Remove field subclass from course
      - Remove field subclassid from course
      - Remove field created from doublebox
      - Remove field generator from doublebox
      - Remove field id from doublebox
      - Remove field sequence from doublebox
      - Remove field short_url from doublebox
      - Remove field skills from doublebox
      - Remove field created from starbox
      - Remove field generator from starbox
      - Remove field id from starbox
      - Remove field sequence from starbox
      - Remove field short_url from starbox
      - Remove field skills from starbox
      - Alter field course_ptr to doublebox
      - Alter field course_ptr to starbox
      - Alter field course_ptr on box

You see management command detects that the child fields still haven't been deleted and that the default value for inserts of the children's parent reference still doesn't exist. Lastly the migration converts the OneToOne field back to a primary key.

Then migrate a final time:

$ python ./manage.py migrate box

Wrap Up

I hope this helps if you need this type of migration. It may look a little complicated at first, but all it amounts to is:

Step 1. Remove abstract inheritance and add temporary fields to the parent class for identifying each subclass's records in the parent table when migrating the data.

Step 2. Migrate the child data to the parent class with the subclass composite keys. Use new parent primary keys to migrate tables with foreign key that have moved to the parent class.

Step 3. Drop columns used in migration on the parent and child tables.

Let me know if you've found other/better solutions!

Django REST Registration with django-rest-auth and django-allauth

2014-10-26T03:02:00-05:00

I'm creating a mobile app for my agilitycourses website and I'm using django-rest-framework to provide a REST API for use by the client application. In order to provide authentication and registration I'm using django-allauth. Lastly I use django-rest-auth to provide REST resources for authentication and registration.

I implemented and tested django-rest-framework and then added in django-allauth. But when I went to integrate django-rest-auth POSTing to the /rest-auth/registration/ resource was generating a traceback:

Exception Type: TypeError at /rest-auth/registration/
Exception Value: add_message() argument must be an HttpRequest object, not 'Request';.

It turns out django-allauth's allauth.account.adapter.DefaultAccountAdapter uses Django's messaging middleware to give feedback to users when HTML templates are used. When rest-auth invokes the view it is is passing in a Request. I took a look at the rest-auth demo application and saw that it's settings.py file had django.contrib.messages disabled. Which keeps this traceback from happening.

Disabling messaging is a reasonable thing to do if the service is only handling REST data. For now I'd like to use the same service for both HTML and REST traffic. So I needed a way to disable messaging in django-allauth.

I found django-allauth allows configuring/replacing the account adapter so I subclassed DefaultAccountAdapter and stubbed out the add_message method. I put it in my "glue" application (called main) in a file called adapters.py:

class MessageFreeAdapter(DefaultAccountAdapter):
    """
    django-allauth's `allauth.account.adapter.DefaultAccountAdapter` uses Django's messaging middleware to give feedback to users. When using django-rest-auth for registration/login JSON-REST requests a traceback is generated when the `HTTPRequest` is passed into `django.contrib.messages.add_messages` when a `Request` is expected:

    Exception Type: TypeError at /rest-auth/registration/
    Exception Value: add_message() argument must be an HttpRequest object, not &#39;Request&#39;.

    If messaging cannot be disabled (it is used by other applications) using this subclass
    disables messaging for allauth/django-rest-auth.

    In settings.py add ACCOUNT_ADAPTER = 'main.adapters.MessageFreeAdapter'
    """
    def add_message(self, request, level, message_template,
                    message_context={}, extra_tags=''):
        pass

Then I set the ACCOUNT_ADAPTER variable in settings.py to use this new adapter:

ACCOUNT_ADAPTER = 'main.adapters.MessageFreeAdapter'

So now I can continue to use Django messaging and use django-allauth in the same Django project.

Integrated Coverage Analysis with Coveralls

2013-12-15T17:00:00-06:00

When I converted django-periodicals to use cookiecutter-djangopackage I was running coverage.py in my Travis-CI .travis.yml to report the coverage results to the command line log. The coverage results were interesting but didn't really alter my development practice much.

Over the years I've had differing options about coverage testing/analysis. Like any programming metric you can "cook the books" and pump up the metric while not actually improving the quality or maintainability of the code being measured. Minimally coverage testing can uncover unexercised corners of the code, especially error handling code. Nothing is worse than crashing an application with faulty error handling code.

coverage.py is trivial to run and it generates reports in various formats to make finding unexercised code simple. It is so easy there is no reason not to run it.

So my local coverage testing showed 7% of my code wasn't exercised - "good enough" right?

Then I discovered Coveralls. Coveralls integrates with Travis and collects coverage data for each buid and displays it on their website. It was trivial to setup:

Create a login on Coverall and enable your Travis-CI project.
Add coveralls to the project's test requirements.txt.
Then add after_success: coveralls to .travis.yml.

The next time the project is tested on Travis-CI the coverage results appear on coveralls.io. You can view the untested code in each file and Coveralls will track the increase/decrease of coverage in each file each time you check-in/test.

They also have badges showing the percent coverage that you can embed in your reStructuredText documentation on GitHub and ReadTheDocs. And that's the insidious part of integrated open source development...

The cookiecutter-djangoproject produces an application GitHub page that shows the Travis-CI test status and the PyPi package version. With the addition of the Coveralls badge it can now show the coverage percentage. That turned out to be a little bit of programming peer group pressure that made 93% coverge no longer "good enough"!

So a few minutes and a few tests later I had tests that did exercise the full code base including error handling. That gave me a happy little green badge that displayed "coverage 100%".

So not only is Coveralls fully integrated with the GitHub - CI - Open Source infrastructure, and dead simple to use, it got this developer to push into all the corners of his code before releasing. And I now have the peace of mind that Travis and Coveralls will be watching my back.

cookiecutter-djangopackage - Do the Right Thing

2013-12-09T23:02:00-06:00

In preparation for upgrading and enhancing Googility.com I've started breaking out reusable applications, upgrading them, and open sourcing the code on GitHub. I wanted to follow development best practices and create high quality applications including these features:

A full set of tests.
Near 100% code coverage.
Continuous Integration running on each check-in via Travis CI.
Documentation in Sphinx on Read the Docs (RTD).
Packaging/versioning compatible with PyPi.
Packaging building/testing on multiple python versions using virtualenvs via tox.

I had started researching each aspect and was getting a little frustrated that there wasn't a best practice for tying everthing together. Then I came across Audrey Roy's cookiecutter and Daniel Greenfeld's cookiecutter-djangopackage. cookiecutter is a utility to create project directory structures and files from the command line. cookiecutter-djangopackage is a template for creating a reusable Django application.

Yes there are other similar projects, and Django provides startproject and startapp commands that can take template arguments. But since I've never used Travis, RTD or tox I really wanted to leverage more experienced developers' knowledge so I could set them up in a "smart" way.

That's what I liked about cookiecutter-djangopackage it came with sane defaults that worked out of the box [1] and did smart stuff like wiring the version from the package's __init__.py in to the documentation and setup.py. The requirements.txt used by pip is wired in to tox and the README.rst is used in the setup.py and included in the Spinx docs.

And there are other integrations that make it easy to release a professional Django application. In fact that's my long winded point - it makes it hard to not do the right thing! I might have skipped using one or more of these support technologies, but cookiecutter-djangopackage made it easy for me to use them and focus on writing code, tests and documentation.

So that is what I want to stress: with cookiecutter-djangopackage you can create a packaged application, whose code is tested on multiple python/Django versions, tested for installation, installable via PyPi and nicely documented without much additional effort.

So take look at my nearly released django-periodicals application to see how it all works on GitHub, RTD and Travis.

[1]	I submitted a very minor pull request and found a related issue with application names that don't match their imported package name. (i.e. `django-periodicals` is the application name and `periodicals` is the package that is imported).

Moved To Pelican

2013-03-17T12:00:00-05:00

Way back in 2004, when the content in this blog was a category in the main AglityNerd blog, I used the Perl Blosxom application to serve the blog. When my dog agility readers complained/were confused by the sprinkling of tech postings I split off tech.agilitynerd in another Blosxom instance.

Once modern web hosted blogs came in to existence I wanted to see how they worked and moved the tech content to posterous. That was only OK, the source code formatting was painful but it did support email and web based content creation. I was reasonably content, not bugged enough to move again.

Then posterous announced it is closing on April 30 so I finally had to do something. This site really only needs a few features:

easy format in which to write content
source code formatting
Atom/RSS feeds
comments

I use Disqus for comments on my other sites so that meant I could go with a statically generated site. Also since I primarily code in Python I wanted a platform to which I could contribute. I came across Pelican and it fit the bill nicely.

I exported the posterous posts, imported them through a temporary wordpress.com site using their posterous importer and finally re-exported them in wordpress format. After a quick bug fix to the pelican wordpress importer I had reStructured Text files for each post. After a few days of editing the embeded source code in the content files and fixing long broken links, I had the blog running and not looking bad at all.

I wanted a Twitter Bootstrap based responsive layout and I found azizmb's pelican-bootstrap-responsive-them which had a very pleasing layout. I tweaked it to have category and tag feeds and some other enhancements to get what you see today. I will see if Aziz is interested in pulling any of my changes back.

So far I like using Pelican and I've made the content of the blog and my edits to the theme available on github in case anyone is interested.

Google Analytics for JQuery Mobile With Internal and AJAX Pages

2013-01-21T02:00:00-06:00

I added google analytics to my agilitycourses.com website and was having problems getting the mobile version of the site to log analytics data after the first page. Google found this very helpful blog post: Using Google Analytics with jQuery Mobile and it got me very close to my final solution.

Since that time jQuery Mobile has changed to pageinit events and jqmData for page selectors, so I had to change the first line below to match the recomendations for the 1.x version of jQuery Mobile that I'm using:

$("div:jqmData(role='page')").live('pageinit',
  function (event, ui) {
    var url = '';
    var hash;
    try {
      /* urls begin with locale but track urls independent of locale: strip off leading locale */
      url = location.pathname.replace(/^/[-a-zA-Z_]{2,5}//, '');
      hash = location.hash;
      if (hash) {
        url = '/' + hash.substr(1); /* strip # from hash */
      }
      _gaq.push(['_setAccount', 'UA-XXXXXXX-X'],
                ['_setDomainName', 'example.com'],
                ['_setSiteSpeedSampleRate', 100],
                ['_trackPageLoadTime'],
                ['_trackPageview', url]);
    } catch(err) {}
});

My mobile site uses both internal pages and AJAX loaded pages. So unlike the blog post's solution, I needed to use URLs with and without hashtags. Per the Google docs on _trackPageview I also added a leading slash to the hashed URL.

Another change for my site is urls are localized, they begin with the locale for the user's language via django-localeurl. So I strip the leading "/es/", "/ca-FR/" or "/en_GB/" from the URL before sending it to Google.

There are a couple other variables I push on the _gaq array. I log my mobile site (m.agilitycourses.com) as a subdomain of my desktop site so I followed the settings from the Google analytics and set the domain name. I also enabled client side page load and site speed settings.

So I put the code in the head of the original post in the head of my pages and the code snippet above in the foot of the pages.

Hope this is helpful for others.

Create Posterous Posts from Google Forms

2012-05-12T23:57:00-05:00

I've been organizing dog agility bloggers for Dog Agility Blog Action Days and wanted to automate updating our group's blog when each blogger posted their article on the event/action days. For past events I had the bloggers email me with the information and manually created each entry in the blog.

Since posterous supports posting via email I knew I could create a form on one of my web servers and have the page format the data so posterous could turn it into a post. Here are the details for formatting emails for posterous: how to get the most out of posting by email.

Instead I stumbled upon an article on using Google Forms to send emails via the Google infrastructure: Get Form Data from Google Docs in an Email Message [Video Tutorial].

So all I had to do was create a form for my bloggers to fill out and then have the script run by their submission format and send an email to posterous to post.

Here's my modified version of the form script:

function sendFormByEmail(e) {
  // create a draft post - posterous will send you an email
  // when the draft is created.  var email = "draft@YOURBLOGNAMEHERE.posterous.com";
  // e.namedValues is a dictionary where the keys
  // are the names of your form's questions. So you can pick
  // them out and combine them as you wish.
  //
  // My form contained five questions:
  // 1. 'Title of your blog as you want it to appear'
  // 2. 'The URL of your blog as you type it in your browser'
  // 3. 'Title of your article'
  // 4. 'The URL of your article copied from your browser'
  // 5. 'Short description/teaser about your article'
  var v = e.namedValues;
  var subject = v['Title of your article'].toString();
  subject += "((tag: attitude))"; // add tags as you wish
  var message = '<a href="' + v['The URL of your blog as you type it in your browser'].toString() + '">';
  message += v['Title of your blog as you want it to appear'].toString();
  message += '</a > wrote:';
  message += "nn";
  message += v['Short description/teaser about your article'].toString();
  message += '< a href="' + v['The URL of your article copied from your browser'].toString();
  message += '"> Read the full article < /a >nn';
  message += "n#end"; // so your email signature won't cause a problem
  // This is the MailApp service of Google Apps Script
  // that sends the email. You can also use GmailApp here.
  MailApp.sendEmail(email, subject, message);
  // Watch the following video for details   // http://youtu.be/z6klwUxRwQI
  // By Amit Agarwal - www.labnol.org
}

In the script you can see how I've accessed the form fields by the full name and woven them into an email with links embedded - just have to watch the use of double quotes within single quotes. Which produces blog entries similar to this one I created manually.

So now I can just send the link to the Google form to participants and they'll create drafts that I can tweak and/or just submit. It took longer to write this post than it took to modify and test the script!

My Favorite ORM and Python Anti-Patterns

2012-05-12T23:57:00-05:00

At work I was looking at improving the performance of one of our slower web pages. It can be rewarding to find a little piece of code that can be easily optimized. This time there were several functions that were adding 10+ sec to the page in worst case. It wasn't a problem for most clients, but when clients with who are related to many other clients hit the page they'd experience terrible performance. Here's pseudo code for the combination of anti-patterns that caused the problem:

# Projects have users and users are in different organizations
# (project can contain multiple organization's users)
activeOrganizationProjectUsers = [x for x in project.users
        if x.active and x.organization == organization]
if activeOrganizationProjectUsers:
    # do something *NOT* using activeOrganizationProjectUsers

There are two main problems with this code:

It ignores the fact the project, users, and organization are backed by an ORM
The list comprehension is being used to find all matching elements when only a single element is needed.

Ignoring the ORM

The code above wouldn't be too bad if these were just lists of objects in memory. But being objects that are instantiated by an ORM a number of database queries will be issued. In this particular case (w/o eager loading across user to the organization table) the following queries were executed:

Join project to user and get all users for the project's id
For each user load their organization (one by one) if the user is active

So in the case where there were hundreds of users on a project there were hundreds of queries executed and hundreds of User and Organization instances were instantiated. Depending on the size of the objects (and the ORM's behavior) it can take "real time" to fetch and instantiate all these large objects.

This code base has this kind of code sprinkled through out it. At one time during it's development the developers were encouraged to treat ORM backed objects as though they were Plain Old Python Objects (POPOs). The developer wouldn't necessarily see the performance degradation using small data sets either. This is one of the reasons why I like to tail the database log (or use django-debug-toolbar if I'm using Django) to see the queries go by.

Using List Comprehensions When a Single Value is Needed

To make this situation worse, the activeOrganizationProjectUsers list wasn't actually used. This is a combination of a Python anti-pattern and the ORM anti-pattern. What was required was to determine if a single active organization user existed.

I believe the original developer(s) used the list comprehension solution in a combination of ignorance and syntactic sugar. They didn't want to write a new function to do the query and put it in the User class so they used the existing class's API. The syntactic sugar was using the list comprehension to get more values than the one that was needed. If this wasn't a (potentially) expensive ORM backed operation the original code could have been:

activeOrganizationProjectUsers = False
for x in project.users:
    if x.active and x.organization == organization:
        activeOrganizationProjectUsers = True
        break
if activeOrganizationProjectUsers:
    # do something

But this solution could still query all possible user/organization combinations. The other question would be: which set is larger the organization users or the project users? It is likely looping over the organization's users looking for active ones would be more efficient anyway.

Remember the Underlying Representation

When performance matters remembering the objects are ORM backed is important. So in this case a single query was all that was required (SqlObject pseudo syntax):

activeOrganizationProjectUsers = Users.selectBy(project=project,
                                                active=True,
                                                organization=organization).count() > 0

If abstracting out the ORM's methods is important this new function could be added to the appropriate class as a method. In my case making a change to use a query resulted in cutting the page load time by two orders of magnitude.

Reducing the Cost of Client Side Analytics

2012-01-12T22:10:00-06:00

I read Andy McKay's blog post on timing user experience on the Mozilla Webdev blog the other day and it reminded me of an idea I was thinking about for measuring client side timings at work. I had been toying with the idea of rolling our own library to capture JavaScript rendering time for our JS heavy pages (grids of hundreds of lines of data).

Andy's post mentions the boomerang JavaScript library and when I was reading it's docs they pointed out potential for abuse/load on the URL used to report the timings. For each instrumented page boomerang can hit the "beacon" URL to report the statistics it collects. So in the worst case you could double your page hits - although for specific pages/samples recording a few statistics shouldn't be too costly for low volume sites.

One solution is to only sample the pages/users of interest; selecting the sample could occur on the server and/or client. But another solution would be to collect statistics across multiple pages and periodically send batches of analytics to the beacon URL.

I've been playing with mobile web development for agilitycourses.com lately and will soon let users store the courses they create in localStorage on their browser. That got me thinking that sessionStorage could be used to store analytics across pages and then periodically send the stats to the server. This would reduce the number of hits on the beacon, allowing deployment to a larger sample of clients. It also gets flushed once the session is closed and (if kept small) doesn't prompt the user to approve storing the data.

A lot of modern browsers support session storage and for my purposes only ones with support would be relevant - due to our browser support policy at work.

The other problem the boomerang docs discuss isabuse of the beacon (accidental or malicious). A solution would be to piggyback reporting of analytics into application form post payloads. This is trickier to implement and it suffers from coupling analytic reporting into the application itself.

To try to solve it some what generally... The client side JS library could add a hidden field to any/some/specific forms into which it writes the analytics data collected thus far. If it hooked the form submit callback it could know if the form was successfully submitted and clear the session storage.

Server side middleware could detect the hidden analytics field in the form and extract/store the data. It could also remove the field before passing the request data along to the app server.

All in all a fair amount of twiddling to keep from exposing a recording URL to the outside world.

Of course if an authenticated session was being used then abusers would have to have a valid session to post to the beacon URL.

I don't know if I will have time to play with the sessionStorage idea but I think it might be a worthwhile extension to boomerang or other client side analytics capture libraries.

YellowGrass - Web Based Issue Tracking

2011-06-07T12:24:00-05:00

I was doing some reading on mobl and saw that they are using a free web based service called YellowGrass for issue tracking. It has some nice features and seems easy to use. Everything is tag based. I think I'll try to use it for tracking enhancements to agilitycourses.com …

Django Shrink The Web django-stw 0.2.0 Released

2011-04-23T13:32:00-05:00

Shrink The Web has announced a new API for free users using their new preview verification feature. This change required changes to my django-stw package.

The changes (lifted from the CHANGELOG.txt):

Changes to the shrinkthewebimage template tag:

The shrinkthewebimage template tag is NOT backward compatible with version 0.0.1. The alt argument is no longer accepted.
The shrinkthewebimage template tag is now intended for use by free accounts, it adds the required preview feature. It can also be used by PRO account users wanting the preview functionality.
The shrinkthewebimage template tag now accepts PRO key-value arguments in the same manner as the stwimage tag. This functionality is shown in theexample template but may not yet be fully implemented by the STW web service.

Changes to the stwimage template tag:

The stwimage can now only be used for PRO features.

Common changes:

Template tags now throw exceptions in their constructors instead of in the render function so configuration errors are visible during development.
django-stw defines a key 'lang' for the SHRINK_THE_WEB dictionary that can be passed along as a default to the preview tag. Alternately a 'lang' keyword can be supplied in each template tag invocation. django-stw defaults it to 'en'. This functionality is not yet implemented by the STW web service.

The v 0.2.0 package is available on PyPi, as a source download on github, or via git clone.

Python dict.get's Default Value is Always Evaluated

2011-04-05T17:16:00-05:00

This is a gotcha I ran across in some production code that is obvious in retrospect. I was profiling the code to find places where we were calling an_expensive_database_function and came across code like this:

def doit(*args, **kwargs):
     value = kwargs.get('key', an_expensive_database_function())

The original author probably assumed that if 'key' was present in the kwargs dictionary an_expensive_database_function wouldn't be called; that it would be short circuited in the same manner as Boolean expressions. But since get is a function the arguments are always evaluated on the way into the function. So in this case even if the value of an_expensive_database_function was already present in the kwargs dictionary the database function would be called again.

Here is a "look before you leap" solution:

def doit(*args, **kwargs):
    value = kwargs.get('key')
    if value is None:
         # assuming default value None isn't a valid value
         value = an_expensive_database_function()``

Here is the "easier to ask forgiveness than permission" solution:

def doit(*args, **kwargs):
    try:
        value = kwargs['key']
    except KeyError:
        value = an_expensive_database_function()``

Blosxom Plugin for Generating Facebook Comment xids

2010-11-23T05:00:00-06:00

I've been using Blosxom to power my dog agility blog for over 6 years. In the past year or so I've enabled Facebook comments in addition to my site's own comment plugin. I ran into a problem using Facebook's comments: if the user enters a comment on a page and the URL has any additional URL parameters then the comment is only associated with the page when accessed with those parameters, others hitting the page w/o parameters won't see the comments.

This behavior is documented by Facebook when the xid attribute isn't set in fb:comments HTML element. I didn't think I'd encounter this situation since my blog post URLs don't contain any parameters. However, when people link to one of my articles within Facebook, Facebook appends various parameters to the base URL.

The solution is to specify an xid attribute in the fb:comments element containing the URL encoded URL of the page (Facebook's default xid). This causes existing comments to show up and causes comments created when the page is loaded with URL parameters to use the same encoded URL.

So I created a simple Blosxom plugin to perform the encoding so the encoded URL can be placed in the story.html template:

# Blosxom Plugin: urlencode -*- perl -*-
# Author: Steve Schwarz <http://agilitynerd.com/>
# 2010-NOV-22    0.1 initial version.package urlencode;
# puts the urlencoded string of the URL for this page
# into $urlencode::url without any params# use this in fb.comments xid to give the same xid
# even when query params are provided
# -------------------
use CGI qw/:standard/;
use URI::Escape;

sub start {
    return 1;
}
$url = '';

sub story {
    my ($pkg, $path, $filename, $story_ref, $title_ref, $body_ref) = @_;
    $urlencode::url = uri_escape("$blosxom::url/${blosxom::path_info}");
    return 1;
}
1;

Then it is used in the story template:

<fb:comments width="550" numposts="5" xid="$urlencode::url"></fb:comments>

Now my readers won't have to worry that their comments won't show up.

Obtain Short URLs and QR-Codes for Django Apps

2010-10-22T04:00:00-05:00

Lately I've been interested in improving the interaction of my agilitycourses website for mobile users. One such improvement is to add QR Codes (aka 2D barcodes) representing the page URLs to the printed representations of pages served as PDFs.

I found that developers have reverse engineered the "api" of the goo.gl URL shortening web site. In my brief testing it is very fast. What makes that service extra useful is by adding ".qr" to a shortened URL it returns a PNG image of the QR Code for the shortened URL. That made it perfect for providing both short text and QR Code URL representations for my printed documents.

I threw together a few functions and put them in a module to make it easy to shorten a long URL, obtain the QR Code PNG and store it using Django's Storage functionality:

import osimport urllib
from django.utils import simplejson

def googl_shorten_url(long_url):
    """
    Returns goo.gl shortened url for the provided long_url.
    Code taken from: http://djangosnippets.org/snippets/2220/
    Parameters:
    - `long_url`: the url to supply to goo.gl to be shortened.
    """
    params = urllib.urlencode({'security_token': None, 'url': long_url})
    f = urllib.urlopen('http://goo.gl/api/shorten', params)
    return simplejson.loads(f.read())['short_url']

def googl_qrcode(googl_url):
    """
    Return file containing qr code image file for the given goo.gl url.
    Parameters:
    - `googl_url`: url from which to obtain the qr code.
    """
    return urllib.urlopen(googl_url + '.qr')

def get_url_qr_code_image(long_url, storage, storage_image_file_path=''):
    """
    Return goo.gl shortened url and storage name of qr code corresponding to
    the shortened url for the supplied full url. Contacts goo.gl to shorten
    the supplied long url then downloads and stores the qr code image file
    in the storage instance using the file path and the shortened url name
    as the storage name.
    Parameters:
    - `long_url`: the url to shorten.
    - `storage': a Django storage instance into which to store the qr code
    image.
    - `storage_image_file_path`: file system path to prepend to shortened
    url. This path must exist prior to calling this function.
    """
    try:
        googl_url = googl_shorten_url(long_url)
        qr_file_name = googl_url.split('/')[-1] + '.qr'
        qr_code_name = os.path.join(storage_image_file_path, qr_file_name)
        if not storage.exists(qr_code_name):
            qr_buffer = storage.open(qr_code_name, 'wb')
            qr_buffer.write(googl_qrcode(googl_url).read())
            qr_buffer.close()
    except:
        googl_url = None
        qr_code_name = None
    return googl_url, qr_code_name

Yes, it has a nasty bare try/except. For my uses this is optional functionality so I never want a failure to stop the main functionality of the views that use it. Add exception handling appropriate for your needs.

The main entry point is get_url_qr_code_image(). Here is an example of its use (assuming you save the code in googl.py):

>>> import googl
>>> from django.core.files.storage import default_storage
>>> short_url, qr_code_storage_name = googl.get_url_qr_code_image('http://google.com', default_storage)>>> short_urlu'http://goo.gl/mR2d'
>>> qr_code_storage_nameu'mR2d.qr'
>>> default_storage.path(qr_code_storage_name)u'/home/dev/agilitycourses/static/mR2d.qr'
>>> default_storage.url(qr_code_storage_name)u'mR2d.qr'
>>>

Hope you find this useful.

Adding pyrsvg to a virtualenv created with --no-site-packages

2010-10-12T03:02:00-05:00

I set up my development and deployment environments on Ubuntu with virtualenv with the --no-site-packages option to isolate them from packages in the system installation. My application uses pyrsvg and it is installed by default as a system package. Consequently I had to link the shared libraries it installs (w/in gtk) into my virtualenv.

Here are the links I created (workon and cdsitepackages are virtualenvwrapper shell aliases):

$ workon project
$ cdsitepackages
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/rsvg.so .
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/gobject .
$ ln -s /var/lib/python-support/python2.6/gtk-2.0/glib .

Mobile Web Site Redirects in Django

2010-10-05T14:34:00-05:00

For the mobile version of agilitycourses.com I wanted to follow the approach Google appears to be using on some of its sites:

If the user views agilitycourses.com from a desktop browser they should see the standard/desktop version of the site.
If the user views agilitycourses.com from a mobile browser they should be redirected to a mobile domain (m.agilitycourses.com).
The mobile version of the website includes a link to the standard version.
If the mobile user chooses the standard website they should "stick" on that site and not be redirected to the mobile site.

I wanted to run two different websites but share templates and have the templates and css change for the mobile site. That meant that I'd need to set a variable(s) in the request to use to generate the appropriate HTML. So I found the simplest mobile device detector minidetector and initially used that. I later found Chris Drackett's fork has a number of useful enhancements and switched to it.

But minidetector didn't provide the ability to redirect to another site. I found Scott Newman's article on using multiple templates which had a section on performing the redirect and storing the user's selection in the session. So I forked Chris' minidetector and modified it to include the redirect and session storage. At the same time I decided to store all the minidetector variables into the session and add them, via middleware, to the request so the raw request wouldn't have to be parsed each time. My fork is available here with details on the new configuration options.

I'm using two domains so I can track analytics for the mobile and non-mobile sites separately and allow users to bookmark the desired site's pages. I use Google Analytics (via django-google-analytics) and Awstats for analytics.

Since I'm using two separate domain and sharing everything else I'm using a setup similar to the one described by Dustin Davis. I have a settings.py file and a mobile_settings.py that only overrides the features I need:

from settings import *
SITE_ID = 2
CACHE_MIDDLEWARE_KEY_PREFIX = "m.ac-"

I use a different memcached key prefix so the cached pages for the mobile site don't clash with those for the desktop site.

I setup m.agilitycourses on my server using the same Gunicorn setup I used for agilitycourses.com with the only changes being specifying the --bind address/port and the name of the mobile settings file:

#!/bin/sh
GUNICORN=/home/user/virtualenvs/myapp/bin/gunicorn_django
ROOT=/home/user/source/myapp
PID=/var/run/myapp.pid
if [ -f $PID ]
    then rm $PID fi
cd $ROOT
exec $GUNICORN --bind 127.0.0.1:8001 -c $ROOT/gunicorn.conf.py --pid=$PID $ROOT/mobile_settings.py

If my templates/content start to diverge more significantly between the mobile and desktop sites I may set the TEMPLATE_DIRS differently in the mobile_settings file. Or I can move to Dustin's approach and create a new application containing the urls.py and views.py specific to my mobile deployment. I would think diverging further would call for a refactoring of the common functionality to its own application which could be imported into separate code branches for each domain.

Debug Site for Website Redirects By Referer String

2010-09-29T18:42:00-05:00

I'm adding an "m" subdomain to agilitycourses.com to provide a better mobile browsing experience. I'm using the referrer string in Django middleware (currently using minidetector) to detect whether the client is mobile and redirect them to the mobile site. Since it is likely that some folks will/won't get appropriately redirected I was looking for an easy way for them to tell me when they were incorrectly redirected. I'd need to know their referer string.

A little googling turned up a nice one purpose website: www.whatismyreferrer.com/

Configuring Runit for Gunicorn and Django Installed in a Virtualenv on Ubuntu

2010-09-08T03:08:00-05:00

I couldn't find any documentation that covered all the pieces for configuring my latest Django site so I hope this helps someone else out.

I had used mod_wsgi under Apache for my other Django sites. But now I'm using different python versions for the sites (until if/when I update the older sites) and I wasn't getting the correct versions of some python libraries (even though virtualenv apeared to be putting the appropriate python packages at the start of the sys.path). So I decided to configure Apache to ProxyPass to Gunicorn so I could run my Django app in its virtualenv without it getting any other python modules.

Installing Gunicorn

I installed Gunicorn into the virtualenv for my application, which simplifies using gunicorn from the command line. Assuming /home/user/virtualenvs/myapp is the location of the virtualenv:

$ source /home/user/virtualenvs/myapp/bin/activate
$ pip install gunicorn

# or
$ easy_install gunicorn

This copies gunicorn_django to the /home/user/virtualenvs/myapp/bin directory. Test gunicorn with your app, assuming your Django app is located at /home/user/source/myapp, as follows:

$ source /home/user/virtualenvs/myapp/bin/activate
(myapp)$ cd /home/user/source/myapp
(myapp)$ gunicorn_django

Gunicorn starts myapp using the settings.py file in the current directory on 127.0.0.1:8000. Ctrl-C to stop the process.

Installing Runit on Ubuntu

There are two runit packages. You want the one that only runs services you add to it:

$ sudo apt-get install runit
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  runit-run socklog-run
The following NEW packages will be installed:
  runit0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/113kB of archives.After this operation, 537kB of additional disk space will be used.
Selecting previously deselected package runit.
(Reading database ... 209845 files and directories currently installed.)
Unpacking runit (from .../runit_2.0.0-1ubuntu2_i386.deb) ...
Processing triggers for man-db ...
Setting up runit (2.0.0-1ubuntu2) ...
runsvdir (start) waiting
runsvdir (start) startingrunsvdir (start) pre-start
runsvdir (start) spawned, process 9575
runsvdir (start) post-start, (main) process 9575
runsvdir (start) running, process 9575

You'll want to create a directory for the application and a run script in /etc/service:

$ sudo mkdir /etc/service/myapp
$ sudo vi /etc/service/myapp/run
# enter the run script I'll show below
$ sudo chmod +x /etc/service/myapp/run
# stop runit from trying to run gunicorn until we are ready
$ sudo sv stop myapp
ok: down: myapp: 0s, normally up

The example run script checked into Gunicorn had some syntax errors and wasn't quite what I wanted. Here's my version:

#!/bin/sh
GUNICORN=/home/user/virtualenvs/myapp/bin/gunicorn_django
ROOT=/home/user/source/myapp
PID=/var/run/myapp.pid

if [ -f $PID ]
    then rm $PID
fi

cd $ROOT
exec $GUNICORN -c $ROOT/gunicorn.conf.py --pid=$PID

You can create a configuration file for gunicorn to use or just create an empty file for now:

$ touch /home/user/source/myapp/gunicorn.conf.py

If you have multiple appserver you'll need to run gunicorn on different ports, you can put the configuration in the gunicorn.conf.py file:

bind = "127.0.0.1:8111"

Putting it Together

Now you can test that the run script works when run as root:

$ sudo /etc/service/myapp/run

Gunicorn should start and start the appserver. If it fails you can debug the script via:

$ sudo bash -x /etc/service/myapp/run

Tell runit to start and keep gunicorn running:

$ sudo sv start myapp
ok: run: myapp: (pid 7540) 0s
$ sudo sv status myapp
run: myapp: (pid 7540) 1s

MOTD

2010-08-31T20:21:00-05:00

I've always preferred maintainable code over clever code:

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Brian W. Kernighan

Kernighan had it right all those years ago.

Confidently Refactoring Django URLs, Views, and Templates

2010-08-22T05:12:00-05:00

Googility.com is my first Django website and under the covers the oldest code looked like it. I had originally written it with the sole intent of allowing people to enter dog agility businesses and websites into a database that I could use to create a Dog Agility Google Custom Search Engine. The primary mistake I made was making the "project" (in Django speak) effectively equivalent to the primary application. In other words I didn't divide the major features of the site into standalone applications (which would allow them to be more easily reused, extended and tested).

As I continued to work on it I learned more about organizing Django projects. When I added the periodical search to the website I created it as a standalone application. I recently split out my django-shrinktheweb application from the main code base.

The Custom Search Engine (CSE) functionality is a worthwhile application that I'm planning on releasing as its own reusable application. I had already created an application directory called "cse" into which I had placed my models, views, urls, and tests specific to the CSE functionality. But I wanted to make the following changes:

Move CSE templates into a cse template subdirectory
Name the templates to match the views that use them
Name the urls in the urls.py prefixed with the application name ("cse_")
Covert all reverse() calls in the views and url template tags to use the named urls

Those are enough changes that I was concerned that I might miss something that would fail either in the view code or in rendering of the templates.

The Django test client makes it easy to test the forward and reverse url matching, calling the view and rendering the template. It is kind of a coarse grained test but the changes I was making were perfect for this tool. Given a urls.py:

urlpatterns = patterns('cse.views',
                    url(r'^site/view/(?P<id>d+)/$', 'view', name='cse_view'),)``

and a view:

def view(request, id, template='cse/view.html'):
    """Display an end user read only view of the site information"""
    site = get_object_or_404(Annotation, pk=id)
    return render_to_response(template,
                          {'site': site,
                           'labels': get_labels_for(site, cap=None),
                           },
                          context_instance=RequestContext(request))``

I then wrote a test class to create the required test instances and tests for each url to verify that the url can be found by name (via reverse()), the url maps to a view, the view invokes the desired template(s), and the {% url %} calls within the template can all be resolved:

from django.test import TestCase
from django.test.client import Client
from django.conf import settings
from django.core.urlresolvers import reverse
from cse.models import Label, Annotation

class ViewsTestCase(TestCase):
    def setUp(self):
        self.client = Client()
        self.ROOT_URLCONF = settings.ROOT_URLCONF
        # can provide a custom urls.py for testing so the tests can be run when
        # the application is incorporated into another project
        # settings.ROOT_URLCONF = 'cse.tests.cse_test_urls'
        # override the template context processors if there are special ones in place
        # that either you want to test or want to avoid
        self.TEMPLATE_CONTEXT_PROCESSORS = settings.TEMPLATE_CONTEXT_PROCESSORS
        settings.TEMPLATE_CONTEXT_PROCESSORS = ()
        # Create some instances on which we can invoke views
        self.label = Label(name='name', description='description')
        self.label.save()
        self.annotation = Annotation(comment='Site Name', original_url='http://example.com/')
        self.annotation.save()
        self.annotation.labels.add(self.label)
        self.annotation.save()

    def tearDown(self):
        # put settings back so the next tests aren't effected
        settings.ROOT_URLCONF = self.ROOT_URLCONF
        settings.TEMPLATE_CONTEXT_PROCESSORS = self.TEMPLATE_CONTEXT_PROCESSORS

    def test_view(self):
        response = self.client.get(reverse('cse_view', kwargs={"id":self.annotation.id}))
        self.assertEquals(200, response.status_code)
        self.assertTemplateUsed(response, 'cse/view.html')

The normal unittest asserts are available in the tests. I'm using one of the special asserts provided by the Django test Client to verify that the template I expected was used. All the templates used (due to template inheritance) are collected by the client and can also be verified.

I used these tests in a TDD-ish manner, I wrote the test for a view, ran the tests and kept resolving errors in the templates as I made the changes in my bullet list. It made a tedious job simple and gave me good confidence that I'd found all the renamed urls, views, and templates.

Haystack Search Result Ordering and Pre-Rendering Results

2010-08-10T03:41:00-05:00

I use Haystack and the Python Whoosh project to provide search over ~3400 articles in my Googility.com database. I had originally implemented the search in the "simplest way that works". I was making some other enhancement to Googility and noticed the search result page had two undesirable behaviors:

The ordering of results was basically random for all matching articles. For the domain of magazine article search having a bias toward the most recent publications would be more desirable.
Looking at the django-debug-toolbar output each element in the search results was hitting the database twice (once for the Article instance and again for its corresponding Periodical). So a single result page was making as many as 60 database selects.

Haystack provides mechanisms to help with both of these issues.

Imposing an Order on the SearchQuerySet

Haystack models search using an API based on Django's QuerySet. The only thing to remember is it performs its queries over the Haystack SearchIndex subclass(es) you create instead of over the Django ORM. So you define a SearchIndex subclass that contains the data from the application's model overwhich you'd like to search. You can also define additional fields that can be used to modify the results of the query. Here is my magazine Article search index:

from haystack.sites import site
from haystack import indexes
from periodicals.models import Article

class ArticleIndex(indexes.SearchIndex):
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='issue__pub_date')

site.register(Article, ArticleIndex)``

The text field contains the "document" over which the search engine (Whoosh) will actually perform the search. I'm using the template feature that allows me to use Django templates to format the data presented to the search engine.

I added the pub_date field to the index to allow the matching search results to be ordered by the pub_date field. The 'issue__pub_date' syntax mirrors the Django QuerySet syntax and means extract the "pub_date" attribute of the Article's "issue" attribute (it joins Article to Publication and get's the Publication's published date).

Then the urls.py is modified to change the SearchQuerySet passed to the default haystacksearch view to order by the ArticleIndex's pub_date attribute:

from haystack.views import SearchView
from haystack.query import SearchQuerySet

# query results with most recent publication date first
sqs = SearchQuerySet().order_by('-pub_date')
urlpatterns = patterns('',
                       url(r'^search/',
                           SearchView(
                               load_all=False,
                               searchqueryset=sqs,
                               ),
                           name='haystack_search',
                           ),<snip>``

Pre-Rendering Result HTML

Since I have only a few thousand records I decided to follow the Haystack Best Practices for Not Hitting the Database. This solution trades space in the Whoosh index files by generating the HTML that will be displayed when each article matches along with the data used by Whoosh to match articles to search keywords. The changes were pretty simple. In the ArticleIndex:

from haystack.sites import site
from haystack import indexes
from periodicals.models import Article

class ArticleIndex(indexes.SearchIndex):
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='issue__pub_date')
    # pregenerate the search result HTML for an Article
    # this avoids any database hits when results are processed
    # at the cost of storing all the data in the Haystack index
    result_text = indexes.CharField(indexed=False, use_template=True)site.register(Article, ArticleIndex)``

The use_template keyword requires you to create a Django template file that is used during index creation to build the HTML that will be displayed. The only peculiarity I found was figuring out where the template should live. On my system it was at templates/search/indexes/periodicals/article_result_text.txt. I understand the periodicals/article_result_text part but I haven't looked into where the search/indexes is generated from. I imagine a reverse() to find the url for the view and "indexes" is appended to that...

The final change is the template used to display the search results. In order to not hit the database the object list generated by the haystack SearchView is placed into the context used by the template and only the result_text attribute should be accessed:

{% if page.object_list %}
<div class="search-results-title">Results <b>{{page.start_index}}</b>  - <b>{{page.end_index}}</b> for <b>{{query}}</b></div>
<div class="search-results-list">
{% for result in page.object_list %}
  {{result.result_text|safe}}
{% endfor %}
<div class="pagination">
    <span class="step-links">
      {% if page.has_previous %}
          previous
      {% endif %}
      <span class="current">
          Page {{ page.number }} of {{ page.paginator.num_pages }}
      </span>
      {% if page.has_next %}
          next
      {% endif %}
    </span>
  </div></div>{% else %}<h2>No matching articles found.</h2>
{% endif %}

The actual result is placed in the template via {{result.result_text|safe}} the safe filter is required since the HTML doesn't need to be escaped again - it was escaped by Django when it was placed into the SearchIndex.

So now my search results are in reverse chronological order and they render using only 3 database queries and at least 10x faster than before.

Improving Google Ads and Google Search Descriptions

2010-08-03T01:46:00-05:00

I was looking at the google search results for my Googility web site and noticed that the descriptions shown underneath the title often contained text from my navigation links instead of content from the body of the page:

I did some searching and found the Google Webmaster blog post about description meta tags. Since almost all of the pages on Googility are generated by fewer than a dozen Django templates I edited the templates and inserted meta tags and filled the description in with data from each database entry. This avoids boilerplate information that would be ignored by Google and improves the descriptions shown to Google searchers. Some of my pages have already been reindexed:

Yahoo and some other search sites use a class robots-nocontent on any page elements it should ignore for it's index, Unfortunately, Google doesn't follow this standard. So I might end up making that edit to the templates also. Looking at my site's log files it appears the Yahoo spider is hitting my site more frequently than Google's and the Yahoo index is more up to date. Looking at my analytics reports though Google refers far more readers to my site than Yahoo...

I also noticed that the ads served on pages containing mostly links appeared to be using words in my navigation or other boilerplate instead of the few lines of valuable content. More searching to the rescue and I found this Google Adsense article on section targeting. Once again editing the dozen or so templates I used were easy to edit to add in these HTML comment tags. Checking back a couple days later showed improvements in the ads being generated for those pages. I keep an eye on my Adsense click rate and see if there is any increase in ad clicks.

So a couple simple edits made noticeable improvements not bad for a couple hours investigation and implementation.

Firediff: track and save CSS edits in Firebug

2010-07-25T16:36:00-05:00

When I'm making fiddly changes to a web page I like to tweak the CSS using the Firefox Firebug plugin. It has the advantage of letting you try changes quickly and see the effect. The downside has always been that you had to then change the source CSS file to include your changes. Which increases the risk that you forget to include a change.

I was thinking about that recently when I changed the color scheme on my AgilityNerd blog and went searching for a way to at least identify the changes. Turns out there is a nice plugin that provides exactly that functionality. The Firediff plugin not only tracks changes to the CSS it also allows you to save those changes for overwriting or diff'ing into you CSS file(s).

Like other firebug related extensions you need to enable this feature per page by clicking on the arrow next to the new Changes tab.

Here is a screenshot of the plugin in action:

You can see I've changed the line-height attribute - it shows the previous and final values. A right mouse menu lets you save those changes to a file or revert them.

Another feature that can be configured in the Changes menu is whether to track changes made by other JavaScript to the page. This is an interesting feature if you were wondering how a JQuery or other JavaScript modified the page for a given effect.

This is a great plugin that I'll continue to use regularly.

Initial Release of django-stw

2010-07-11T15:47:00-05:00

I have been using the free website thumbnail service from Shrink The Web on my dog agility search website Googility since I launched it. It is quick and easy to use and it adds a lot to the look of the pages.

I had created a simple Django template tag for inserting the little snippet of HTML needed by their service.

Recently they asked me to add support for their advanced features to my template tag. I used this opportunity to convert my templatetag to a Django application. This mostly makes it a lot easier to install but it also let me to bundle tests and an example template with the template tag.

I kept the existing shrinkthewebimage template tag and added a new tag called stwimage to enable the new features.

I'm hosting the example page included in the package here so you can see how the template tags work.

I've hosted the project source on github and uploaded the initial release to the CheeseShop for easy installation.

Embedding JSON Within Generated HTML

2010-07-08T21:03:00-05:00

Ran into an interesting problem at work this past week that had a simple and pleasing resolution. We have an in house developed JavaScript grid on some of our pages and when users entered some text strings we'd generate invalid JSON payloads that would give the user an error page. If they entered strings that looked like an HTML Entity i.e. &#13 which (with the addition of a trailing ; ) is a non-visible HTML character (carriage return) the text wasn't displayed in the widget. To further complicate things some of the content displayed in the grid is HTML which is inserted into the grid as is and can contain escaped HTML characters.

The grid gets its content as a JSON payload from within a hidden div in the HTML which is generated via a template mechanism. Heres a portion of the template where <%= and %> stringifying of the value of the Python variable(s)/code they surround:

<div style="display:none;" id="grid-init-args-<%= count %>">
  <textarea>
  <!-- this is the JSON payload loaded via the grid JavaScript -->
  <%= [ columnsIndex, indexColumns, columns, rowBuffer, footerRows, formulas] %>
  </textarea>
</div>

This approach has a number of problems:

By using the template mechanism to create the JSON payload this template was relying on the similarity of the string representation of Python objects to JSON. After some testing I found the following scenarios: If a string contained a single quote character the string representation was a double quoted string around the text and the single quote; a valid JSON string. If the string contained a double quote character the string representation was a single quoted string around the text and the double quote; an invalid JSON string. If the string contained both a single and a double quote the string representation would be a single quoted string containing a slash escaped single quote and the double quote; an invalid JSON string. Depending on the browser (of course) the JSON string would fail to parse correctly when the double quote was encountered within the single quoted string.
The JSON payload had to be HTML encoded (converting <, >, ", and &) since it was parsed by the browser as HTML.
The HTML encoding would encode or double encode HTML to be inserted directly into the grid's DOM.

The variation in single/double quoting was an easy fix, I changed to simplejson.dumps() which correctly double quotes key/values in dicts and escapes embedded double quotes (single quotes don't need to be escaped). I didn't time it but with the C extension it may be faster than the template engine for our larger datasets.

I played around with (not) encoding various portions of the payload and then it hit me that I should change the grid to get its payload from a non HTML element so that only HTML destined for insertion into the DOM would be HTML encoded (which is as you'd expect for normal HTML handling). I started changing the payload to be stored in JavaScript generated in the template but didn't like the impact the change would have on all the existing templates. So I started Googling and found Ben Nadel's blog post on using script tags as data containers.

So here's my solution:

<div style="display:none;" id="grid-init-args-<%= count %>">
<script type="application/json">
<%= simplejson.dumps([ columnsIndex, indexColumns, columns, rowBuffer, footerRows, formulas]) %>
</script>
</div>

There were two changes:

Used simplejson.dumps to correctly double quote and escape double quotes within the variables in the payload.
Change the textarea to a script element.

By converting to a script tag within the hidden div the HTML parser no longer parsed the content of the JSON payload. so the JSON payload only needed to HTML encode HTML elements that were being inserted into the DOM created by the grid.

This change also meant I was able to delete the unnecessary HTML encoding of non-HTML JSON payload data. Got to love solutions that involve deleting code.

Ultimately, we'll convert to loading the JSON payload as a separate AJAX request from the page to the server, but for now this simplifies the markup and handles all types of user input and HTML encoded characters correctly.

Django Shrink The Web Template Tag Updated

2010-07-01T23:00:00-05:00

I recently updated my Django template tag for simplifying the use of Shrink The Web images. They recently announced a CDN based distribution of images and they took the opportunity to modify their API.

The updated template tag is on django snippets.

The STW folks have asked be to extend my template tag with support for their PRO features. With luck I'll make that available sometime this weekend.

Multiple YouTube Videos per page using VideoLightBox

2010-06-04T22:58:00-05:00

I decided to stop displaying the default YouTube video players within posts on my AgilityNerd blog and I started looking for a light boxed player. Their were two main reasons. The smallest video playback window provided by YouTube for HD videos is too wide for my two column layout and now that I'm posting more videos the load time of the page is delayed by the communication with all the off site webservers; serving the YouTube static image of the video will be much faster/lighter weight.

I looked around and really liked the lightbox containing the default YouTube player provided by VideoLightBox and started playing around with their demo. VideoLightBox (VLB) has an interesting approach. You download an application (PC or Mac), configure how you want your video(s) to look and it generates a directory of files on your local disk (or uploads files to your website via FTP) along with an index.html file from which you copy the code to put in the <head> and <body> of your web page. For YouTube it also downloads a static image for each selected video which is used as the image link within the HTML page. Straight forward and works well.

For my purposes there was a problem with their approach, its locates the image used to launch the light box using an element id. This assumes a single video or gallery of videos per web page. On my blog's main page or the category pages there will be multiple videos (possibly multiple videos within a single post). I figured a little bit of CSS and JQuery hacking would solve the problem and it did.

I decided to modify their HTML/CSS/JS to use a CSS class instead of an element id to allow for multiple videos per page. At first I just modified the generated files. Then I saw that VLB has template files in their installation. So I started modifying the templates to output the new code. Two hours later I bailed. Using procmon it looks like the client app reads the template files but then doesn't actually use the files to generate the output files(?). I was only able to modify one of the three template files that needed to change and have it effect the generated files.

I'm going to provide my edits to the VLB developers in case they are interested.

So the solution is to edit one of the template files and then edit two of the generated files; not ideal but once you put the generated files on your webserver you'll probably not touch them unless you are changing CSS styles. The modifications aren't hard but you need to be careful and typos will definitely break things. You should backup the VLB directory before your start or be prepared to uninstall and reinstall from their installation program.

Navigate to the VideoLightBox directory (i.e. C:Program Files (x86)VideoLightBox)
Change the permissions on the templates subdirectory to give your user full access to overwrite the files
For each directory in the templates subdirectory open the videolightbox.js file in an programming editor (a keyboard macro makes this trivial):
1. Globally replace $("#videogallery a[rel]") with $(".videogallery a[rel]").each(function(idx){$(this)
2. Go to the end of the line and add });
3. Save the file

Then generate the output files using the VLB executable for one or more videos, saving the results to your local file system

Navigate to the output directory
Open the index.html file in an programming editor
1. Globally replace #videogallery with .videogallery
2. Globally replace id="videogallery" with class="videogallery"
3. Save the file
In the engine/css subdirectory open the videolightbox.css file in an programming editor
1. Globally replace #videogallery with .videogallery
2. Save the file

Then you can copy the files just as specified by the VLB installation instructions.

The other change I'll be making for my deployment is to rename the video images. They are named 0.png, 1.png, etc. I'm going to put them all in a directory on my resource webserver so I'll rename the files and their references in the code copied from the index.html to use the YouTube video id.

I'll be changing my existing web posts over to this new scheme over time...

Blosxom hitcounter, favorites and lastread plugins for MongoDB

2010-05-17T10:48:00-05:00

When I wrote my plugins for tracking the visits to each page (hitcounter), presenting the counts by popularity (favorites), and displaying the most recently read posts (lastread) I used Perl's Storable module as a simple way to serialize the data structure to and from disk. At the time I wasn't too concerned with performance, my blog agilitynerd.com didn't suffer under the load of the disk reads/writes.

But periodically I get DOS'd and my website becomes unresponsive. Not too suprising since Blosxom has to run as a plain CGI script; but I'm certain the disk writes of these plugins weren't helping. Occasionally the Storable on disk would become corrupted which would take down the site (not enough exception handling in the plugins).

I decided to go with Mongo as the backing data store to remove the disk reads/write from the cgi script. It supports an increment operation (ala memcached) so multiple threads can update the hit count and MongoDB will "do the right thing". It also has a query API that made rewriting favorites and lastread trivial.

I haven't tested the performance but my website, especially the favorites page, seems faster. Once I write a JavaScript client to update the hitcounts I'll be able to move the website to a fully cached deployment while still tracking hits which should make the load of running the agilitynerd blog very low.

I've hosted the plugins on github: http://github.com/saschwarz/blosxom-mongodb-plugins The mongohitcounter plugin can be configured to automatically import existing count data from the hitcounter plugin's Storable. So conversion to the new plugins just requires updating the template's variables to the new plugin names.

Using django-sitemap with django-tagging

2009-11-27T17:33:00-06:00

I was adding django-sitemap to googility.com yesterday and found that Tags don't implement get_absolute_url(). Which makes sense since the site developer would want to decide how to expose them in the URL space.

It is also arguable that links to pages displaying the tag view already exist in the page for models that are already in the sitemap so they don't need to be put in the sitemap explicitly. For example, a page for an Article might be at /article/django-11-release and that page would contain the links to pages linked with the tags for that article e.g. /tag/django/ and /tag/python/

But I figured having the tag pages indexed by Google would be useful. It also allows a different priority to be specified for the pages. So I made a little class that derives from GenericSitemap that allows the url and suffix for the Tag name to be specified:

class SlugSitemap(GenericSitemap):
"""Use for objects that don't implement get_absolute_url
   but have a slug field used in creating their url"""

def __init__(self, info_dict, priority=None, changefreq=None):
    GenericSitemap.__init__(self, info_dict,
                            priority=priority,
                            changefreq=changefreq)
    self.url = info_dict.get('url', '/')
    self.slugfield = info_dict['slugfield']
    self.suffix = info_dict.get('suffix', '')

def location(self, obj):
    return "%s%s%s" % (self.url,
                       getattr(obj, self.slugfield),
                       self.suffix)

Here's how I use it:

sitemaps = {'tag_detail': SlugSitemap({'queryset':Tag.objects,
                                       'url':'/tag/',
                                       'slugfield':'name',
                                       'suffix':'/'},
                                       changefreq='monthly',
                                       priority='0.5'),
}

The urls for tags are at /tag/slugname/ where /tag/ is prepended to tag.name and / is appended to the end

This class can be used to create sitemap entries for any url parameterized on a single field of an instance returned by the QuerySet.

Firefox Web Developer Add-on AJAX NON-Caching Problem

2008-12-30T04:08:00-06:00

I ran across some "interesting" behavior of the Firefox Web Developer Add-on today. When editing JavaScript code I normally leave Web Developer set to "disable cache". This causes all images and, significantly for my purposes, the JavaScript files to be downloaded from the server on every page request. But I ran into a unexpected problem with this setting that cost me a couple hours today.

I was working on a legacy page today that uses an AJAX POST and the response is sent to an IFRAME. The POST was successful, the results were valid and present in the IFRAME. The problem was as soon as the content of the IFRAME was eval'd() a GET was being sent to the server for the URL of the page. This GET was a problem because it had a side effect of clearing the session data (which contained the POSTed AJAX data) for the current page. So when the page's FORM was finally POSTed the server couldn't find the data in the session.

After trying numerous code changes including disabling event handlers and events, it finally occurred to me that the cache disable feature of the add-on might be causing the unexpected GET. Sure enough, once I re-enabled caching the GET stopped being sent.

It was strange that the eval() caused the GET to be requested. All the other code around processing the IFRAME's data didn't trigger it, only the eval. Strange but true.

So just something to keep in mind when strange behavior occurs on AJAX pages where session data is being used.

reCAPTCHA in Django

2008-12-23T22:04:00-06:00

I first read about ReCAPTCHA in this article in Wired magazine last year.

reCAPTCHA provides a free CAPTCHA web service that pairs together two words from OCR scanned books. One of the words is known and the other couldn't be recognized. The user types in both words not knowing which is unknown to the system. As reCAPTCHA collects the responses for the unknown word they get human verified character recognition. So the millions of users of the system are clearing up millions of unrecognized words. It is a very clever human "cloud computing" system using only seconds of human effort for each use of the system.

I'm using a FIGLet based ASCII CAPTCHA on my websites since it was easy to integrate into the Blosxom writeback plugin. But I wanted to give reCAPTCHA a try while converting my Googility site to Django. John DeRosa made my job trivial by writing up the steps with a clear example.

So I followed his directions which involved installing the recaptcha-client python library on my dev and production systems and obtaining a free public/private license key from the reCAPTCHA site. Then I updated my Django view and template files for the one form that needed CAPTCHA protection. It was dead simple and working within minutes. The only minor addition I'd make to John's article is of course you need to pass the captcha_error variable from your view to the template:

return render_to_response('edit.html', {'form': form, 'captcha_error':captcha_error})

So give reCAPTCHA a try for your next project. It was so easy to do I might even convert my Blosxom blogs to use it via Lars Engel's recaptcha plugin.

Improving Site Navigation - Add Titles to Blosxom Pages With storytitle

2008-06-20T22:03:00-05:00

I've always wanted my AgilityNerd blog to serve as a reference for interesting dog agility subjects. Consequently, I'm interested in making it as easy as possible for readers to locate, explore, and learn more about the sport. So improving navigation is a strong interest to me.

I have been reading Designing Web Navigation by James Kalbach. In the section discussing Browser Mechanisms Kalbach discusses the back and forward buttons and the Session History drop down. The session history is usually a menu that drops down from the back button (or near it). It lists the pages the user has visited in reverse order showing their page titles. Kalbach states: "Session history is a good reason to supply meaningful browser titles". That reminded me that for my blog the session history has always just displayed "AgilityNerd" for each page visited on my site. I was leaving out a valuable aid to my readers who use the session history to find where they had been previously.

Another benefit of setting the title in pages is in tracking site statistics. It had always bothered me that GoogleAnalytics never showed statistics for my pages by page title.

So these two things finally moved me to take some action. I had been thinking about writing a Blosxom plugin to set the title element in the HTML head section but I've always been too busy. Luckily I didn't have to.

I found out about the storytitle plugin from Nick Leverton's Serendipity blog. He had made some changes and released version 0.7 of the plugin. It was trivial to setup, I just followed the directions in the plugin and had this site and agilitynerd.com/blog working within minutes.

So if you are a Blosxom user just grab this plugin. Otherwise, take the time to set your page title sections so you can help your readers and help analyzing your own site's statistics

Blosxom Plugin lastcommented Enhancement

2008-02-29T18:50:00-06:00

I had a comment entered on my dog agility blog that I subsequently deleted and of course my lastcommented plugin recorded the comment as the most recent comment. Since the plugin uses Storable to store the data I couldn't manually edit the file to remove the entry.

So I modified my lastcommented plugin to allow you to specify the, base zero, index of the entry on the URL and delete the entry. For example to delete the second entry in the list (index 1):

http://example.com/blog/?lastcommentremove=1

To keep mischief makers from deleting all your comments the plugin has a variable in its configuration section called lastcommentremove that you set to a non zero value to enable this feature. Since this is probably a relatively infrequent activity (if you have blacklisting or wbcaptcha comment spam protection enabled) I figured this was a sufficient mechanism.

You can download the latest version of this plugin here.

Minor Additional Mods to Blosxom moreenties Plugin

2008-02-24T16:24:00-06:00

I had previously extended Jason Clark's moreentries plugin for Blosxom to allow adding images to links to the previous and next group of articles/entries in the head or foot of a Blosxom weblog. While attempting to have valid HTML on my blogs I found that I had left an img tag unclosed. I also found that it wasn't as easy to change the styling of the links as I had originally planned.

So I've closed the open img tags and added the configuration variable $selfstyle into which you can place any CSS style information you'd like added to the td containing each page link. For example:

# Set to the CSS style for $moreentries::links
$selfstyle = 'style="padding: 0 3px;"'; # set to '' for no style

New Scheme Using YouTube to Get "Google Juice"?

2008-02-15T04:32:00-06:00

In the last couple days my YouTube account has fallen victim to a new scheme by porn websites attempting to garner Google Juice. It is pretty straightforward. A generic YouTube account is created with a user name like "babiegirl123xyz" and the "user" has a single line profile with the line "all my pics are at the link below". The link is to an adult sex/dating site.

All by itself creating a bogus YouTube account with a link to your site won't get a site listed any higher in a Google search. But the clever part of this approach is the "user" subscribes to many YouTube channels. Since YouTube users often have channel pages with links to all of their subscribers, the more subscriptions the bogus user creates; the more incoming links they have to their profile and consequently to their adult website. But I'd think the Google algorithm wouldn't weigh links from YouTube too heavily anyway...

I came across this scheme because I got several subscription emails from variants of the babiegirl accounts all with the same template of a profile page and all linking to the same website. It doesn't really matter much to me that they go to this trouble, but it is strange since I don't display a list of my subscribers on my channel so they don't get any real benefit.

I guess another way this scheme could help increase traffic would be through the subscription emails. Heck I clicked on the links to see what the site was about. It could be there are enough folks who do the same in response to the subscription email. If only a few end up becoming members of the adult sites it could make this a worthwhile means for these sites to get more page hits/subscribers.

Sigh...

Web Site Themes for Techies

2008-02-04T05:34:00-06:00

I'm no graphic artist so when it comes time to create or change the look of my website I always have a difficult time. So I've discovered some tools and tips that might help other non artists create reasonable looking websites.

Here's the general approach:

Select an existing template or layout to use as a framework
Select a photo or logo that represents the theme/color scheme of the website
Create a background or background image using a sample of the photo/logos color
Sample colors from the background, photo or logo for highlighting other elements of the website

I'll present how I applied this approach to my latest website makeover and I'll discuss a some websites/tools I found helpful in these areas.

Selecting a Layout

Unfortunately I don't have any suggestions for templates for specific blog engines (apart from searching with Google). But I'll try to make up for that with my next discovery.

If you are familiar with using CSS for website customization Alessandro Fulciniti has created a set of 40 one, two, and three column layouts on his website. They are straight forward to follow if you've done any CSS coding. If you haven't please skip ahead to the next section for some other tips that might be useful.

I have to admit to having some problem retrofitting my old CSS design. My issue was with the rendering of the right hand column. When the negative offset for the column was larger than the right margin for the containing div the right column would be rendered at the bottom of the page. It would look OK in Firefox but not in IE.

Of course it is important to test any new layout in at least Internet Explorer and Firefox. Now that Safari is available for Windows it is easy to test on that browser as well.

Select a Photo or Logo

If you already have a color logo that you use for your business then you can help reinforce that logo by using those colors in your site design. Since my logo is currently black and white that didn't help me much.

I was looking to change my site to reflect the winter season so I went to StockXCHNG and found a nice royalty free snow scene by photographer ijsendoorn. There are thousands of interesting photos suitable for use in blogs hosted on Stock.XCHNG, it is a great resource and I heartily thank the photographers who post their work.

I then used the free Gimp image editing software to select a section of the appropriate size for the header of my site. The Gimp is easy to use and there is a fair amount of online help available if you are doing something tricky. Of course you can use Photoshop or a similar program if you have it.

Create a Background

This is an area where I was really at a loss. I've never been good at creating graphics, and a tileable image is even harder to create. There are some tutorials you can find via Google but I just wasn't up for the challenge. Of course you can use a complementary solid color background by sampling your photo or logo (see the next section).

Fortunately I came across a software program called Gliftex - An Infinity of Tileable Graphics Designs. This is an interesting program that can create a multitude of different images suitable for tiling across the background of a web page. Gliftex has a number of algorithms for choosing the form, color scheme, and interpretation and generating images. Here is a good overview of the product along with some examples.

Gliftex has a feature that makes it perfect for creating a tileable background image for a website, it can sample an image and use the colors in the image as its palette of colors. So I took my snow scene section and had Gliftex sample it. I pressed the Form and Interpretation buttons a number of times and after a while came up with these understated and simple tileable images:

As far as I'm concerned the $ 50 US for the Gliftex Home download was a great investment. They have a free demo version available for download if you want to play around before you buy.

Sample Colors for Highlights

So now I had an image for the header and a repeating image for the background. I just needed to add some matching highlights for some other areas of my blog to tie the whole theme together. I decided to color the headings of my right and left sidebars to match one of the darker colors of the header image. But how to pick out colors from the image?

Gimp to the rescue again. I loaded my image again and selected Tools > Color Picker. You are then presented with an eye dropper pointer. I just clicked on interesting colors in my image to obtain a color value that I could type into my theme's style sheet for the background colors. In the picture below you can see the color value is "bec7d0" which you would write as #bec7d0 in the style sheet.

Results

While there are an many additional changes to my theme I could make I am pretty happy with the cool, blue-ish, wintry theme I've created:

Take a look at AgilityNerd to see it in action or, depending on when you read this, to see my next seasonal theme. I hope some of these tips might be helpful for helping you create your next theme.

Stock.XCHNG - Free Stock Photos

2008-01-19T19:41:00-06:00

I was looking at templates for ZenCart and came across some templates that use images from Stock.XCHNG. Over an hour ago I started browsing the images and am really impressed with the quality and breadth of photos that artists are making available for free use.

I'll definitely use some of these images to spice up my sites.

Feed2JS - Adding AgilityNerd Headlines to Your Website

2008-01-11T22:24:00-06:00

Updated 16-Aug-2006 to reflect Feed2JS's move to http://feed2js.org

I never thought anyone would want me to syndicate my blog and put links to my latest articles on their website. But Marj Kibby in Australia with her very nice dog raising and training blog Choose and Raise a Puppy did just that. So I started thinking about how I might provide an HTML feed that could be included into blogs using client-side JavaScript as not all web/blog writers are developers who can integrate this data on the server side.

Well Google to the rescue: the Feed To JavaScript Open Source project provides a really great service that exactly fits the bill. If you have a website or blog and wish to list the latest articles from another website you can just fill out a form on the Feed2JS site and get the HTML to paste into your website. All you do is supply the RSS feed from the source site to Feed2JS, select the options and click Preview or Generate JavaScript to get the code you need. Feed2JS does all the "heavy lifting".

Here is an example of the generated HTML as it would appear in your website using the current data from this site:

The code for the above is a single (long) line of code:

<script src="http://feed2js.org/feed2js.php?src=http%3A%2F%2Fagilitynerd.com%2Fblog%2Findex.rss10&chan=y&desc=0&date=n" type="text/javascript"></script>

There are many other ways to configure/display the information from AgilityNerd on your site. Just go to: http://feed2js.org/index.php?s=build and enter: http://agilitynerd.com/blog/index.rss10 in the URL text entry box. Then answer the other questions.

Feed2JS kindly caches the latest RSS feed from the source site you specify and replies with the above HTML each time someone visits a page on your site. This has the added advantage of not taxing the webserver of the source site to feed your website; which is nice for sites like this one with a small monthly bandwidth quota.

Feed2JS also has a number of ways to style the HTML that is delivered so you can change the look of the links to match your site. Of course, web developers fluent in CSS can just style the HTML directly.

For Blosxom bloggers I've been playing around developing a new plugin based on the headlines plugin to provide this same functionality. I hope to post this new plugin when/if I get it working. Either way I strongly recommend Feed2JS, it is a great project developed in the true spirit of the Open Source movement.

New hitcounter, favorites Blosxom Plugin Features

2008-01-09T13:25:00-06:00

After fixing a bug in these plugins I decided to add some new features:

The links generated by the favorites plugin used to display the path and filename of each popular article. It now displays the article title and the filename is displayed when hovering over the link. This makes for a much nicer looking and more useful display. Thanks to Doug for the suggestion.
The hitcounter plugin now tests for existence of articles before adding them to its hash of counts.
Requesting a page which no longer exists deletes the page and its counts from the hash of counts. Since previous versions of this plugin would record non-existent pages they could appear in the favorites listing. So now you can delete them by clicking on the link in the favorites page.
As pages previously stored in the hitcounter hash of counts are visited their titles will be added so they will appear in the favorites' generated links.

Both plugins must be upgraded together and will upgrade the existing hitcounter database as pages are accessed. The new versions can be downloaded here:

Notes on Configuring Postfix on Ubuntu Gutsy to Send Email via Google Apps

2008-01-05T23:53:00-06:00

Here are some notes I took on configuring my Slicehost Ubuntu Gutsy installation to use Postfix to send emails via Google Apps. I am far from an expert on postfix configuration but maybe these notes will be helpful to others needing this configuration.

These sites contain the key information:

I basically followed the Behind My Screen tutorial (read the comments too) with the updates from The Prancing Tarantula and the following changes.

My Ubuntu server install didn't have the Thawte certificates installed by default so I installed them:

sudo aptitude install ca-certificates

Then you can append that file to your /etc/postfix/cacert.pem If you "sudo su" before doing the append to the file you won't get messed up by the shell:

$ sudo su
# cat /etc/ssl/certs/Thawte_Premium_Server_CA.pem >>
/etc/postfix/cacert.pem

Since I have Google Apps setup for my domain I don't just want to relay email as "user@gmail.com", I want the email to be sent as though it came from my domain ("me@agilitynerd.com"). This requires some simple changes to the config files.

In my transport file I have:

agilitynerd.com smtp:[smtp.gmail.com]:587

In my generic file I have:

demo@myservername.agilitynerd.com me@agilitynerd.com

Where "demo" is the login name and "myservername" is my slicename. In my sasl_passwd file I have:

[smtp.gmail.com]:587 me@agilitynerd.com:me_gmail_account_password

After restarting postix you can test sending email from your server:

$ sudo aptitude install mailx
$ mailx -s "test email" someotheraccount@gmail.com <
~/sometestfile_to_send

Check your logfiles for errors/warnings:

sudo tail /var/log/mail.\*

I hope these notes might help folks "get over the hump" if they are setting up the same configuration.

hitcounter, favorites, lastcommented and lastread Blosxom Plugin Updates

2008-01-01T18:44:00-06:00

I found a bug in my hitcounter, favorites, lastread and lastcommented plugins due to my not using the correct Perl Storable functions. For some reason I didn't use the locking versions of retrieve() and nstore(). So when my old webhost had some problem causing long page load times (and many simultaneous requests) I ended up having my favorites data file written as zero sized.

So I changed to Storable::lock_retrieve() and Storable::lock_nstore() for all four plugins. The new versions are here:

I never heard any bug reports about these plugins, but I'm sorry if anyone ran into any difficulties due to these bugs.

Taking Better Photos - Building a Light Box

2007-12-31T19:15:00-06:00

I have often been disappointed with the quality of photos of objects I've posted on my site. My biggest problems have been providing good lighting and a nice background (see this for example). So I was Googling to find a photographer's light box to build when I found Bill Huber's great directions for building your own Light Box. So I built my own light box based on Bill's design.

I got two 10 foot lengths of 1/2 PVC and the fittings Bill outlines. Nancy picked up a couple yards of white cloth from the fabric store. I went to a craft store and bought 10 different colored sheets of poster board about 20" by 26". I already had some goose neck desk type lamps into which I put some compact fluorescent bulbs.

I figured I'd size the box so that I could use the poster board oriented in either direction. That made my pipes for the width and depth 28" by 22" (to leave room for the pipe going into the fittings). I made two sets of legs at 12" and 16" long. I got to use my PVC Cutter to quickly cut the pipe. I didn't bother gluing any of the pipe together since the friction fit is plenty tight enough. I also wanted to be able to easily make the box smaller if I over estimated the size I would need. It is a pretty sizeable box.

Light Box with Backgrounds and Cover Sheet Pulled Back

So if you are looking to take photos for your blog or for selling items on eBay this is a simple and inexpensive project. We'll see if it helps my photos look nicer.

Blog Spammers Using URL Encoding

2007-12-29T23:16:00-06:00

I was getting hit by comment spammers in the last week who were using URL-encoding of their addresses to get around the comment blacklisting filter I use. By replacing regular characters with the multi-character encoded representation of those characters within the URL the spammers were able to post comment spam with links to casino and porn websites to my blog.

This type of spam is a reversal of the same method for hiding your own email address in a web page so it won't be harvested by email spammers (see for example Chip Rosenthal's Blog entry).

I just added a regular expressions in my blacklist file to block the use of URL encoded characters in all links. If your blog comment software supports this approach you might want to do the same thing.

Blosxom - Default Flavour Plugin - Fixes Unknown Flavour Error

2007-12-29T23:16:00-06:00

I previously blogged a solution for the Unknown Flavour error that involved modifying the blosxom.cgi script itself. I recently found my modification caused requests for the RSS 0.91 feed and atom feeds to return the default flavour instead of the desired feed. I suspect any plugin using a flavour that inserted itself into the Blosxom template hash table without corresponding flavour file(s) probably didn't work correctly after my modification.

While investigating a fix for that bug I noticed that a better solution would be to override the Blosxom template subroutine. So I created the defaultflavour plugin. This plugin fixes the bug by first applying any registered flavour templates if the requested template files are not found. If that registered flavour templates don't exist the plugin finally applies the default flavour template files.

Download version 0.1 of the defaultflavour plugin here.

No configuration of the plugin is required. Just copy it into the plugin directory.

Blosxom - Hit Counter and Favorites Plugins

2007-12-29T23:16:00-06:00

I added a Favorites page to the side menu of my AgilityNerd site. Since then I've been updating it manually about once a month based on the AWStats reports from my web hosting provider (HostMagix). This task is unnecessarily complicated because AWStats keeps page counts independently for each URL, which includes the Blosxom flavour (filename extension). For my purposes I don't want to distinguish between say HitCounterFavorites.html and HitCounterFavorites.htm

So I decided to write a plugin to track the hits per page and "lump" together counts independently from the file extension. This plugin allows me to display the hit count of visitors for each page on each page. I wrote a second plugin to use the hit count data to automatically generate my Favorites page.

Hitcounter Plugin

I based the hitcounter plugin on the categories plugin written by Todd Larason. The plugin has a couple features of interest:

The $reset_count flag within the plugin lets you provide a starting count value for any page. For example to set a page's count to 10 append "?count=10" to the page's URL. Disable this flag once you've set your counts to avoid mischevious count setting by outsiders.
The $retrieve_only_flavour flag within the plugin can be set to a flavour you want to use for retrieving counts without incrementing the count. Use this flavour to view the counts for URLs of interest.
You can add filters to the start() subroutine to exclude certain requests from updating your counters. I exclude RSS and Atom feed requests from my counts.
As of version 0.5 you can filter out loading and incrementing page counts for specific user agents via the ignore_agents array.

The hitcounter plugin stores the $blosxom::path as the key in a hash whose value is the count of hits. The hash is stored in a file in the data directory. The same hash is used for the entire site. As each page is "hit" the value is incremented and stored in the $hitcounter::count variable. I put this variable in the footer of my pages.

Favorites Plugin

My favorites plugin uses the data file containing the hash of paths and counts as its input. This plugin generates an HTML unordered list of the most visited URLs with the following configuration options:

The $num_entries variable controls how many URLs are listed.
The $include_counts variable controls if directories will be included in the list of entries.
The $include_counts variable controls whether or not the number of counts is appended to each list entry.
The $anchor_format variable controls if the full path or only the filename is displayed in the list entry.
The $anchor_link variable controls whether or not the entry is wrapped in an anchor <a>.
The $excludes variable controls which pages are excluded from consideration. I exclude the main URL for my site from showing up in the list.
The $groups array holds regexps that will generate separate unordered lists grouping together pages matching each regexp of $num_entries list elements. See the configuration section for an example. I use this feature for my Favorites page to group pages for agility and tech categories into separate lists.
The $url_flavour variable lets you specify the flavour to use for anchors to non-category URLs. It defaults to to $$blosxom::default_flavour.
You can add filters to the start() subroutine to exclude certain requests from doing the work to generate the HTML. For example, I only run this plugin for my Favorites page.

When this plugin runs the $favorites::count variable is populated with the HTML. I then put this variable in my Favorites.txt page.

Download version 0.5 of the hitcounter plugin here.

Download version 0.1 of the favorites plugin here.

Comments/bug reports are welcome.

Blosxom Plugins: lastcommented and lastread

2007-12-29T23:16:00-06:00

In preparation for an overhaul of my website I wrote two new Blosxom plugins. The lastread plugin allows me to create a list of links to the last n articles that were viewed by readers. The lastcommented plugin allows me to create a list of links to the last n articles that had comments added by readers.

I took pains with these plugins so they only write their state files to disk when a single article is read (not when the index pages are hit) or when a comment is actually posted. I've also made it optional to not rewrite the state file if an already existing entry in the list is visited/commented again. There are a lot of Blosxom plugins that do a lot of work on every page access and I didn't want to add to the server load.

The configuration of each plugin is pretty straightforward. Both plugins require the Storable plugin for storing the state files. Only the lastcommented plugin requires any coding; whatever comment plugin you use needs to set a variable in order for lastcommented to do any work.

Download the lastread plugin

Updated 2007-06-23 to version 0.3 to fix unshift() error and ignore unknown file requests and filter out user defined HTTP Agents.

`Download the lastcommented plugin`_ Updated 2007-06-23 to version 0.2 to fix unshift() error.

Hopefully I'll have my new site design using these plugins available in a few weeks.

Blosxom - Removing the Unknown Flavour Error

2007-12-29T23:16:00-06:00

After my recent Referer Spam attacks I've been checking my logs and AWStats reports daily. I noticed that occasionally I'd get people requesting pages with incorrect suffixes. i.e. foo.ht or bar.|id| (some of these are probably spammers too). A "feature" of Blosxom is that it will still serve the page with the default layout or flavour. This feature causes a not too helpful and ugly error message to be displayed on the top of the page:

Error: I'm afraid this is the first I've heard of an "ht" flavoured
Blosxom. Try dropping the "/+ht" bit from the end of the URL.

Of course removing the suffix from the URL doesn't usually work; the client is served with an empty (but flavoured) page. So I though a better solution would be to have Blosxom serve my default_flavour whenever the flavour couldn't be identified. A little Googling later turned up this Blosxom mod by James Vasille. James explicitly loads a flavour file named "default" whenever the requested flavour can't be found.

For my site it is more appropriate to serve the $default_flavour when the requested flavour can't be found. That way I didn't have copy/create head.default, foot.default, etc files. So my code change is almost the same as James':

$template =   sub {
  my ($path, $chunk, $flavour) = @_;
  do {
    return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$flavour");
  } while ($path =~ s/(/*[^/]*)$// and $1);
  # Begin added code
  do {
    return join '', <$fh> if $fh->open("< $datadir/$path/$chunk.$default_flavour");
  } while ($path =~ s/(/*[^/]*)$// and $1);
  # End added code
  return join '', ($template{$flavour}{$chunk} || $template{error}{$chunk} || '');
};

So now a less ugly page will be provided when an unknown suffix/flavour is requested.

Update 19-Apr-2005 - this algorithm has a bug that has been fixed by the defaultflavour plugin.

Comment Spam and wbcaptcha Plugin Enhancements

2007-12-29T23:16:00-06:00

I have been fighting automated comment spam robots (see also refererblock update, and hit counter changes) with my modified writeback plugin that uses a blacklist of spam words and URLs. However, I'm getting tired of updating the blacklist and removing spam comments that get through the filter. So I decided to install the wbcaptcha plugin written by Pasi Savolainen. His plugin uses the FIGlet program to generate an ASCII "image" of random letters:

              _
  ___  _ __  | |__    __ _
 / _ \| '_ \ | '_ \  / _` |
|  __/| | | || | | || (_| |
 \___||_| |_||_| |_| \__,_|

A person leaving a comment then enters the letters (enha for the image above) as a "key" along with their comments. The plugin then validates that the key matches the generated image before it allows the comment to be posted. This is similar to the approach used by other blog software to distinguish humans from automated spam programs.

Pasi's plugin worked by executing figlet in a subshell whenever someone wanted to post a comment. I wanted to change this approach since I don't have the ability to install figlet on my web host and I didn't want to pay the performance cost of calling a program through the shell.

So I found the Text::FIGlet Perl package on CPAN and made a few small edits to Pasi's plugin to allow it to invoke the FIGlet Perl code without the shell. This simplified the plugin and sped up it's performance. I've sent my changes to Pasi in case he wants to incorporate them in a future release of his plugin.

I've decided to leave my blacklist in place, since I'm sure I'll still be facing human comment spammers looking to promote their websites. The last change I can think to make to further discourage spammers will be to change all <a> elements in comments to include the rel="nofollow" attribute supported by most portal/search websites.

I also took this opportunity to add some name anchors to my foot.htm and foot.html files so clicking on the "Comments" link on my index pages now takes you directly to the comments section of the individual article. Similarly, selecting "Add Your Comment" and Posting a comment also show the writeback for and the last added comment respectively. This makes for less scrolling on long articles and multiple comments.

Unfortunately, I don't think this will actually reduce the spammer hits on my site since these are due to automated "bots". I doubt the spammers actually check if their comments get posted, they are just hoping for a comment to "get through" occaisionally so their sites can can get some "Google Juice". So they'll keep pounding on my site although hopefully fewer spam comments will get through.

Please email me at the address shown in the right hand menu bar if you have any problems posting comments.

Directing Search Engines to Blosxom Content Pages

2007-12-29T23:16:00-06:00

I've noticed that folks who come to my site by querying through a search engine often end up on my main page and the article they are searching for has already rolled off the front page. This is because the search engine robots tend to choose the index pages as more relevant than the content pages linked to by the index pages. This is especially frustrating to me since I want my blog to serve as a reference for other Agility enthusiasts - so making my articles easily retrievable from search engines is important to me. It turns out this is a pretty common problem faced by many blogs.

I found that Jason Clark had run into this problem with his Blosxom blog and resolved it by using the following meta element on his index pages: <meta name="robots" content="noindex,follow" >. His post describes how he conditionally includes these robot meta tags through his head flavour files (using the interpolate_fancy plugin and his storystate plugin) only on his index pages and not on other pages.

For my site it is even easier. I use separate head.index pages to style my index pages differently from my htm/html pages. So I just directly inserted the <meta name="robots" content="noindex,follow" > into my head.index pages.

There are a number of other robots meta tags that I might consider adding in the future. Websnob has a thorough explanation of the tags and the robots that use them.

Hopefully over the next few months these changes will cause search engines to give their users links directly to the articles in my site.

Image Theft Ranting And Blocking

2007-12-29T23:16:00-06:00

I was really disappointed yesterday when I checked my blog's statistics and found that someone from a Hungarian Agility discussion board had directly linked course images from my website. Direct linking of an image is placing the URL of the image directly in a webpage hosted on another server. Each time a browser loads that page the direct link causes the other server to send the image to the client's browser. So each time the forum is viewed my server has to send the images to the forum's viewer.

I purposely try to make my image sizes small to make it possible for dial up visitors to view my site without too much delay. So direct linking doesn't impact my bandwidth costs too much; so far it is about 1Mb a day. The additional burden it puts on the webserver should be small too, but since I don't own the shared server I don't really know.

Ranting

The aspect of this that most irritates me is that someone would copy my work without attribution. That is the only thing I ask of visitors to my site who wish to reuse my content. The license link at the bottom of each page should make this clear. I really don't think this is it too much to ask.

I take copyright infringement very seriously. To me it isn't just the legal requirements of using other's material in accordance with their wishes that is important, taking credit, explicitly or implicitly, for another's work is just wrong. This is one of those All I Really Need To Know I Learned In Kindergarten concepts: Don't take things that aren't yours. As a friend of Nancy's says "Some people don't have good home training".

Lastly, the Agility community is still a small community and the online Agility community even more so. I guess I am naive, but I hoped that the members of our community wouldn't do things like this.

Blocking

My first step was emailing the webmaster of the site. The site was entirely in Hungarian so it is possible the webmaster may not have understood my English request. In any event, after 24 hours they hadn't removed the links.

So I went to look for a technical solution. There are a few well known technical solutions for this problem. A search of Google for blocking direct linking or blocking hotlinking will turn them all up. The most useful solutions include:

Rename the direct linked images

This mean updating all posts one your site to match the new name. But if you only have a few images and/or posts to them you can do this on a per direct link basis.

Randomly generate image file names that change over time

This is usually used for photo galleries where there is no text referring to each image.

Serve images through a script

This script would reject requests for images based on information in the request. This can be a compute intensive approach since it causes requests even from your own site to go through the script. For sites where you don't have control over the webserver this may be required.

Use a Rewrite rule to serve a different image to non-local referers

This is the technique I used.

Based on this altlab.com article I originally added these rules to my .htaccess file on my server:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www.)?agilitynerd.com/.\* [NC]
RewriteRule .\*.(jpg\|jpeg\|gif\|png\|bmp)$ /images/nodirectlink.g [L,NC]

I then created an image called nodirectlink.g shown below.

Redirected Image

Don't use the same filename suffix as one of the real image filenames you use or you'll loop the rewrite engine.

I had forgotten about images direct linked by RSS feed readers that access the root. Rather than rewrite those requests I moved my Rewrite rules into the .htaccess file in the images directory. I also decided to not send the image after all, I'll just fail the request. There is no sense in even wasting the bandwidth, the clients will now get the broken image icon from their browser. The official Apache URL Rewriting Guide describes this in the Blocked Inline-Images section. So here is my final solution:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www.)?agilitynerd.com/.\*$ [NC]
RewriteRule .\*.(jpg\|jpeg\|gif\|png\|bmp)$ - [F]

I took this opportunity to modify some of my very first articles and move their images into the /images directory. Those images were being served by the Blosxom binary plugin back when I thought it was a good idea. Now I won't pay any extra processing cost for those images.

Conclusion

So now I have a technical solution to a human problem. There is a chance that this change may cause some viewers of this site to not see images on this site. But hopefully that should be a very, very small number of people. But if you think you are getting my blocking image incorrectly Please email me: steve at agilitynerd.com.

I'm still disappointed I was forced to resort to this change.

Modifications to Blosxom entriescache PluginHandles Articles With Future meta-creation_date

2007-12-29T23:16:00-06:00

Fletcher Penney created the original entriescache Blosxom plugin. It is designed to cache all the articles and their creation dates in a Blosxom blog to avoid the overhead of scanning the file system for new/modified files. He merged the meta plugin functionality into the plugin to allow authors to date their entries without relying on the file creation date (this makes it easy to move from one web hosting provider to another without having your article dates get reset).

Unfortunately, articles posted to a Blosxom blog with meta-creation_date's in the future would be displayed immediately. I had wanted to modify Blosxom to allow me post articles with future creation dates and have them hidden until that date. After a request for this functionality on the Blosxom email list, I looked into the required modifications and found it wasn't too difficult to make them.

You can download the modified plugin here. Let me know if you have any questions or comments.

Refererblock Version 0.2

2007-12-29T23:16:00-06:00

I came up with two improvements to the first release of my refererblock plugin:

If the referer string matches the site's URL it passed immediately and isn't checked against the blacklist.
The blacklist.txt file was being read even if the referer string was empty. Now it is only read if the referer string is not empty and it isn't for the site's URL.

These optimizations do improve the performance of the plugin. My testing on a PIII 800MHz running Fedora Core 3 Linux with Apache 2.0 showed the following average latencies:

1.5 ms - Empty referer string or current domain.
2.0 ms - Referer string matching the first regex of the example blacklist file.
3.0 ms - Referer string matching the final regex of the example blacklist file.

I was kind of surprised at how little additional time was required to load the blacklist file and process the regular expressions. This is probably due to the file remaining in the disk cache for subsequent requests. Of course your mileage may vary.

Download version 0.2 of the plugin here.

See my original plugin description for installation, configuration, and testing information. Please let me know if you use this plugin or if you have comments or suggestions for improving it.

Those Darn Spammers Blosxom - Hit Counter and Writeback Changes

2006-08-11T19:00:00-05:00

Over the past year I'd noticed that comment and trackback spammers had been hitting the same dozen or so pages of my blog multiple times daily. (It is probably the same person/group who took a snapshot of the articles on my front page at that time and just reuses those URLs for all of their different domains). Last I calculated, about 50% of my overall website traffic is due to spammers. This constant barrage of hits skews my AWStats statistics and, more importantly, skews the results on my Favorites page. So I took a little time to work on this problem.

FWIW I'm also starting to see a lot of spam coming from "blogs" being setup on BlogSpot that are a single page all of whose links point to the real site. Some of these bogus blogs use the BlogSpot temmplate which contains a flag used to alert BlogSpot admins to content in violation of their Terms of Service. This allows anyone to report the bogus blog by just clicking on the flag. Other "blogs" just use their own HTML and BlogSpot support would have to be sent an email with the offending site's URL.

So anyway, I've taken the following steps:

Disabled trackback comments entirely using the configuration variable in the writeback plugin. I've never received a legitimate trackback ping.
Modified my modified version of the writeback plugin to set a variable $rejected with a 1 if the comment was rejected or if trackback was attempted.
Modified my HitCounter plugin to read the $writeback::rejected variable and then not increase the counter for the spammed page.
I had to change the ordering of the hitcounter plugin to run after writeback so the variable would be set correctly when hitcounter ran.
Set the $hitcounter::reset_count variable and reset the counts of the spammed pages back to "reasonable" counts.

Another couple hours wasted messing around against spammers.

I've previously written how I've been using comment content blacklisting to reject comment, trackback, and referer (sic) spam. My current blacklist file has over 40 regular expressions containing over 250 words and patterns. I update it whenever a spam comment slips through. So that is an ongoing almost daily effort. I might just have to go to the trouble of getting a CAPTCHA plugin to work.

How Safe is Your Personal Information in the Hands of Website Developers?

2005-06-23T19:00:00-05:00

I was going through the webserver statistics for this site to see if any new sites had linked to any of my articles (it is always nice to see that what I have to say is useful to someone). Anyway, I ran across someone who had come to my site through a Google query (I won't mention what the query was for reasons you'll soon see). I ran the same query on Google to see what else came up since it was a rather unique query. Another Google link was for a site that looked like it had raw data - not your usual HTML pages.

When I went to the site I found what looked like a website developer's development directory wide open to the internet. There were at least three company's websites sitting in subdirectories. The file referred to in the Google result page was a backup of an SQL database dump file. Not just any database file - a backup of all the customer information for running one site's shopping cart database. It included names, addresses, email addresses, and phone numbers! (I didn't poke around to see if it had any more sensitive data).

I was able to figure out the original data owner's domain name from some info in the header of the file. So I just sent them an email letting them know that their customer information is posted for all to see on someone else's website. It will be interesting to see if they respond. I hope it is just their website developer who has a test server running and accidentally left this SQL dump in a publicly accessible area of their webserver. I'd hate to think this data was stolen from the real website and being used for spamming purposes.

As a software developer I've read numerous cautionary tales of accidental (and malicious) data theft occurring when real customer data is used in test systems. I just never imagined I'd stumble across such an egregious privacy violation. So this experience makes me wonder about all the online systems into which we type our personal information. All it takes is one careless developer (not even a malicious one) to expose our private information to a much wider audience...

Blosxom Plugin to Block Referer Spam

2005-02-12T20:00:00-06:00

Like so many other bloggers who allow comments on their websites and blog articles, I was facing increasing comment spam as my blog got noticed by more spammers. The size of this problem is illustrated by this Google query for "comment spam" that returned 1.5 million hits. For the uninitiated comment spam is like email spam for blogs; the spammer inserts fake comments in a blog where either the comment text contains from one to dozens of links to the spammer's websites. When web search sites "spider" the blog the links to the spammer's site are treated as "endorsements" of the spammer's sites and the spammer's sites are raised to the top of the search site's result lists.

There is another growing type of blog spam called referer spam (yes it is officially misspelled). When a web surfer clicks on a link in a web page that sends them to another web page most web browsers fill in the URL of the referring page into the request called the HTTP_REFERER. Some websites and blogs capture that page link information when they are on the receiving end of a web request. These sites might have a section on each page indicating the sites that link to that page. These links are referer links.

Referer spam uses the same mechanism as comment spam to raise the search sites ranking of the spammer's websites. But referer spammer's don't post comments; they post fake referrals to a website. The are hoping that the website or blog displays links of the sites that refer to them. So when the website is spidered the search ranking is raised.

Blosxom Plugins Addressing Spam

Like so many bloggers once I started getting comment spam I was able to manually delete them as they occurred. But that got old fast. After some Googling I discovered Doug Alcorn's Blosxom writeback blacklist plugin. I had been using the original writeback plugin. Doug's improvements provided enough protection (so far) with less than a dozen regular expressions removing all my comment spam.

Referer spam started hitting me three weeks ago. What was most infuriating was that I don't display any links of referring sites on my site at all. So all these spammers were succeeding in doing was skewing my site statistics and using my bandwidth with their fake referer attacks several times a day.

Of course all the referer spam site's addresses contained one or more of the same dozen blacklisted words I had already configured for comment spam. About the same time I saw Jason Clark post his deferer plugin to return a 301 permanent redirect for the IP address of one particular referer spammer who was attacking his site. I thought that by combining Doug's blacklisting plugin with Jason's immediate redirect plugin I could reduce the referer spam from my log files.

This plugin hasn't removed the entries entirely from my logs, since the initial request is still logged with a 301 status. But it has stopped the subsequent downloading of images for the pages whose content is now not served. Now that the log contains 301 status messages for these requests they are ignored by my host's statistics program (Advanced Web Statistics - AWStats).

On a re-reading of Jason's blog entry for deferer he also mentions the idea of white and black lists for referer filtering. So I might have subliminally remembered his idea and implemented my refererblock plugin based on his idea. In any event, I fully credit Jason and Doug for giving me the ideas and code with which to put together my plugin.

Referer Block Plugin

The refererblock plugin's tar file can be downloaded here. It contains the refererblock plugin already named 000refererblock so that it runs before all other plugins (you want to discard the blacklisted requests before all legitimate requests). A sample blacklist.txt file is provided and contains some example regexs. It uses the same blacklist file format and file name as Doug's writeback modification (I took the code from his plugin with only cosmetic changes). See Doug's website for links to the Movable Type and other blacklists.

The only configuration variable you can set is $log_blacklisted. If set to a full path file name the script logs the UTC date/time, referer string, and the page to which they were referring. You could use the frequency of words in the rejected referer strings to fine tune the content and ordering of your blacklist.txt file to match the spammers hitting your site. Be aware that this file isn't trimmed so you might want to keep an eye on its size.

Lastly, the zip file contains a simple Perl script you can use to test the plugin. Execute it as:

referer_test.pl http://example.com http://referer-spam.com
where the first URL is your website and the second URL is the referer

to be sent in the request. This script uses HTTP::Request to send the request. The script returns the status "200 OK" for the requested page if it isn't blacklisted and "301 Moved Permanently" if the referer is blacklisted.

Please let me know if you use this plugin or if you have comments or suggestions for improving it.

Online Button Creation Tool

2004-12-08T20:00:00-06:00

I was reading the Blosxom email list tonight and came across an interesting photoblog by Rob Park. After enjoying some of his photos, I noticed his post about an online form for creating the XML, RSS, etc. buttons popular on so many websites. Adam Kalsey has created a simple (dare I say elegant?) form that gives you just enough inputs to completely control the look of the buttons. The image is generated in PNG format.

Had I known of this site before it would have saved me a couple hours messing around with Fireworks creating the buttons on my AgilityNerd Feeds page...

AgilityNerd Site Enhancements

2004-10-29T23:16:00-05:00

I've spent some time over the past couple nights trying to improve the access to information on my site. As a software developer it is often more fun to play around with the technology than it is to do "real" work.

I've made the following enhancements:

Added Quick Links to the right hand navigation to allow folks to get to the three main Agility sections of the website without going to the Navigate page. Used the [[Blosxom]] file plugin to include the new HTML in all pages.
Changed my AgilityGlossary page to have a preface with an alphabetical listing ofdefinitions. I ended up using three new Blosxom plugins to make thiswork: config, postheadprefoot and sort_order.

For other Blosxom users I've updated my site pluginspage with the details.

Hopefully these changes along with the page links and previous/nextpage arrows at the bottom of the page will give everyone enough ways to move easily around the sight.

Modifications to Blosxom moreenties PluginLinks to a Configurable Number of Index Pages

2004-10-21T19:00:00-05:00

Jason Clark created the moreentries plugin for Blosxom to allow adding links to the previous and next group of articles/entries in the head or foot of a Blosxom weblog. He solved a problem that most weblogs have; it can be difficult for visitors to browse older articles. Although he calls the solution a hack, he nicely solved an important problem.

I had been using the morentries plugin but found the text links didn't make it obvious to some visitors that there were more pages of entries. So I added images to the links to make them stand out. But I decided it might be more intuitive to use the same model used by search sites; create a graphical and textual link for each page of articles along with previous and next links.

So I decided to modify Jason's plugin to provide this feature. If you take a look at the bottom of my AgilityNerd home page page with more than $blosxom::num_entries entries and scroll to the bottom) you'll see grey dots, arrows, and page numbers each linking to a page of articles.

You can download the modified plugin here. The prolog of the file describes the configuration options. I've included the public domain icons I use on my site in the zip file for your experimentation. Since there are a few configuration options I've put some examples below so you can see how the new parameters control the html stored in $moreentries::pagelinks.

With these settings:

$numpagelinks = 10;$textlinks = 1;                # set to 1 to enable text links 0 to disable
$imagelinks = 0;        # set to 1 to enable image links 0 to disable

A screenshot of the HTML generated when the fifth page is selected:

With these settings:

$numpagelinks = 10;
$textlinks = 0;                # set to 1 to enable text links 0 to disable
$imagelinks = 1;        # set to 1 to enable image links 0 to disable
$prevlinkimage = "/images/left.gif";
$nextlinkimage = "/images/right.gif";
$currentpageimage = "/images/ball.red.gif";@pageimages = qw( /images/ball.gray.gif );

A screenshot of the HTML generated when the fifth page is selected:

With these settings:

$numpagelinks = 10;
$textlinks = 1;                # set to 1 to enable text links 0 to disable
$imagelinks = 1;        # set to 1 to enable image links 0 to disable
$prevlinkimage = "/images/left.gif";
$nextlinkimage = "/images/right.gif";
$currentpageimage = "/images/text.gif";@pageimages = qw( /images/ball.gray.gif /images/ball.red.gif );

A screenshot of the HTML generated when the fifth page is selected:

Thanks again to Jason for creating this plugin. It was fun to modify and I hope it is useful for other Blosxom webloggers.

See More... Added to Article Display on Index Pages

2004-09-26T19:00:00-05:00

I know my Agility readers will give me a hard time about this non-Agility post, but hey I'm "thinking about technology" too. I was limiting to five the number of articles/posts on my main and index pages to minimize the download time for non cable/DSL visitors. But I've noticed some searches hit my site for which an article was moved off my main page and I'm sure they didn't hunt around for the missing post. So I wanted to show more articles without increasing the download time.

The Blosxom plugin seemore automatically adds a link labeled "See More..." in place of a special label in each article. Selecting the link takes you to the full text of the article. This useful plugin was developed by Todd Larason, a blogger who's developed a number of powerful Blosxom plugins.

I made a minor modification to seemore, which I've forwarded to Todd, to allow showing the entire article for a configurable number of articles on a page. This lets you see the full text and diagrams for the latest articles and only seeing the first couple paragraphs of the older articles.

If you are a Blosxom blogger and would like my modifications you can download my modified version here. Unzip it and put it in your plugin directory. By default it should work the same as Todd's original version. A new configuration variable $show_all_of_stories_until can be set to the number of stories on each index page that you'd like to show in full.

So I've modified most of my posts to include the seemore label and increased the number of articles shown on each index page to $blosxom::num_entries. I like this new site configuration; let me know if you do too.

Backup Your Data Lately?

2004-08-20T19:00:00-05:00

A few weeks ago I bought a Ximeta NetDisk 120 GB external hard drive to back up the data on my home computers. There are a number of vendors making hard drives that support USB connections; this model is unique in that it also supports direct ethernet connections. Unlike more expensive Network Attached Storage devices (more than $ 1000) this unit only cost a little more than the hard drive itself; about $ 150. The only downside is that it uses it's own proprietary network protocol which requires installation of a driver on any computers using the drive.

My goal was to put this drive on my network switch and back up data from my XP, WinME, RedHat 8.0, and Win95 machines. It turns out this was a little trickier than I expected. So I spent the better part of the morning on a cool, grey Chicago day making this all work.

The disk comes formatted for NTFS but in order for it to be shared on the older Windows platforms and my Linux machine it needs to be reformatted to use a FAT32 file system. Windows XP doesn't support formatting drives for FAT32 so I had to install the Ximeta driver software on my WinME machine and install the NetDisk on my network.

Running fdisk and reformatting the hard drive is the only "scary" aspect of the installation. Choosing the wrong drive or partition would be a bad thing. This document gives a good step by step description.

I was skeptical that running fdisk over the network would work correctly, but it did. At this point I was able to view the drive on both my WinME and XP systems and copy data to the drive as if it was locally connected.

I was pretty sure making the drive work for Linux would be difficult. Unfortunately, the PDF documents from the Ximeta website are unreadable as they require installing the Korean Acrobat extensions... thankfully Google has its "View has HTML" facility which let me read the RedHat instructions. The docs on the install CDROM are viewable (but don't include the RedHat docs).

I'll spare you all the trial and error but after downloading the driver RPM from the website and installing it I couldn't configure and connect to the drive on the network. It could be that ports required for their protocol aren't opened on my Linux machine, but Ximeta doesn't give any information on what ports are used by their driver. The admin tool gives some cryptic error messages that Googling and the docs didn't explain (the docs recommend reinstalling the drivers for any errors...). I ended up connecting the drive directly via USB and was able to mount the drive and backup my user and system accounts to the disk.

So it looks like my goal of leaving the drive on the network and copying to it from any computer will only work for Windows machines. But at least I have a mechanism for backing up all my machines, that is easy enough that I'll use it all the time. My next step is looking into configuring rsync or a similar mechanism to only backup the changed files to the NetDisk.

In summary, I'd recommend this drive for anyone who is using Windows XP; it is plug and play for that operating system. If you are computer savvy you can make this hard drive play with other systems too.

However you do it, backup your computer!

Centering DIVs in CSS

2004-08-15T00:00:00-05:00

I took a look at this site on an 800x600 resolution monitor and didn't like how the original layout looked. Having the menu box on the left hand side with a 20px margin to the left used up too much screen space. So I decided to move the menu box to the right of the main body of the page.

I could have hard-coded the box position; but decided that I wanted to have the main body remain centered regardless of browser size. That would complicate the positioning of the menu box (which is it's own div not contained with the main page). Once again A List Apart had a good article describing how they got their CSS centering to work.

The ALA article had links to these two pages describing the methods I used: Auto Width Margins and NegativeMargins.

The AgilityNerd site has two main divs: #page for the main body and#sidebar for the box containing the menu. I used the Auto Width Margins method to center the main body:

#page { font-size: 13px; background: white; border: 1px solid black;
top: 0px; width: 640px; margin: 0px auto; margin-bottom: 50px;
text-align: left;}

The two lines of interest are margin: 0px auto;` and ``text-align: left;. Thelater is required because I used the IE 5.x workaround of specifying text-align: center; in the body element.

So now that the main body of the site was centered, I wanted the menu to be floating off to the right of the main body. The Negative Margins method let me horizontally center the divand also offset it horizontally by any amount. I offset it by half the(fixed) width of the main body plus a little space. The additionalspace keeps IE 5.x browser's miscalculation of sizes from causing theborders to not meet correctly:

#sidebar { background: white; font-size: 12px; position: fixed;
width: 140px; height: auto; top: 70px; left: 50%; margin-left:
330px; text-align: center; border: 1px solid black; }

Unlike the article, I used position: fixed instead of absolute. This allows the menubox to float in place during scrolling (but only for CSS compliant browsers like: Mozilla/Netscape).

CSS for Print Media

2004-08-14T23:16:00-05:00

In keeping with my desire to have this site's layout be easy to read,I wanted it to look good in print too. The layout uses CSS instead of HTML tables. This lets me position the divscontaining various text for the screen and it allows me to positionand hide text differently for the printed media.

Any CSS book covers the basics of changing the display, position, margins, and padding along with the use of separate style sheets forprint and screen media. But I ran into problems with printing mysite's pages with the Mozilla/Netscape browsers. Only the first page of my main div would print. Alittle Googling turned up this excellent article on A ListApart. ALA is a great resource for web developers.

The only clarification I have is that since both the screen and the print CSS files are included, the print CSS file's elements may needto set additional properties as well as reseting properties that existin the screen media file. Otherwise the definitions for the screen will be used. I had to set some div's position properties to static to get them back into the document flow.

AgilityNerd Blosxom Plugins

2004-08-10T19:00:00-05:00

Updated 20-Jun-08 for addition of storytitle

I'm really impressed with the Blosxom plugin mechanism. Rael Dornfest, author of Blosxom, put hooks at strategic points in the page generation process. At each hook point the main script iterates all the plugins present in the plugin directory. Plugins that implement the hook function can then modify the state or add additional data for use in the page presentation.

Consequently, a plugin only implements the hook functions it needs invoked to provide its features. Some powerful plugins need only implement one or two hook functions and are implemented with a couple dozen lines of Perl code.

When I started building this site I found it a little confusing to sort through all the plugins and understand their functionality. The ease of creating and modifying plugins has allowed a lot of developers to contribute to the Blosxom plugin registry. I found it helpful when Blosxom sites listed the plugins that implemented their site's features. So here is the list of plugins I use.

atomfeed - Provides the AgilityNerd Atom feed. Point your browser to http://agilitynerd.com/blog/index.atom

breadcrumbs - Creates the click-able trail to the current position in the weblog's path shown at the top of all .index pages and at the bottom of each article.

binary - Used to supply images from the directory in which an article exists. Modified to use LWP::MediaTypes since MIME::Types isn't installed on this web server. I also commented out the interpolate() function and added it to the one in interpolate_fancy to make this plugin coexist with interpolate fancy.

I no longer use binary because it is faster to just have Apache serve binary files directly from one or more directories outside of the Blosxom src tree.

I've stopped using this plugin since it requires Blosxom to look at each file to determine whether or not it should be skipped. So I'm starting to put all binary files in my images directory outside of Blosxom. This has the added advantage that I can just put "/images" and "/video" in by robots.txt file so my images and videos don't get indexed by search engine robots.

categories - Displays the tree of paths/categories of articles on the navigate page. Slightly modified to remove the root directory from the display. Added a hide feature to hide specified directories entirely - used this to prune subdirectories for my side bar menu. Removed flavour ending from all generated links since all paths are valid links, are shorter for people emailing links around, and would always use the default flavour. Modified to not calculate breadcrumbs since I use the breadcrumbs plugin for that feature.

config - Allows any Perl variable to be set differently in any category/directory. I use this to change the sort order for my glossary page to be alphabetical order by changing the sort_order plugin's sorder variable.

entries_cache - Speeds up processing of the site by caching the article index to avoid scanning directories until a configured amount of time has elapsed. Also caches article creation time to allow for editing of articles without changing their timestamps.

favorites - Used to automatically generate my Favorites page based on the visitor hit counts recorded by my hitcounter plugin.

file - Used to include the navigational side div into all pages without copy/paste of the same HTML.

find - Provides search capabilities for all articles, comments, and linkbacks on the web site. HTML slightly modified to remove the advanced search link. Also set the hidden "path" input to the empty string so the entire site is searched regardless of where in the hierarchy the search was initiated.

foreshortened - Ends articles at the end of the first sentence. Used to provide the shortened version of articles for use in RSS and Atom feeds.

headlines - Used to determine all the entries in my glossary category and create links in alphabetical order. Requires interpolate_fancy.

hide - Excludes directories of articles from the index views (like the agilitynerd.com main page) but still allows them to be searched by find.

hitcounter - Used to automatically generate the number of visitors displayed in the footer for each page on my site. These hit counts per page are used by my favorites plugin to generate my Favorites page.

interpolate_fancy - Used to remove the space allocated for the results of find and the breadcrumbs plugin when their variables aren't defined. I think I'll use this more in the future to dynamically change the page format.

lastcommented - Used to automatically generate the list of the last 10 articles that were commented shown in the side bar on each page of the site.

lastread - Used to automatically generate the list of the last 10 articles that were read shown in the side bar on each page of the site.

moreentries - Creates the "Previous" and "Next" links at the bottom of each index or search page when there are more than $blosxom::num_entries entries. I've modified this plugin as I described here.

postheadprefoot - Allows each category/directory to have unique text inserted after the header and/or before the footer. I use this on my glossary page to add a category specific subsection. I had to modify the script to have paths resolve correctly.

redirect - Unmodified version of this plugin. I had to use it when I moved a directory within my tree so search engine and email list links would redirect correctly.

refererblock - A blacklist driven plugin to permanently redirect referer spam. The redirect probably has little effect on the receipt of referer spam but the 301 redirect code keeps the spammer's sites out of my site statistics.

rss10 - Provides the extra information to provide a valid RSS 1.0 feed for syndication. Slightly modified to work with the writeback plugin.

seemore - Adds "See more..." link within the text of a post when the post is shown in an index page. The location of the link is controlled by placing a special tag in the posting source file. I have modified this plugin to show the first "n" entries of an index page in full. Furthe

r modified to always create a link to my html flavoured pages.

sort_order - Allows sorting by date, directory names, or file name. I modified it to not read the URI parameters and just use a configuration variable. Then I used the config plugin to change the sort order for my glossary page to sort by file name instead of in reverse chronological order.

storytitle - sets the <title> element in the HTML <head> with the title of the article or the name of the category. I use it to aid in navigation and site statistics]].

timezone - Adjusts timestamps automatically created or entered using meta-creation_date: to be offset correctly to my timezone; which is different from the web server's timezone.

I no longer use timezone since I can set the timezone on my Slicehost server.

defaultflavour - Provides the user with the default Blosxom flavoured feed when an unknown flavour is requested. This keeps the user from getting the Unknown Flavour error across the top of the page.

wbcaptcha - Provides an ASCII image via FIGlet when visitors enter comments to stop spam bots from saturating my blog comments. See this article for my modifications to this plugin.

wikiwordish - Provides WikiWord-like linking to Wiki articles, local article file names and modified to replace WikiWords with HTML <a> links. This plugin saves me from having to enter links to common sites to which I always refer. I had to name this file 00wikiwordish so that WikiWords would be correctly replaced in RSS and Atom feeds.

writeback blacklist plugin - Provides comment and TrackBack capability. This is Doug Alcorn's blacklist modified version to help fight comment spam. I've slightly modified mine to try to protect comment poster's from spam by obfuscating their email addresses.

In case anyone is trying to get these plugins to "play" together I have them named as follows:

000refererblock
001redirect
002defaultflavour
005wikiwordish
007google_highlight
008wbcaptcha
01atomfeed
01breadcrumbs
01categories
01config
01entriescache
01favorites
01file
01find
01foreshortened
01fullcategories
01headlines
01hide
01moreentries
01postheadprefoot
01rss10
01seemore
01sort_order
01storytitle
01writeback
02hitcounter
02lastcommented
02lastread
02recentwritebacks
50interpolate_fancy