Frank Hecker - Blosxom

Extensionless URIs for Blosxom entries

2006-07-23T16:37:00Z

This post introduces the Blosxom plugin "extensionless". Ever since reading the essay "Cool URIs don't change" I've wanted to change my web site to conform to some of its recommendations, including the recommendation to omit file extensions (e.g., ".html") on URIs. Unfortunately standard Blosxom requires that a file extension be present in a URI for an individual entry (e.g., http://www.example.com/foo.html) to distinguish it from a URI for a category (e.g., http://www.example.com/foo). How to fix this?

In my ignorance I originally thought it would be necessary to patch Blosxom itself to recognize extensionless URIs for entries. However I hadn't fully grasped the power of Blosxom plugins, nor had I had a good look at the Blosxom plugin registry, where I would have found not one but two plugins to implement a "cool URI" scheme.

Unfortunately those plugins are overkill for my own personal needs, since I'm not interested in doing date-based permalinks as implemented by those plugins. As a result I decided to implement my own plugin to provide the more limited functionality that I wanted.

(I refrained from naming the plugin "cooluri3" or something similar since it really doesn't implement the full date-based "cool URI" scheme recommended by the W3C and implemented by the cooluri and cooluri2 plugins; at best it implements "semi-cool" URIs.)

See the plugin code itself for the full documentation. In most cases you should just be able to copy the plugin into your Blosxom plugin directory. Note that you should not have to change any other plugins; the extensionless plugin will fix up an entry's URI internally so that it has the proper file extension for its flavour as expected by Blosxom and Blosxom plugins.

If you encounter problems with the plugin (or if you just use it and like it) please send me email.

UPDATE: Thanks go to Stu MacKenzie for providing a patch to allow the plugin to work for extensionless URIs where the entry name starts with a digit.

UPDATE 2: The original version of the extensionless plugin would not work with Blosxom versions 2.0.1 and higher. I fixed the plugin to correct this problem; current versions of the extensionless plugin should work with all Blosxom versions.

The feedback plugin, an alternative to writeback

2006-07-20T05:41:00Z

When I originally put up my blog one of the major things lacking was support for comments and TrackBacks. After looking at the various alternatives (the writeback plugin, the writebackplus plugin, and so on) I decided to embark on a complete rewrite of the writeback plugin in order to support my particular requirements for a comments system. After much struggle I created an initial version of my feedback plugin for publication and use on my site; since that time I've upgraded the plugin and incorporated bug fixes suggested by various people.

Features

I did a pretty much complete rewrite of writeback and the various writeback derivatives, both because they didn't support particular features of interest to me and also because I didn't understand exactly why they did certain things the way they did. Here are the key things I wanted, all of which I managed to implement in one form or another:

no need to use a special writeblack flavour
different formatting for comments vs. TrackBacks
correct formatting of basic plain-text comments (in particular, if you enter paragraphs separated by blank lines then the resulting comment should actually display as multiple paragraphs, without requiring you to use HTML tags)
comment previewing
comment and TrackBack moderation
~~basic~~ spam blacklist checking using Akismet
basic security measures against code injection attacks
Markdown support (although I'm not using it right now)
clean enough code structure to allow easy addition of new features
minimal dependence on other plugins

Non-features

There were a number of other features that I didn't care about and decided to leave out:

Captcha support. I omitted this in favor of a combination of Akismet spam checking plus (optional) moderation; unlike captchas this protects against TrackBack spam, not just comment spam.
HTML tags in comments (even just a subset). I think Markdown is a better approach for those who want links and more complicated formatting.
Comment threading (e.g., as implemented in the comments plugin). This just wasn't that important to me.
A writeback compatibility mode (i.e., the ability to use the feedback plugin as a drop-in replacement for thr writeback plugin, using legacy writeback templates and variables). The plugin could probably be enhanced to support this, but I'll leave it to someone else to do this.
Correct display of comment and TrackBack counts (e.g., displaying "1 comment" vs. '2 comments"). I left this as a future task.
A generalized API for extending features through feedback-specific plugins, e.g., as implemented by Doug Alcorn for writeback. While an interesting concept, I felt this was a bit heavyweight for what I wanted to do.

There were also several features that I initially implemented and then later pulled out in the interest of simplicity:

Ability to display comments and TrackBacks on index and archive pages. (Right now the plugin displays comments and TrackBacks only on individual story pages.) I've seen some Blosxom-based blogs that do this, but it wasn't something I was interested in. (For anyone who wants to do this, it's a trivial patch.)
Support for "comments_head" and "comments_foot" templates to provide additional content to be displayed before all comments and after all comments (with analogous "trackbacks_head" and "trackbacks_foot" templates for TrackBacks). In the end I found I could use variable interpolation (as supported by the interpolate_fancy plugin) in the story and/or foot templates to achieve the look I wanted.
Support for a separate "preview template" used for previewed comments; in the end I just reused the comment template for this.
Support for HTML email for moderation and notification messages. I decided that plain text email worked fine and was arguably better from a security point of view.

Implementation

Finally, here some implementation details that might be of interest to people using or (especially) writing writeback-like plugins:

The feedback plug-in contains all the default templates needed that are specific to the plug-in; you should only have to modify your existing story and foot templates to reference the plugin variables (as described in the documentation at the bottom of the plugin).
I used the same basic file structure to store comments and TrackBacks as is used by writeback and friends. (In fact, I think you may be able to use existing writeback files with this plugin, but I haven't tested this.)
I moved processing of submitted comments from the start subroutine into the story subroutine. This seemed to simplify the implementation, especially when it came to deciding whether or not comments or TrackBacks should be closed after a certain time. (To do this properly you need the modification date/time of the story, which you don't have until the date subroutine runs, right before the story subroutine.)
Comment previewing was pretty straightforward to implement: It's basically a matter of keying off the particular submit button clicked, formatting a special "preview" comment separate from the main comments, and then pre-filling the comment form fields with the previewed field values.
Moderation was also pretty straightforward (after a couple of false starts): Write the moderated comment or TrackBack not to the main feedback file for the story, but to a separate file whose name contains a randomly-generated 8-character alphanumeric string. To support approval or rejection of the comment or TrackBack, create moderate=approve and moderate=reject URLs referencing that string as a query parameter (e.g., feedback=mi3g9qcl4) and send those URLs to the moderator as part of an email message notifying them of the comment/TrackBack. When the moderator clicks on the appropriate URL and the plugin processes the GET request, it will either append the temporary file to the main feedback file (if the request was approved) or just delete it (if the request was rejected). The use of a random string minimizes the possibility of unauthorized persons trying to approve their own requests. (This doesn't protect against eavesdropping attacks, of course; doing that would require some use of cryptography.)
The plugin seems to support non-ASCII posts (e.g., in Japanese, etc.), except in the notification/moderation email messages. I suspect this is just be an artifact of the particular version of Perl I'm using; I certainly didn't do any work to support anything but 7-bit ASCII, and it may be that some things are still broken even in my case.

Anyway, I hope this may be of interest to some people. As always, please feel free to re-use the code or ideas as you wish. See the plugin itself for the full documentation on how to configure it. If you encounter problems with the plugin (or if you just use it and like it) please send me email.

UPDATE: Added mention of Akismet support. Thanks go to Kevin Scaldeferri for the Akismet support (adapted from his version of writebackplus) and to Keith Carangelo, Gustaf Erikson, Matthijs Kooijman, and Michael Lamertz for various bug fixes and related suggestions for improvement.

Patch for atomfeed plugin (UTC dates)

2005-02-20T08:06:00Z

I recently experienced a strange problem with the Atom feed on my weblog. My weblog server is running on U.S. Eastern time as the basic time zone, but the story dates in the Atom feed should be expressed in UTC/GMT; the atomfeed plugin has code that supposedly should do any necessary conversions. On my local test blog (running under OS X 10.3 using Perl 5.8.1) this worked fine, but on my real blog (running on Red Hat Enterprise Linux 3 using Perl 5.8.0) the dates in the Atom feed were incorrect; they were five hours earlier than what they should be, suggesting that they didn't get converted to UTC/GMT. After some investigation this turned out to be due to non-portable code in the atomfeed plugin.

More specifically, the atomfeed plugin attempts to convert the modification time of each entry file (expressed as a Unix time, i.e., in seconds since the epoch) into a UTC date by calling the subroutine blosxom::nice_date, which in turn uses the ctime function defined by Time::localtime. Since ctime normally converts Unix time into a local time (i.e., using the time zone currently in effect), the original atomfeed plugin attempts to coerce ctime into returning a UTC/GMT time by setting the TZ environment variable to the value 'GMT' prior to calling nice_date, and then restoring TZ to its original value afterwards.

Unfortunately, as noted in "Writing Portable Perl", this won't necessarily work on all systems:

The system's notion of time of day and calendar date is controlled in widely different ways. Don't assume the timezone is stored in $ENV{TZ}, and even if it is, don't assume that you can control the timezone through that variable.

I appear to have hit one of the cases where this doesn't work.

In any case the solution was simple: I just changed the atomfeed plugin to convert the entry's modification time using the built-in Perl function gmtime, which converts a Unix time into a time expressed as UTC/GMT. Like the nice_date subroutine the gmtime function returns an array; the returned values require a bit more reformatting than those from nice_date, but nothing that complicated.

For the exact code changes see the atomfeed UTC patch itself.

Patch seemore plugin for full text feeds

2005-01-18T09:50:00Z

I use the seemore plugin by Todd Larason to show only excerpts of entries on my main blog page, index pages for categories, and archive pages, while displaying the entire article on an individual entry's page. It's worked well, with one exception: When I created my RSS and Atom feeds I wanted the feeds to contain the full text of all entries, for the convenience of people using news readers. (Many of these applications display article text directly in the reader, removing the need to open a browser window to read the article.)

To do this I made a minor patch to the seemore plugin, which I thought others might find of interest as well. The patch essentially bypasses seemore processing for selected Blosxom flavours (in my case, the 'rss' and 'atom' flavours).

Patch for entries_cache_meta plugin (meta values)

2005-01-17T23:57:00Z

I've been using the entries_cache_meta plugin by Jason Thaxter, mainly for the convenience of specifying the modification date within the entry file. After a while I decided I'd like to also use its "meta" capability, i.e., the ability to specify arbitrary variables in the entry header along with the modification time, e.g.,

The entry title
meta-mtime: 2005/01/17 12:18:00
meta-foo: Whatever you want

The entry text begins here...

and then reference the variables as, e.g., $meta::foo within the story template (as is possible with Rael Dornfest's original meta plugin). Unfortunately, I couldn't get this to work at all.

After a bit of debugging I managed to find out what the problem was: The plugin code for the story subroutine attempts to take the cached meta values for the current story and stuff them into the "meta" namespace, so that they can be accessed as, e.g., $meta::foo. However the code doesn't correctly access the cached set of meta values for the entry; it uses as a cache key the value of the variable $filename as passed to the story subroutine, but this variable is simply the basename of the entry file (e.g., "foo"). What it should be using is the full absolute pathname of the entry file, e.g., /blosxom/data/abc/foo.txt.

The fix is very simple: use the standard Blosxom variables $blosxom::datadir and $blosxom::file_extension and the variable $path (also passed to the story subroutine) along with $filename to recreate the entry file's absolute pathname. For full details see the one-line patch itself.

A final note: The documentation for the entries_cache_meta plugin claims as a special feature that

By combining the meta-tag functionality with the entries cache, it becomes possible to write or use plugins that access meta-values outside the story hook. For example, you could use this plugin to write another one showing the most recent entry for each author defined in meta tags.

The documentation doesn't expand on how this would be possible, and in my confusion I was thinking that this was done using $meta::foo variables; this is not correct. The problem is that entry-specific meta variables have a unique meaning and value only in the context of a single entry. For example, one entry file might have a line meta-foo: abc and another file a line meta-foo: xyz; hence $meta::foo would have the value 'abc' in the context of the first entry (e.g., when processing a story template for that entry) and the value 'xyz' in the context of the second entry. Outside the context of those two entries (e.g., when processing a head template) it doesn't make sense to refer to $meta::foo.

Now having said that, with the entries_cache_meta plugin it is in fact possible to make use of cached meta values outside the context of an entry, since the variable %entries_cache_meta::cache is populated as soon as the entries subroutine is run. For example, a plugin could loop over all the entries to be displayed and determine how many entries were by a particular author, as determined by a "meta-author" field in the entry files:

my %num_by;                   # number of entries for each author
for my $entry (keys %entries_cache_meta::cache) {
    $entries_cache_meta::cache{$entry}{'author'}
        and $num_by{$entries_cache_meta::cache{$entry}{'author'}}++
}

The results could then be used to customize the head section of the page (e.g., to identify the most prolific author).

Enforcing proper use of trailing slashes

2005-01-11T10:50:00Z

I've previously blogged about my canonicaluri plugin that checks to see whether the requested URI is in the canonical form for the type of page being requested, and if necessary does a browser redirect to the canonical form of the URI. However the canonicaluri plugin may be overkill for some people, for example, it presumes use of the extensionless plugin, so that canonical URIs for individual entries do not have file extensions for the default flavour. A simpler alternative to the canonicaluri plugin is the slashredir plugin, which only enforces proper usage regarding trailing slashes.

In particular, the slashredir plugin enforces the following rules:

URIs for individual entry pages should not have a trailing slash.
URIs for all other pages (including the blog root, category index pages, and archive index pages) should have a trailing slash.

For example, if you request the URI

http://www.example.com/blog/foo

where "foo" is a category, this plugin will force a redirect to the canonical URI

http://www.example.com/blog/foo/

Similarly, if you request the URI

http://www.example.com/blog/foo.html/

where "foo.html" is an individual entry, this plugin will force a redirect to the canonical URI

http://www.example.com/blog/foo.html

Note that this plugin depends on having URI rewriting rules in place (e.g., in the Apache configuration file) to enforce the restriction that a URI should never have more than one trailing slash. The plugin as presently written can't handle this case properly (even though the code attempts to do so) because it depends on the path_info() function to get the URI path, and the path_info() value has already been stripped of any excess trailing slashes that might have been present in the original URI.

See the plugin code itself for the full documentation. If you encounter problems with the plugin (or if you just use it and like it) please send me email.

Patch for atomfeed plugin ("modified" element for feed)

2005-01-09T13:55:00Z

The "official" atomfeed plugin does not generate valid feeds for the current version (0.3) of the Atom specification because the output does not have a "modified" element for the feed as a whole, just "modified" elements for each story. Obviously the modification date/time for the feed can be interpreted as the date/time modified of the most recent story, so then it's just a matter of generating the proper output for the MODIFIED tags.

Jason Clark already looked at this and created a patched version of the atomfeed plugin. However his patch requires the use of the lastmodified plugin. While I'm using a rewritten version of the lastmodified plugin, I don't want to depend on it being present in order to get Atom feeds to work properly.

Prior to discovering Jason Clark's atomfeed patch I had already patched atomfeed myself. My atomfeed patch has the advantage of not requiring a separate plugin. However the downside is that I had to generate the "modified" element for the feed in the foot section after the "entry" elements for the stories themselves, right before the FEED end tag. Jason's patch generates the "modified" element for the feed in the head section, which I think is more aesthetically pleasing if nothing else.

Putting the "modified" element for the feed at the end after the "entry" elements doesn't appear to affect the validity of the feed output; it validates fine according to feedvalidator.org. However I don't know if this might cause problems for any Atom-aware feed readers out there. (The only ones I've tested with are NewsFire and NetNewsWire for OS X; both seem to work fine.)

The lastmodified2 plugin

2005-01-09T13:43:00Z

In a previous post I discussed the general problem of validating and caching dynamic content. In order to implement the strategy outlined in that post I decided to create a new version of the lastmodified plugin originally created by Bob Schumaker. The lastmodified plugin was a good base to build on; however it didn't do exactly what I wanted to do, and hence I couldn't resist trying to improve on it.

The following material documents the lastmodified2 plugin that I created, including my notes on how I implemented page validation according to my interpretation of the HTTP 1.1 specification.

The strategy revisited

As you may recall, in my previous post I outlined an overall strategy for how to support validating and caching dynamic content. Here's a recap of that strategy, with additional detail added on the subject of validation:

When sending responses to requests, add a Content-length header to identify the total number of bytes in the response.
When sending responses to requests, add an ETag header to identify the "version number" (entity tag) for this particular version of the page, and/or a Last-Modified header to identify the date/time the page was last modified. These are computed as follows, depending on whether weak or strong validation is used:
- For weak validation, the Last-modified header should reflect the date/time modified of the most recently-updated "semantically-significant" component of the page. (For example, for Blosxom we consider entries to be semantically significant, but not flavour templates.) The ETag header can then supply a weak etag directly derived from this date/time.
- For strong validation, the ETag header should change if even a single bit on a page changes; for example, it could be derived from the MD5 or SHA-1 digest of the page. A Last-modified header value could then be determined by consulting a cached copy of the ETag and Last-modified values for the URI; if there is a cache match then the Last-modified value can taken from the cache, otherwise it can be arbitrarily assigned to be a date in the recent past.
When sending responses to requests, also add Cache-control and Expires headers to the response to provide a "use by" date/time to clients doing caching.
When processing requests, look for the If-none-match andIf-modified-since headers. If one or both are present, return the full page in the response only if necessary: if the version of the page currently available is different than the version requested in the If-none-match header, or if the page has been modified since the date in the If-modified-since header.

Implementation overview

This section and the next describe in more depth how I implemented the above strategy.

First, the plugin is designed to have its behavior easily modifiable using configurable variables, as is done with other Blosxom plugins. In particular, it is possible to specify whether the plugin should do strong or weak validation ($strong boolean variable) and whether it should generate an ETag header, Last-modified header, or both ($generate_etag and $generate_mod boolean variables). By default the plugin is configured to be a "plug-compatible" replacement for the lastmodified plugin, doing weak validation and generating both ETag and Last-modified headers.

The basic plan of the lastmodified2 plugin is as follows:

start subroutine: Read in the cached information containing the previous ETag and Last-modified values for this URI.
filter subroutine: Get the information necessary for weak validation by traversing the list of entries to be displayed on the page and determining the date/time any of the entries was most recently modified. Use this last-modified date/time to create a weak ETag value.
skip subroutine: For weak validation interpret any If-none-match or If-modified-since headers and determine whether or not we need to send a full response. If not we can skip the actual story processing after setting Status to 304 (Not Modified) and generating any other headers appropriate for a 304 response.
last subroutine: For strong validation generate an MD5 digest of the page and use this to create the ETag value. Create the Last-modified header by using the cached Last-modified value if the new ETag value matches the cached ETag value, otherwise assigning a new Last-modified value in the very recent past. Then interpret any If-none-match or If-modified-since headers in the request and determine whether or not we need to send a full response. In either case send the appropriate headers.

Note that there is also a story subroutine in the lastmodified2 plugin, but its purpose is restricted to setting output variables (e.g., for use in flavour templates) for compatibility with the lastmodified plugin. It does not affect the actual caching and validation processes.

Implementation details

Like the lastmodified plugin, this version of the plugin looks for and acts upon the If-modified-since header itself, instead of letting the underlying web server deal with it. Note that the 1.3 and 2.0 versions of Apache in common use today have a feature whereby the underlying web server will handle If-modified-since checks as long as the CGI script simply sets the Last-modified header; this can be used to easily implement simple validation. (Previous versions of this plugin relied on this feature.)

The plugin also looks for and acts upon the If-none-match header. (Apache does not do this for CGI scripts, so the plugin has no choice but to do it itself.) Note that for weak validation (generate_etag set to 1 but $strong set to 0) we generate entity tag values using the date/time modified of the most recent entry, so both the If-modified-since check and the If-none-match check can be done as soon as we compute the Last-modified value, which is done in the filter subroutine. This allows the plugin to save processing time by skipping story processing (i.e., using the skip subroutine) when it does not need to return a full response to a conditional GET with If-modified-since or If-none-match.

For strong validation using ETag ($generate_etag and $strong both set to 1) the ETag value is computed as an MD5 digest of the entire page as it will be returned to the user, in order to distinguish changes that affect even a single bit of the page. We can't skip story processing in this case since we need the complete output (including the results of interpolating variables) in order to compute the correct MD5 digest.

For strong validation using Last-modified ($generate_mod and $strong both set to 1) we also compute an MD5 digest of the entire page as it will be returned to the user, and we compare that value against a cached MD5 digest computed for the page on previous requests. If they match then we know that no changes have occurred since the previous requests, and we set the Last-modified value to the value cached with the MD5 digest. Otherwise we know that some change has occurred since the time of the previous requests, but do not know exactly when that change occurred; we therefore arbitrarily set the Last-modified value to a time just prior to the time of the current request. Note that we can't skip story processing in this case either, since again we need the complete output (including the results of interpolating variables) in order to compute the correct MD5 digest.

Note that for weak validation ($strong set to 0) the Last-modified header does not necessarily provide the date/time at which the actual (bit for bit) contents of the page last changed; instead it provides the date/time at which the meaning of the page last changed, i.e., because the contents of at least one entry on the page were changed. It is possible for other elements on the page such as headers, footers, or comments to change without changing the meaning of the page in this sense, so in this case the Last-modified value is only a "weak validator" as defined by section 13.3.3 of the HTTP 1.1 specification. When $strong is set to 0 the entity tag provided by the ETag header is derived from the Last-modified value and hence is also only a weak validator, and we explicitly mark it as such by prefixing it with "W/", as described in section 3.11 of the HTTP 1.1 specification.

The net effect is that with weak validation if browsers, web caches, and news aggregators caching the page send a conditional GET request (i.e., with an If-none-match and/or If-modified-since header) to check the current status of the page, they will be given a brand new copy of the page only if there have been "semantically significant" changes to the page (in the words of the HTTP 1.1 specification). With strong validation they will get a new copy of the page if the page has changed in any way, no matter how slight.

Generation of the Cache-control and Expires headers is relatively straightforward: We use the value of $freshness_time directly with the max-age directive of the Cache-control header, and add it to the current date/time to create a date/time in the future for the Expires header. If $freshness_time is set to 0 then we instead send the no-cache directive with the Cache-control header and set the Expires header to a date in the past.

Generation of the Content-length header is also straightforward: We simply use the length of $blosxom::output. (This assumes of course that no other plugin will subsequently be changing that output.) Note that for HEAD requests Apache will not actually send the output but will send the Content-length header if set; in that case the Content-length value reflects the length of the output that would have been sent for a GET request, in compliance with section 14.13 of the HTTP 1.1 specification.

Note that the plugin does not generate the Last-modified and Content-length headers for a 304 (Not Modified) response, in accordance with section 10.3.5 of the HTTP 1.1 protocol specification.

Finally, for upward compatibility the lastmodified2 plugin supports the following features present in the original lastmodified plugin:

Optionally checking the %others hash for last-modified dates: Checking %others is one way to detect changes other than changes in the entries themselves; in particular it can be used to detect changes to flavour files used in creating the page. Unfortunately some of the entries in %others are not relevant for the page being created (e.g., flavour files for flavours other than the one currently being generated) and may cause the Last-modified time to be computed incorrectly. Also, checking %others will not detect page changes due to interpolating variables into flavour files (e.g., for comments). Finally, some plugins that replace the default Blosxom entries subroutine (including the entries_cache_meta plugin in particular) do not create the %others hash at all.

For the above reasons this feature is deprecated; you should not use it unless you need it for upward compatibility with your current lastmodified configuration. If you want to check for changes to a page outside the entries themselves then you should simply enable strong validation.
Exporting variables with the last-modified time and other times in RFC 822 and ISO 8601 formats: The lastmodified2 plugin computes these variables essentially in the same way as the lastmodified plugin; see the code and the plugin documentation below for more information.

Note that the variables $latest_rfc822 and $latest_iso8601 always refer to the date/time modified for the most recently updated entry, regardless of whether weak or strong validation is being used. The problem with interpreting $latest_rfc822 or $latest_iso8601 as a Last-modified value is that when using strong validation we wouldn't have values for these variables until we completed generating output for the page, too late for the variables to be of any use.

If you want to use the lastmodified2 plugin to replace an existing configuration of the lastmodified plugin, change the plugin's filename and Perl package name (i.e., in the package statement at the beginning of the code) to "lastmodified" and set the $generate_mod, $generate_etag, and $use_others configurable variables to match your current values. All other configurable variables can be left as is.

Description

This section and the succeeding ones contain more in-depth documentation of the lastmodified2 plugin to supplement the material included in the plugin itself.

The lastmodified2 plugin enables caching and validation of dynamically-generated Blosxom pages by web browsers, web proxies, news aggregators, and other clients by generating various cache-related HTTP headers in the response and supporting conditional GET requests, as described below. This can reduce excess network traffic and server load caused by requests for RSS or Atom feeds or for web pages for popular entries or categories.

The plugin generates an ETag header to identify the particular version of the page, as well as a Last-modified header based on the plugin's determination of when the contents of the page were most recently modified. The plugin also recognizes and properly acts on an If-none-match and/or If-modified-since header in a request, enabling a client to check whether the page has changed since it last requested the page. This reduces network traffic for the site, because the server can skip returning a copy of the page if in fact it has not changed.

The plugin can also optionally generate Cache-control and/or Expires headers to specify how long copies of a page should be retained by caches. This reduces server load for the site, because web proxies and other caching clients can use a cached copy of the page and avoid sending additional requests for the page (including conditional GET requests) to the site's server for as long as the page remains fresh. Alternatively you can use the Cache-control and Expires headers to specify that pages should not be cached at all under any circumstance. This helps ensure that users always get the most up-to-date content, at the expense of increased server load.

Finally, the plugin also generates a Content-length header containing the length in bytes of the content ("entity body" in HTTP 1.1 jargon). Providing a Content-length header supports persistent connections for clients that use the HTTP 1.0 "keep-alive" mechanism (as documented in section 19.7.1 of RFC 2068); this can reduce the number of connections to the site in some cases.

Note that at present this plugin can be used as a replacement for the lastmodified plugin and its default configuration is essentially equivalent to that of the lastmodified plugin, as discussed below.

Installation and configuration

To install the lastmodifed2 plugin copy the plugin file into your Blosxom plugin directory. You should not normally need to rename the plugin; however see the discussion below.

Configurable variables specify how the plugin handles validation ($generate_etag, $generate_mod, and $strong), caching ($generate_cache, $generate_expires, and $freshness_time), whether or not to generate any other recommended headers ($generate_length), and whether to implement features from the lastmodified plugin for compatibility ($use_others and $export_dates).

For validation the most common configurations are the following:

No validation: $generate_etag and $generate_mod both set to 0. The plugin does not generate ETag or Last-modified headers, and does not check If-none-match and If-modified-since headers in the request. Use this configuration if you plan to allow caching of responses (as discussed below) but for some reason you don't want to do validation.
Weak validation: $generate_etag and $generate_mod both set to 1, $strong set to 0. The plugin generates both ETag and Last-modified headers based on the most recent time that any entry on the page was modified; it checks for If-none-match and/or If-modified-since headers in the request, and sends a 304 (Not Modified) response with no output when it can do so. Use this configuration if changes to your pages are only (or at least primarily) due to changes to the entries themselves. This is the default configuration, for compatibility with the lastmodified plugin.
Strong validation: $generate_etag, $generate_mod, and $strong all set to 1. The plugin generates both ETag and Last-modified headers based on the current page's contents and our estimate as to when the contents were last modified; the plugin checks for If-none-match and/or If-modified-since headers in the request, and sends a 304 response when it can do so. Use this configuration if your pages contain comments or other material that is updated more frequently than the entries themselves.
Strong validation using ETag only: $generate_etag and $strong set to 1, $generate_mod set to 0. The plugin generates only an ETag header (not Last-modified) and checks only for an If-none-match header in the request (not If-modified-since). Use this configuration if you want to support strong validation but don't want the performance overhead of caching Last-modified values as previously described. Note that this configuration does not support validation for HTTP 1.0 clients or other clients that do not support validation using If-none-match.

Note that if you set $generate_mod and $strong to 1 then you might as well set $generate_etag to 1 as well, since correctly using Last-modified as a strong validator requires that we generate and cache MD5 digests of the page in order to detect any changes, and these digests are also what we use to generate ETag values.

For caching the most common configurations are the following:

No caching: $generate_cache and $generate_expires both set to 0. The plugin does not generate either a Cache-control or Expires header, and thus web proxies and other clients will typically not cache returned pages. This is the default configuration; use it if you don't care about caching.
Caching allowed: $generate_cache and $generate_expires both set to 1, and $freshness_time set to a positive integer value. The plugin generates Cache-control and Expires headers that allow for caching of returned pages for up to $freshness_time seconds from the time of the request. Use this configuration if you'd like to allow caching by proxies and other clients to reduce server hits due to GET requests (whether conditional or not), and set $freshness_time to a value comparable to the frequency with which your site is updated.

(By default $freshness_time is set to 3,000 seconds, long enough to provide some benefit through caching by web proxies, especially during periods of heavy load, but short enough to ensure that news aggregators doing hourly polling will always use up-to-date copies of feeds.)
Caching prohibited: $generate_cache and $generate_expires both set to 1, and $freshness_time set to 0. The plugin generates Cache-control and Expires headers that specifically prohibit caching of returned pages. Use this configuration if you want all clients to always see the most up-to-date content.

Note that if you set $generate_cache to 1 then you might as well set $generate_expires to 1 and vice versa, in order to properly support both HTTP 1.1 and HTTP 1.0 clients; there is no performance penalty for doing so.

The other configurable variables are as follows:

$generate_length controls whether or not generate a Content-length header. The default is to generate the header; you can disable this by setting $generate_length to 0. Note that support of HTTP 1.0 persistent connections using Content-length requires that your web server be configured to support persistent connections in the first place; for Apache this is done using the KeepAlive On directive in the Apache configuration file.

Also note that HTTP 1.1 clients can use persistent connections even if the Content-length header is not present, if (like Apache) the underlying web server supports HTTP 1.1 persistent connections for CGI scripts using the Connection header and chunked transfer coding. However we generate a Content-length header by default because it's recommended by section 14.13 of the HTTP 1.1 specification.
$use_others controls whether changes to flavour files and other non-entry files in the Blosxom data directory should also be considered semantically significant for weak validation. Note that this feature is provided only for compatibility with the lastmodified plugin and its use is deprecated; by default it is disabled.
$export_dates controls whether or not the plugin should set the following variables for use in flavour templates and other plugins:
- $now_rfc822 and $now_iso8601: Current date/time, in RFC 822 and ISO 8601 formats respectively. These variables can be used in any flavour template.
- $latest_rfc822 and $latest_iso8601: Date/time modified of the most recently modified entry to be displayed on the page, in RFC 822 and ISO 8601 formats respectively. These variables can be used in any flavour template.
- $others_rfc822 and $others_iso8601: Date/time modified of the most recently modified non-entry file in the Blosxom data directory, in RFC 822 and ISO8601 formats respectively. These variables can be used in any flavour template, but are set only if $use_others is set to 1.
- $story_rfc822 and $story_iso8601: Date/time modified of the current entry, in RFC 822 and ISO 8601 formats respectively. These variables can be used in the story and date templates.
Note that the ISO 8601 format produced is the complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ssTZD (e.g., 1997-07-16T19:20:30+01:00).
You can set the variable $debug to 1 or greater to produce additional information useful in debugging the operation of the plugin; the debug output is sent to your web server's error log.

This plugin supplies filter, skip, story, and last subroutines. It needs to run after any other plugin whose filter subroutine changes the list of entries included in the response; otherwise the Last-modified date may be computed incorrectly. It needs to run after any other plugin whose skip subroutine does redirection (e.g., the canonicaluri plugin) or otherwise conditionally sets the HTTP status to any value other than 200. Finally, this plugin needs to run after any other plugin whose last subroutine changes the output for the page; otherwise the Content-length value (and the ETag and Last-modified values, if you are using strong validation) may be computed incorrectly. If you are encountering problems in any of these regards then you can force the plugin to run after other plugins by renaming it to, e.g., 99lastmodified2.

Bugs

Several of the following items are not in fact bugs, but the behaviors in question may cause confusion in some cases; hence their inclusion here:

As discussed above, with weak validation the Last-modified header generated may not always reflect the date/time at which the bit-for-bit contents of the page most recently changed, and the contents of the page may change without changing the ETag value. In particular, if changes are made to flavour files used in generating the page or comments are added to a page via variable interpolation (e.g., as done by the writeback plugin and others) then a user will not necessarily see such changes without forcing an full reload of the page (i.e., using an unconditional GET request). This should be considered a feature and not a bug; if you are not comfortable with this behavior then you should set $strong to 1 to enable strong validation.
When ETag generation is enabled and Last-modified disabled (or vice versa) and a request includes both an If-none-match and If-modified-since header, the plugin will not return a 304 response under any circumstances. This is not a bug, but rather complies with section 13.3.4 of the HTTP 1.1 specification: "An HTTP/1.1 origin server, upon receiving a conditional request that includes both a Last-Modified date (e.g., in an If-Modified-Since or If-Unmodified-Since header field) and one or more entity tags (e.g., in an If-Match, If-None-Match, or If-Range header field) as cache validators, MUST NOT return a response status of 304 (Not Modified) unless doing so is consistent with all of the conditional header fields in the request."

In other words, if a conditional request contains both tests and we can't perform one of the tests (because we're not generating the header value used in the test) then we can't return a 304 regardless of the results of the other test.
If the Cache-control and/or Expires headers are enabled then a user requesting to view a page will not necessarily see updates to that page even if the underlying entries have been changed since the last time the user viewed the page. This should be considered a feature and not a bug; if you are not comfortable with this behavior then you should not enable generation of the Cache-control and/or Expires headers, or you should explicitly prohibit caching by setting the freshness time to 0.
When using the Expires header to prohibit caching, for strict consistency with the HTTP 1.1 specification (section 14.21) the date/time sent with the Expires header should be equal to the date/time sent with the Date header. However we don't necessarily know what the exact Date value is (at least not for Apache, where it is generated by the server itself), and it's possible that the current date/time as measured in the plugin itself may be a little bit later than the time in the Date header, so instead we set the Expires value to be a minute before the current date/time (as measured in the plugin itself).

This should produce correct behavior for HTTP 1.0 clients relying on the Expires header, per section 10.7 of the HTTP 1.0 specification, as well as for HTTP 1.1 clients in the absence of a Cache-control header, per section 14.9.3 of the HTTP 1.1 specification.
As noted previously, if we're doing strong validation using Last-modified and we don't have a cached Last-modified value then we have to make up one; we arbitrarily set it to 5 seconds prior to the current time. Since our value for the current time may be later than that used in the Date header (as noted in the previous item), it's possible that the Last-modified value generated may be in the future relative to that sent with the Date header, especially if the CGI script takes a long time to run (e.g., because of heavy load). This violates the HTTP 1.1 specification (see section 14.29).

The probabability of this happening could be lessened by setting an earlier Last-modified time; however this increases the possibility of having two updates occur within the n-second time window between the ostensible Last-modified time and the current time, and there may be race conditions associated with this that could cause other problems, such as sending a Last-modified value that's earlier than one sent previously for the same URI.
With strong validation using Last-modified it's possible that the plugin may attempt to update the cache file while another plugin invocation (resulting from a simultaneous request) may attempt to read it; more seriously, two plugin invocations may attempt to both update the validator cache file simultaneously. I've tried to minimize problems relating to this by having the plugin write out cache data to a temporary file and then rename it to the real file; if the rename is an atomic operation then this should eliminate the problem of a plugin invocation trying to read from a partially-written validator cache file.

As for simultaneous updates, presumably the worst that can happen is that one of the plugin invocations will fail to update the cache entry for its URI (since its changes will be overwritten by the second plugin invocation); however this simply means that the plugin won't be able to send a 304 on a subsequent conditional GET for that URI, and will then have to update the cache file again.
As noted above, you may experience problems if you install this plugin with other plugins that set HTTP status in the skip subroutine. Blosxom stops executing skip subroutines as soon as one returns a true value, so whichever plugin is first in the execution order will get to set the final HTTP status.

To do

Here are some ideas for ways in which the lastmodified2 plugin could be enhanced and extended:

Support selective use of strong or weak validation depending on the flavour. For example, weak validation would probably work fine for RSS and Atom feeds, since they typically contain content only for entries; however strong validation may be needed for the HTML flavour of individual entry pages (and, to a lesser extent, HTML index pages) in order to pick up changes due to comments.

Note that doing this would be perfectly compatible with the HTTP 1.1 protocol specification, since different flavours correspond to different URIs; any given URI (or set of URIs) could be either strongly validated or weakly validated independent of any other URIs.
Support specifying different freshness times for different types of content, e.g., for different flavours, for individual entries vs. entry index pages, and/or for current index pages vs. archive index pages.
Try to make the filename for the validator cache temporary file more unique to minimize the possibility of name collisions by simultaneous plugin invocations. (Perhaps use Time::HiRes to get subsecond times?)
For completeness, support the case where the If-none-match header has the value '*' (which matches any entity). See section 14.26 of the HTTP 1.1 specification for the desired behavior in this case.
For completeness, support conditional GETs using the If-match and/or If-unmodified-since headers within the plugin itself, in addition to If-none-match and/or If-modified-since. However note that this doesn't appear to be necessary for Apache, since it appears to correctly make these checks as long as ETag and/or Last-modified headers are returned by the CGI script.

Validating and caching dynamic content

2005-01-09T05:25:00Z

One of the things I enjoy about setting up my own blog with the Blosxom software is learning about the deep details of web protocols and formats that I've never worried about before. (This might have been the case if I'd used another blogging system, but the hackable nature of Blosxom inspires, nay, almost demands it.) Lately I've been educating myself about HTTP conditional GET requests and validation and caching of dynamically-generated content.

In this post I discuss the subtleties of validating and caching dynamic content in general, and then in a separate post I tell how I created the lastmodified2 plugin for Blosxom, a rewrite of the lastmodified plugin.

I'm writing this really for my own education more than anything else (under the theory that you don't really understand something until you can explain it), but others may find it useful as well. My goal is to explain how the HTTP protocol actually works in this context (as opposed to just saying "do this" without explaining why) while at the same time avoiding "protocol geekery" that's irrelevant to the problem at hand.

The problem

Suppose that you have a blog (or other web site) whose content is generated dynamically in response to incoming requests. (In other words, you are not using Blosxom in static mode, or another blogging system like MovableType that normally generates static pages.) In practice there are several types of web clients that might access such a site, of which the following are the most common:

web browsers used by humans viewing web pages
web proxy servers supporting browser users by accessing and caching web pages on their behalf
search engines "spidering" a site: downloading pages, following links to find more pages, and indexing the results
news aggregators downloading RSS and/or Atom feeds, typically on a periodic basis

If we generate a full dynamic response for each and every request (as is the case with standard Blosxom, for example) then this produces a lot of network traffic and server load, some of which we could avoid. In the blogosphere most of the attention has been on traffic generated by feed aggregators, but (as pointed out by Charles Miller and others) there's really no need to treat RSS feeds as a special case, at least initially. (Some people have proposed more advanced techniques specifically tailored to RSS/Atom feeds, but these techniques presuppose use of the techniques I describe here.)

The tools at hand

We have three goals in dealing with web clients: to minimize network traffic to and from our site, to minimize server load for our site, and to deliver up-to-date content to the various users of the site; as discussed below, these three goals are often in conflict with each other, but we can usually implement a reasonable trade-off.

We have at least three approaches we can take to help achieve these goals: We can reduce the need for clients to make multiple network connections to the site, we can provide ways for clients to validate whether they need to re-download a page that they've previously downloaded, and we can provide clients "freshness" information telling them how long they can keep copies of pages without having to revalidate them. (There is also a fourth possible approach, namely to use compression techniques to reduce the size of page data returned to the client; I hope to discuss this in a future article.)

Persistent connections

Reducing the number of needed connections can be done through support of so-called "persistent connections". The original HTTP 1.0 protocol required the client to open up a new network connection for each and every request; for an HTML page with lots of included images this might amount to a dozen or more connections, each of which the server has to accept and then close after responding. Support for persistent connections allows a client to open up one connection and make several requests over that connection, either one after the other or nearly simultaneously ("pipelining").

A form of persistent connections was introduced as an extension to HTTP 1.0 and described in section 19.7.1 of RFC 2068, an earlier version of the HTTP 1.1 specification. Using this scheme a client sends a Connection: Keep-Alive header in the request to the server, with the server then keeping the connection open after sending its response. However the client needs some indication from the server as to when the response is actually complete; this is provided by a Content-length header in the response that provides the size in bytes of the response (more correctly, the "entity-body" part of the response, after the headers).

Providing a Content-length value requires determining the length of the output prior to sending it. This is easy to do for static files but more difficult for dynamic content (e.g. CGI output), and hence many content generation tools (including Blosxom and other blogging systems) do not produce Content-length headers by default.

In HTTP 1.1 a new scheme for persistent connections was introduced; in this scheme the response can be broken up into multiple "chunks", with each chunk accompanied by an indication of its size. When Apache is configured to support persistent connections (using the KeepAlive On directive) then it can automatically handle persistent connections for dynamic content without the need for a Content-length header.

Validation

Validation can be done in the HTTP protocol by using a so-called "conditional" GET request, which is in turn implemented by using one or both of two special HTTP headers sent with the request: If-modified-since and/or If-none-match. (The If-modified-since header is supported in both HTTP 1.0 and HTTP 1.1, while the If-none-match header is only in HTTP 1.1. In practice servers can and should recognize and properly handle either of them.)

For example, using the If-modified-since HTTP header a client can tell the server, "Give me this page, but only if it's been modified since 9:08 am on December 17, 2004". This date could represent the last time the client downloaded the page; alternatively it could represent the last time the page was actually modified, as identified in a Last-modified HTTP header returned by the site as part of the response to a previous page request from that client.

Similarly, using the If-none-match HTTP header a client can tell the server, "Give me this page, but only it's different from the version 'foo' I already have". The version is identified using an "entity tag" (or "etag") assigned by the server to each new version of the page; the entity tag value is contained in an optional ETag HTTP header previously returned by the site as part of the response for that page.

Note that some people have suggested using HTTP HEAD requests for validating pages: Send a HEAD request, check the Last-modified or ETag value in the response, and then send a GET request (unconditional) if the page appears to have changed. However this approach is inferior to using a conditional GET, for at least two reasons:

Using HEAD for page validation requires two HTTP requests and responses to accomplish the same purpose as a single conditional GET request and response. This leads to increased network traffic and longer network latency relative to using conditional GETs.
A response to a HEAD request is supposed to contain the exact same HTTP headers as would the response to a GET request for the same URI; the only difference is that a response to a HEAD request doesn't contain any actual content (i.e., an entity-body). For dynamic content this often means that to properly satisfy a HEAD request you have to generate exactly the same content you would for a GET request, only to discard the content after creating the headers; for example, this is true if you're generating the Content-length header, and is also true for certain approaches to generating ETag and Last-modified values, as discussed below. This leads to increased server load relative to using conditional GET requests.

A site based on dynamic content should be able to properly respond to HEAD requests (as required by the HTTP specifications), but should support conditional GET requests as the primary mechanism for page validation.

Freshness and Caching

Validation can reduce the network bandwidth used by your site, since the site does not always need to send back full copies of the pages; however the clients are still hitting the site, if only to validate pages, and this still puts a load on the server. To reduce this load the site can also indicate how long a response should be considered "fresh"--in other words, how long clients can wait before having to return to the site to check for a new version of the page.

Freshness tests can be done in the HTTP protocol using one or both of two special HTTP headers sent with the site's response to a request: Expires and/or Cache-control. The Expires header is supported in HTTP 1.0, while the Cache-control header was introduced with HTTP 1.1; sites can and should support both headers.

The Expires header is like the "use by" date on a perishable item in a grocery store: For example, a site can tell a client, "If you keep a copy of this page, throw it away after 7:00 pm on December 20, 2004; if you need a copy after that please check to see if there's a new version available". The Cache-control header can be used similarly, except expressing the "use by" date in terms of a time relative to the time of the request: "Don't keep a copy of this page longer than 12 hours from now".

Regardless of whether the Expires or Cache-control header is used, the net effect is the same: Clients downloading the page are instructed to keep a copy of the page for a specified period of time and reuse it as necessary during that time, and to avoid contacting the site again to request that page (even with a conditional GET) until that time period is over.

The strategy

Based on the techniques available, a suitable strategy for a dynamically-generated site is then as follows:

If possible, use a web server that supports HTTP 1.1 persistent connections for CGI output. In addition, when sending responses to requests add a Content-length header to identify the total number of bytes in the response, in order to support HTTP 1.0 persistent connections.
When sending responses to requests, add an ETag header to identifiy the "version number" (entity tag) for this particular version of the page, and/or a Last-Modified header to identify the date/time the page was last modified.
When sending responses to requests, also add Cache-control and Expires headers to the response to provide a "use by" date/time to clients doing caching.
When processing requests, look for the If-none-match and If-modified-since headers. If one or both are present, return the full page in the response only if necessary: if the version of the page currently available is different than the version requested in the If-none-match header, or if the page has been modified since the date in the If-modified-since header.

What's new?

The strategy outlined above seems simple enough, but we've glossed over a crucial and surprisingly difficult question: How do we determine if and when a page has changed and a new version has been created?

The creators of the HTTP protocol specification suggested two different approaches to this question, with two correspondingly different ways to use the ETag header. (For various reasons too geeky to go into, ETag is a better example here than Last-modified.)

The strict approach is to consider a page to be changed if even one bit on the page changes. For example, in the context of Blosxom if you made even a single-character correction to a flavour template then any page using that template would be considered to have changed. When sending an ETag header for such a page you would then be duty-bound to update the entity tag value identifying the version for that page. Under this approach the ETag header is considered to be a "strong validator" in HTTP jargon.

A looser approach is to consider the page to be changed only if the essential "meaning" of the page changes, where you as the site author get to decide what that meaning actually is in this context. For example, if you are primarily concerned with RSS feeds then you might decide that the response sent to news aggregators and other clients should be considered changed only if there were content changes to any of the entries included in the response. You would then be free to keep the entity tag value sent in the ETag header the same as long as the underlying entries didn't change; here you're using the ETag header as a "weak validator".

Having a strong validator as described above is important in cases where knowing about bit-level changes is absolutely required. The most common example of this is downloading large binary files where the download might be interrupted for some reason and the client wishes to resume from the point at which it was interrupted (as opposed to restarting the download from the beginning). For this purpose HTTP provides a mechanism whereby clients can request a range of bytes for a resource, so that (for example) a client can tell the server "give me bytes 737878-1643324 for this resource (I already have the others)".

In order for this to work properly, when the client goes back to the server to pick up the rest of the file the client has to know that the version of the file for which it's getting the new set of bytes is exactly the same as the version for which it got the first (interrupted) set of bytes; otherwise it will end up with a corrupted copy. The ETag header can provide the necessary version information, but only if it's a strong validator as described above.

However using the etag as a weak validator is arguably a better approach for a typical blog, both because it better fits the nature of the content (most people care more about the prose content of the page than about the exact bytes making up the page) and also because correctly implementing strong validation for dynamic content can be more difficult and time-consuming (in terms of server load), at least for Blosxom.

However implementing weak validation has its own problems as well. In particular, for Blosxom there are changes which at least for some would arguably change the "meaning" of a page but which can be difficult to detect in practice without checking for byte-for-byte changes; the most important examples are having new comments for an individual entry's page or a new number of comments for an entry listing on an index page. In these cases the change is typically introduced through interpolating variables when processing flavour templates, and hence you can't use the date/time modified for either the entry file or the flavour template as a guide to when the change occurred.

(You could potentially look at the date/time modified for comment-related files as stored in the Blosxom plugin state directory or elsewhere, but this would require knowing exactly what plugin is being used to generate comments, and how it stores comment-related information. This is one of the drawbacks to Blosxom's minimal approach to blogging, in which a comments capability isn't a standard feature of the software but has to be implemented by add-on software, with different sites using different comments plugins.)

I discuss this and other implementation issues in more detail in my next post.

For more information

Here are some useful reference documents and related material I consulted while researching the issue of validating and caching dynamic content in the course of creating the lastmodified2 plugin. The main documents of interest are the following:

"Caching Tutorial for Web Authors" by Mark Nottingham is the best introductory document I've found on page validation and caching.
The HTTP 1.1 protocol specification (RFC 261, "Hypertext Transfer Protocol -- HTTP/1.1") is the ultimate authority for how validation and caching should work. See in particular sections 10.3.5 ("304 Not Modified"), 13 ("Caching in HTTP"), 14.9 (Cache-control header), 14.13 (Content-length header), 14.19 (ETag header), 14.21 (Expires header), 14.25 (If-modified-since header), and 14.26 (If-none-match header).

You may also find the following documents of interest:

"Conditional GET for RSS Hackers" by Charles Miller is a basic tutorial on implementing conditional GETs in the context of a blog. However it lacks an in-depth discussion of strong vs. weak validation and why the distinction matters.
The post "Joel's RSS Problem" on Phil Ringnalda's blog is a good example of various views on how to address the problem of blog's being overloaded by aggregator requests, including links to related blog posts and articles.
Chapter 16 of the book Practical mod_perl has some good in-depth information on the issue of validation and caching of dynamic content and strong vs. weak validators.
Though it's been superceded by RFC 2616, the original version of the HTTP 1.1 protocol specification (RFC 2068, "Hypertext Transfer Protocol -- HTTP/1.1") is worth consulting for its description (in section 19.17) of the HTTP 1.0 "Keep-Alive" extension for persistent connections.
The original HTTP 1.0 protocol specification (RFC 1945, "Hypertext Transfer Protocol -- HTTP/1.0") is mainly of historical interest. (The HTTP 1.1 specification addresses backwards compatibility for HTTP 1.0 clients.)

Emptymessage patch for Apache compatibility, etc.

2005-01-08T13:15:00Z

When stock Blosxom sees a URL that doesn't correspond to an existing entry or list of entries, it simply puts up a "normal" page (i.e., using the standard heat and foot templates for that flavour) that doesn't have any actual content. I really don't like this behavior, and thus I decided to try out the emptymessage plugin created by Fletcher Penney. Unfortunately I wasn't entirely happy with its behavior either, and so I decided to patch it.

First, I didn't like the standard "404 Not Found" message produced by the plugin; it was reminiscent of the standard message produced by Apache, but different enough that it got on my nerves. My first patch was therefore to make the 404 message produced by the emptymessage plugin look exactly like the standard message produced by Apache 2.0, even down to including the same information about the server version, etc. (You can compare for yourself by trying a bogus URL for my blog vs. another bogus URL for a static site hosted on the same system.)

My second patch wasn't to fix an actual problem but rather to make the emptymessage plugin work better with another plugin I'm creating that also wants to set the HTTP Status header in the response. In the original emptymessage code the HTTP status is set by doing an actual print operation to output the Status header; unfortunately doing it this way means that other plugins have no way of detecting that the status was thus set, in case they want to modify their own behavior.

To address this problem I patched the emptymessage plugin to set the status header using $blosxom::header->{'Status'}; other plugins can then check that variable to determine what the HTTP status will be. At the same time I also changed the emptymessage plugin to produce a Content-length header (by setting $blosxom::header->{'Content-length'}) for further compatibility with Apache.

The above patches worked fine in testing (which I do using a stand-alone Apache server located on my laptop) but when I deployed the emptymessage plugin to my production site it didn't seem to be working at all: The returned response for an empty/non-existent page was exactly the same as if the plugin were not installed.

After doing some debugging I discovered the problem: The data directory on my production site has a period in the pathname (e.g., /usr/local/foo.bar) and the emptymessage plugin was mangling the pathname in its attempts to take the Blosxom $currentdir variable, prepend the data directory pathname, and then strip off any flavour suffix (i.e., after a '.'). I patched this by changing the order of operations so that the flavour suffix was stripped before prepending the data directory.

For more details on the above changes see the patch itself.

A final note: The emptymessage plugin still doesn't address one important case, namely when a request is made for a date-based archive page (e.g., http://www.example.com/blog/2005/01) and there are no entries for that date. I may look at providing a patch for that later.