L'Alpiniste

Getting an Early Warning of Low Disk Space

2010-01-21T23:51:57Z

Keeping an eye on Disk Space

Running out of disk space is not a good situation to be in. Services that run out of disk space can cause your server to blow up in a very short space of time. As a result of that it's a good idea to get an early warning that your machine is on it's way to running out of disk space. This script is a noddy way to get that insight with minimum effort.

# -*- coding: utf-8 -*-

"""
Disk Space
Simple class for working out the free disk space on a system

by Stuart Colville http://muffinresearch.co.uk
License: http://www.opensource.org/licenses/mit-license.php

"""

import os
import math

class DiskSpace():

    """Free Disk Space"""

    def __init__(self, path="/"):
        """Init class and retrieves disk space info"""
        self.disk = os.statvfs(path)

    def has_free_space(self, limit):
        """Bool returns true if remaining space is above limit %"""
        if float(self.percent_free()) < float(limit):
            return False
        else:
            return True

    def percent_free(self):
        """Gets the amount of space left as a percentage"""
        return (math.ceil(float(100) / float(self.bytes_capacity())
                                                * self.bytes_free()))

    def bytes_capacity(self):
        """Returns the total capacity in bytes"""
        return self.disk.f_frsize * self.disk.f_blocks

    def bytes_free(self):
        """Returns the free space in bytes"""
        return self.disk.f_frsize * self.disk.f_bavail

    def bytes_used(self):
        """Returns the used space in bytes"""
        return self.disk.f_frsize * (self.disk.f_blocks -
                                                   self.disk.f_bavail)

    @staticmethod
    def humanize_bytes(bytes, kilo=1024):

        """Humanizes bytes

        See http://en.wikipedia.org/wiki/Kilobyte for info on the
        different ways to interpret whether a kilobyte is 1,024 bytes
        or 1,000 bytes

        """

        if kilo == 1024:
            label = ["KiB", "MiB" , "GiB", "TiB"]
        else:
            # fallback
            kilo = 1000
            label = ["KB", "MB" , "GB", "TB"]

        if bytes < kilo:
            return "%d" % bytes
        elif bytes > kilo and bytes < math.pow(kilo, 2):
            return "%.2F%s" % (float(bytes/kilo), label[0])
        elif bytes > math.pow(kilo, 2) and bytes < math.pow(kilo, 3):
            return "%.2F%s" % (float(bytes/math.pow(kilo, 2)), label[1])
        elif bytes > math.pow(kilo, 3) and bytes < math.pow(kilo, 4):
            return "%.2F%s" % (float(bytes/math.pow(kilo, 3)), label[2])
        elif bytes > math.pow(kilo, 4) and bytes < math.pow(kilo, 5):
            return "%.2F%s" % (float(bytes/math.pow(kilo, 4)), label[3])

Example

Here's a quick example of how this can be used in practice

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""Simple example command line usage:

Usage: ./check_disk_space.py <percentage free space>

e.g:

$ ./check_disk_space.py 30
Running low on disk space. 20.0% remaining (Warning Threshold: 30%)
Total: 115.13GB
Used:  92.35GB
Avail: 22.77GB

The idea would be to put this in a Cron Job and have it email you when
the free disk space is lower than "n" percent.

"""

import sys

sys.path.insert(0, '../')

from diskspace import DiskSpace

ds = DiskSpace()

perc_arg = len(sys.argv) > 1 and sys.argv[1] or None
percent_limit = perc_arg or 20

if not ds.has_free_space(percent_limit):
    print "Running low on disk space. %s%% remaining "\
       "(Warning Threshold: %s%%)" % (ds.percent_free(), percent_limit)
    print "Total: %s" % ds.humanize_bytes(ds.bytes_capacity(), 1000)
    print "Used:  %s" % ds.humanize_bytes(ds.bytes_used(), 1000)
    print "Avail: %s" % ds.humanize_bytes(ds.bytes_free(), 1000)

This last file can be saved as check_disk_space.py and made executable with:

chmod +x check_disk_space.py

All you then need to do is add a crontab entry and define the limit at which you want to be warned of impending disk space running out.

MAILTO=systems@yourdomain.com
0 */2 * * * /path/to/file/free_disk_space 15

This example will check the space every 2 hours and will email you when the remaining disk space drops below 15%.

A Branch of these files is available here: http://code.projectfondue.com/diskspace/trunk/files/head:/diskspace/

Suggestions for improvement are always welcome.

Loggerhead and mod_wsgi

2009-11-27T09:14:52Z

Currently loggerhead defaults to using pasteserver out of the box, which means you have to proxy this with Apache if you don't want to use pasteserver directly. Instead of doing that we went the route of setting up Loggerhead under apache with mod_wsgi.

Here's the wsgi file to set-up the application.

BZR_SERVE_DIR = 'THE_DIR_CONTAINING_YOUR_BRANCHES_HERE'

import sys
import os
sys.path.insert(0, os.path.dirname(__file__))

from paste.httpexceptions import HTTPExceptionHandler
from loggerhead.apps.transport import BranchesFromTransportRoot
from loggerhead.apps.error import ErrorHandlerApp
from loggerhead.config import LoggerheadConfig
from paste.deploy.config import PrefixMiddleware
from bzrlib.plugin import load_plugins

load_plugins()
config = LoggerheadConfig()
app = BranchesFromTransportRoot(BZR_SERVE_DIR, config)
app = PrefixMiddleware(app, prefix='/')
app = HTTPExceptionHandler(app)
application = ErrorHandlerApp(app)

I save this file as loggerhead.wsgi in the checkout of loggerhead.

Next all you need is a basic apache config and a WSGIAlias directive pointing at the wsgi file. Obviously it goes without saying this example is just for illustrative purposes.

<VirtualHost *:80>
    ServerName loggerhead

    WSGIScriptAlias / /PATH_TO_LOGGERHEAD/loggerhead.wsgi

    <Directory /PATH_TO_LOGGERHEAD>
        Order allow,deny
        Allow from all
    </Directory>

</VirtualHost>

Note

The following assumes you have a dns entry for whatever value you give ServerName - In my example I use loggerhead which I have set-up on my local machine to point to 127.0.0.1 so I have a local loggerhead running to view my checkouts.

Change the values of the paths and the ServerName as necessary
Save this file as loggerhead (or whatever you want to call it) in /etc/apache2/sites-available

Then all you need to do is enable it with

sudo a2ensite loggerhead
sudo /etc/init.d/apache2 force-reload

That's it!

You can view our code repository at http://code.projectfondue.com/ and a branch for these files is here: http://code.projectfondue.com/loggerhead-wsgi/trunk/files

Python script for Text Link Ads

2009-11-04T15:25:28Z

To counter this I've written a simple Python script which replicates the functionality of their PHP script. It should be pretty solid and we've been using it on one of our sites for a while but, as this is the first bit of proper Python I've written, please go easy on me if you spot mistakes!

Usage

The script is pretty easy to use. Start by including the library.

import tla

The quickest way to test out the library is probably with the python command line interpreter. This will allow you to introspect its properties and methods with out initially worrying about how to make it sit alongside your own code.

Next create an instance of the TLA class passing the inventory code that was supplied to you by TLA.

tla = TextLinkAds('ENTER_TLA_INVENTORY_KEY_HERE')

The class has methods to supply the raw data or that same data formatted as an HTML unordered list.

data = tla.get_data()

print tla.get_html()

If you're using a templating engine such as jinja2 you'll probably need to make sure you mark the TLA output as "safe" to prevent to the engine from escaping the HTML tags returned.

Requirements

The script uses a couple of external libraries - httplib2 and memcache - httplib2 is required but memcache is optional and is, by default, disabled.

You can install both libraries with the following:

sudo easy_install python-memcached httplib2

The script also write temporary files to a directory specified within the script (by default /tmp). You'll need to make sure that it has permissions to write to this directory or change it to one it can write to.

Source Code

The source code for the library is available from our public bazaar repository. You can also download a zip containing all files and we'll be adding it to PyPi shortly.

Unit Tests

Stuart's written a number of unit tests for the library. Test coverage currently stands at around 70% which is pretty good. We'll obviously be looking to expand this coverage in due course.

Learning Python

Being new to Python, I learned quite a lot whilst writing the script. Stuart and Cyril provided plenty of suggestions which resulted in a number of rewrites. Much of this feedback was gathered via a lengthy code review, managed with Google's excellent code review tool - rietveld.

Here's a few nuggets of information you might find useful, if like me, you're just starting out.

httplib2 is much better than urllib2. It does far more out of box resulting in shorter, cleaner code.
Python provides excellent facilities for logging information - be it to a text file, by email or over HTTP. I found it a little confusing initially but perseverance definitely pays off. The Python docs have detailed information about what's possible and of course my script demonstrates relatively simple but effective usage.
Python uses Exceptions for every type of error. This may seem a little strange if you're used to PHPs rather hectic approach to handling errors. You'll need to get used to using try..except blocks everywhere in Python. Also, be careful to be explicit about what errors your catching. Naked excepts (those that catch every conceivable error) make it harder to track down problems later.
Building up a string by appending to a list which you later join is a common idiom in Python. As strings are immutable, concatenation results in a copy of the string being created every time.
Python supports calling functions and methods with named parameters as well as **kwargs - a dictionary from which named function parameters can be retrieved without the need for them to form part of the actual function definition. I wish PHP supported similar functionality.

Google Chrome Frame — good or bad for the web?

2009-10-02T20:35:38Z

Here is Google's description of what Chrome Frame is:

Google Chrome Frame is an early-stage open source plug-in that seamlessly brings Google Chrome's open web technologies and speedy JavaScript engine to Internet Explorer. With Google Chrome Frame, you can:

Start using open web technologies - like the HTML5 canvas tag - right away, even technologies that aren't yet supported in Internet Explorer 6, 7, or 8.

Take advantage of JavaScript performance improvements to make your apps faster and more responsive.

—http://code.google.com/chrome/chromeframe/

I'm intrigued by Google Frame; on one hand it's a pretty neat way of providing a more advanced experience to users of Internet Explorer, but I'm also concerned that it's providing a prop to browsers that really should be moving towards deprecation. If a browser isn't capable of displaying a site wouldn't it make more sense to point people in the direction of browsers that are more advanced?

Is this also saying that Google Wave will need Chrome Frame because it will only be meaningful to users with the most up to date technology. If you aren't up to date is there no base-line experience? If that's the case, a future that ignores progressive enhancement has a nasty aroma to it.

Update: 24th September 2009

According to this post on the Google Wave Blog, it seems IE 6,7 and 8 users will not be able to use Google Wave unless they install the Google Frame Plugin.

Google Frame and Accessibility

Early investigations by Steve Faulkner have shown that the inclusion of Google Frame locks out users of assisitive technologies completely.

Clearly this is an early release and assistive technology support could be improved in a future release. Whilst I'm happy to be proved wrong, I somehow doubt this will change given the epic technical challenges involved. That said, if anyone can do it, Google is a company that is best placed to make it happen.

Elsewhere

Apache: Moving from prefork to worker

2009-08-26T12:59:58Z

Differences between MPM Versions

So what are the differences between Apache MPM ("multi-processing module") types?

MPM Prefork doesn't use threads and instead spawns child processes to serve requests. MPM Worker on the other hand uses multi-threaded processes where each child process utilises a number of threads:

By using threads to serve requests, it is able to serve a large number of requests with less system resources than a process-based server. Yet it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.

—Apache 2 Documentation

Something to bear in mind is that you probably won't want to switch to worker if you are using PHP unless you are ok with using PHP as FastCGI and even then the PHP manual doesn't recommend using MPM worker.

For us that means we are now running our PHP apps using prefork and mod_php on a box dedicated to running PHP. This is better in a lot of ways because it means we can tune Apache for each specific application rather than running a mixture of PHP and Python apps on the same hardware.

Converting to mpm-worker

Actually carrying out the conversion on an Ubuntu box is as simple as running

sudo apt-get install apache2-mpm-worker

That will remove prefork if you have it installed and will install worker in its place and trigger a restart. That's all there is to it. (Though of course YMMV)

What Difference does it make?

In our case used memory levels dropped a reasonable amount as you can see in the last 6th of this graph:

CSS Sprite Generator v4.0 available on Launchpad

2009-09-01T11:03:28Z

This is useful for sprites which contain gradients that need to be repeated in a specific direction. There's also been some suggestion, particularly from the Yahoo! performance guys, that building sprites horizontally results in a smaller overall image size.

Arranging the images in the sprite horizontally as opposed to vertically usually results in a smaller file size.

—Yahoo! Exceptional Performance Team

Other Changes

Generated CSS

Displaying the generated CSS within a textarea field (rather than a scrolling div) to enable select all and easier copy and paste.

Component Spacing

A reduction in the default spacing (horizontal and vertical) between component images in the sprite. This makes the tool kinder to our servers and means, with a default generation, the browser has to handle a smaller uncompressed image in memory. This should be of particular benefit to the iphone which doesn't seem to handle large sprites too well.

"Be mobile-friendly" and don't leave big gaps between the images in a sprite. This doesn't affect the file size as much but requires less memory for the user agent to decompress the image into a pixel map. 100x100 image is 10 thousand pixels, where 1000x1000 is 1 million pixels

—Yahoo! Exceptional Performance Team

There's a balance to be had here, of course. The usual reason for including space between component images is to allow for text sizes increases which would otherwise result in partial display of the next image in the sprite. You can see how this would be a particular problem on something like the navigation of the Yahoo! home page.

Language Selection

Conversion of the language selection interface from a drop down box to a list of links to make it easier for someone to switch to their native language, particularly where they don't understand the currently displayed language. There's an interesting analysis of the use of drop downs versus links in the W3C i18n FAQ with the relative benefits and pitfalls of each approach.

Looking Forward

We've got a few other bugs that need fixing. We'll be working on those shortly and, in due course, we'll also be developing a command line version of the tool. This should make it easy to incorporate into an automated build script. Check back regularly for further details and do drop us a line if you've got any suggestions.

Get the Code

You can get the source code here on Launchpad. As always comments, suggestions, bugs and patches are welcomed.

Try Version 4 Online

CSS Sprite Generator version 4 is available online where you can dive right in and try out the latest features.

Raclette: Web App Utils for Python

2009-07-14T10:58:36Z

The goals for Raclette are to provide well tested middlewares and libraries that can be used to facilitate web development. We're also looking to keep the quality of code at a high standard as much as possible whilst balancing that with getting code out there so developers can try it out and provide feedback.

raclette.gbs: Graded Browser Support Middleware

The first app off the blocks is a WSGI middleware which provides a solution to implementing Yahoo's Graded Browser support strategy see http://developer.yahoo.com/yui/articles/gbs/ for more information.

Installation

sudo easy_install raclette.gbs

To use the middleware simply wrap the init of the middleware around your WSGI application

from raclette.gbs import GradedBrowser
...
application = GradedBrowser(application)

raclette.gbs works by analyzing the user-agent string and returning a grade stating that the browser is C, A or X grade. Yahoo's own strategy only provides details of which browser's are A grade so the C grade matrix is pretty much created based on guess-work around the notion that C-Grade browsers should be Antiquated and rare.

Ultimately for the purposes of this middleware only C-grade browsers really need to be identified as A and X grade are generally treated the same. However to be able to give the end user more scope to do different things with browser grades in development or production C and A grades are explicitly identified. It's up to the user to modify the browser matrix should they wish to add or drop support for any particular browser.

Known Issues

There's clearly some work to be done in terms of extending the data used to provide the decisions on which browsers are and aren't supported. With that in mind we're happy to get feedback on which browsers you think need to be indentified. Clearly it doesn't have to be something that's in Yahoo's matrix of support as it's clear that anyone using this may wish to define their own matrix of support.

Caching is not currently implemented. As the UA string is parsed on every request it's fairly crucial that caching is added - expect several caching solutions to be made available in subsequent releases.

Future Possibilities

Something that came up early on was an idea around extending support to include some notion of graded mobile browser support in the sense that you might have MA, MX and MC. At this point it's not clear if having any separate grades for mobile devices is a good thing or not, but certainly being able to detect common mobile browsers would be useful.

Getting the code

The code is available on launchpad.net at https://launchpad.net/raclette.gbs -- Contributions and patches gratefully received. All code is licensed under the modified BSD license.

Welcome!

2009-07-12T10:48:31Z

We've been hanging out in Paris for the weekend for a mini sprint/planning session. Naturally we made a point of getting out for Fondue which as always was pretty awesome.

One of the things we've been up to is spending some time bugfixing and putting out the v3.0 release of CSS Sprite Generator which Ed's been working on recently. Ed's added a feature for Vertical and Horizontal layout of sprites which has been requested a few times. Translations are now available via dedicated subdomains which is a distinct improvement. In addition a Russian Translation contributed by Pavel Sorokin and a Turkish Translation provided by Kerem Can Caliskan are now live.

The time in Paris gave us a time to finalise this blog and we've also planned out future changes to our architecture and systems which has been invaluable to help keep everything running smoothly whilst minimising all manual repetitive tasks to next to nothing.

We'll have plenty to talk about in the future; from the details on development of existing applications, as well as exciting things that are coming up. Most importantly for us is that this can be a two-way conversation and we look forward to hearing your thoughts, comments and suggestions on everything at Project Fondue.

L'Alpiniste

Getting an Early Warning of Low Disk Space

Keeping an eye on Disk Space

Example

Loggerhead and mod_wsgi

Python script for Text Link Ads

Usage

Requirements

Source Code

Unit Tests

Learning Python

Google Chrome Frame — good or bad for the web?

Google Frame and Accessibility

Elsewhere

Apache: Moving from prefork to worker

Differences between MPM Versions

Converting to mpm-worker

What Difference does it make?

Related Reading

CSS Sprite Generator v4.0 available on Launchpad

Other Changes

Generated CSS

Component Spacing

Language Selection

Looking Forward

Get the Code

Try Version 4 Online

Raclette: Web App Utils for Python

raclette.gbs: Graded Browser Support Middleware

Installation

Known Issues

Future Possibilities

Getting the code

Welcome!