I realized I haven’t really explained anything yet, so let’s look at a suitably contrived example.

A network manager

Suppose you’re writing a network manager type of application (I actually tried that once). You might have a class called NetworkIface. And the class has an attribute ip. So how does ip get its value? Well, it can be set statically, or via dhcp. In the latter case there is a method dhcp_request, which requests an ip address and assigns to ip.

# <./main.py>
class NetworkIface(object):
    def __init__(self):
        self.ip = None
 
    def dhcp_request(self):
        self.ip = (10,0,0,131) # XXX magic goes here
 
 
if __name__ == '__main__':
    iface = NetworkIface()
    iface.ip = (10,0,0,1)
    iface.ip = (10,0,0,2)
    iface.dhcp_request()

Download this code: main.py

Now suppose you are in the course of writing this application, and you need to do some debugging. It would be nice to know a few things about NetworkIface:

1. The dhcp server seems to be assigning ip addresses to clients in a (possibly) erroneous manner. We’d like to keep a list of all the ips we’ve been assigned.
2. Sometimes the time between making a dhcp request and getting a response seems longer than reasonable. We’d like to time the execution of the dhcp_request method.
3. Some users are reporting strange failures that we can’t seem to reproduce. We would like to do exhaustive logging, ie. every method entry and exit, with parameters.

Now, this kind of debugging logic, however we realize it, is not really something we want in the release version of the application. It doesn’t belong. It belongs in debug builds, and we’re probably not going to need it permanently.

Here we will demo how to achieve the first point and omit the other two for brevity.

Where AOP comes in

Common to these issues is the fact that they all have to do with information gathering. But that’s not necessarily the only thing we might want to do. We might want to tweak the behavior of dhcp_request for the purpose of debugging. For instance, if it took too long to get an ip, we could set one statically after some seconds. Again, that would be a temporary piece of logic not meant to be in the release version.

Now, AOP says “don’t change your code, you’ll only make a mess of it”. Instead you can write that piece of code you need to write, but quite separately from your codebase. This you call an aspect, with the intention that it captures some aspect of behavior you want to inject into your code. And then, during compilation from source code to bytecode (or object code) you inject the aspect code where you want it to go. Compiler? Yes, AOP comes with a special compiler, which makes injection very toggable. Want vanilla code? Use the regular compiler. Want aspected code? Use the AOP compiler.

How does the compiler know where to inject the aspect code? AOP defines strategic injection points called join points. Exactly what these are depends on the programming language, but typically there is a join point preceding a method body, a join point preceding a method call, a method return and so on. (As we shall see, in aopy we are being more Pythonic.) Join points are defined by the AOP framework. But how do you tell it to inject there? With point cuts. A point cut is a matching string (ie. regular expression) which is matched against every join point and determines if injection happens there.

Back to you, John

Enough chatter, the code is getting cold! As it happens, Python has first rate facilities for writing AOP-ish code. We already have language features that can modify or add behavior to existing code:

• Properties let us micromanage assignment to/reading from instance variables.
• Decorators let us wrap function execution with additional logic, or even replace the original function with another.
• Metaclasses can do just about anything to a class by rebinding the class namespace arbitrarily.

We will use these language constructs as units of code injection, called advice in AOP. This way we can reuse all the decorators and metaclasses we already have and we can do AOP much the way we write code already. Let’s see the aspects then.

A caching aspect

The first thing we wanted was to cache the values of ip. For this we have a pair of functions which will become methods in NetworkIface and make ip a property.

# <aspects/cache.py>
class Cache():
    def __init__(self):
        self.values = set()
        self.value = None
cache = Cache()
 
def get(self):
    return cache.value
 
def set(self, value):
    if value:
        print "c New value: %s" % str(value)
    if any(cache.values):
        prev = ", ".join([str(val) for val in cache.values])
        print "c  Previous values: %s" % prev
    if value:
        cache.values = cache.values.union([value])
    cache.value = value

Download this code: cache.py

Cache is the helper class that will store all the values.

A spec

Aspects are defined in specification files which provide the actual link between the codebase and the aspect code.

# <./spec.py>
import aopy
 
import aspects.cache
 
caching_aspect = aopy.Aspect()
caching_aspect.add_property('main:NetworkIface/ip',
    fget=aspects.cache.get, fset=aspects.cache.set)
 
__all__ = ['caching_aspect']

Download this code: spec.py

We start by importing the aopy library and the aspect code we’ve written. Then we create an Aspect instance and call add_property to add a property advice to this aspect. The first argument is the point cut, ie. the matching string which defines what this property is to be applied to. Here we say “in a module called main, in a class called NetworkIface, find a member called ip“. The other two arguments provide the two functions we wish to use in this property.

Compiling

To compile the aspect into the codebase we run the compiler, giving the spec file. And we give it a module (or a path) that indicates the codebase.

$ aopyc -t spec.py main.py
Transforming module /home/alex/uu/colloq/aopy/code/main.py
Pattern matched: main:NetworkIface/ip on main:NetworkIface/ip

Download this code: main.compile

The compiler will examine all the modules in the codebase (in this case only main.py) and attempt code injection in each one. Whenever a point cut matches, injection happens. The transformed module is then compiled to bytecode and written to disk (as main.pyc).

main.pyc now looks like this:

# <./main.py> transformed
import sys ### <-- injected
for path in ('.'): ### <-- injected
    if path not in sys.path: ### <-- injected
        sys.path.append(path) ### <-- injected
 
import aspects.cache as cache ### <-- injected
 
class NetworkIface(object):
    def __init__(self):
        self.ip = None
 
    def dhcp_request(self):
        self.ip = (10,0,0,131) # XXX magic goes here

    ip = property(fget=cache.get, fset=cache.set) ### <-- injected
 
 
if __name__ == '__main__':
    iface = NetworkIface()
    iface.ip = (10,0,0,1)
    iface.ip = (10,0,0,2)
    iface.dhcp_request()

Download this code: main_t.py

Injected lines are marked. First we find some import statements that are meant to ensure that the codebase can find the aspect code on disk. Then we import the actual aspect module that holds our advice. And finally we can ascertain that NetworkIface has gained a property, with get and set methods pulled in from our aspect code.

Running aspected

When we now run main.pyc we get a message every time ip gets a new value. We also get a printout of all the previous values.

c New value: (10, 0, 0, 1)
c New value: (10, 0, 0, 2)
c  Previous values: (10, 0, 0, 1)
c New value: (10, 0, 0, 131)
c  Previous values: (10, 0, 0, 1), (10, 0, 0, 2)

Download this code: main.output

And the yet the codebase has not been touched, if we execute main.py instead we find the original code.

Here the show endeth

And that wraps up a hasty introduction to AOP with aopy. There is a lot more to be said, both about AOP in Python and aopy in particular. Interested parties are kindly directed to these two papers:

1. Strategies for aspect oriented programming in Python
2. aopy: A program transformation-based aspect oriented framework for Python

If you prefer reading code rather than English (variable names are still in English though, sorry about that), here is the repo for your pleasure:

• aopy repo

And if you still have no idea what AOP is and think the whole thing is bogus then you can watch this google talk (and who doesn’t love a google talk!) by mr. AOP himself.

For example, look at that handsome guy there on the right. Is that a gorgeous graph, or what? (Incidentally, if you aren’t yet a fan of graphviz, it’s time to become one, it’s superb!)

Now take a gander at the code snippet below, that’s the python representation of it.

class Node(object):
    def __init__(self, name):
        self.name = name
 
a, b, c, d = Node('a'), Node('b'), Node('c'), Node('d')
a.refs = [b, d]
b.refs = [c]
c.refs = [a]
d.refs = []

Download this code: python_graphtraversal_model.py

But it’s not there just to be stared at, so let’s do something with it! One thing we could do is go over the graph and print out the nodes. But let’s do one better, let’s also show how deep into the graph we are by indenting the output! What we want is this:

a
__b
____c
__d

Download this code: python_graphtraversal_spec.py

We don’t have the nice arrows here, but you can still make out the shape of the graph.

First iteration

def traverse_naive(node, depth):
    print('__'*depth + node.name)
    for ref in node.refs:
        traverse_naive(ref, depth+1)

Download this code: python_graphtraversal_naive.py

Very simple. Print the name of the node at the current level of depth, and then recurse down the outgoing edges. But when we run this something bad happens:

>>> traverse_naive(a, 0)
a
__b
____c
______a
________b
..
RuntimeError: maximum recursion depth exceeded

Download this code: python_graphtraversal_naive_output.py

There’s no sign of d, instead we keep going round and round along the path a-b-c-a. Woops.

Preempting cycles

Traditionally, graph traversal algorithms have set properties on nodes in the graph to indicate to themselves that a particular node had been seen before. For example, we could set node.been_here_before = True every time we enter a node, and then we could check for this property to make sure we don’t re-enter the node later.

But this is not awesome, because we then have to change the graph as we traverse it. What if we want to traverse it again later, do we then need another traversal algorithm to remove all the markers or what?

There is another way to do this, however. We can use a data structure completely outside the graph in which we keep track of what we’ve seen so far (imagine a guy walking around a warehouse with a clipboard, he then doesn’t have to mark any of the merchandise!).

So instead of checking the value of node.been_here_before, we’re going to check the value of cache[node].

Now, there are two main strategies for where to do this check. We can do it right before the recursive call, or right after.

def traverse_check_before(node, depth, cache):
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        # we are about to recurse, first check if the node we want
        # to recurse on is in the cache
        if ref in cache:
            print('Already in cache: %s' % ref.name)
            continue
        # recurse if we reach this point
        traverse_cached(ref, depth+1, cache)
 
 
def traverse_check_after(node, depth, cache):
    # we have just recursed, return if this node is already in
    # the cache
    if node in cache:
        print('Already in cache: %s' % node.name)
        return
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        # recurse without exception
        traverse_cached(ref, depth+1, cache)

Download this code: python_graphtraversal_two.py

Here it doesn’t matter which one we use, so we’re just going to use the second option.

Second iteration

At this point you might be thinking that everything is going swimmingly, but in fact a small problem has crept up. It cannot have escaped your attention that we’ve snuck another parameter into that function definition. This means that we have to call the function like this:

>>> traverse_check_after(a, 0, {})
a
__b
____c
Already in cache: a
__d

Download this code: python_graphtraversal_param.py

The function works, but we really, really don’t want to have to do it like this. There is no reason the caller has to know about the cache, let alone that it is a dictionary. And if we ever decide to change the cache mechanism, we have to update all the client code.

So what can we do? We can’t set up the cache in the function body, because the function is recursive!

This is where you might get that look in your eye that you get when you’re being clever. Hah, but what about this:

def traverse_cached(node, depth, cache={}):
    if node in cache:
        print('Already in cache: %s' % node.name)
        return
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        traverse_cached(ref, depth+1, cache=cache)

Download this code: python_graphtraversal_cached.py

Brilliant! We don’t have to set up the cache neither inside the function body nor at the call site. This cache has to be initialized somewhere between the call and the function being executed, but somehow we’ve found a magical in-between, haven’t we!

Keyword parameters really are a tremendous boon, but unfortunately they will not save our skin this time. Do you know what happens if we do this?

>>> traverse_cached(a, 0)
>>> traverse_cached(a, 0)

Download this code: python_graphtraversal_cached_call.py

Read and weep:

>>> traverse_cached(a, 0)
a
__b
____c
Already in cache: a
__d
>>> traverse_cached(a, 0)
Already in cache: a

Download this code: python_graphtraversal_cached_output.py

Keyword parameters

Keyword parameters don’t do what you thought they did. You thought traverse_cached would get a new dictionary every time it was called without a cache argument. But that’s not what happens. (But isn’t python magical???)

In fact, it works like this. The cache keyword parameter gets initialized once and for all when the function is compiled. Since the value it is set to is an empty dictionary, the dict object is initially empty. But every time you call the function, it’s the same dictionary that gets passed in!

Third iteration

It’s come to this, I’m afraid. The problem is unsurmountable. We can’t not pass the cache parameter. But we definitely don’t want to do it at the call site. Which leaves no other option than…

def traverse_cached_fixed(node, depth):
    def rec(node, depth, cache):
        if node in cache:
            print('Already in cache: %s' % node.name)
            return
        cache[node] = None
 
        print('__'*depth + node.name)
        for ref in node.refs:
            rec(ref, depth+1, cache)
    rec(node, depth, {})

Download this code: python_graphtraversal_fixed.py

I really hate using an inner function just for this. It makes it messier, you have to route the arguments through an extra function call. But that’s the price you pay, I’m afraid.

Postscript

By now you must be fuming at the fact that I’ve come all this way while pretending that the function call didn’t have another parameter that was completely pointless and indeed deserving of the very same criticism.

Well, the difference is that you can set depth=0 as a keyword parameter in the function definition, so that you don’t have to pass it. And this doesn’t break anything. But why??

The answer is found when poking around python. A function is in fact a mutable object. Have a look:

def f(a=0):
    a = 1
 
>>> print f.func_defaults
(0,)

Download this code: python_graphtraversal_keywords1.py

func_defaults is the tuple that stores the values of the keyword parameters which have a default value. The value we see here is the integer 0, and it will not change no matter how many times we execute the function.

But collection types (and your own types too) are different:

def g(a=[]):
    a.append(1)
    print a
 
>>> print g.func_defaults
([],)
>>> g()
[1]
>>> print g.func_defaults
([1],)
>>> g()
[1, 1]

Download this code: python_graphtraversal_keywords2.py

Despite different, it actually works the same way (sorta). The binding of the keyword default is constant. But with an integer, the parameter is bound to the constant 0, whereas with “heavier” objects, it is bound to the object, but the object’s inner properties can very well mutate!!!

Ars Technica is bending over backwards not to make The Big Accusation, namely “if you skip the ads you are stealing”. And yet they still try to guilt trip you saying “well gosh, our business is on the line here, y’know?”.

I don’t know where Al Gore is, but there is a highly Inconvenient Truth here. And it is this: publishers are working from the assumption that the viewer has implicitly agreed to make them money without ever agreeing to it. This is why the Ars Technica article has to do that weird dance around the issue where they’re not calling you a cheat, but then they are, but then they’re not etc. Because if they did just come out and say it, they couldn’t look themselves in the mirror, because they have more self respect than certain muppets.

Ars Technica couldn’t live with themselves if they actually came out and said: Dear viewer, when we built this site we made the assumption that you would help us pay for it by looking at our ads. We never ran this by you, but we take it for granted that you have accepted this deal. Now we expect you to honor it.

There are others who are not so “modest”, like the entertainment industry claiming billions in damages based on sales they would have had if the people who downloaded stuff had paid for it. Here again, the assumption is that the consumer agreed to pay for something, and then refused. So who’s being cheated? The publisher!

But the viewer agreed to no such thing.

I can tell you it’s a real pain to author latex documents this way, I can’t fit both the kile and evince on the screen at the same time. It wasn’t until recently that it hit me what I was doing wrong. There are three processes involved here:

1. Document editor.
2. Compiler (I run a loop that invokes make continuously in the background).
3. Document viewer.

Come to think of it, this applies just as well to coding if you think “running the code” on the last step.

Well, X11 is a display server, for peet’s sake! So you have the editor on the workstation, but then you log in from the other laptop (with the larger screen) and run evince to display there.

Just do:

oldlaptop$ ssh -XYC workstation

Don’t ask me why -Y, I don’t know, but that’s how I get my ubuntu to allow remote connections.

So if there is a person you have to get to you have to do it through the reception. “Yes, I am blah and I need bleh and why don’t you just transfer me to the person I need to talk to, per favore!” It’s not a lot of fun, but this way whoever makes this decision to give out only the number of the reception can also decide who may and may not receive calls. And even on what conditions. If you say the magic word then, yes, you can get Frank on the line, otherwise not. And maybe Steve has been known to say too much and has a history of divulging information he wasn’t supposed to. So no, you can’t talk to Steve.

Well, this is the principle of a firewall. The reception screens calls with the discretion to reject or divert according to the protocol that has been instituted. Some people can be reached anytime, others only at certain intervals. Some are available depending on your request, and some are completely unreachable.

This picture, however, conflicts with the original meaning of the word firewall, which is a wall erected to stop a fire from spreading. Unconditionally.

Well, bash --version tells you about bash, ls --version tells you about coreutils and so on, you can keep going with that. uname will tell you about the platform and the kernel, useful stuff. What about the distro? Distros are diverse, sometimes it helps a lot to know what kind of environment you’re in. Well, strangely enough, there’s no standard answer to that.

They all do their own thing, if they do at all. Some distros use lsb_release to dispense this information. Others have files in /etc that you can check for, if you know what they are supposed to be called. So I decided to try and detect this. I’ve checked a bunch of livecds and it works on those distros that identify themselves somehow.

# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3
#
# <desc> Detect OS (platform and version) of local machine </desc>
#
# <usage>
# source this file in bash, then run `osdetect`
# </usage>
 
 
_concat() {
	local s="$1";shift;
	while [ "$1" ]; do
		s="$s $1"; shift
	done
	echo "$s" | sed "s|^[ \\t]*||g" | sed "s|[ \\t]*$||g"
}
 
_glob() {
	local file=
	local glob=
	local lst=
	while [ -z "$file" ] && [ "$1" ]; do
		glob="$1";shift;
		lst=$(ls $glob 2>/dev/null | grep -v /etc/lsb-release)
		if [ "$lst" ]; then
			file=$(echo "$lst" | head -n1)
		fi
	done
	echo "$file"
}
 
osdetect() {
	# ref: http://linuxmafia.com/faq/Admin/release-files.html
 
	local os=
	local release=
	local machine=
	if ! which uname &>/dev/null; then
		echo -e "${cred}No uname on system${creset}" >&2
		os="N/A"
	else
		os=$(uname -s)
		release=$(uname -r)
		machine=$(uname -m)
	fi
	if [ "$os" = "SunOS" ]; then
		os="Solaris"
		machine=$(uname -p)
	fi
	local platform="$(_concat "$os" "$release" "$machine")"
 
	# prefer lsb_release
	if which lsb_release &>/dev/null; then
		local id="$(_concat "$(lsb_release -i | sed "s|.*:||g")")"
		local rel="$(_concat "$(lsb_release -r | sed "s|.*:||g")")"
		local code="$(_concat "$(lsb_release -c | sed "s|.*:||g")")"
	elif [ -f /etc/lsb-release ]; then
		local id="$(_concat "$(grep DISTRIB_ID /etc/lsb-release | sed "s|.*=||g")")"
		local rel="$(_concat "$(grep DISTRIB_RELEASE /etc/lsb-release | sed "s|.*=||g")")"
		local code="$(_concat "$(grep DISTRIB_CODENAME /etc/lsb-release | sed "s|.*=||g")")"
 
	# find a file or another
	else
		local vfile=$(_glob "/etc/*-rel*" "/etc/*_ver*" "/etc/*-version")
		[ "$vfile" ] && local id=$(cat "$vfile")
 
		# distro specific
		[ "$vfile" = /etc/debian_version ] && [ "$id" ] && id="Debian $id"
	fi
 
	[ "$id" = "n/a" ] && id=
	[ "$rel" = "n/a" ] && rel=
	[ "$code" = "n/a" ] && code=
 
	local version="$(_concat "$id" "$rel" "$code")"
	[ ! -z "$version" ] && version=" ~ ${cyellow}$version${creset}"
 
	echo -e "Platform: ${ccyan}${platform}${creset}${version}"
}

Download this code: osdetect.sh

Let’s use findpkgs as an example here. The function is defined in a separate file and the file is source’d. But this means that every time I start a new shell session the whole function has to be read and loaded into memory. That might not be convenient if there are a number of those. networktest, for instance, defines four function and is considerably longer.

So let’s “compile to bytecode” again:

findpkgs ()
{
    [ -f ~/.myshell/findpkgs.sh ] && . ~/.myshell/findpkgs.sh;
    findpkgs $@
}

Download this code: lazyfuncionload_loadfunc

When the function is defined this way, the script actually hasn’t been sourced yet (and that’s precisely the idea), but it will be the minute we call the function. This, naturally, will rebind the name findpkgs, and then we call the function again, with the given arguments, but this time giving them to the actual function.

Okay, so that was easy. But what if you have a bunch of functions loaded like that? It’s gonna be kinda messy copy-pasting that boilerplate code over and over. So let’s not write that code, let’s generate it:

lazyimport() {
# generate code for lazy import of function
	function="$1";shift;
	script="$1";shift;
 
	dec="$function() {
		[ -f ~/.myshell/$script.sh ] && . ~/.myshell/$script.sh
		$function \\$@
	}"
	eval "$dec"
}

Download this code: lazyfuncionload_lazyimport

Don’t worry, it’s the same thing as before, just wrapped in quotes. And now we may import all the functions we want in the namespace by calling this import function with the name of the function we want to “byte compile” and the script where it is found:

## findpkgs.sh
 
lazyimport findpkgs findpkgs
 
## networktest.sh
 
lazyimport havenet networktest
lazyimport haveinet networktest
lazyimport wifi networktest
lazyimport wifiscan networktest
 
## servalive.sh
 
lazyimport servalive servalive

Download this code: lazyfuncionload_functions

So let’s imagine a hypothetical execution. It goes like this:

1. Start a new bash shell.
2. Source import.sh where all this code is.
3. On each call to lazyimport a function declaration is generated, and eval’d. The function we want is now bound to its name in the shell.
4. On the first call to the function, the generated code for the function is executed, which actually sources the script, which rebinds the name of the function to the actual code that belongs to it.
5. The function is executed with arguments.
6. On subsequent executions the function is already “compiled”, ie. bound to its proper code.

So what happens, you may wonder, in cases like the above with networktest, where several mock functions are generated, all of which will source the same script? Well nothing, whichever of them is called first will source the script and overwrite all the bindings, remember? It only takes one call to whichever function and all of them become rebound to the real thing. So all is well.

I amaze so easily, and so frequently, that amazement has ceased to be special to me. It has become mundane. I need to check my standards for amazement. I need to raise the bar. I need to make amazement once again worth having.

I must stop being amazed, for example, when a man rings my doorbell because he cannot figure out the house numbers on my street. “Is this number thirty”, he asks. I lean out, in a mock gesture, to gaze at the street number opposite my front door. It does not say thirty. I imagine this gesture will suffice to make him understand. Instead he reiterates his dilemma. “Is this number thirty.” No. It is not. I must not be amazed, even if it is a man in his fifties. With gray hair, an elegant tie, and a fancy suit. Who proceeds to reenter his expensive automobile. How does a person like that not know how to read street numbers. I must not be amazed.

I must not be amazed, either, at the communal workers. Who must necessarily have intimate knowledge of such complicated administrative intricacies as are street numbers. Through their work of visiting various addresses everyday. Who still ring my doorbell by mistake.

One wonders how such people can accomplish complicated tasks like air travel, which requires all sorts of documents, procedures, requires remembering important facts and following a timetable. How do they manage it? It seems amaz I’m sure they pull it off somehow.

But what about files that are changing? Files get transfered all the time, but network connections are not always reliable. Have you ever been in the middle of a transfer wondering if it just stopped dead, wondering if it’s crawling along very slowly, too slow, almost, to notice? Or how about downloading something where the host doesn’t transmit the size of the file, so you’re just downloading not knowing how much there is left?

These things happen, not everyday, but from time to time they do. And it’s annoying. A file manager doesn’t really do a great job of showing you what’s happening. Of course you can stare at the directory and reload the display to see if the file size is changing. (Or the time stamp? But that’s not very convenient to check against the current time to measure how long it was since the last change.) Or maybe the file manager displays those updates dynamically? But it’s still somewhat lacking.

Basically, what you want to know is:

1. Is the file being written to right now?
2. How long since the last modification?

And you want those on a second-by-second basis, ideally. Something like this perhaps?

Here you have the files in this directory sorted by modification time (mtime). One file is actively being written to, you can see the last mtime was 0 seconds ago at the time of the last sampling. Sampling happens every second, so in that interval 133kb were written to the file and the mtime was updated. The other file has not been changed for the last 7 minutes.

The nice thing about this display is that whether you run the monitor while the file is being transfered or you start it after it’s already finished, you see what is happening, and if nothing is, you see when the last action took place.

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# <desc> Watch directory for changes to files being written </desc>
 
import os
import sys
import time
 
 
class Formatter(object):
    size_units = [' b', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb', 'zb', 'yb']
    time_units = ['sec', 'min', 'hou', 'day', 'mon', 'yea']
 
    @classmethod
    def simplify_time(cls, tm):
        unit = 0
        if tm > 59:
            unit += 1
            tm = float(tm) / 60
            if tm > 59:
                unit += 1
                tm = float(tm) / 60
                if tm > 23:
                    unit += 1
                    tm = float(tm) / 24
                    if tm > 29:
                        unit += 1
                        tm = float(tm) / 30
                        if tm > 11:
                            unit += 1
                            tm = float(tm) / 12
        return int(round(tm)), cls.time_units[unit]
 
    @classmethod
    def simplify_filesize(cls, size):
        unit = 0
        while size > 1023:
            unit += 1
            size = float(size) / 1024
        return int(round(size)), cls.size_units[unit]
 
    @classmethod
    def mtime(cls, reftime, mtime):
        delta = int(reftime - mtime)
        tm, unit = cls.simplify_time(delta)
        delta_s = "%s%s" % (tm, unit)
        return delta_s
 
    @classmethod
    def filesize(cls, size):
        size, unit = cls.simplify_filesize(size)
        size_s = "%s%s" % (size, unit)
        return size_s
 
    @classmethod
    def filesizedelta(cls, size):
        size, unit = cls.simplify_filesize(size)
        sign = size > 0 and "+" or ""
        size_s = "%s%s%s" % (sign, size, unit)
        return size_s
 
    @classmethod
    def bold(cls, s):
        """Display in bold"""
        term = os.environ.get("TERM")
        if term and term != "dumb":
            return "\\033[1m%s\\033[0m" % s
        return s
 
class File(object):
    sample_limit = 60  # don't hold more than x samples
 
    def __init__(self, file):
        self.file = file
        self.mtimes = []
 
    def get_name(self):
        return self.file
 
    def get_last_mtime(self):
        tm, sz = self.mtimes[-1]
        return tm
 
    def get_last_size(self):
        tm, sz = self.mtimes[-1]
        return sz
 
    def get_last_delta(self):
        size_last = self.get_last_size()
        try:
            mtime_beforelast, size_beforelast = self.mtimes[-2]
            return size_last - size_beforelast
        except IndexError:
            return 0
 
    def prune_samples(self):
        """Remove samples older than x samples back"""
        if len(self.mtimes) % self.sample_limit == 0:
            self.mtimes = self.mtimes[-self.sample_limit:]
 
    def sample(self, mtime, size):
        """Sample file status"""
        # Don't keep too many samples
        self.prune_samples()
        # Update time and size
        self.mtimes.append((mtime, size))
 
class Directory(object):
    def __init__(self, path):
        self.path = path
        self.files = {}
 
    def prune_files(self):
        """Remove indexed files no longer on disk (by deletion/rename)"""
        for f in self.files.values():
            name = f.get_name()
            file = os.path.join(self.path, name)
            if not os.path.exists(file):
                del(self.files[name])
 
    def scan_files(self):
        # remove duds first
        self.prune_files()
        # find items, grab only files
        items = os.listdir(self.path)
        items = filter(lambda f: os.path.isfile(os.path.join(self.path, f)),
                       items)
        # stat files, building/updating index
        for f in items:
            st = os.stat(os.path.join(self.path, f))
            if not self.files.get(f):
                self.files[f] = File(f)
            self.files[f].sample(st.st_mtime, st.st_size)
 
    def display_line(self, name, time_now, tm, size, sizedelta):
        time_fmt = Formatter.mtime(time_now, tm)
        size_fmt = Formatter.filesize(size)
        sizedelta_fmt = Formatter.filesizedelta(sizedelta)
        line = "%6.6s   %5.5s   %6.6s   %s" % (time_fmt, size_fmt,
                                               sizedelta_fmt, name)
        if time_now - tm < 6:
            line = Formatter.bold(line)
        return line
 
    def sort_by_name(self, files):
        return sorted(self.files.values(), key=lambda x: x.get_name())
 
    def sort_by_mtime(self, files):
        return sorted(self.files.values(),
                      key=lambda x: (x.get_last_mtime(),x.get_name()))
 
    def display(self):
        time_now = time.time()
        files = self.sort_by_mtime(self.files.values())
        print("\\nmtime>   size>   delta>   name>")
        for f in files:
            line = self.display_line(f.get_name(),
                                     time_now, f.get_last_mtime(),
                                     f.get_last_size(), f.get_last_delta())
            print(line)
 
 
def main(path):
    directory = Directory(path)
    while True:
        try:
            directory.scan_files()
            directory.display()
            time.sleep(1)
        except KeyboardInterrupt:
            print("\\rUser terminated")
            return
 
 
if __name__ == '__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        print("Usage:  %s /path" % sys.argv[0])
        sys.exit(1)
    main(path)
 

Download this code: peek.py

So why did this come about? Well, in the early 90s television was privatized. There had been cable tv for a number of years already, but in 1992 the channel TV2, the first commercial tv station to broadcast nationally, alongside the state broadcaster NRK, was launched. Effectively, television was let go from under the control of the rather insular NRK and freed to be driven by commercial profit. Naturally, TV2 in short order proceeded to import all of the popular culture, chiefly American, that has informed our lives. Another characteristic of TV2, quite unlike NRK — the absolute sterility of any form of even mild intellectualism. (Aside from NRK which may have as much as 2-3 weekly hours of programming suited to the more discerning viewer, provided the topic is up your street.)

Another big development in the mid and late 90s, of course — wide access to the internet. Here again is a brand new medium with immense amounts of information and culture “beamed” right into our homes.

I feel it was these two developments that formed the impetus behind this fashion within the education system. All of a sudden students would be writing essays using web pages for sources that, believe it or not, were just incorrect. Oh noes, something has to be done! And so it began. The internet is not trustworthy. You can’t believe everything you read. The deeper reasoning behind this is the question of motive. More important than what they are saying; why are they saying it? Back in my junior high days we received the dumbed down version (as, actually, with everything in junior high). The question was framed in terms of sources. This source is reliable, because it’s Encyclopedia Britannica. This source is not, because it’s a web page, or I heard it on tv. Never mind who gets to decide what is reliable and why.

Of course, the truth, as your astute self would have figured out by now, is that nothing is in and of itself trustworthy. It’s not just when you go online that you find garbage. There’s just as much garbage in books, in what your teachers tell you, in what your parents tell you, and above all in what your peers tell you. You need to be critical of all this stuff, not just of those crazy people on the internet.

I’m inadvertently rehashing Jürgen’s argument here. I read his entry and didn’t give it any more thought, but perhaps my subconsciousness has been chewing on it ever since? Thanks, Jürgen.

Naturally, some people are just malicious, but that is not the main problem. Even if you do have a piece of insight that you honestly believe is beneficial to someone, there are still a lot of things that can go wrong:

1. You’re plain mistaken.
2. It works for you, but it doesn’t work for everyone.
3. Even though you have the right idea, you fail to communicate it effectively.

The last one is particularly unfortunate. How many times has someone told you that they’ve just this discovered this new thing and it’s everything they needed, and then you say “but I told you that already a long time ago!”. Well, I guess you didn’t tell me in the words that I needed to hear in order to absorb the information, or in order to be convinced.

Parental advice, of course, is susceptible to the same flaw as those self help books. I’m sure you’ve seen some of those around, the basic premise is always the same — some person has figured out how to do something and wants to tell everyone. The problem is that just because it worked for him, doesn’t mean it will work for you. Especially when you hear it from a secondary source (a relative comes up and says “I read this amazing book, it changed my life, all you have to do is..”). But it’s not science. At best it “sorta works a lot of the time, kind of”.

So over time you develop a filter. “This person is not worth listening to on these topics, this book is written by someone who has no idea what he’s talking about, this website is usually reliable on these issues”. Now everything depends on that filter of yours. You may find one day that you bought into some utter nonsense, or that you discarded good insight.