<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/atom10full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><feed xmlns="http://www.w3.org/2005/Atom" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0"><title>SIGUSR2</title><link href="http://sigusr2.net/" /><updated>2010-06-25T11:27:01Z</updated><author><name>Andrew Gwozdziewycz</name></author><id>md5:ec2c8804a8dac6a866e1a43cfce32fc1</id><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/atom+xml" href="http://feeds.feedburner.com/SIGUSR2" /><feedburner:info uri="sigusr2" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><entry><title>Google Spreadsheets: Game of Life</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/YOJeTM0sZVc/google-spreadsheets-game-of-life.html" /><id>md5:026e9ae419696af7421a97f140a267e1</id><updated>2010-06-25T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Google Docs has become an incredibly useful tool for businesses looking to cut costs on software and infrastructure, and it just keeps getting better.</span></p>

<p>For years, many businesses were tied to Microsoft Windows due to macros that they made use of in Excel and Word, but that's about to change. Recently, Google Spreadsheets released "<a href="http://www.google.com/google-d-s/scripts/scripts.html">Apps Script</a>," which allows you to extend Spreadsheets and/or integrate other Google products like Calendar.</p>

<p>Apps Scripts are written in JavaScript and the environment provides you with a bunch of different <a href="http://www.google.com/google-d-s/scripts/guide_chapter_01.html">services</a>, which allow you to access your contacts, maps, XML, fetch URLs from the Internet, get Google finance information, send email&mdash;all sorts of things that make it possible to create complicated workflows and applications. I can imagine that there will be a marketplace sometime soon, which will be bustling with great apps to buy and use.</p>

<p>For now though, you're on your own.</p>

<p>Anyway, yesterday, Google had a hackathon, and because I had no real business oriented application ideas I thought I'd make a game. The trouble is, all processing is done on the server side, so real time interaction is a bit difficult&mdash;if not impossible. There are timer events, form events and menu events, but you can't bind a key or anything like that&mdash;at least not that I know of.</p>

<p>So, a game like Tetris or Snake was out, but not Conway's classic <a href="http://en.wikipedia.org/wiki/Conway's_Game_of_Life">Game of Life</a>. The Game of Life is a zero-player game in which the initial board evolves to form interesting patterns or disappears into nothing. It's normally drawn on a grid of some sort, so I thought a spreadsheet was appropriate.</p>

<p><img src="http://files.sigusr2.net/images/game-of-life-gosper.png" /></p>

<p>I've put the source code up on <a href="http://github.com/apgwoz/app-script-game-of-life">GitHub</a>, along with some more instructions if you wanna check it out. Fork it.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2010/Jun/25/google-spreadsheets-game-of-life.html</feedburner:origLink></entry>
<entry><title>Code Blogging: ARGF in Python</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/zdxqo9iC2nA/argf.html" /><id>md5:9d5ba6e7df3fe6f44b607a6a0be06952</id><updated>2010-06-23T00:00:00Z</updated><content type="html"><![CDATA[<pre><code class="python">"""ARGF from Ruby in Python.
Released into the public domain by Andrew Gwozdziewycz, 2010
"""

import sys, os


class _ARGF(object):

    def __init__(self):
        self.lineno = 0
        self.file = None

    def __iter__(self):
        return self.next()

    def next(self):
        files = filter(os.path.isfile, sys.argv[1:])
        pairs = [(f, open(f)) for f in files] \
            if files else [('STDIN', sys.stdin)]
            
        for name, fobj in pairs:
            self.file = name

            for line in fobj.xreadlines():
                self.file = 'STDIN'
                self.lineno += 1
                yield line

ARGF = _ARGF()
</code></pre>

<p><strong>Update:</strong> It was pointed out that I didn't handle <code class="inline">lineno</code> correctly; I fixed it.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2010/Jun/23/argf.html</feedburner:origLink></entry>
<entry><title>Maps are Broken, for Some Definition of Broken</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/162FqnXEKiY/maps-are-broken-for-some-definition-of-broken.html" /><id>md5:ee8c78a17c21d8d986ea5f7e3bc78811</id><updated>2010-06-22T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Map datatypes are extremely useful for a variety of tasks in programming. But, they are often painful to use; take for example the following task.</span></p>

<p>In Java, I have a <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/HashMap.html">HashMap</a> and I wish to get a random key. Well, <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/AbstractMap.html">AbstractMap</a> doesn't define a way to get a random key, but it does provide a way to get a <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/Set.html">Set</a> of keys. Does Set have a way to get a random element? No, but you can create an Array from a Set with the <code class="inline">toArray()</code> method on Set.</p>

<p>We end up with the following:</p>

<pre><code>public String randomKey() {
    // Assuming: map = HashMap&lt;String, String&gt;;
    Set&lt;String&gt; set = map.keySet();
    Object[] strings = set.toArray();
    Random random = new Random();
    if (strings.length &gt; 0) {
        return (String)strings[random.nextInt(strings.length)];
    }
    return null;
}
</code></pre>

<p>Now, this isn't necessarily bad, but we have to create a new array, and a new set each time we want a random key. We can of course be smarter about this by caching the array and/or set, but then we run into synchronization issues. We also get screwed when we attempt to implement the <code class="inline">popRandom()</code> operation, which could be implemented like so:</p>

<pre><code>public String popRandom() {
    String key = randomKey();
    if (key != null) {
        String value = map.get(key);
        map.remove(key);
        return value;
    }
    return null; // or more appropriately, throw an exception
}
</code></pre>

<p>So, we're doing all this extra copying, allocating and deleting, when all we really need is an iterator, to solve this generically in <code class="inline">O(n)</code> time.</p>

<pre><code>public String randomKey() {
    // randomKey method in O(n) using imaginary iterator() on AbstractMap
    int size = map.size();
    if (size &gt; 0) {
        int index = new Random().randInt(size);
        Iterator&lt;String&gt; keys = map.iterator();
        while (keys.hasNext()) {
           if (index-- == 0) {
               return keys.next();
           }
           keys.next();
        }
    }
    return null;
}
</code></pre>


<p>This sort of thing isn't necessarily true for dynamic languages like Python which normally have ways to iterate over keys in a map, dictionary or set. They still don't have a way to get a random element from either out of the box without resulting to the <code class="inline">O(n)</code> iteration method, or converting to a list and using a random index approach.</p>

<pre><code>&gt;&gt;&gt; import random
&gt;&gt;&gt; random.choice(set([1, 2, 3]))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/random.py", line 248, in choice
    return seq[int(self.random() * len(seq))]  # raises IndexError if seq is empty
TypeError: 'set' object is unindexable
</code></pre>

<pre><code>&gt;&gt;&gt; random.choice({'1': 'world', '2': 'galaxy', '3': 'universe'})
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/random.py", line 248, in choice
    return seq[int(self.random() * len(seq))]  # raises IndexError if seq is empty
KeyError: 2
</code></pre>

<p>And of course that makes sense given how <code class="inline">random.choice</code> is implemented, since there's not necessarily  an order for the elements of a set or dictionary, so you can't expect to subscript them. However they do provide an order when iterating over them and traversing the structure they exist in, so you could certainly use the same <code class="inline">O(n)</code> approach from above.</p>

<p>If there's some other less obvious way to do this in Java using a <a href="http://en.wikipedia.org/w/index.php?title=Dependency_injection&oldid=260831402#A_code_illustration_using_Java">EnterpriseFactoryObserverFactoryFactoryCreator</a>, please leave a comment.</p>


<p><strong>Update: I overlooked something important, which was pointed out by <a href="http://news.ycombinator.com/item?id=1452619">gojomo</a> on Hacker News. Set, which is returned from <code class="inline">keySet()</code> on HashMap, has an iterator. Thus:</strong></p>

<pre><code>public String randomKey() {
    int index = random.nextInt(map.size());
    for (String key: map.keySet()) {
        if (index-- == 0) {
            return key;
        }
    }
    return null;
}
</code></pre>
]]></content><feedburner:origLink>http://sigusr2.net/2010/Jun/22/maps-are-broken-for-some-definition-of-broken.html</feedburner:origLink></entry>
<entry><title>Pattern Matching with "With"</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/dUZCcOukXZ8/pattern-matching-with-with.html" /><id>md5:862f9d6d39aec112eecc179210926c60</id><updated>2010-04-28T06:29:00Z</updated><content type="html"><![CDATA[<p>
<span class="preamble">When I originally thought about adding pattern matching to Python, in the <a href="http://sigusr2.net/2008/Sep/30/python-type-constructors-like-ocaml.html">OCaml sense</a>, I ended up using a decorator that more or less registed a bunch of callbacks with a dispatch table based on the types of it's arguments.</span>
</p>

<p>That worked out fine, but it didn't really have the feel of <a href="http://en.wikipedia.org/wiki/Pattern_matching">pattern matching</a> like you get with real <a href="http://en.wikipedia.org/wiki/Algebraic_data_type">algebraic data types</a>. If you recall, I was playing with the following example with the decorator approach:
</p>

<pre><code class="ocaml">type astnode = 
| AndNode of astnode * astnode
| OrNode of astnode * astnode
| NotNode of astnode 
| IdNode of bool

let rec eval_node (n: astnode) = 
  match n with
  | AndNode (l, r) -> (eval_node l) && (eval_node r)
  | OrNode (l, r) -> (eval_node l) || (eval_node r)
  | NotNode l -> not (eval_node l)
  | IdNode v -> v

eval_node (AndNode (IdNode true, IdNode false)) (* returns false *)
</code></pre>

<p>The idea of that program was to create a small language to evaluate boolean expressions. In OCaml, it's quite succinct&mdash;too succinct, in all honesty. That's it. Of course it doesn't include a parser, or a lexer, but that's the crux of it.
</p>

<p>Since that original post, I've posted about two other <q>language hacks</q> that I've attempted to create&mdash;both of which use <a href="http://www.python.org/dev/peps/pep-0343/">context managers</a> and the <code class="inline">with</code>-statement, <a href="http://sigusr2.net/2009/Oct/01/python-worlds.html">worlds</a> and <a href="http://sigusr2.net/2009/Mar/04/dispatching-with-with.html">dispatching urls (a la routes)</a>.</p>

<p>Basically, it occurred to me yesterday, that <code class="inline">with</code>'s <code class="inline">as</code> clause did destructuring of tuples, in the same way that the assignment statement does. That is to say:</p>

<pre><code class="python">a, b, c = 1, 2, 3</code></pre>

<p>Will correctly assign <code class="inline">a = 1</code>, <code class="inline">b = 2</code>, <code class="inline">c = 3</code>, in the same exact way that:</p>

<pre><code class="python">from contextlib import contextmanager
@contextmanager
def assign(*args):
    yield args

with assign(1, 2, 3) as (a, b, c,):
    pass
</code></pre>

<p>will assign <code class="inline">a = 1</code>, <code class="inline">b = 2</code>, <code class="inline">c = 3</code>.</p>

<p>I'll admit, that doesn't look very powerful by itself, but when you consider the possibilities, you might come up with something like I did:</p>

<pre><code class="python">with structural_matching((1, 2, 3)) as match:
    with match('list() x y z') as (x, y, z):
        print x, y, z
    with match('tuple() x _ z') as (x, z):
        print "tuple case"
        print x, z
</code></pre>

<p>which looks incredibly close to pattern matching in OCaml. I was super excited&mdash;but it won't work.</p>

<p>See, <code class="inline">match</code> is a context manager that gets returned with the intention that if the <code class="inline">__enter__()</code> method raises a <code class="inline">NoMatch</code> exception, it skips the "body" and goes to the next match. The problem with that thinking however is simple&mdash;there's no way for <code class="inline">__enter__</code> to force skipping the body due to rejected <a href="http://www.python.org/dev/peps/pep-0377/">PEP-377</a>!</p>

<p>In the example above (full source <a href="http://files.sigusr2.net/match1.py">here</a>), raising <code class="inline">NoMatch</code> in the first <code class="inline">match</code> block, results in control being passed back to the <code class="inline">__exit__()</code> of the outer context manager&mdash;<code class="inline">structural_matching</code>. And to think, I got my hopes up!</p>

<p>But nevertheless, I pressed on, and hacked <a href="http://files.sigusr2.net/match2.py">together</a>, a <code class="inline">match</code>, that can destructure the following examples correctly:</p>

<pre><code class="python">with match('[1:3]', [1, 2, 3, 4]) as (a,):
    print a
# [2, 3]

with match('[1:]', "hello world") as (a,):
    print a
# ('e', 'ello world')

with match('str() x y', 'hello world') as (h, e):
    print 'h = ', h, ',',
    print 'e = ', e
# h = h, e = e

with match('x y z', [1, 2, 3]) as (x, y, z,):
    print z, y, x
# 3 2 1

class obj(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

with match('obj() .x .y', obj('x-ity', 'y-ity')) as (x, y):
    print 'x = ', x, ',',
    print 'y = ', y
# x = x-ity, y = y-ity

with match('x y _', [1, 2, 3]) as (x, y):
    print x, y
# 1, 2
</code></pre>

<p>It's much less useful considering you can't put it in the <code class="inline">structural_match</code> block ,like you would in a <em>real</em> <code class="inline">match</code> statement, but it's all we've got.</p>

<p>Back then, I concluded with <q>This is as close to OCaml like  pattern matching that we're going to get, at least as far as I know how to get, but it's sort of cool, and definitely a hack.</q> Today, I'll conclude the same way.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2010/Apr/28/pattern-matching-with-with.html</feedburner:origLink></entry>
<entry><title>The Real Disappointment</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/C3v0rI4m41Q/the-real-disappointment.html" /><id>md5:14b52b115b36632711ab9050b7832862</id><updated>2010-01-28T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">It's no <a href="http://mashable.com/2010/01/27/apple-ipad-downsides/">secret</a> <a href="http://i.gizmodo.com/5458382/8-things-that-suck-about-the-ipad">that</a> <a href="http://www.pmptoday.com/2010/01/27/apple-ipad-tablet-disappoints-ibookstore-inspires-wonder/" rel="nofollow">Apple's</a> <a href="http://search.twitter.com/search?q=ipad+failhttp://search.twitter.com/search?q=ipad+fail" rel="nofollow">newly</a> released <a href="http://apple.com/ipad" title="iPad" rel="nofollow">iPad</a> is a big disappointment for many. For me the disappointment isn't in the device itself, but rather the <a href="http://www.defectivebydesign.org/ipad" title="Sign the petition">crippling control</a> that Apple has over it.</span></p>

<p>What I don't like about this, is that there are obvious uses of this thing outside of the basic computing needs Jobs demoed.</p>

<p>Imagine being able to use this thing as a next generation "mouse." Perhaps it would allow multiple windows to be tossed around the screen, or moved around <a href="http://en.wikipedia.org/wiki/Three-card_Monte">"Three-card Monte"</a> style. It'd be revolutionary.</p>

<p>I dare you to name a piece of software intended for creation that couldn't benefit from a multi-touch input pad&mdash;perhaps even customizable based on the context your in within the app? Photoshop sliders? Aperture light tables? Scaling and carving models in Cinema 4d or Maya? The list goes on.</p>

<p>Now, imagine 2 of them working in tandem. On the left, you have a hunking half-keyboard for the Q-W-E-R-T half, and on the right the Y-U-I-O-P half. Suddenly, we have a multitouch split keyboard (without haptic feedback of course), which could be the next generation of <a href="http://makezine.com/pub/tool/FingerWorks_TouchStream_LP">FingerWorks TouchStream</a> style devices, which Apple, for reasons unbeknownst to me, <a href="http://en.wikipedia.org/wiki/FingerWorks">ceased the operations</a> of. (If I had to guess why they killed off FingerWorks, it'd be that Apple thinks the market for ergonomics is oodles smaller than mobile multi-touch devices&mdash;and I reckon they're right.)</p>

<p>The real disappointment with the iPad isn't that it can't run multiple apps, or that it'll be another $29.99 a month for 3G support for iPhone users. It's not even that the pixel density of the thing might make it tough to read books on. The real disappointment is the fact that the device is locked down and we'll never get the chance to use it to innovate the way we interact with our <em>real</em> computers.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2010/Jan/28/the-real-disappointment.html</feedburner:origLink></entry>
<entry><title>Remote Objects</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/t8Kt2B4pFHY/remote-objects.html" /><id>md5:c613f9f24fe2f9428acc9780d5c39573</id><updated>2009-11-20T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Back in May, I was tasked with migrating a few years worth of <a href="http://roundup.sourceforge.net/">Roundup</a> data to <a href="http://www.activecollab.com/">activeCollab</a>&mdash;a more featured project management and collaboration tool.</span></p>

<p>The move was justified, as Roundup no longer matched the way our organization wished to conduct business. It was a good move, but as with any migration there are bound to be some hiccups along the way.</p>

<p>Right away I hit one. The data models for these 2 very different pieces of software are insanely different (who knew?). Roundup makes <em>use of</em> a <a href="http://en.wikipedia.org/wiki/SQL">SQL</a> database, but not in a traditional way. And, activeCollab combines almost all of its entities into one table. This of course makes querying with SQL incredibly difficult, and a SQL to SQL translation close to impossible.</p>

<p>Luckily, in moving to activeCollab, we inherited a somewhat <a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">RESTful</a>, API that allows for the modification and creation of entities. As a result, it became apparent that the logical way to migrate this was to pull the data out using the Roundup's libraries and make the appropriate API calls with the translated data.</p>

<p>To make it easy on myself, I developed a quick <a href="http://en.wikipedia.org/wiki/Object_relational_mapper">ORM</a>-like interface to the APIs. I had the idea that if I could make it work for activeCollab, it must be generalizable enough to work for other services and APIs as well, which I could do later.</p>

<p>And, I was right. It could be more generalized, and it could be useful. In fact, the idea was so useful that 3,000 miles away in the San Francisco office someone else was already secretly working on the same idea!</p>

<h3>Enter Remote Objects</h3>

<p>The result of the secretive effort, that I became aware of, is called <a href="http://github.com/sixapart/remoteobjects/">Remote Objects</a>, which identifies itself as <q>An object RESTational model.</q> <a href="http://sixapart.com/">Six Apart</a> has graciously released it as part of the requirements to run <a href="http://www.typepad.com/go/motion">TypePad Motion</a>, which is also available, under a <a href="http://en.wikipedia.org/wiki/BSD_License">BSD License</a>, on <a href="http://github.com/sixapart/typepad-motion">github</a>.</p>

<p>But, just because its main intentions were for being used against the <a href="http://developer.typepad.com/api/rest.html">TypePad APIs</a>, doesn't mean that it <em>only</em> works with them.</p>

<p>In fact, the developers had the foresight to think that this would be useful in other cases too, just as I did, as can be seen in the <a href="http://github.com/sixapart/remoteobjects/tree/master/examples">examples</a> that come with the package.</p>

<p>Needless to say, I abandoned my implementation of this idea and will adopt Remote Objects in the future. Have a look. Hopefully it is useful in your toolbox too.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2009/Nov/20/remote-objects.html</feedburner:origLink></entry>
<entry><title>Python Worlds</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/0a1csaS6Ktg/python-worlds.html" /><id>md5:8ec25d2f452904fcb32829303656bf92</id><updated>2009-10-01T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Last year, I was introduced to a paper via <a href="http://lambda-the-ultimate.org/node/3040">Lambda the Ultimate</a> about <q>worlds,</q> a language construct which allows one to control the scope of side effects while programming.</span></p>

<p>Worlds allow you to capture the current scope of a program in a first-class way. All updates to the current state (i.e. local variables, global variables) happen in a non-commiting way. In other words, you can back out of any changes at any time.</p>

<p>Consider the following example (taken from the <a href="http://www.vpri.org/pdf/rn2008001_worlds.pdf">Warth paper</a>):</p>

<pre><code class="javascript">A = thisWorld; // thisWorld is always the current world
r = new Rectangle(4, 6);

B = A.sprout(); // sprout creates a new world with it's parent set to A
in B { r.h = 3; } // side effects within this `in' occur only in the world B.

C = A.sprout();
in C { r.h = 7 }; // in A's world r.h = 6 still.

C.commit(); // only now does r.h = 7 in world A.
</code></pre>

<p>If you follow along in the comments I've appended to the example, you'll start to see why this idea is interesting, even from this little example.</p>

<p>The astute Scheme programmer, however, will notice almost certainly that this construct could be created with <code class="inline">call/cc</code>, which is certainly true. The problem with this fact is that not all programming languages are Scheme (unfortunately), and of course not all languages support first-class <a href="http://en.wikipedia.org/wiki/Continuation">continuations</a>.</p>

<p>The question I asked myself, however, is this: <q>Can I hack worlds into Python?</q> To which I came up with the short answer after some thinking, <q>sort of.</q>

<p>I guess I should explain what's going on more clearly in the example above. The first thing to note is that <code class="inline">A</code> represents the current scope; the current state of all variables in the program. <q>Sprouting</q> a new world from an existing world means that any changes that occur when using the sprouted world, do not affect the world who sprouted the current world, unless the new world <em>commits</em> the changes made to the original world.</p>

<p>Which is to say, changes that occur in an <code class="inline">in</code> block acting on world <code class="inline">X</code> do not propagate to the parent (the world <code class="inline">X</code> was sprouted from) of <code class="inline">X</code> unless <code class="inline">X.commit()</code> is called.</p>

<h3>Enter Context Managers</h3>

<p>A few months ago, I wrote a blog post about <em><a href="http://sigusr2.net/2009/Mar/04/dispatching-with-with.html">Dispatching With "with"</a></em>, in which I explained <a href="http://www.python.org/dev/peps/pep-0343/">context managers</a> in Python, and how they can be exploited to create a less separated mapping from URLs to request handlers (something that definitely has its place in the small web-app world).</p>

<p>The basic idea of this was that in the <code class="inline">__exit__</code> method of the web-application object, the current frame was inspected and references to functions that represent HTTP methods would be collected, stored and tied to the last regular expression passed to the <code class="inline">expose</code> method in the application object. This simple solution allowed us to express a web application succinctly like so:

<pre><code class="python">app = web.application()
with app.expose('/'):
   def get(): 
       return "Hello World"
app.run()
</code></pre>

<p>For worlds, I also exploit context managers, though mostly for the <code class="inline">in</code>-like syntax, and for managing the current <code class="inline">thisWorld</code> variable.</p>

<p>The quick<a href="#f1">[1]</a> solution that I came up with for <a href="http://github.com/apgwoz/python-worlds">implementing worlds</a> can be used like so:</p>

<pre><code class="python">with Universe(): # establishes new world, assigns to local variable `thisWorld'
   thisWorld.r = True # must assign <em>directly</em> in the world. LIMITATION
   new = thisWorld.sprout() 

   with new:
       new.r = False

   with new.sprout():
       thisWorld.r = 15
       thisWorld.commit() # now new.r is 15, but the original r is still True

   print thisWorld.r # => True
   new.commit()
   print thisWorld.r # => 15
   thisWorld.commit() # have to commit to the actual scope LIMITATION
   # r is now part of the local variables where this universe exists
   print r # => 15 
</code></pre>

<p>Looking at this example, it's already apparent that the Warth implementation of worlds is superior, just in the amount of code needed to take advantage of it. You might also see that I didn't even attempt to port the rectangle example  from above. That is because there isn't anything smart going on under the hood when it comes to container objects (such as lists, tuples, objects, dicts), and I'm not yet sure how to get there.</p>

<p>With simple immutable objects such as booleans, integers and strings, using copy-on-write semantics works wonderfully. Then, on <code class="inline">commit</code> of the world, the code just copies all of the changes into its parent. I haven't tackled the case of mutable container objects just yet, as there are complications in the API<a href="#f2">[2]</a>, as well as the implementation.</p>

<p>The interaction with this is sort of annoying though. In order to take advantage of worlds in Python, you have to touch virtually every line of code in the function you are trying to <q>worldize</q>, because you must assign explicitly to a world. The world's context manager sets up <code class="inline">thisWorld</code> for you, but you still have to do <code class="inline">thisWorld.<em>variable</em></code> to get any sort of benefit.</p>

<p>My inclination is to get into some bytecode hacking to modify all assignments within the <code class="inline">with</code> block to be assignments to <code class="inline">thisWorld</code> automatically, but bytecode hacks are neither pleasant to maintain, nor are they portable across implementations.</p>

<p>It's also possible in the Warth version to <q>worldize</q> functions and any other first class object. Maybe the solution is simple and I just haven't seen it yet. Whatever hacks, that I come up with though, will be just that, hacks, as there is no <em>easy</em> way to add worlds to Python in the same way that Warth added them to JavaScript<a href="#f3">[3]</a>.</p>

<p>We are in an age of programming where mainstream programming languages are unable to adapt to our needs as programmers. We are unable to bend them at our will like we can with <a href="http://en.wikipedia.org/wiki/Scheme_%28programming_language%29">Scheme</a>, <a href="http://common-lisp.net/">Lisp</a> and even <a href="http://clojure.org/">Clojure</a>. Attempts to bring about change on this front have not been met with enthusiasm from most groups. Whether it's a lack of marketing, evangelism or just that the general population doesn't view <q>unbendability</q> as a problem, I'm not sure. But, I for one like the idea of being able to <em>easily</em> add worlds, and other ideas, as <em>true</em> language features to languages that by practicality, I'm forced into using. That would make me a much happier, and effective programmer.</p>

<ol class="footnotes">
    <li id="f1">By quick, I do mean quick. This was 2 hours of work and sketching. Surely there is lots of work to be done to make it a true solution.</li>
    <li id="f2">The same strategy could be used as for simple values like booleans, if the API used a method, say <code class="inline">assign</code> instead of the more natural assignment operator. Consider, <code class="inline">thisWorld.assign('obj.height.inches', 30)</code> vs. <code class="inline">thisWorld.obj.height.inches = 30</code>.</li>
    <li id="f3">The Worlds prototype was written in <a href="http://tinlizzie.org/ometa/">OMeta</a>, which is a solution to the "unbendable" languages problem. Note: I didn't attempt to write worlds in PyMeta, but, it may be possible to do.</li>
</ol>
]]></content><feedburner:origLink>http://sigusr2.net/2009/Oct/01/python-worlds.html</feedburner:origLink></entry>
<entry><title>Bitten... Really Hard</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/dVof9dWn8cE/bitten-really-hard.html" /><id>md5:56c6a1146108e40e674d0df15cd6c5b2</id><updated>2009-09-09T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Last Friday I was bitten really hard by something I would not have, in my wildest dreams think would bite me, Python's <a href="http://docs.python.org/library/urllib2.html"><code class="inline">urllib2</a></code>.</span></p>

<p>It is the ever so common case that you always hear, <q>It works fine here, it must be your fault.</q> And, boy did I come up with some interesting theories as to why it was not the fault of my code.</p>

<p>Basically, I'm POSTing some data to a resource that is protected by <a href="http://en.wikipedia.org/wiki/Basic_access_authentication">HTTP Basic Authentication</a>, and then reading the response for further processing. It is a basic task, and one that I've done many times in the past.</p>

<p>In fact, I've even used <code class="inline">urllib2</code> to do this before using the following method:</p>
<pre>
<code class="python">headers = {'User-Agent': useragent}
req = urllib2.Request(url, data, headers)
 
passmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmgr.add_password(None, url, username, passwd)
handler = urllib2.HTTPBasicAuthHandler(passmgr)
opener = urllib2.build_opener(handler)
connection = opener.open(req)
</code>
</pre>

<p>In this case, the following behavior is seen:</p>

<ol>
  <li>Client (urllib2) POSTs data leaving out the <code class="inline">Authorization</code> header</li>
  <li>Server responds with a <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2">401</a>. Client (urllib2) never sees the 401 error, and instead gets a <a href="http://en.wikipedia.org/wiki/SIGPIPE">SIGPIPE</a></li>
  <li>Client attempts to reconnect, going right back to step 1</li>
</ol>

<p>Naturally, the only way I discovered this was to fire up <a href="http://en.wikipedia.org/wiki/Tcpdump">tcpdump</a> and look at what was being sent. It quickly became obvious that there was never an <code class="inline">Authorization</code> header being sent, and therefore the 401 was completely justified. The broken pipe seems to have come from the server responding before reading all of the posted data to thwart off <a href="http://en.wikipedia.org/wiki/Denial-of-service_attack">denial of service</a> attacks.</p>

<p>My only explanation for why this works perfectly fine on my laptop and not in either of the data centers I tried running it from is latency, though I'm shocked, and not yet convinced, by that result.</p>

<p>Incidentally, I solved the problem by sending the <code class="inline">Authorization</code> header on first request, rather than waiting for the confirmation that, yes indeed, the resource needs the Authorization header sent, as <code class="inline">urllib2</code> likes to do.</p>

<code class="python">headers = {'User-Agent': useragent, 
         'Authorization': 'Basic ' + base64.b64encode(username + ':' + passwd)}
req = urllib2.Request(url, data, headers)
connection = urllib2.urlopen(req)
</code>

<p>... and now it works fine.</p>

<p>This whole experience will make me think twice before laying the blame anywhere until I have uncovered the real truth, something I think all programmers should learn to do, if they don't know already.</p>
]]></content><feedburner:origLink>http://sigusr2.net/2009/Sep/09/bitten-really-hard.html</feedburner:origLink></entry>
<entry><title>The Case of the Unusable Reusable</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/zJ7YK3cPeK8/case-of-the-unusable-reusable.html" /><id>md5:1790e770e90b7ed37ba8eeeda62034a7</id><updated>2009-07-22T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">The <a href="http://www.djangoproject.com/">Django</a> web framework has a huge following that releases lots of simple reusable apps that can be plugged into your website. However, sometimes they are too simple, and not "plugin-able" quite enough.</span></p>

<p>Take for instance <a href="http://code.google.com/p/django-favorites/" title="django-favorites on Google Code">django-favorites</a>. By itself it's a great package that does exactly what you want&mdash;it allows a user to mark things as being a favorite. Notice that I said "things." "Things" here means any model, because it uses the <a href="http://docs.djangoproject.com/en/dev/ref/contrib/contenttypes/#id1" title="Generic Relations">generic relationships</a> framework that Django provides.</p>

<p>This is a huge win on its own because it means that I don't have to define a new favorites model for blog posts, one for photos and another for music I'm listening too. But, there's a downside to this as well.</p>

<p>How do I check to see if an item is a favorite? Well, for each blog post I select back from the database, I have to make another query to see whether or not it was a favorite or not. Alternatively, of course, I could be smarter and do a bulk query using an <code class="inline">IN</code> clause, making the query count only 2. But, if I wasn't using an <a href="http://en.wikipedia.org/wiki/Object-relational_mapping" title="Object Relational Mapper">ORM</a>, my SQL would use an <code class="inline">OUTER JOIN</code>, or a sub-query to select back whether or not it was marked a favorite for the current user.</p>

<p>Django can do this using the <a href="http://docs.djangoproject.com/en/dev/ref/models/querysets/#queryset-methods-that-return-new-querysets"><code class="inline">extra</code></a> method for <code class="inline">QuerySet</code>s.</p>

<p>Ok, so we can use <code class="inline">extra</code> every time we want to 
select back whether or not an item is a favorite, no big deal.</p>

<code class="python">content_type = ContentType.objects.get_for_model(Photo)
SQL = """SELECT 1 FROM favorites_favorite f
WHERE f.content_type_id = %(content_type)d and
      f.object_id = photos_photo.id and
      f.user_id = %(user_id)d
"""
attrs = {'content_type': content_type.id, 'user_id': user.id}
entries = Photo.objects.extra(select={'is_favorite': SQL % attrs})
</code>

<p>Except that you have to do this every time. So, you then create a custom manager for <code class="inline">Photo</code> that includes a <code class="inline">check_is_favorite</code> method, which adds the <code class="inline">is_favorite</code> pseudo-column and everything is good.</p>

<p>That is until you have to do it for <code class="inline">Entry</code>, and <code class="inline">Song</code>, and <code class="inline">User</code>.</p>

<p>The solution however is simple. Reusable apps should include a "ManagerMixin", if it makes sense to make reusing the app as painless as possible. The django-favorites application that I've been using in my example would be complete in my eyes if it had something like this defined in it:</p>

<code class="python">class FavoritesManagerMixin(object):
    """ A Mixin to add a `favorite__favorite` column via extra 
    """
    def with_favorite_for(self, user, all=True):
        """ Adds a column favorite__favorite to the returned object, which
        indicates whether or not this item is a favorite for a user
        """
        content_type = ContentType.objects.get_for_model(self.model)
        pk_field = "%s.%s" % (qn(self.model._meta.db_table),
                              qn(self.model._meta.pk.column))

        favorite_sql = """(SELECT 1 FROM %(favorites_db_table)s 
WHERE %(favorites_db_table)s.object_id = %(pk_field)s and
      %(favorites_db_table)s.content_type_id = %(content_type)d and
      %(favorites_db_table)s.user_id = %(user_id)d)
""" % {'pk_field': pk_field, \
           'db_table': qn(self.model._meta.db_table), \
           'favorites_db_table': qn(Favorite._meta.db_table), \
           'user_id': user.pk, \
           'content_type': content_type.id, \
           }

        extras = {
            'select': {'favorite__favorite': favorite_sql},
            }

        if not all:
            extras['where'] = ['favorite__favorite == 1']

        return self.extra(**extras)
</code>

<p>I have yet to run a bench mark against this to determine whether or not the sub-query here is less efficient than doing an <code class="inline">OUTER JOIN</code>. My gut says it would be, but for a first go at it, I'll keep it like this.</p>

<p>Anyway, then to make use of this you create a custom manager that uses <code class="inline">FavoritesManagerMixin</code> as one of its base classes:</p>

<code class="python">class SongManager(models.Manager, <strong>FavoritesMixinManager</strong>):
    pass

class Song(models.Model):
    title = models.CharField(max_length=255, null=False, blank=False)
    artist = models.ForeignKey('Artist')
    album = models.ForeignKey('Album')

    <strong>objects = SongManager()</strong>
</code>

<p>And then to make use of it, we do:</p>
<code class="python">all_songs_with_favorites_marked = Song.objects.with_favorite_for(user)
only_favorite_songs = Song.objects.with_favorite_for(user, all=False)
</code>

<p>The intention of this post isn't to point out a flaw in django-favorites, an otherwise great reusable application, BTW, but instead is meant to give a way that this idea of reusable can actually be usable.</p>

<p>I put my django-favorites on <a href="http://github.com/apgwoz/django-favorites/">github</a>. Feel free to flame me for going against what this post stands for and creating yet another "favorites" reusable app, without submitting a patch to the original, I wanted some practice in Django development.
</p>
]]></content><feedburner:origLink>http://sigusr2.net/2009/Jul/22/case-of-the-unusable-reusable.html</feedburner:origLink></entry>
<entry><title>The Hacker's Utility Belt: SSH</title><link href="http://feedproxy.google.com/~r/SIGUSR2/~3/7wxRc_-VpLw/hacker-utility-belt-ssh.html" /><id>md5:1c30eb53e322a859bb17913f525bd6ef</id><updated>2009-05-07T00:00:00Z</updated><content type="html"><![CDATA[<p><span class="preamble">Most hackers<a href="#f1">[1]</a> aren't <a href="http://en.wikipedia.org/wiki/Batman">Batman</a>, but aside from <a href="http://en.wikipedia.org/wiki/Batman%27s_utility_belt">Batman's utility belt</a>, the Batcave and Alfred, he's no different than you and me.</span></p>

<p>He's got strength and training on his side, but I could probably take him if he didn't have his belt and was blindfolded. Give him his utility belt though, and I'd be doomed. That's not because his belt has super powers&mdash;instead his belt contains tools that are useful in common situations of distress, and of course detective work. The Bat-grappling hook for instance allows him to scale buildings and walls, and tranquillizer darts allow him to temporarily disable a foe. Batarangs, with practice, can be used to disable an opponent by aiming for a body part, or to cut down a hanging chandelier, creating yet another obstacle for foes to fight through.</p>

<p>Hackers have similar tools, but we carry them in "/usr/bin/" (though how cool would it have been if the <a href="http://en.wikipedia.org/wiki/Unix">Bell Labs folks</a> named it "/usr/belt/" instead). The tools <em>we</em> use, generally allow us to solve problems quickly and efficiently&mdash;just like Batman.</p>

<p>The tool that I've come to rely on quite a bit recently is <a href="http://en.wikipedia.org/wiki/Secure_Shell">SSH</a>. Most hackers use SSH for working remotely on servers, and for copying files over <a href="http://en.wikipedia.org/wiki/Secure_copy">SCP</a> or <a href="http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol">SFTP</a>. Some may even use <a href="http://en.wikipedia.org/wiki/Filesystem_in_Userspace">FUSE</a> and <a href="http://en.wikipedia.org/wiki/SSHFS">SSHFS</a> to mount file systems over SSH. Its potential uses are endless, but the most useful uses for me lately have been tunneling and <a href="http://en.wikipedia.org/wiki/SOCKS">SOCKS</a> proxies.</p>

<h3>Tunneling</h3>
<p>As a programmer who on occasion works from home (more so currently), it's often a battle to connect to all of the company resources I need in order to perform my job. Firewalls are good things, and are configurable to allow remote people access to walled gardens, but with dynamic <a href="http://en.wikipedia.org/wiki/IP_address" title="Internet Protocol Address">IP addresses</a> being the common norm, opening up resources to many different addresses gets hard for sysadmins to track. Besides, they've got a ton on their plate already.</p>

<p>To combat this problem, <a href="http://en.wikipedia.org/wiki/VPN" title="Virtual Private Network">VPN</a>s were created. Basically, a VPN allows one to connect networks together as if they were local to each other. When I use my companies VPN, it's as if I'm sitting in the office.</p>

<p>Often though, this isn't enough. When dealing with networks that clients own, they may provide us access through only one entry point, a single development server for instance, which is locked down.</p>

<p>This is where SSH comes in handy. In the <a href="http://www.openssh.com/">OpenSSH</a> implementation of the SSH client tools, the <code class="inline">-L</code> option allows one to setup forwarding of traffic to another host. Suppose, I wanted to connect to a client's Oracle server, listening on port 1521:</p>

<code>$ ssh -N -L 9999:clients.oracle.server.name:1521 user@your.companies.host
</code>

<p>Now instead of pointing our SQL*Plus client to <em>clients.oracle.server.name</em>:1521, we point it to localhost:9999, and traffic is forwarded via <em>your.companies.host</em> as if we were connecting directly from <em>your.companies.host</em>.
</p>

<p>This is extremely powerful, and simple.<a href="#f2">[2]</a></p>

<h3>SOCKS Proxy</h3>

<p>Similar to the tunnelling example above, OpenSSH can act like a <a href="http://en.wikipedia.org/wiki/SOCKS">SOCKS</a> proxy, allowing you to forward outbound traffic to a trusted source, which will then carry out the request on your behalf. This is <em>great</em> for browsing in public <a href="http://en.wikipedia.org/wiki/Wi-Fi" title="Wireless Fidelity">WI-FI</a> spots where attackers might be sniffing for passwords being sent in plain text over HTTP, or some other unencrypted protocol.<a href="#f3">[3]</a></p>

<p>And, of course, setting it up is as easy as passing the <code class="inline">-D</code> option to ssh when connecting to you@remoteserver.</p>

<code>$ ssh -N -D 8000 user@remoteserver</code>

<p>Then, you can setup your computer (or web browser) to use a SOCKS proxy, and point it to localhost:8000.</p>

<p>This isn't without its problems though. For one, the network could block all outbound traffic except HTTP/HTTPS. One obvious workaround is to have a remote server listening for SSH connections on port 143 or port 80, instead of the protocol default of 22. But, you'll of course need your own box, or <a href="http://en.wikipedia.org/Virtual_private_server" title="Virtual Private Server">VPS</a> for this, as your shared hosting account will probably not allow you to change sshd's listening port.<a href="#f4">[4]</a></p>

<p>Of course, since SSH is a protocol for sending encrypted network traffic, there are <a href="http://www.ssh.com/support/documentation/online/ssh/adminguide/32/X11_Forwarding.html">many</a> <a href="http://unixwiz.net/techtips/ssh-agent-forwarding.html">other</a> uses for it. These are just the two alternatives I find the most useful lately.</p>

<ol class="footnotes">
<li id="f1">
I use the <a href="http://www.ccil.org/jargon/jargon_23.html#TAG833">jargon file's</a> definition of hacker, not the commonly confused term <a href="http://www.ccil.org/jargon/jargon_18.html#TAG365">cracker</a>.
</li>

<li id="f2">
It can of course get much, much more complicated
</li>

<li id="f3">
Unencrypted WIFI traffic is extremely easy to capture, with a tool like <a href="http://www.tcpdump.org">tcpdump</a>
</li>

<li id="f4">
The thought of a "loopback" SOCKS proxy, "loopback" tunnel or some combination of the two cannot work, as it would only ever forward traffic to your localhost's sshd, which would then forward traffic to the destination, unencrypted, over the local network&mdash;this is the exact situation we were trying to avoid. Plus, if you're trying to forward traffic in this way, over a port that's being blocked, it's still blocked.
</li>
</ol>
]]></content><feedburner:origLink>http://sigusr2.net/2009/May/07/hacker-utility-belt-ssh.html</feedburner:origLink></entry>

</feed>
