schettino72

python AST vs XML

schettino72 — Tue, 16 Sep 2014 01:46:57 GMT

One of the things that I really like in python is its introspection capabilities. It goes as far as exposing its own syntax tree with the ast module.

AST

The ast module is usually used to do progammatically analysis, generation, refactoring/transformation of python code. But is it the right tool for the job?

API

The ast provides a very simple API with NodeVisitor subclasses being called for every node in the tree. It also provides a similar NodeTransformer to modify nodes in-place. This API feels a lot like SAX (Simple API for XML), an event sequential access parser API.

SAX style is simple, efficient and very useful in some situations, but not very powerful compared to DOM or other API's that support XPath. XPath is a very expressive query language for selecting nodes. In python there are several libraries that support it (on stdlib, lxml, ...)

Comments and Formatting

The ast throws away every information that is not important for the compiler, like code comments and formatting (white-space, new-lines, etc). This makes it very hard to modify existing code without messing up with other parts of the original source code.

The node tree

The AST tree is designed to be used by the compiler. It might no be optional for other uses...

py2xml

py2xml is a tool to covert python code into XML. It is already good enough to do lose-less (preserve formatting and comments) round-trip conversion of python to XML and back to python.

py2xml is still WIP (work in progress). The next step is to define a XML structure to make it easy for querying and transforming source code. Development is going on github.

Merging dicts using singledispatch

schettino72 — Sat, 14 Dec 2013 16:00:00 GMT

I am working on some code that reads config values from different sources. I would like to merge them in a way that values that are themselves dicts get the items from both sources, something like:

source_1 = {'foo': 10, 'options': {'a':1, 'b':1}
source_2 = {'foo': 20, 'options': {'a':2, 'c':3}

# merging source_1 and source_2 would give me:
{'foo': 20, 'options': {'a':2, 'b':1, 'c':3}

Notice how this is different from a simple dict.update() because options contains an item 'b'. Apart from that I would like list to be merged, that is merging [1, 2] and [3, 4] would result in [1 ,2, 3, 4].

I found some implementations but it seems that everybody has a different use-case for what "merge" should exactly do. So I decided to create a MergeDict that can be easily extended/configured.

singledispatch

The merge operation is specific depending on the type of the item. That makes a perfect opportunity to use singledispatch. Python 3.4 added support for single dispatch (see PEP 443). I am not using python 3.4 yet but there is also a package on pypi.

A single-dispatch is form of generic programming where you register different functions to be executed depending on the type of the first argument.

Here is a simple example:

from singledispatch import singledispatch

# the function to be executed by default gets a singledispatch decorator
@singledispatch
def fun(arg):
    print("default: {}".format(arg))

# register alternate function when argument is an int
@fun.register(int)
def _(arg):
    print("int: {}".format(arg))

# register alternate function when argument is a list
@fun.register(list)
def _(arg):
    print("list: {}".format(" ".join(str(a) for a in arg)))


fun('hi')
# default: hi

fun(1)
# int: 1

fun([1,2,3])
# list: 1 2 3

MergeDict

A MergeDict defines a merge() method. Each item is merged using the merge_value() function, by default it works the same as update() just replacing the value...

class MergeDict(dict):

    def merge(self, other):
        """merge other dict into self"""
        class Sentinel: pass
        for key, other_value in other.items():
            this_value = self.get(key, Sentinel)
            if this_value is Sentinel:
                self[key] = other_value
            else:
                self[key] = self.merge_value(this_value, other_value)

    @staticmethod
    def merge_value(this, other):
        """default merge operation, just replace the value"""
        return other

Applying singledispatch to class methods is a bit tricky, but ideally users would sub-class from MergeDict and define new merge() methods with a decorator similar to singledispatch.

The idea is to just annotate the methods but really register the dispatch only on object initialization.

class MergeDict(dict):

    def __init__(self, *args, **kwargs):
        super(MergeDict, self).__init__(*args, **kwargs)
        # register singlesingle dispatch methods
        self.merge_value = singledispatch(self.merge_value)
        for val in self.__class__.__dict__.values():
            _type = getattr(val, 'merge_dispatch', None)
            if _type:
                self.merge_value.register(_type, val)


    class dispatch:
        """decorator to mark methods as single dispatch functions."""
        def __init__(self, _type):
            self._type = _type
        def __call__(self, func):
            func.merge_dispatch = self._type
            return func

An example of a merge that sum up int values:

from mergedict import MergeDict

class SumDict(MergeDict):

    @MergeDict.dispatch(int)
    def merge_int(this, other):
        return this + other

a = SumDict({'a':2, 'b':3})
a.merge({'a':2, 'c':5})
print(a)
# {'a':4, 'b':3, 'c':5}

TL;DR;

Check mergedict on pypi.

github pull request workflow

schettino72 — Mon, 22 Jul 2013 02:00:00 GMT

A quick reference so I don't need to re-learn these steps every time I do a pull-request...

clone the repo in github
clone the repo on your machine

git clone xxx

create a branch

git branch fix-something

use the branch

git checkout fix-something

hack something then commit

git commit -a

push new branch

git push origin fix-something

on github click create pull request (or something like that)

generating HTML with javascript

schettino72 — Tue, 29 Jan 2013 12:00:00 GMT

I often hear people saying something like "HTML must be created using a template engine, anything else is a bad practice". I will try to demystify this dogma...

HTML

HTML is a text document. But browsers don't handle the text directly. It is first converted to DOM (document object model). So HTML is more like a declarative way to describe a user interface program.

This is a very important distinction, one is text the other is an object in a system.

Most components (HTML tags) simply describe how to display some data but other are actual UI components. For example a link (a tag) has an associated interaction with the user, but being part of the standard the browser already knows how to handle it. You don't need to write code for it, you just need to declare its href...

server-side templates

At some point the web moved from static HTML pages to dynamically generated HTML by the server side. After some time people realized that mixing text generation in the code could get quite messy. So they started using template engines.

A template engines combine a template with some data source to create a HTML text document.

Nowadays it is pretty much an established consensus that when generating HTML on the servers-side you should use a template engine, and I agree that's the right thing to do.

javascript templates

As the web evolved the code started to move from server to client. And the templates came together...

But wait. The server has no access to the DOM it can only create HTML text. Javascript has direct access to the DOM!

Should we always programmatically write some text in a declarative interface to create some UI components? Why not directly create these UI objects?

comparison

Lets compare different ways to modify the DOM using javascript to generate the content <div>Hello there</div>.

1) using DOM API

var myDiv = document.createElement("div");
var myContent = document.createTextNode("Hello there!");
myDiv.appendChild(myContent);
document.body.appendChild(myDiv);

Note this is not writing HTML text (although sometimes we may call it this way). It is manipulating the DOM directly. Oh, yes, it sucks :)

2) using jQuery

$('body').append($('<div>Hello there!</div>'));

That's funny, even that we are manipulating the DOM directly you must write HTML markup.

3) another DOM API (hoe.js)

$('body').append(div('Hello there!'));

That looks more reasonable.

4) using templates (Handlebars)

template = Handlebars.compile("<div>{{content}}</div>")
$('body').append(template({content: 'Hello there!'}));

This example is too contrived, normally you would put the template in a script tag.

declarative vs imperative

So which one is better? The answer is, of course, it depends...

Declarative is good for data. If you are just laying out the structure for some data, and using some existing UI component (HTML-tag). Using a template you will get a much more readable code.

example (handlebars):

<ul class="people_list">
  {{#each people}}
  <li>{{this}}</li>
  {{/each}}
</ul>

Declarative is also good if you are just using some custom component. In the example bellow a framework/library would look for elements with a class attribute indicating the widget to be used and the data- attributes to configure it.

<ul class="widget-XYZ" data-opt-a="5">
   some content
</ul>

Template has a big disadvantage, it produces text and loses the direct access to the DOM that is available from javascript.

When the web "moved" from server to the client the default UI elements provided by HTML were not enough to provide a rich user experience. So developers often need to create new UI components in javascript.

If you are creating a new UI component where you need to specify the behavior for a click and other actions you must manipulate the object in the DOM because you need to attach events.

example (hoe.js):

var myComponent = div('click me');
myComponent.click(function(){alert('Hello there!')});

So in one way or another you will have to get back a reference to the created object. If you use a template this is usually done by a CSS selector.

Using a CSS selector is bad for many reasons. It might be slow (need to search in the DOM). The code is error prone because it depends on attribute values that are far away from the code. The selectors have two meanings (styling with CSS and reference in javascript) that can gets confusing.

Another choice would be that your template/HTML reference the javascript code. For very simple things this was always possible using onclick and other attributes for events. Some frameworks provide more sophisticated mechanism like associating with a "controller", etc. The problem with this approach is that it doesn't scale as your component evolves. When it gets more complex you try to stuff more logic in the template. When your template can't handle it anymore you need to throw away your template and re-write it in javascript...

conclusion

If you are creating an UI component, manipulating the DOM from javascript is probably your best option.

If you are just filling some data and using existing components, templates are probably better for you.

mongodb setup for deployd on heroku

schettino72 — Sat, 26 Jan 2013 16:00:00 GMT

I am using deployd to prototype an applicaton. It's been great, really helped me focus on what matters for the prototype.

Today I reached the first milestone and decided to deploy it in heroku.

I had a small problem because deployd uses an object to config mongodb, but heroku provides only a URL to mongodb server... So here is the script I am using to run deployd on heroku:

var deployd = require('deployd');

// on heroku must use port from env
var port = process.env.PORT || 3000;

var url = require('url');
var db_url = url.parse(
    process.env.MONGOHQ_URL || "mongodb://:@localhost:27017/my_db_name");

var options = {
    port: port,
    db: {
        "host": db_url.hostname,
        "port": parseInt(db_url.port),
        "name": db_url.pathname.slice(1),
        "credentials": {
            "username": db_url.auth.split(':')[0],
            "password": db_url.auth.split(':')[1]
        }
    }
};

var server = deployd(options);
server.listen();

server.on('listening', function() {
  console.log("Server is listening on " + port);
});

server.on('error', function(err) {
  console.error(err);
  process.nextTick(function() { // Give the server a chance to return an error
    process.exit();
  });
});

strace & build-tools

schettino72 — Thu, 10 Jan 2013 21:00:00 GMT

strace is a utility to monitor system calls. I.e. it can report all files open by a process.

fabricate.py is a build-tool based on strace. The idea is pretty cool. It will execute all commands through strace. Than it can automatically figure out the dependencie and targets by looking at the strace output. So you don't need to explicitly specifying the dependencies and targets.

But these approach has a few problems:

strace will slow down the execution

(not only because of strace itself but also becaue it will have to parse strace output)

not always correct. i.e. run('cp', '-r', 'src', 'out')

If you just add a file to the src it can not figure out that a "dependency" was added.

confuse targets & dependencies

It checks the mode files were open, if open in write mode it is target. The problem is that some programs have their own cache system, so a target might be taken as a dependency.

Where these limitations are a problem or not depends on your use-case...

doit & strace

doit takes a very different approach to dependency handling. All dependencies must be explicitly defined.

doit doesn't support any kind of implicit dependency but it now comes with a strace command that lets you easily check which files are being used.

The idea is that you can use this feature while developing your tasks and make sure you are setting the dependencies correctly.

Example:

def task_o():
    return {'actions': ['cp abc abc2', 'touch xyz']}

$ doit strace -f traceme.py o
.  o
.  strace_report
R /xxx/abc2
R /xxx/abc
W /xxx/abc2
W /xxx/xyz

For more details check the docs.

python code coverage & multiprocessing

schettino72 — Sat, 29 Dec 2012 22:45:00 GMT

I wanted to get code coverage for my python code that uses the multiprocessing module.

The default way of coverage measuring in subprocess is to set the python interpreter to turn on code coverage right on start-up. But this technique won't work if the process being measured forks (as it does in multiprocessing). See issue.

I solved this by monkey-patching the multiprocess to programmatically start the coverage. Only the method Process._bootstrap needs to be monkey-patched. It is just wrapped with some code to start the coverage and save it when it is done.

from multiprocessing import Process

def coverage_multiprocessing_process(): # pragma: no cover
    try:
        import coverage
    except:
        # give up monkey-patching if coverage not installed
        return

    from coverage.collector import Collector
    from coverage.control import coverage
    # detect if coverage was running in forked process
    if Collector._collectors:
        class Process_WithCoverage(Process):
            def _bootstrap(self):
                cov = coverage(data_suffix=True)
                cov.start()
                try:
                    return Process._bootstrap(self)
                finally:
                    cov.stop()
                    cov.save()
        return Process_WithCoverage

ProcessCoverage = coverage_multiprocessing_process()
if ProcessCoverage:
    Process = ProcessCoverage

Note that the monkey-patch is only applied when the original process was being covered. This is done by checking if there are any Collector.

When running the coverage you need use the parallel-mode and than combine the results before creating report:

$ coverage run --parallel-mode my_program.py
$ coverage combine
$ coverage report

doit task creation

schettino72 — Fri, 14 Dec 2012 21:45:00 GMT

On my previous post I explained one way to create tasks for doit in a higher level than the default plain python dictionaries.

I just pushed a change that will allow the creation of not only decorators to generate tasks but also classes and instance objects.

The idea is very simple. Apart from collect functions that start with the name task_. The doit loader will now also execute the create_doit_task callable from any object that contains this attribute.

Note you will need the developement version of doit to run this code.

decorator example

##########################################################
## the decorator

def task(*args, **meta):
    def decorated(func):
        def create():
            task_dict = {'basename': func.__name__, 'actions': [func]}
            task_dict.update(meta)
            return task_dict
        func.create_doit_tasks = create
        return func

    if args:
        # decorator without parameters
        return decorated(args[0])
    else:
        return decorated


###########################################################
## task definition

DOIT_CONFIG = {'verbosity': 2}


@task
def simple():
    print "ho"

@task(file_dep=['dodo.py'])
def hello():
    print "hi"

class example

This interface was suggested by Thomas here.

class Task(object):
    @classmethod
    def create_doit_tasks(cls):
        if cls is Task:
            return # avoid create tasks from base class 'Task'
        instance = cls()
        kw = dict((a, getattr(instance, a)) for a in dir(instance) if not a.startswith('_'))
        kw.pop('create_doit_tasks')
        if 'actions' not in kw:
            kw['actions'] = [kw.pop('run')]
        if 'doc' not in kw:
            kw['doc'] = cls.__doc__
        return kw



class hello(Task):
    """Hello from Python."""
    targets = ['hello.txt']

    def run(self):
        with open(self.targets[0], "a") as output:
            output.write("Hello world.")

class checker(Task):
    """Run pyflakes."""
    actions = ['pyflakes sample.py']
    file_dep = ['sample.py']

Object example

class TaskHello(object):
    REG = [] # save instances

    def __init__(self, name):
        self.name = name
        self.REG.append(self)

    def say_hello(self):
        print "hello", self.name

    @classmethod
    def create_doit_tasks(cls):
        for inst in cls.REG:
            yield {
                'basename': inst.name,
                'actions': [inst.say_hello],
                }

######################

DOIT_CONFIG = {'verbosity': 2}

for name in ('foo', 'bar', 'spam'):
    TaskHello(name)

doit task decorator

schettino72 — Tue, 11 Dec 2012 03:45:00 GMT

doit uses plain python dictionaries to define tasks. Many, many people ask for a decorator based syntax. So here it is:

##########################################################
## the decorator

def task(*args, **meta):
    """decorator to attach task metadata to a function
    decorated function will become the task's action
    """
    def make_task(func):
        func._doit_task = True
        func._doit_meta = meta
        return func

    if args:
        # decorator without parameters
        return make_task(args[0])
    else:
        # decorator with task metadata
        return make_task


###########################################################
## task definition

DOIT_CONFIG = {'verbosity': 2}


@task
def simple():
    print "ho"

@task(file_dep=['dodo.py'])
def hello():
    print "hi"




############################################################
### boilerplate - convert decorated stuff to default doit style tasks

def task_all():
    for name, obj in globals().iteritems():
        if getattr(obj, '_doit_task', False):
            task_dict = {'basename': name, 'actions': [obj]}
            task_dict.update(obj._doit_meta)
            yield task_dict

Note that this will work with any version of doit there is no need to have any modification in doit internal code.

doit is a generic tool that aims to by different kinds of applications. There is no one task definition interface that will make everyone happy. doit provides the most basic and flexible one based on dicts...

I must agree that this decorator interface looks more readable :) But it has many limitations and can be used only for trivial tasks.

Limitations:

only one action per task
no support for command line (shell) actions
not easy to create several tasks with same actions

You could go one step further and create a custom task loader, so you could get rid of the task_all. I might add something like this for next doit release...

power up your tools

schettino72 — Thu, 06 Dec 2012 08:36:33 GMT

Goal

pyflakes is a static checker for python. It is great but it lacks a few features:

no parallel multi-processing execution
no cache to avoid checking files that were not modified
no option to have a long running process that watches the file system and automatically re-executes the checker when files are modified

These features are not specific to a static checker and are nice to have to many other tools. This post shows how to create an application adding those features. It uses pyflakes as an example but it could be applied to other tools also.

static checker tasks

doit is an "automation tool" that can execute tasks and provide the features which we are interested into. So the first step is to transform pyflakes operations into doit tasks.

In doit a task is composed of actions, and some other metadata.

actions

The action describes what the task does (some code to be executed). It can be a python function...

For pyflakes the function pyflakes.scripts.pyflakes.checkPath checks a file and returns the number of flakes or warnings found, so it is successful when zero flakes were found.

The action's return value is used to indicate if the execution was successful or not. So the action must return True if it has zero flakes

pyflakes checker action:

from pyflakes.scripts.pyflakes import checkPath

def check_path(filename):
    return not bool(checkPath(filename))

dependencies

A dependency indicates something that is required for a task execution.

For the pyflakes task, the file being checked is a file dependency for the task. That is, the task uses the file as input for its execution.

Explicit information of task dependencies are important for two factors:

1) cache results, if dependencies are not modified since last time the task was executed. It can use the results saved in a cache instead of re-executing the task.

2) execution order, when dealing with the execution of several tasks, the dependencies contains information on the order that tasks should be executed and whether they can be executed simultaneously (in parallel).

task creation

In doit usually task creation is done in a python module. This module functions that return/yield new tasks as a dict with metadata. It can also contain some extra configuration.

Something like:

import os
import glob

from pyflakes.scripts.pyflakes import checkPath


DOIT_CONFIG = {
    # output from actions should be sent to the terminal/console
    'verbosity': 2,
    # does not stop execution on first task failure
    'continue': True,
    # doit itself should not produce any output (use only actions output)
    'reporter': 'zero',
    # use multi-processing / parallel execution
    'num_process': 2,
    }


def check_path(filename):
    """execute pyflakes checker"""
    return not bool(checkPath(filename))

def task_pyflakes():
    """generate task for each file to be checked"""
    for filename in glob.glob('*.py'):
        path = os.path.abspath(filename)
        yield {
            'name': path, # name required to identify tasks
            'file_dep': [path], # file dependency
            'actions': [(check_path, (filename,)) ],
            }

If you drop the content above in a file called dodo.py in a folder. Executing doit from the command line you would execute pyflakes in all python modules in that folder.

It already has multi-process support. And it would execute only tasks that were failing or which checked module file was changed.

To execute in a long running process that watched the file system and re-execute tasks you could execute it as doit auto.

creating a new tool

Creating tasks in dodo.py works fine if you are working on your own project. But sometimes you just want to create a new application that you can easily distribute to other users without requiring them to add any special file into their project.

doit latest release (0.18) has exposed some of its internal API so you can create new applications and still use its task execution model.

The idea is that instead of loading tasks from a dodo.py module, the application itself create tasks and execute doit.

custom task loader

To create a custom task loader you should subclass doit.cmd_base.TaskLoader and implement the method load_tasks

pyflakes command line is extremely simple. It doesn't even support a --help option. It only takes positional parameter that specify python modules or folders containing python modules.

The first change from the previous example using dodo.py is to get a list of files in the same way pyflakes does instead of just getting all modules from a folder...

So on __init__ the loader will find the list of files to be checked.

from doit.cmd_base import TaskLoader

class FlakeTaskLoader(TaskLoader):
    """create pyflakes tasks on the fly based on cmd-line arguments"""

    def __init__(self, args):
        """set list of files to be checked
        @param args (list - str) file/folder path to apply pyflakes
        """
        self.files = []
        for arg in args:
            if os.path.isdir(arg):
                for dirpath, dirnames, filenames in os.walk(arg):
                    for filename in filenames:
                        if filename.endswith('.py'):
                            self.files.append(os.path.join(dirpath, filename))
            else:
                self.files.append(arg)

        for path in self.files:
            if not os.path.exists(path):
                sys.stderr.write('%s: No such file or directory\n' % path)

Than we need to change the function generating tasks to use the computed files instead of using glob.

    @staticmethod
    def check_path(filename):
        """execute pyflakes checker"""
        return not bool(checkPath(filename))

    def _gen_tasks(self):
        """generate doit tasks for each file to be checked"""
        for filename in self.files:
            path = os.path.abspath(filename)
            yield {
                'name': path,
                'file_dep': [path],
                'actions': [(self.check_path, (filename,))],
                }

Usually doit creates a file named .doit.db to store some information that will be used to determine which tasks are up-to-date. This file is created in the same location path as the dodo.py file. Since our new application works independent from the current path we specify the store file (dep_file) to be saved in the user's directory.

    DOIT_CONFIG = {
        'verbosity': 2,
        'continue': True,
        'reporter': 'zero',
        'dep_file': os.path.join(os.path.expanduser("~"), '.doflakes'),
        'num_process': 2,
        }

And finally we implement the TaskLoader interface:

    def load_tasks(self, cmd, params, args):
        """implements loader interface, return (tasks, config)"""
        return generate_tasks('pyflakes', self._gen_tasks()), self.DOIT_CONFIG

The command line implementation we just need to pass the positional parameters to the custom task loader. We also need to add a -w command line option to use the watch mode where the process keeps waiting for modifications in the file system.

from doit.doit_cmd import DoitMain

if __name__ == "__main__":
    cmd = 'run' # default doit command
    args = [] # list of positional args from cmd line
    for arg in sys.argv[1:]:
        if arg == '-w': # watch for changes
            cmd = 'auto'
        else:
            args.append(arg)
    doit_main = DoitMain(FlakeTaskLoader(args))
    sys.exit(doit_main.run([cmd]))

The full code can be found here doflakes.py .

power up your tools!

Using doit you can easily add some nice features to external tools or leverage its power in the tools you are the author!

doit has a very flexible dependency system that can be used to perform tasks much more complex than simple static checkers.

For more information check doit website.