Recent notes from b.93z.org

Running CPython under Memcheck

2017-08-01T00:00:00+00:00

Recently I’ve been doing some things that involve use of ctypes. Running CPython under Memcheck (part of Valgrind) is known to have some quirks. Nevertheless, it is still useful for my particular case: finding memory leaks in a program that uses ctypes to call malloc but sometimes does not call free. Here’s a contrived example. It is a Python script (test.py) that allows introduction of deliberate memory leak:

import ctypes
import argparse


libc = ctypes.CDLL("libc.so.6")


def allocate_and_maybe_free(must_free):
    mem = libc.malloc(ctypes.c_size_t(1234))
    if must_free:
        libc.free(mem)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-f", "--no-free", default=False, action="store_true")
    args = parser.parse_args()
    allocate_and_maybe_free(not args.no_free)

When --no-free command-line argument is passed, args.no_free becomes True, so must_free (not args.no_free) in allocate_and_maybe_free becomes False, therefore libc.free(mem) is not called.

To use Valgrind (and its tools, like Memcheck) with CPython (3.6 in my case), debug-enabled build of the latter is needed:

$ ./configure --prefix /home/user/build --with-pydebug --with-valgrind
$ make install

Three options are passed above:

--prefix is usual installation prefix
--with-pydebug is recommended option for building debug-enabled interpreter
--with-valgrind makes interpreter automatically disable pymalloc memory allocator when running under Valgrind

After building & installation successfully finishes, python3 binary is installed into /home/user/build/bin/ directory (due to --prefix /home/user/build). Now let’s see what happens when memory is allocated and then correctly freed in test.py:

$ valgrind --leak-check=full --show-possibly-lost=no --show-reachable=no ./build/bin/python3 test.py
...
==...== LEAK SUMMARY:
==...==    definitely lost: 0 bytes in 0 blocks
==...==    indirectly lost: 0 bytes in 0 blocks
==...==      possibly lost: 1,993,410 bytes in 9,066 blocks
==...==    still reachable: 6,179 bytes in 18 blocks
==...==         suppressed: 0 bytes in 0 blocks
...

Memcheck reports no “definitely lost” memory. There is “possibly lost” & “still reachable”, but that’s CPython. But if deliberate memory leak is introduced by passing --no-free to test.py, Memcheck complains about “definitely lost” memory:

$ valgrind --leak-check=full --show-possibly-lost=no --show-reachable=no ./build/bin/python3 test.py --no-free
...
==...== 1,234 bytes in 1 blocks are definitely lost in loss record 22 of 34
==...==    at 0x...: malloc (in /usr/lib/valgrind/vgpreload_memcheck-....so)
...
==...==    by 0x...: ffi_call (in /usr/lib/.../libffi...)
...
==...==    by 0x...: _PyFunction_FastCall (ceval.c:4891)
==...==
==...== LEAK SUMMARY:
==...==    definitely lost: 1,234 bytes in 1 blocks
==...==    indirectly lost: 0 bytes in 0 blocks
==...==      possibly lost: 1,993,277 bytes in 9,065 blocks
==...==    still reachable: 6,179 bytes in 18 blocks
==...==         suppressed: 0 bytes in 0 blocks
...

As you can see, “definitely lost” are 1234 bytes allocated by libc.malloc(ctypes.c_size_t(1234)) that due to --no-free are not freed. So, Memcheck may help catch memory leaks in Python programs under CPython.

How to redirect to primary domain

2017-07-11T00:00:00+00:00

Quite often I encounter such redirects in Nginx configuration:

server {
    listen 80;
    server_name www.example.org;
    rewrite ^(.*)$ $scheme://example.org$uri;
}

I see three problems with this approach. Let’s fix them one by one.

First, use of regular expressions for such a simple case is unwarranted. return seems good enough:

server {
    listen 80;
    server_name www.example.org;
    return 302 $scheme://example.org$uri;
}

Second, in such cases we usually want permanent redirect. The rewrite use as in first snippet (rewrite ^(.*)$ $scheme://example.org$uri;) generates 302 Found. Let’s generate 301 Moved Permanently:

server {
    listen 80;
    server_name www.example.org;
    return 301 $scheme://example.org$uri;
}

Finally, $uri is not necessarily original URI. For redirects it may be better to use “full original request URI (with arguments)”—$request_uri:

server {
    listen 80;
    server_name www.example.org;
    return 301 $scheme://example.org$request_uri;
}

For example, my static blog engine (that powers this site) generates configuration with permanent redirect and $request_uri as I recommend here.

Unit-testing usage examples in README.rst

2017-01-27T00:00:00+00:00

Automatic testing of code examples in README is an area that is often neglected. I’d like to share some experience I got when was integrating doctests from README.rst of paka.cmark into existing unittest-based test discovery.

Let’s say there are following files:

somelib-project/
    README.rst
    somelib/
        ...
    tests/
        test_readme.py

To run tests from tests/ dir following command is executed from inside somelib-project/ dir:

$ python3 -m unittest discover --start-directory tests/

Inside test_readme.py file doctest examples are parsed from README.rst and then are exposed via load_tests protocol:

import doctest
import unittest


def load_tests(loader, tests, pattern):
    suite = doctest.DocFileSuite("../README.rst")
    tests.addTests(suite)
    return tests

This might be enough in general. Now for the specifics. Tests and code of paka.cmark have to support both Python 2.7 and 3.5 with single source. As paka.cmark.to_html accepts and returns unicode in 2.7 and str in 3, use of repr in doctests will fail for one version of language, and succeed for other. That is, not using “u” string prefix will cause failure under 2.7:

>>> cmark.to_html(u"Hello,\n*World*!")
'<p>Hello, <em>World</em>!</p>\n'

Failed example:
    cmark.to_html(u"Hello,\n*World*!")
Expected:
    '<p>Hello, <em>World</em>!</p>\n'
Got:
    u'<p>Hello, <em>World</em>!</p>\n'

And using “u” string prefix will result in fail under 3.5 (but tests will pass for 2.7):

>>> cmark.to_html(u"Hello,\n*World*!")
u'<p>Hello, <em>World</em>!</p>\n'

Failed example:
    cmark.to_html(u"Hello,\n*World*!")
Expected:
    u'<p>Hello, <em>World</em>!</p>\n'
Got:
    '<p>Hello, <em>World</em>!</p>\n'

I use a “hack”—wrap each call with print:

>>> print(cmark.to_html(u"Hello,\n*World*!"))
<p>Hello, <em>World</em>!</p>

But just wrapping them is not enough, as resulting HTML has newline at the end:

Failed example:
    print(cmark.to_html(u"Hello,\n*World*!"))
Expected:
    <p>Hello, <em>World</em>!</p>
Got:
    <p>Hello, <em>World</em>!</p>
    <BLANKLINE>

With doctest directive that problem is solved:

>>> print(cmark.to_html(u"Hello,\n*World*!"))  # doctest: +NORMALIZE_WHITESPACE
<p>Hello, <em>World</em>!</p>

If you don’t want to add such directives to each example individually, you can use optionflags argument of doctest.DocFileSuite to pass doctest.NORMALIZE_WHITESPACE (and thus use that directive for all examples):

import doctest
import unittest


def load_tests(loader, tests, pattern):
    suite = doctest.DocFileSuite(
        "../README.rst", optionflags=doctest.NORMALIZE_WHITESPACE)
    tests.addTests(suite)
    return tests

Adding hard line breaks to paka.cmark

2017-01-26T00:00:00+00:00

Recently I’ve been working on support for CMARK_OPT_HARDBREAKS in paka.cmark, and I’ve been thinking about design of API for paka.cmark.to_html. In particular, how to conveniently allow library user to specify hard breaks via breaks keyword argument without breaking compatibility with existing interface that accepts True (to render “softbreak” elements as line breaks) or False (to render “softbreak” elements as spaces) for breaks.

There are three option specification cases related to line breaks and HTML in cmark (C library):

When no line break-related options are specified, “softbreaks” are rendered as line breaks.
When CMARK_OPT_NOBREAKS is specified, “softbreaks” are rendered as spaces.
Finally, on CMARK_OPT_HARDBREAKS, “softbreaks” are rendered as <br>s.

In paka.cmark, first case is handled by breaks=True, and the second case by breaks=False. So the question is, what’s the best way (in this case) to handle third one. I don’t want to get rid of booleans, they are natural for “yes” and “no”, so let’s see how range of possible argument values can be augmented.

Enum

In addition to to_html user will have to import Breaks class (probably a subclass of enum.Enum):

>>> from paka.cmark import Breaks, to_html
>>> to_html(..., breaks=Breaks.soft)
>>> to_html(..., breaks=Breaks.no)
>>> to_html(..., breaks=Breaks.hard)

Though constants (e.g. HARD_BREAKS, NO_BREAKS) can be used instead, they’d still have to be imported, so there would be no real benefits comparing to use of enumeration.

String

Like “enum” variant, but no need to import anything except to_html:

>>> from paka.cmark import to_html
>>> to_html(..., breaks="soft")
>>> to_html(..., breaks="no")
>>> to_html(..., breaks="hard")

Combined approach

There’ll be Breaks class with two members:

class Breaks:
    soft = "soft"
    hard = "hard"

Breaks.no will not exist (so it will not be member of Breaks), as False is good enough, no need to reinvent it :) Then in to_html this will be handled in a way that does not set any cmark options when Breaks.soft is used:

opts = lib.CMARK_OPT_DEFAULT
if breaks:
    if breaks == "hard":
        opts |= lib.CMARK_OPT_HARDBREAKS
else:
    opts |= lib.CMARK_OPT_NOBREAKS

“Combined” variant will allow for enough flexibility, giving library user freedom to choose most suitable way to use paka.cmark:

>>> from paka.cmark import to_html
>>> to_html(..., breaks="hard")

>>> from paka.cmark import Breaks, to_html
>>> to_html(..., breaks=Breaks.hard)

So this is, roughly speaking, what I have implemented.

Automatic HTML escaping in Mako

2017-01-02T00:00:00+00:00

Some developers I know think that Mako templating library is “less secure” than, for example, Jinja2 (and, therefore, “must not be used”). Actually, Mako is as “secure” as you’ll configure it.

Though by default Mako does not HTML-escape rendered expressions, it can be configured to do that. When you create lookup (an instance of mako.lookup.TemplateLookup), you may pass arguments: list of directories to search for templates, encoding of templates, etc. Usually it looks like this:

import mako.lookup


lookup = mako.lookup.TemplateLookup(
    directories=[os.path.join(..., "templates")],
    input_encoding="utf-8",
    filesystem_checks=False,
    strict_undefined=True)

So, how to make lookup, as constructed above, escape rendered expressions? In addition to aforementioned arguments, TemplateLookup accepts default_filters—a list of filters to use on expressions by default:

import mako.lookup


lookup = mako.lookup.TemplateLookup(
    directories=[os.path.join(..., "templates")],
    input_encoding="utf-8",
    filesystem_checks=False,
    strict_undefined=True,
    default_filters=["h"])

For example, pyramid_mako uses default_filters=["h"] by default, so if you’re user of Pyramid framework you may have one less thing to worry about.

Why not default_filters=["str", "h"]? It may seem to work, but not for objects with __html__ method: these will be treated as text that needs escaping, not HTML that should be rendered as-is. Default filters are applied from left to right, so ${form.somefield()} will be transformed, roughly speaking, into h(str(form.somefield()))—which is, basically, h(form.somefield().__str__()). As you can clearly see, form.somefield().__html__ will not be called if default_filters are defined as ["str", "h"].

How to create .tar.gz reproducibly (putting it together)

2016-11-12T00:00:00+00:00

Though I recommend you to read .tar and .gz notes, where internals and features of these file formats and relevant modules from CPython standard library are discussed, you are free to skip them it you are in rush and need just code, because here we’ll do just that: code :)

Yes, here we are to make a script (mktgz.py) that’ll take paths to files and dirs and make .tar.gz—compressed archive.

#!/usr/bin/env python3

import os
import gzip
import tarfile
import argparse


def tgz(top_paths, *, mtime, dest, reltop, filename, verbose):
    relpath = lambda p: os.path.relpath(p, start=reltop)
    def _get_to_add():
        for top_path in sorted(top_paths):
            if os.path.isdir(top_path):
                for dirpath, dirnames, filenames in os.walk(
                        top_path, topdown=True):
                    dirnames.sort()
                    yield (dirpath, relpath(dirpath))
                    for filename in sorted(filenames):
                        filepath = os.path.join(dirpath, filename)
                        yield (filepath, relpath(filepath))
            else:
                yield (top_path, relpath(top_path))
    with open(dest, "wb") as _dest_file:
        with gzip.GzipFile(
                fileobj=_dest_file, mode="w",
                filename=filename, mtime=mtime) as dest_file:
            with tarfile.open(fileobj=dest_file, mode="w") as archive:
                for path, arcname in _get_to_add():
                    if verbose:
                        print(path, "->", arcname)
                    tinfo = archive.gettarinfo(path, arcname=arcname)
                    if mtime is not None:
                        tinfo.mtime = mtime
                    if tinfo.isreg():
                        with open(path, "rb") as file:
                            archive.addfile(tinfo, fileobj=file)
                    else:
                        archive.addfile(tinfo)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("paths", nargs="+")
    parser.add_argument("--reltop", default=os.getcwd())
    parser.add_argument("--mtime", type=float, default=None)
    parser.add_argument("--filename", default=None)
    parser.add_argument("--verbose", action="store_true", default=False)
    parser.add_argument("dest")
    args = parser.parse_args()
    tgz(
        args.paths, mtime=args.mtime, dest=args.dest, reltop=args.reltop,
        filename=args.filename, verbose=args.verbose)

Few quick notes about code. First, same mtime value is used for both “tar” and “gzip” parts of code. Second, gzip.GzipFile is used instead of gzip.open because the latter does not accept filename and mtime arguments. Oh, and if you are puzzled about reltop, please read the note about tar, reltop’s purpose is explained there.

Example usage:

$ ./mktgz.py dir1/ file2 dir2/ file1 out.tar.gz --mtime 0 --filename '' --verbose
dir1/ -> dir1
dir1/1 -> dir1/1
dir1/1/a -> dir1/1/a
dir1/1/b -> dir1/1/b
dir1/1/c -> dir1/1/c
dir1/1/d -> dir1/1/d
dir1/2 -> dir1/2
dir1/2/aa -> dir1/2/aa
dir1/2/bb -> dir1/2/bb
dir1/2/cc -> dir1/2/cc
dir1/2/dd -> dir1/2/dd
dir1/3 -> dir1/3
dir1/3/aaa -> dir1/3/aaa
dir1/3/bbb -> dir1/3/bbb
dir1/3/ccc -> dir1/3/ccc
dir1/3/ddd -> dir1/3/ddd
dir2/ -> dir2
file1 -> file1
file2 -> file2

As we expect, out.tar.gz is a gzip file:

$ file out.tar.gz
out.tar.gz: gzip compressed data, max compression

And tar also works with out.tar.gz as expected:

$ tar tvf out.tar.gz
drwxrwxr-x u/u             0 1970-01-01 03:00 dir1/
drwxrwxr-x u/u             0 1970-01-01 03:00 dir1/1/
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/1/a
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/1/b
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/1/c
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/1/d
drwxrwxr-x u/u             0 1970-01-01 03:00 dir1/2/
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/2/aa
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/2/bb
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/2/cc
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/2/dd
drwxrwxr-x u/u             0 1970-01-01 03:00 dir1/3/
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/3/aaa
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/3/bbb
-rw-rw-r-- u/u             5 1970-01-01 03:00 dir1/3/ccc
-rw-rw-r-- u/u             7 1970-01-01 03:00 dir1/3/ddd
drwxrwxr-x u/u             0 1970-01-01 03:00 dir2/
-rw-rw-r-- u/u            10 1970-01-01 03:00 file1
-rw-rw-r-- u/u            17 1970-01-01 03:00 file2

How to create .gz reproducibly

2016-11-11T00:00:00+00:00

Essentially, gzip file consists of “members”, each represented by following diagram:

+---+---+--+---+---+---+---+---+---+--+     +=========================================+
|___|___|__|FLG|     MTIME     |___|OS| ... |...original file name, zero-terminated...|
+---+---+--+---+---+---+---+---+---+--+     +=========================================+

Member parts not relevant to our discussion are not named, shown or described, but, if you want specifics, you can look at GZIP file format specification version 4.3 (RFC 1952) and gzip module implementation.

FLG is a flag byte that may indicate presence of original file name; as done in Python’s gzip module (more precisely, in GzipFile._write_gzip_header):

if fname:
    flags = FNAME
# ...
if fname:
    self.fileobj.write(fname + b'\000')

MTIME is “the most recent modification time of the original file being compressed” and “if the compressed data did not come from a file, MTIME is set to the time at which compression started”. In Python optional mtime argument (POSIX timestamp) is taken and set to self._write_mtime in constructor of GzipFile. Then self._write_mtime is used in _write_gzip_header method:

mtime = self._write_mtime
if mtime is None:
    mtime = time.time()

Finally, OS indicates the type of file system, and in Python’s gzip it is set to 255—“Unknown”:

self.fileobj.write(b'\377')

If you are not sure why 255, b'\377' is 0xff, which is indeed 255:

>>> b'\377'
b'\xff'
>>> 0xff
255

I guess “Unknown” value for OS is hardcoded in gzip library to ensure portability.

So, when you are making a gzip file, there are two result-influencing factors: original filenames and timestamps that are kept in members of file (type of filesystem—OS—is set by Python’s gzip library, so no need to worry about it).

Implementation

Our goal is to build a script (mkgz.py) that creates gzip file containing file pointed to by path passed as a command-line argument.

#!/usr/bin/env python3

import gzip
import shutil
import argparse


def gz(path, *, mtime, filename, dest):
    with open(path, "rb") as src_file, open(dest, "wb") as _dest_file:
        with gzip.GzipFile(
                fileobj=_dest_file, mode="w",
                filename=filename, mtime=mtime) as dest_file:
            shutil.copyfileobj(src_file, dest_file)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("path")
    parser.add_argument("--mtime", type=float, default=None)
    parser.add_argument("--filename", default=None)
    parser.add_argument("dest")
    args = parser.parse_args()
    gz(args.path, mtime=args.mtime, filename=args.filename, dest=args.dest)

So, having tar archive out1.tar, we could use ./mkgz.py out1.tar out1.tar.gz command to create gzip file out1.tar.gz, but it would contain timestamp and filename (command to get such output is file out1.tar.gz):

out1.tar.gz: gzip compressed data, was "out1.tar", last modified: Fri Nov 11 00:00:07 2016, max compression

As you may have noticed, mkgz.py can accept --mtime and --filename command-line arguments: they are passed to gzip.GzipFile constructor. Let’s use these arguments:

$ ./mkgz.py out2.tar out2.tar.gz --mtime 1.23 --filename ''
$ file out2.tar.gz
out2.tar.gz: gzip compressed data, last modified: Thu Jan  1 00:00:01 1970, max compression

As we can see above, timestamp (“last modified”) is set to one we passed as --mtime 1.23. filename is set to empty string.

Let’s make two more gzip files where only filename will vary:

$ ./mkgz.py out3.tar out3.tar.gz --mtime 1.23
$ ./mkgz.py out3.tar out4.tar.gz --mtime 1.23 --filename ''

If you’ll compare outputs of hd out3.tar.gz | sed 2q and hd out4.tar.gz | sed 2q, you’d see that out3.tar.gz indeed contains filename of out3.tar (while out4.tar.gz does not):

00000000  1f 8b 08 08 01 00 00 00  02 ff 6f 75 74 33 2e 74  |..........out3.t|
00000010  61 72 00 ed 9b 41 6e c2  30 10 45 bd f6 29 72 83  |ar...An.0.E..)r.|

00000000  1f 8b 08 00 01 00 00 00  02 ff ed 9b 41 6e c2 30  |............An.0|
00000010  10 45 bd f6 29 72 83 7a  6c 8f 7d 9e 24 a6 2a 52  |.E..)r.zl.}.$.*R|

Now that we looked at tar and gzip separately, we can proceed and combine them.

How to create .tar reproducibly

2016-11-10T00:00:00+00:00

Given that in tarfile default for format argument of tarfile.TarFile is GNU_FORMAT, I will use GNU tar, not POSIX, but in aspects I care about for this note they are very similar.

Tar archive is comprised of sequence of “file entries”. Each entry consists of header (metadata) and contents (data). Header contains mtime field, that represents file modification time at the time of archivation. In Python’s tarfile archive entries are represented by instances of tarfile.TarInfo, and mtime field of file entry header is represented by public field of tarfile.TarInfo instance. This is all specifics you need to know about tar archive format, but if you want non-simplified description, consult, for example, the relevant section of GNU tar manual.

So, when you are making a tar archive, there are 2 ways in which you can influence end result:

add files in particular order;
set added files’ modification time (mtime).

If (for your goals) you consider two identical files with differing mtime different, you don’t have to worry about it (but still you may want to add files in order). Otherwise, you’ll have to set it to some fixed value to make it irrelevant.

I am talking about realistic scenario, so it should be possible to add not just regular files, but also directories (recursively). Given that in such case we still care about file order, I will not use TarFile.add: it uses the os.listdir, that returns a list of dir entries in arbitrary order—and TarFile.add does not sort it nor allow us to do that:

for f in os.listdir(name):
    self.add(...)

Instead for for recursing into dirs I’ll use os.walk. Though it uses os.scandir internally (os.listdir in CPython < 3.5), which does not guarantee any ordering too, it is possible to affect walk so it does its work in particular order:

When topdown is True, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is False has no effect on the behavior of the walk, because in bottom-up mode the directories in dirnames are generated before dirpath itself is generated.

Thus, code that always traverses all dirs and files in top_dir is same order will look like this:

for dirpath, dirnames, filenames in os.walk(top_dir, topdown=True):
    dirnames.sort()
    # ...
    for filename in sorted(filenames):
        # ...

To add file entries (that is, regular files, dirs, etc.) into archive I’ll use TarFile.addfile instead of TarFile.add. Therefore, in general, tarfile-related code will look like this:

with tarfile.open(name="out.tar", mode="w") as archive:
    for path, arcname in ...:
        tinfo = archive.gettarinfo(path, arcname=arcname)
        tinfo.mtime = 123123.1
        if tinfo.isreg():
            with open(path, "rb") as file:
                archive.addfile(tinfo, fileobj=file)
        else:
            archive.addfile(tinfo)

Before getting to actual implementation, let’s discuss one more concept: arcnames. Arcname is name of file in archive, and it can, actually, be path: that is, it can be control.txt as well as somedir/1.txt or /home/user/some/file. For our purposes, having absolute file paths in archive is undesirable, so we have to “modify” them in a way that ensures arcnames do not include parents of “top dirs” we intend to add into archive.

Implementation

The goal is to build a script (mktar.py) that creates tar archive containing files and directories: paths are passed as command-line arguments.

#!/usr/bin/env python3

import os
import tarfile
import argparse


def tar(top_paths, *, mtime, dest, reltop, verbose):
    relpath = lambda p: os.path.relpath(p, start=reltop)
    def _get_to_add():
        for top_path in sorted(top_paths):
            if os.path.isdir(top_path):
                for dirpath, dirnames, filenames in os.walk(
                        top_path, topdown=True):
                    dirnames.sort()
                    yield (dirpath, relpath(dirpath))
                    for filename in sorted(filenames):
                        filepath = os.path.join(dirpath, filename)
                        yield (filepath, relpath(filepath))
            else:
                yield (top_path, relpath(top_path))
    with tarfile.open(name=dest, mode="w") as archive:
        for path, arcname in _get_to_add():
            if verbose:
                print(path, "->", arcname)
            tinfo = archive.gettarinfo(path, arcname=arcname)
            if mtime is not None:
                tinfo.mtime = mtime
            if tinfo.isreg():
                with open(path, "rb") as file:
                    archive.addfile(tinfo, fileobj=file)
            else:
                archive.addfile(tinfo)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("paths", nargs="+")
    parser.add_argument("--reltop", default=os.getcwd())
    parser.add_argument("--mtime", type=float, default=None)
    parser.add_argument("--verbose", action="store_true", default=False)
    parser.add_argument("dest")
    args = parser.parse_args()
    tar(
        args.paths, mtime=args.mtime, dest=args.dest, reltop=args.reltop,
        verbose=args.verbose)

So, for example, to create archive out.tar with mtime pre-set to 123, you’d use ./mktar.py dir1/ file1 dir2/ file2 out.tar --mtime 123. If, say, dir{1,2} and file{1,2} are in reproducible_tar/, and you want it to be “root” inside an archive, you could use --reltop ../ to modify “relative top dir” (reltop), so arcnames will be built like this (command is ./mktar.py dir1/ file1 dir2/ file2 out.tar --mtime 123 --reltop ../ --verbose and is run from inside reproducible_tar/):

dir1/ -> reproducible_tar/dir1
dir1/1 -> reproducible_tar/dir1/1
dir1/1/a -> reproducible_tar/dir1/1/a
dir1/1/b -> reproducible_tar/dir1/1/b
dir1/1/c -> reproducible_tar/dir1/1/c
dir1/1/d -> reproducible_tar/dir1/1/d
dir1/2 -> reproducible_tar/dir1/2
dir1/2/aa -> reproducible_tar/dir1/2/aa
dir1/2/bb -> reproducible_tar/dir1/2/bb
dir1/2/cc -> reproducible_tar/dir1/2/cc
dir1/2/dd -> reproducible_tar/dir1/2/dd
dir1/3 -> reproducible_tar/dir1/3
dir1/3/aaa -> reproducible_tar/dir1/3/aaa
dir1/3/bbb -> reproducible_tar/dir1/3/bbb
dir1/3/ccc -> reproducible_tar/dir1/3/ccc
dir1/3/ddd -> reproducible_tar/dir1/3/ddd
dir2/ -> reproducible_tar/dir2
file1 -> reproducible_tar/file1
file2 -> reproducible_tar/file2

Now we have working implementation of reproducible tar building, and we can look at gzip.

Reproducible builds and django.utils.feedgenerator

2016-06-20T00:00:00+00:00

Recently I’ve been working towards making builds of this blog reproducible. My goal was to allow use of regular diff for spotting differences between resulting files.

But there was a problem. Atom 1.0 feeds (e.g., /notes/feed/, /tags/django/feed/) are generated with my fork of django.utils.feedgenerator. Both original and fork use xml.sax.saxutils.XMLGenerator (implementation of ContentHandler interface) subclass called SimplerXMLGenerator for XML generation, and both pass elements’ attributes into startElement and startElementNS as regular dicts. This caused random ordering of XML elements’ attributes in resulting feeds: textually feeds were changing, while semantically they were not. Despite being inconvenient for my use case (when use of specialized tools for XML comparison is undesirable), such behavior is, according to specification, valid:

Note that the order of attribute specifications in a start-tag or empty-element tag is not significant.

So I decided to take advantage of that. Both startElement and startElementNS methods assume that attrs argument is an object that behaves like mapping (see lines 170 and 195 of Lib/xml/sax/saxutils.py). OrderedDict is a mapping (like dict is), therefore it is possible to provide attrs (attributes of XML element) as an instance of OrderedDict to preserve order of attributes. Least intrusive change—it’s a fork, after all—is to override two mentioned methods and sort attrs there (_order_attrs) before passing to implementation of superclass (XMLGenerator):

import operator
import collections
from xml.sax.saxutils import XMLGenerator


_order_attrs_key = operator.itemgetter(0)


def _order_attrs(attrs):
    return collections.OrderedDict(
        sorted(attrs.items(), key=_order_attrs_key))


class SimplerXMLGenerator(XMLGenerator):

    def startElement(self, name, attrs):
        return super().startElement(name, _order_attrs(attrs))

    def startElementNS(self, name, qname, attrs):
        return super().startElementNS(name, qname, _order_attrs(attrs))

    # ...here goes the rest (already present in feedgenerator)

That’s what I did. Now my blog engine does not randomly reorder attributes in feeds, and the latter are still perfectly valid XML :).

How to determine file where particular function is defined

2016-04-05T00:00:00+00:00

...at execution time, given that all you have is an instance of function object:

>>> type(func)
<class 'function'>

Knowing where it is defined may save you some time during debugging session: say, when you are not sure that function you are dealing with is imported from “correct” location. Despite being easy, such sanity check may help.

Most obvious option is to use co_filename attribute of function’s code:

>>> func.__code__.co_filename
'/home/user/.../something.py'

This also may work with methods:

>>> class SomeClass:
...     def some_method(self):
...         pass
... 
>>> obj = SomeClass()
>>> obj.some_method.__code__.co_filename
'<stdin>'
>>> obj.some_method.__func__.__code__.co_filename
'<stdin>'

Usually it’s better to use inspect.getsourcefile (that uses inspect.getfile, which uses something similar to above, but in more consistent way—for many different types):

>>> import inspect
>>> inspect.getsourcefile(func)
'/home/user/.../something.py'

But if func is defined in CPython extension (that is shared library which uses C API), inspect.getsourcefile will fail with TypeError:

>>> inspect.getsourcefile(func)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.../lib/python3.4/inspect.py", line 571, in getsourcefile
    filename = getfile(object)
  File "/home/user/.../lib/python3.4/inspect.py", line 536, in getfile
    'function, traceback, frame, or code object'.format(object))
TypeError: <built-in function func> is not a module, class, method, function, traceback, frame, or code object

In such case it is still possible to get file path of .so:

>>> inspect.getfile(inspect.getmodule(func))
'/home/user/.../lib/python3.4/site-packages/something.cpython-34m.so'