Sandro Tosi

Mercurial: what's the tag containing this changeset?

2012-04-14T16:42:00.001+02:00

In software development, you often need to know: when was this feature released? in which releases was this changeset included?

Typical example is: you're debugging a problem, you find out what's the changeset introducing it, now you want to know when that changeset was firstly released (i.e. included in a tag) - how to do that?

hg log -r "cset:: and tag()"

where cset is the changeset you're looking for.

So know I know that Sphinx new way to load searchindex.js was introduced in the this changeset, and released for the first time in 1.0b2:

$ hg log -r "423faa03c908:: and tag()"
changeset: 2261:b494009dccf1
tag: 1.0b2
user: Georg Brandl <georg@python.org>
date: Sun May 30 19:52:08 2010 +0200
summary: Release preparation for 1.0b2.
...

awesome!

PS: why did I need to know that? because Matplotlib is still using the old way, but Debian needs the new one.

Get the lines unique on the first field(s)

2011-12-01T16:58:00.000+01:00

uniq is a great tool, since it returns the unique (adjacent) lines of the given input. But it has a limitation: it can't check for uniqueness only the first N fields (while it allows to ignore them, weird).

So, what to do if you have a long file, and lines with several fields, but you're only interested in getting the line with the different first 2 field (but all the rest of the line content? awk to the rescue!

$ awk '!x[$1]++' file

will print the (complete) lines of file that has the first field unique. You can set $1$2 to have lines unique on the first 2 fields, and so on. Thanks to this forum post, but there's some other interesting articles.

Trick of today: find -daystart

2011-11-03T09:33:00.003+01:00

What to get the files older than today? Run

find /path/ -type f -daystart -mtime +0

it will return only the files older than today, no matter the time the command is executed (by default, -mtime counts multiples of 24 hours from now). Kinda nice when you want to archive yesterday log files.

Print a NUL-terminated string with awk

2011-09-19T14:46:00.001+02:00

I thought it would have been easier to print a NUL-terminated string in awk (mawk as it's the default in Debian), but after some trial-and-fail I was able to come up with this kinda ugly solution:

$ echo -e "123\n456" | awk 'BEGIN { ORS="" } { print $0 ; printf("%c", "") }' | xargs -n1 -0 echo
123
456

That:

set the Output Records Separator (ORS) to the empty string (default is the \n, new line)
print the input line (do your elaboration there, if you need)
print the null character, as explained in the mawk manpage: "mawk cannot handle ascii NUL \0 in the source or data files. You can output NUL using printf with %c, and any other 8 bit character is acceptable input."
show that it's actually emitting NUL-terminated strings

There's really no better way to do that?

Mercurial: how to completely remove a named branch

2011-08-11T03:53:00.001+02:00

I like so much the git feature branch workflow, that in the early days of development on Python with Mercurial I created some named branches; well, that is something you should not do.

In Mercurial, the changeset contains the branch name, so you cannot develop on a separated (named) branch and then merge on default and hope that branch goes away, because it will stay.

What do I do now? Python Mercurial repository is quite big (around 200Megs) so I wanted to avoid to re-check it out. Thanks to the help of the folks on #mercurial (on freenode IRC network) I found my solution: strip the branch!

Please note that strip is dangerous. Use it only as last resort, and mind you can lose data with it. That said, it's a very powerful tool :) My main aim was to remove completely those named branches, leave no traces, and lose the changes I made on them. Another important aspect is that I didn't merged those branches on default.

So, how to get rid of a named branch:

$ hg strip "branch(${BRANCHNAME})"

and re-iterate for all the branches you have, that's it. Now, to be completely sure they were removed and no spurious changes are in the repository, you can:

$ hg pull -u
$ hg outgoing

and if it says "no changes found" you're sure that those branches are really gone.

And what am I now? A Python Core Developer!

2011-08-02T00:13:00.000+02:00

Yeah, since a couple of hours I'm officially a Python Core Developer (and this confirms it, so I'm not dreaming!)

I'm now in that mixed state in between the happiness and the fear I'll do stupid mistakes and I'll be ashamed of myself. But hey, it's only those who do nothing that make no mistakes.

Interesting days ahead, a lot of procedures to learn and get used to, hopefully also a lot of bugs fixed :) That's for sure, I'll go step by step, following the better be safe than sorry rule.

At the end, I'd like to thank all the people at Python that made this possible, they are quite a number, so if I'd named them, I surely forgot someone, and it would be unfair! So well, you know who you are, and this big THANK YOU is yours :)

Read multiple variables from stdin in a bash script

2011-06-15T09:43:00.000+02:00

I procrastinated to look up this for such a long time.. today was enough. What I want to achieve is reading a line from stdin and split it among several variables, naively I'd do:

$ echo "A BB CCC" | read a b c
$ echo $a $b $c

(no output.) As well explained here the above code doesn't work, but this one does:

$ read a b c < <(echo "A BB CCC")
$ echo $a $b $c
A BB CCC

as a hint, <(cmds) is called process substitution.

I'm going to EuroPython 2011

2011-05-19T01:10:00.000+02:00

I just got confirmation my company will sponsor me for EuroPython 2011 (thanks Register.it), so I'll be able to attend the whole week; a lot of amazing talks and the code sprints in the weekend: this is going to be a great time!

Are you coming?

Python: group a list in sub-lists of n items

2011-04-11T16:15:00.000+02:00

A long list, and you want to process its items n at a time; easy, but how to split that list in sublists of n elements (except the last one, of course)?

I looked a bit into the stdlib but it doesn't seem to exist anything I could use (oh, did I say I'm still on 2.4?) so I directed my research to Google, and found a nice recipe at ActiveState, but it has the problem it discards the last list, if it has less than n items.

Searching again, I got more lucky with this article: it's a generator of tuples from a list, splitting every n elements and optionally return the last semi-full tuple. I slightly modified it to obtain:

def group_iter(iterator, n=2):
    """ Given an iterator, it returns sub-lists made of n items
    (except the last that can have len < n)
    inspired by http://countergram.com/python-group-iterator-list-function"""
    accumulator = []
    for item in iterator:
        accumulator.append(item)
        if len(accumulator) == n: # tested as fast as separate counter
            yield accumulator
            accumulator = [] # tested faster than accumulator[:] = []
            # and tested as fast as re-using one list object
    if len(accumulator) != 0:
        yield accumulator

How would you have done it?

MySQL master/slave chain

2011-03-02T09:54:00.000+01:00

Have you ever needed to create a MySQL databases replication chain like A->B->C where B is slave of A and master of C? Me neither, until yesterday.

Since it took us about an afternoon to make it works (along with our DBAs, so we're not alone ;)) let's share some knowledge.

A very brief recap of how MySQL replication works:

slave I/O thread connects to the master, gets the new information from the binlog files and stores them in the relay log;
slave SQL thread reads the relay log and applies the changes to the slave database, without changing the slave binlog files.

That said, B replicates correctly from A, but C is unable to replicate from B because B doesn't change its binlog files with updates coming from A, because there's no changes done directly on B.

In order to make the chain works, you need to add the parameter log-slave-updates on B configuration: that will reply the changes from relay log to binlog, and so C will see the changes it needs to correctly replicate.

PS: mysqldump --master-data (executed on the slave server against the master) would help you set up the correct information for replication.

FTWCA Shani

2011-02-27T22:03:00.000+01:00

As you probably know, Shani is our pet hedgehog. We finally managed to setup a blog just for her (sorry, it's in italian only), and also we uploaded tons of her photos.

You don't remember her? Here's a cute reminder:

EuroPython 2011 @ Florence, IT - it's coming!

2011-01-21T08:40:00.004+01:00

Just when I was looking for a contact email to ask news about EuroPython 2011 dates... I noticed they are already there!! June 19 to 26 !

Book your flights, reserve your vacations, hope to see you all there!

PS: this year, PyCon Italia joins EuroPython in a single conference - the bigger the funnier

Re: Converting date to epoch

2010-12-29T09:37:00.002+01:00

Alexander,did you even consider that I might need to convert a date to epoch in a python script? :)

Convert a date to epoch

2010-12-28T23:00:00.002+01:00

it seems like an easy quest, ain't it? well, it took me far too long to get it done, so let just write it down, so that maybe I won't forget in 2 seconds:

>>> import time
>>> str = '2009-03-04'
>>> format = '%Y-%m-%d'
>>> time.mktime(time.strptime(str, format))
1236121200.0
>>> int(_)
1236121200

and here you have your epoch (if you just need seconds, use int()).

Welcome Shani!!

2010-12-11T23:46:00.005+01:00

It's been two weeks since our "family" has a new element: an african hedgehog. Her name is Shani, that means wonder in swahili.

she is so cute, isn't she?

bts-link has a new home!

2010-12-01T20:00:00.004+01:00

Since (already) several weeks, bts-link has a new home: busoni.debian.org! That's one of the machines where BTS is running, and it helps to let bts-link complete its job in about 30 mins each time. Thanks goes to Don Armstrong and Peter Palfrader for their support in the migration (from merkel).

Once again, let me encourage you to contact the bts-link developers mailing-list to request your project to be added to the tool in case it's not (we usually do some checks and adds from time to time, but if you notify us, you'll get it faster :) ).

"multi-strace"

2010-11-05T16:27:00.003+01:00

Thanks to a colleague, we hereby present the multi-strace:

strace $(pgrep httpd | awk '{ print "-p " $1 }' | tr '\n' ' ')

(it's left as exercise to the reader to understand what it does) if you want to save the output and the play with it, just add '-o <file>'.

Welcome, new handy tool!

Scribe (& thrift) for Debian Squeeze

2010-10-22T09:19:00.002+02:00

Some time ago, I asked what are the common solutions for services remote logging. It was suggested scribe, an application log server developed at Facebook and later released as an open source project, and we finally managed to get this ready for Debian squeeze.

Scribe requires a couple of other tools to work: thrift, a framework for services development (born at Facebook too), and thrift-fb303, a piece of thrift used as a bassline layer for Facebook services.

Searching the web for some work-in-progress of a Debian packaging, I found the wonderful work done by SimpleGeo; since I have some changes to do (and also take the occasion to play a bit with github), I forked their repos in these ones: scribe, thrift and thrift-fb303.

I've prepared a repository with those 3 packages, if you would like to try scribe, give it a look. They are for squeeze because, well, after all I need it for work and since it requires Python 2.6, using squeeze was the easiest choice (backporting 2.6 to lenny could be a mess, even if there are packages out there) and so I could also see how the upgrade from lenny would be.

Let's now dive into the doc, and some examples too, to find out the several configuration options and decide what's best for us.

URL shortcuts in Chromium

2010-10-22T09:04:00.004+02:00

If you like the Firefox feature to create special bookmarks that are actual templates for specific urls, well, here's how I got the same for Chromium.

That's actually how WordReference is suggesting, so it's defining new search engines: right click on the URL bar, Edit seach engine, and then Add; now you can define what you like, f.e. :

and so now typing "pts package" in the URL bar you'll get to the package PTS page.

That's one of the core points I had to get fixed in order to migrate from FF to chromium, one step closer then.

Feeling welcome: Python, you're doing it right!

2010-10-20T20:54:00.002+02:00

It was for a very long time (I think back from PyconIT 2) that I wanted to contribute to python. About a month and a half ago I started reading all the documentation I could about development processes & workflows, mercurial stuff (I don't have commit right, so I prefer working on a DVCS than on SVN), bugs management and so.

After all of this, less than month ago I started contributing very small patches; then when looking for "something to do" I found bugs that could be closed, in particular because they were already been fixed, and then last night R. David Murray proposed to give me "tracker privs", that moments after were granted \o/

What can I say? Thank you! that's something making me feel like I'm doing something useful to the language I love, and even if the privs are "not that much", it's a sign of trust, which really makes me happy!

As of now, I feel like I'm between "better be safe than sorry" and "it's easier to ask forgiveness than permission" mood, but we'll see how it goes.

Python, you're making me feel welcome, and so, once again, thank you!

HTTP requests specifying the Host header (in Python)

2010-09-17T15:53:00.002+02:00

Since it took me some time to find this solution, I think it might be worth to share it.

When you have a web server listening on a single IP address but serving several domains, it's quite common to run:

$ curl -H "Host: domain" http://ip_address/path/to/the/page.html

if you need to view that domain directly pointing to the web server (so avoiding any balancers or network "magics" you might have in place).

Well, the question is: how to do that in Python? I find my answer in the httplib module:

import httplib
conn = httplib.HTTPConnection("ip_address")
conn.putrequest("GET", "/", skip_host=True)
conn.putheader("Host", "domain.ext")
conn.endheaders()
res = conn.getresponse()
print res.read()

HTH

My nerd score is...

2010-04-20T13:20:00.000+02:00

oh my...

select count(bugs) in reportbug where status is open = 91

2010-04-14T18:30:00.006+02:00

well, the actual select (against UDD) would be

udd=> select source, all_bugs from bugs_count where source = 'reportbug';
source   | all_bugs
----------+----------
reportbug |       91

but the result is the same: less than 100 bugs against reportbug! Yay!

As you can see from the bugs graph, the Christmas and Easter holidays helped quite a lot to squash those bugs. When I first step into the reportbug world, the bugs count was around 250, and I first imagined in my mind to brake the 200 barrier, than the 150 and finally the less-than-100 one: while the first 2 limits where the hardest one (new to the codebase, other interest) the last one is the one I'm most proud of.

use SOAP to query the BTS (no comment here please...)
use the new.822 to query NEW or incoming and all other sub-queues
a proper unittest (and I hope the talk @ PyCon Italia will help on this) and check suite
code refactoring and clean-up of the several dead-branches laying in it
several other smaller things but still interesting

They seem quite nice-to-have stuff but recently I've tried to concentrate on reply fast to new report, fix as many reported crashes as possible and implement some of the long-awaited features.

Additionally, sometimes "No, sorry" is the proper reply to a bug and then it can be closed: it's hypocrite to keep bugs opened while internally you think it's a wontfix but you're not closing them because of a fear of possibly upset the reporters.

There will be more in the future, that's for sure, but I think this is the reportbug we'll see in squeeze, modulo some new crashes or very interesting feature we want it, so expect minor releases from now until squeeze+1

I'm going to Pycon Italia 4

2010-03-26T10:41:00.003+01:00

Yep, I'll be there this year too (sadly I can't say it's my fourth time, since I missed Pycon1).

I'm not that excited about the proposed talks, hey we don't need that much of Django :) I hope the invited speakers will surprise me, I'm confident the organization will do that.

See ya there!

Check Nagios from the desktop: nagstamon

2010-03-17T12:23:00.003+01:00

I just discovered nagstamon and all the team fallen in love with it!

I tried to use Nagios Checker, the Firefox plugin to notify of any Nagios alert, but that doesn't play nicely with several opened windows (alerts are multiplied for the number of windows, since it seems everyone does the checks, not just one) and it tends to slow down Firefox, that's already quite slow per se :)

The upstream author provides a Debian package, so promptly I wrote to him asking if he can consider maintain the package in Debian, with me as mentor/co-maintainer and so (it's in Python so I can that :) ); let's see how it goes.

Give it a try, it's really simple and awesome!