Matteo Landi

Windows tips: remap Ctrl to Escape

2014-02-11T18:00:00.000+01:00

The following is by far one of the most useful AutoHotkey snippet around:

Ctrl UP::Send {Escape}
Ctrl & F13::

Smells like black magic, doesn't it? In practice this guy is making the Ctrl key act like it was the Escape one, but only when pressed and released on its own.

Isn't that beautiful? -- I am talking to you, Vim users out there.

Oh, and it works even with your re-mapped Capslock!

What a joy, that made my day!

PS. It's a bit like xcape, but for Windows users.

Bunny1: the untold productivity tool

2014-02-05T14:19:00.002+01:00

How could I have missed Bunny1?

What the hell is that?

In essence Bunny1 is a one-page webserver in charge of redirecting all the input requests to external services supposed to be the best ones capable of handling them -- the requests.

For example, you send Bunny1 a request containing g lorem ipsum, it will parse the leading g, notice that g is an alias for google and answer with a redirect to Google.

Likewise, if you send it a request containing yt where the hell is matt, you will be redirected to Youtube.

Note that all the queries that cannot be managed by Bunny1 internally will be automatically redirected to an external service, YouNub, which is supposed to work the way Bunny1 does but better -- it should manage more external services.
Are you concerned about proxying all of your requests thru a Facebook server?

Host your Bunny1 server locally, its code is available on GitHub.

Do you find any of the proposed redirects wrong or not up to date?

Extend Bunny1! For example I did it to add support for Wikipedia:

def wikipedia(self, arg):
    """Searches Wikipedia or goes there"""
    if arg:
        return "http://en.wikipedia.org/wiki/Special:Search?search=%s&go=Go" % qp(arg)
    else:
        return "http://en.wikipedia.org"
# make wi an alias for wikipedia
wi = wikipedia

StackOverflow:

def stackoverflow(self, arg):
    """Searches StackOverflow or goes there"""
    if arg:
        return "http://stackoverflow.com/search?q=%s" % qp(arg)
    else:
        return "http://stackoverflow.com"
# make so an alias for stackoverflow
so = stackoverflow

And even for Javascript -- default direct seems to be broken:

def javascript(self, arg):
    """Search StackOverflow[Javascript] or goes there"""
    return self.stackoverflow('[javascript] ' + arg)
# make js an alias for stackoverflow[javascript]
js = javascript

I Am a Pentadactyl maniac, how could I use Bunny1 with it?

Add the following lines to your .pentadactylrc -- thanks Google Groups:
```
:bmark -title="bunny1 query" -keyword="bu" www.bunny1.org/?___=%s
:set defsearch=bu
```

jQuery Perspective

2014-01-31T08:56:00.002+01:00

Tonight I am gonna show you a very small project which kept me busy for the last week-end. Everything came out from this video (don't have to watch it all, 12 seconds should be enough):

Have you spot it? Did you notice what happens to pictures when the page is scrolled? Have you seen them moving up and down creating that perspective-like effect?

Well, I had to try and reproduce it, and that's how perspective was born.

Theory

Let me try and explain how that effect works with the aid of some pictures.

Let's start with the beautiful landscape we would like to take a picture of:

Now image to enjoy that landscape thru a window: it seems like our view is less detailed than before, isn't it?

It is not true: what happens indeed if you bend down a little bit? New details appear..

And if you jump on a chair? The same, some details appear, some details go away.

Very remarkable isn't it? Joking apart, we should have now all the pieces we need synthesize that perspective effect.

Practice

With the outcome of these observations at hand we are going to mimic that effect by means of a jQuery plugin. In particular, such plugin should:

take the pictures -- the landscape -- on which to apply the effect
embed them into cropping containers -- the window -- enabling us to hide or reveal details on demand
move the pictures up and down depending on the center of the point of view -- this is the twist: in the real world we are the ones moving in respect of the window while here we stand still and move the windows up or down

Collect the pictures

That is the easiest of the steps: create a new '.js' file containing the standard boilerplate code needed to extend jQuery:

(function ($) {
    $.fn.perspective = function() {
        return this.each(function() {
            // Logic to execute on each selected element
        });
    };
}(window.jQuery));

This should enable us to do something like:

$('img').perspective();

Or even better:

$('.perspective').perspective();

Crop selected items

Even this one shouldn't be an hard task -- at least for all the HTML5/CSS gurus out there; unfortunately I am not one of those, but with the help of Google I spot this interesting blog article.

When you know the trick, the rest is easy: let's create a new Perspective object (?!) in charge of creating the cropping element around each selected item.

(function ($) {
    var $window = $(window);

    var perspectives = [];

    var Perspective = function(element, options) {
        this.$element = $(element);
        this.options = $.extend({}, Perspective.defaults, options);
        this.$cropper = this.$element.wrap(
            $('<div>').addClass('perspective-cropper').css({
                position: 'relative',
                overflow: 'hidden',
                width: this.$element.width() + 'px',
                height: (this.$element.height() - 2 * this.options.maxOffset) + 'px'

            })).parent();
    };

    Perspective.defaults = {
        maxOffset: 30
    };

    $.fn.perspective = function(options) {
        return this.each(function() {
            var data = new Perspective(this, options);

            perspectives.push(data);
        });
    };
}(window.jQuery));

Offset images

We finally arrived to the last piece of the puzzle: set the offset of the cropped image given the position of the cropper container in respect of the visible portion of the page -- I wage you a dollar if you understood a bit of what I said.

Let me try again:

Is the cropper container in the bottom part of the visible area? Then shift the cropped image up.
Is the cropper container in the top part of the visible area? Then shift the cropped image down.
Is the cropper container in the middle of the visible area? Then do nothing.

More or less this translates into the following:

    ...

    Perspective.prototype.scroll = function(e) {
        var docViewHalfHeight = $window.height()/2;
        var docViewMiddle = $window.scrollTop() + docViewHalfHeight;

        var cropperHeight = this.$cropper.height();
        var cropperTop = this.$cropper.offset().top;
        var cropperMiddle = cropperTop + cropperHeight/2;

        var offset = cropperMiddle - docViewMiddle;

        var elementTop = cropperTop;
        // Make sure image is centered while in the middle of the view
        elementTop -= this.options.maxOffset;
        // Add an additional offset to create the perspective effect
        elementTop += this.options.maxOffset * offset / docViewHalfHeight;

        this.$element.offset({top: elementTop});
    };

    $.fn.perspective = function(options) {
        return this.each(function() {
            var data = new Perspective(this, options);

            perspectives.push(data);

            data.scroll();
        });
    };

    ...

Attach an handler to the 'scroll' event of the window:

    ...

    $(window).on('scroll', function(e) {
        $.each(perspectives, function() {
            this.scroll(e);
        });
    });
}(window.jQuery));

Done!

Showtime

I prepared a GitHub page to show you perspective in action: unfortunately it doesn't match the expectations -- the movement is not that smooth and to enjoy it best I suggest you scroll the page by using the browser scrollbar.

Anyway, it's a thing: enjoy it!

HackItaly Camp 2013: Barduino

2013-08-03T11:18:00.000+02:00

480 Hackers
2187 Red Bull
100 KG of Pasta
120 KG of Fish
138240 Minutes of Hacking
50 Projects

Our project? Barduino.

What is it? Have you ever heard about Bartrendo, the cocktail dispenser robot? Well, we decided to recreate our own version of it, because quoting Richard Feynman:

What I cannot create, I do not understand.

The hardware we had handy was not alike, but what made our product different was the way people could reserve their drink, that is leaving a message on a Facebook event wall.

How does it work? It is very simple:

Create a private Facebook event for the party and invite all the guests
Bring your Barduino to the venue and link it to the Facebook event using the admin interface (we used a small webapp running on the laptop communicating with the hardware)
Tell the guests the list of available drinks (i.e. the strings recognized as valid drinks)

That's all. Want a Negroni? Post a new message on the event wall containing the word negroni and Barduino will respond you with the number of queued drinks. When it is your turn it will tell you to come to the cocktail station, and grab your drink.

Here is the video we used for the pitch:

We know, we are not good at filming - and pitching too - too much focus on the admin interface and the Facebook interaction workflow rather than hardware closeups, but you know, when time is running out and you are feared not to be able to pitch anything at all, you start doing things chaotically.

Life is learning, enjoy your drinks!

RPN

2013-01-12T18:28:00.000+01:00

The very first time I heard about Reverse Polish Notation was at University, during the class of Digital Electronics.

There was that fool professor, a genius in reality, so pedantic while correcting your exam exercises and so rigorous with students [1], that one day begun talking about that crazy calculator (using reverse polish notation obviously!) he always had in his pocket.

"You don't need brackets.." he said, and again "..you don't even need to enter the full expression first to evaluate the result; rather you can start typing in operands and operators and the evaluation will be done automatically, when possible".

You know, from a student's point of view, he was too crazy to catch your attention, so, as the vast majority of the students attending that class, I started ignoring his small talk and reverse polish notation just became a fairy tale in my mind.

Time passed, and one day I stepped into a page of a guy talking about operators and their different notations. What really caught my attention was postfix notation (e.g. 1 2 + to get 3); and you know why? Because it recalled to my mind the reverse polish notation tale.

So I started digging on the Internet and found some interesting pages about this crazy calculator, about practical implications of that fool notation and even about possible implementations, in different programming languages, of RPN parsers.

I really liked the idea of a stack used to keep track of the operands of the expression to evaluate, and the fact that computations could be executed as as soon as an operator was read by the parser, and not at the end.

I immediately thought: How hard will it be to build a naive evaluator of reverse polish notation expressions? (with naive I mean at least to be able to process the four most important operators +, -, * and /)

Well, after a bit of tinkering I come up with the following straightforward solution:

import operator

OPERATORS = {'+': operator.add, '-': operator.sub,
             '*': operator.mul, '/': operator.div}

def rpn(it):
    def rpnreduce(operands, value):
        if value in OPERATORS:
            return operands + [OPERATORS[value](
                        *reversed([operands.pop(), operands.pop()]))]
        else:
            return operands + [float(value)]
    return reduce(rpnreduce, it, [])[0]

And it works too!

assert 3 == rpn('1 2 +'.split())
assert 9.5 == rpn('10 2 * 4 5 - + 2 /'.split())

Can you come up with a shorter or more elegant solution than mine? ;-)

[1] If during his lessons he had given us 15 minutes of rest and after 10 minutes all the students were already come back in class to carry on the lecture, he would have waited in front of the students for 5 interminable minutes before start speaking again!

Automate: from Vim to Blogger

2012-11-03T12:35:00.001+01:00

Our days are made of 24 hours, of which lot of them are rightly spent sleeping, eating, and carrying on social relations etc. I usually spend the remaining time in front of a monitor, randomly typing keys on the keyboard and doing lot of ... repetitive tasks.

I don't want to waste my time doing repetitive tasks; computers are so good at it, so why don't educate them to do the boring stuff for us?

As an example I will show how, with Vim, a couple of key mappings and one hell simple Vimscript function saved me a lot of time while writing new blog entries.

A case study: me

You know it, my blogging platform is Blogger, and for those of you who never used it, it offers two different ways to compose/edit posts: via a WYSIWYG editor, or using a plain textarea in which you can see and edit the HTML of the page to be published.

I use Vim on a day-to-day basis and I use it to write blog posts too; HTML is a very tedious language to work with, so I usually write new blog entries in Markdown, type a couple of commands to produce HTML, and finally copy the generated code into the prior mentioned Blogger textarea.

But, what are those couple of commands I type before switching to Blogger to paste the content in the textarea?

Generate HTML sources

For example, generate the HTML for the blog entry, and this is done with a simple:

$ markdown blog.mdown > blog.html

But that's not all! If you put one ore more headers into your source Markdown file (e.g. # Section, ## Subsection and so forth), after the conversion, those will be transformed into HTML header elements (e.g. <h1>Section</h1>, <h2>Subsection</h2> ...). Do you see the problem now?

Blogger embeds blog entries into a page which already contains one or more HTML header elements, and if you just copy and paste the generated HTML into the Blogger editor you will end-up with a page containing two h1 elements, maybe two or more h2 elements and so forth.

Consequently, before the copy and paste thing I have to edit the file and shift all the HTML headers 4 positions ahead, so that a generated h1 element will become a h4, a generated h2 element will become a h5, etc.

Don't ask why I go for 4 positions: I have always been used to it, and it works great! (at least for my theme)

Images

One good thing about Markup is that you can easily embed HTML code in it: markdown will automatically ignore HTML tags and will place them in the right position of the generated file.

Why am I saying it? Because unless you link images using URLs pointing to other web pages, Blogger wants you to upload files on their image hosting service and after a successful upload it will automatically link the image to your blog entry in a form of div + a + img tags.

As you can see, once you are satisfied with all the images of your post you have to switch to Blogger, upload all the images and finally copy all the automatically generated tags from the textarea back to your Markup file. Unfortunately, this is a necessary step, and till now I have not found yet a way to automate it.

So go ahead, and live with it!

Preview work in progress

Another key point of my workflow is previewing blog entries. I often want to have a look at how the post looks like in a browser (plain HTML, without Blogger stylesheets), and to do that I usually pipe the generated HTML code into programs like bcat or browser, so that a new tab with the blog entry will be automatically created a presented to me.

$ markdown blog.mdown | browser

Educate the editor

How much of the workflow presented so far could be automated? A lot, let's see how.

First of all, configure Vim by creating a custom filetype, blogger, and make it automatically loaded for *.blogger files (remember, the syntax for these files should be markdown):

augroup ft_blogger
    au!

    au BufNewFile,BufRead *.blogger setlocal filetype=blogger
    au Filetype blogger setlocal filetype=markdown

    au Filetype blogger setlocal spell
    au Filetype blogger setlocal norelativenumber nonumber

augroup END

Note that even if we globally bind the blogger filetype to all the files with a .blogger extension, internally we still want to use markdown as our language type so that we can use its syntax rules and its key mappings.

Now let's start to tackle all the problems presented before.

Previews

This is easy; simply create a key-mapping to invoke markdown and pipe its output to browser:

au Filetype blogger nnoremap <buffer> <localleader>p :!markdown % \| browser<cr>

You can change the mapping to whatever you like, but I think p (short for preview) shouldn't be too hard to learn for your muscle memory.

If you are wondering why I chose browser instead of bcat, here is the story: I have not been able to properly display pages containing link to local images, and I guess the problem is due to the temporary webserver which gets shut down before all the page content is actually fetched by the browser. I even tried -p and --persist switches with no success.

So I switched to browser, but unfortunately, it had its limitations too. The script basically pipes its input into a temporary file which is then opened with the browser. The main problem with it is that all the temporary files are created inside the /tmp folder, consequently if the page contains relative references to local images, all of them will end up being screwed up.

That's why I used my forked version of browser.

Generate, transform, copy

Now the tricky section; create a mapping for a procedure in charge of:

Generate HTML from Markdown
Shift header numbers
Copy everything to the clipboard

If you are a bit familiar with Unix command line tools, you should have noticed that every single step of the previous list can be done using a simple shell command (i.e. markdown, sed and xclip): we just need to find the right dash options!

The first point of the list is the easiest one; we just need to invoke markdown on the current file:

!markdown %

Now it is the turn of sed: with it, we have to shift all the HTML headers by 4 positions:

sed
\       -e "s:\(</*h\)1\(>\):\14\2:g"
\       -e "s:\(</*h\)2\(>\):\15\2:g"
\       -e "s:\(</*h\)3\(>\):\16\2:g"

What's happening here? We are passing sed three different patterns to transform strings like <h1> or </h1> into <h4> and </h4> respectively, strings like <h2> and </h2> into <h5> and </h5> and so forth: it is all about some marked subexpression magic!

Finally we want to copy the generated HTML into the OS clipboard. Once you have xclip installed on your machine, this step is really straightforward:

xclip -selection clip-board

Wondering why I am using -selection clip-board instead of copying everything in the X selection? Simple, not to lose everything the moment I open the Blogger page and try to delete all the content of the textarea: if you select all the text and then press Del, the X selection will then contain the deleted text, and all your previously generated HTML code will get lost!

Finally, create a function chaining all those three commands together:

function! GenerateTransformAndCopy() " {{{
    !markdown %
        \ | sed
        \       -e "s:\(</*h\)1\(>\):\14\2:g"
        \       -e "s:\(</*h\)2\(>\):\15\2:g"
        \       -e "s:\(</*h\)3\(>\):\16\2:g"
        \ | xclip -selection clip-board
endfunction " }}}

And create add a mapping for it (c short for copy):

au Filetype blogger nnoremap <buffer> <localleader>c :call GenerateTransformAndCopy()<cr>

Wrapping up

After putting all those things together, here is how my .vimrc file looks like:

" Blogger {{{

augroup ft_blogger
    au!

    au BufNewFile,BufRead *.blogger setlocal filetype=blogger
    au Filetype blogger setlocal filetype=markdown

    au Filetype blogger setlocal spell
    au Filetype blogger setlocal norelativenumber

    " Markdown the current file and pipe the output to bcat/browser
    au Filetype blogger nnoremap <buffer> <localleader>p :!markdown % \| browser<cr>

    " Markdown the current file, prepare it for Blogger, and finally copy it to
    " clipboard
    function! GenerateTransformAndCopy() " {{{
        !markdown %
            \ | sed
            \       -e "s:\(</*h\)1\(>\):\14\2:g"
            \       -e "s:\(</*h\)2\(>\):\15\2:g"
            \       -e "s:\(</*h\)3\(>\):\16\2:g"
            \ | xclip -selection clip-board
    endfunction " }}}
    au Filetype blogger nnoremap <buffer> <localleader>c :call GenerateTransformAndCopy()<cr>
augroup END

}}}

And here we are: no more switch back and forth between VIM and your favourite terminal emulator to preview the current file or prepare the generated code to be pasted into your favorite blogging platform.

A couple of keystrokes and your time will not be wasted anymore. And remember: focus on the worthy tasks only!

Ternary Search

2012-11-01T11:59:00.001+01:00

Today I would like to talk you about a simple but at the same time very powerful algorithm handy to minimize (or maximize) unimodal functions: Ternary Search.

The origin of its name comes from the fact that the search domain is reduced by one third on each iteration : it is not as fast as binary search but remember that (1) they serve different purposes and (2) it has still logarithmic complexity, so it should be fast enough!

I ain't good at explaining theoretical stuff so rather than struggle trying to give you an approximate definition of the algorithm, I will point point you out to the Wikipedia page and then spend the rest of the time presenting you a visual explanation of the algorithm and finally tell you where I happened to use it to solve non-trivial tasks.

Just a bit of theory

For the lazy readers who haven't clicked the previous link, the following quoted paragraphs should be enough to understand what is coming next:

A ternary search algorithm is a technique in computer science for finding the minimum or maximum of a unimodal function (function that is either strictly increasing and then strictly decreasing or vice versa).

And:

A ternary search determines either that the minimum or maximum cannot be in the first third of the domain or that it cannot be in the last third of the domain, then repeats on the remaining two-thirds.

In particular, the last paragraph tells us that in order to find the minima of the function, we have to iteratively split the input domain into three separate regions and discard the one for which we know for sure the minima won't be in there.

Seems easy, isn't it? But how can we decide which of the three sub-domains is the one to be discarded? How can we compare the output of our function on those boundary points? Let's try to analyze all the possible scenarios:

func(left_third) > func(right_third)

What can we assert knowing that? Well, if we combine it with the fact that we are trying to find the minimum value of a unimodal function, for sure we can assert that we won't find that minima in between the range [left, left_third]. Why are we so confident about that? Because if the minima was there, func(left_third) should have been smaller than func(right_third) (remember, the function is unimodal)! Summarizing, we define the new search domain as: [left_third, right].

func(left_third) < func(right_third)

If you are still with me, you would have noticed that this is the opposite of the previous case; this time however, we know for sure that the minima won't lie in between [right_third, right] and consequently we will cast our search domain into [left, right_third].

func(left_third) = func(right_third)

That is the easiest scenario: the minima is in between left_third and right_third.

Wrapping up, the idea is very simple: you have to minimize a function for a given domain and you know that in such domain the function is unimodal; you split the current domain in three parts and discard the one for which you know for sure the minima won't lie in there; finally you carry on the process until the size of current search domain lessens a pre-defined threshold (e.g. desired precision).

Practice

Let's start with some practical stuff: imagine we want to minimize the function 3 * (x - 0.25)**2 - 1 in the domain [0, 1].

The algorithm tells us to split the function domain in three slices and to evaluate the output of the function on those cut-off points.

Because we are going to implement the algorithm step-by-step using the Python interpreter, let's start defining our function to minimize and its initial domain:

>>> from __future__ import division
>>>
>>> def func(x):
...     return 3 * (x - 0.25) ** 2 - 1
...
>>> (left, right) = [0, 1]

Now calculate the cut-off points used to create the three domain sections:

>>> left_third  = ((2 * left) + right) / 3 # left + ((right - left) / 3)
>>> left_third
0.3333333333333333
>>> right_third = (left + (2 * right)) / 3 # left + ((right - left) * 2 / 3)
0.6666666666666666

Finally, evaluate the function in such points:

>>> func(left_third)
-0.9791666666666666
>>> func(right_third)
-0.47916666666666674

As you can see, func(left_third) is smaller than func(right_third), consequently we need to strip the range [right_third, right] from our search domain and then carry on the processing:

>>> right = right_third
>>> left_third  = ((2 * left) + right) / 3
>>> left_third
0.2222222222222222
>>> right_third = (left + (2 * right)) / 3
>>> right_third
0.4444444444444444
>>> func(left_third)
-0.9976851851851852
>>> func(right_third)
-0.8865740740740741

OK, same situation as before, but I think you got the point.

Why don't we create a function that implements all the reasoning explained so far? On Wikipedia there is already an implementation of the algorithm based on recursion, so I wrote my own iterative version:

>>> def minimize(func, a, b, precision):
...    (left, right) = (a, b)
...    while True:
...        if right - left <= precision:
...            return (left + right) / 2
...        left_third = ((2 * left) + right) / 3
...        right_third = (left + (2 * right)) / 3
...        if func(left_third) > func(right_third):
...            (left, right) = (left_third, right)
...        else:
...            (left, right) = (left, right_third)
...
>>> minimize(func, 0, 1, 10e-7)
0.24999996832273685

If you are wondering why the result is not a sharp 0.25 (as expected), bear in mind that we are looking for the minima using an iterative approach and not a mathematical approach: imprecisions are likely to be there! However, you can always use strings formatting magic to beatify the result:

>>> print("{0:.7f}".format(minimize(func, 0, 1, 10e-7)))
0.2500000

Bingo! What if, instead of looking for the minima, we wanted to maximize the function? There is a straightforward solution for that: closures. We can use them to change the sign of all the values returned by the function we want to maximize, and then use minimize() to do all the dirty job:

>>> def neg(func):
...    def inner(*args, **kwargs):
...        return -1 * func(*args, **kwargs)
...    return inner
...
>>> def func1(x):
...    return -abs(x - .5)
...
>>> minimize(neg(func1), 0, 1, 10e-7)
0.5000000933420725

Or if you prefer, we can just create a maximize() function doing all the wrapping for us:

>>> def maximize(func, a, b, precision):
...     def neg(func):
...         def inner(*args, **kwargs):
...             return -1 * func(*args, **kwargs)
...         return inner
...     return minimize(neg(func), a, b, precision)
...
>>> maximize(func1, 0, 1, 10e-7)
0.5000000933420725

Other applications

The first time I read about this algorithm was during the Google Code Jam edition of 2009. I spent so much time trying to implement the right math algorithm to solve the puzzle Center of Mass, and when the contest analysis was out, I found out that a stupid-simple ternary search algorithm could have been used to solve that.

The same happened to me three years later during the Google Code Jam edition organized for the EuroPython 2012. This time the name of the puzzle was Shoot the Target and as happened before, I wasted all my time trying to implement a solution which was completely wrong (fuck!). I even thought about using ternary search to solve the problem, but I was so sure about the correctness of my solution that I spent all the available time trying to find rounding bugs and error like those (duh). One week later, when the contest was over, I tried to solve the problem using ternary search and in a handful of minutes I had a solution running.

Always strive for the simplest solution that could possibly work. Once you have something working, if necessary, start to optimize it!

Gimme the code

All the code I presented so far can be downloaded from this gist.

Cheers.

Linux and the Apple wireless keyboard: the insert key

2012-09-23T11:46:00.000+02:00

Are you a user of the Apple Wireless keyboard?

I am!

Are you bored with the missing Insert key?

I am!

Are you a Linux user?

I am!

Then

Please, use Xmodmap to map the combination Shift+BackSpace to the Insert key and go on with pasting things in the terminal without using your mouse!

How to do that?

Add the following to your Xmodmap configuration file (i.e. ~/.Xmodmap):

keycode  22 = BackSpace Insert

Then start/restart Xmodmap:

$ xmodmap ~/.Xmodmap

How it works?

What Xmodmap will do under the hoods is to map the BackSpace key (which on my keyboard correspond to keycode 22) to the BackSpace keysym if used without modifiers, and to the Insert keysym while used together with the Shift modifier.

How do I find the specific keycode for my keyboard?

The answer is xev.

Open a terminal and type xev | grep keycode, then look at the output of the program while typing keys on your keyboard:

$ xev | grep keycode
state 0x10, keycode 26 (keysym 0x65, e), same_screen YES,
state 0x10, keycode 22 (keysym 0xff08, BackSpace), same_screen YES,

As you can see the output contains both the information of the keycode (e.g. 26 and 22) and the name of the associated keysym (e.g. e and BackSpace).

That's all folks!

Facebook login on top of webpy

2012-09-15T14:18:00.000+02:00

Have you ever tried to add a Facebook login link to your webpy application? If not, today I will try and explain how to do that, without the aid of any external module.

I found this useful page describing the steps to implement the Facebook Authorization hand-shake on top of the Google App Engine framework, so I adapted it to to make it work with webpy.

Authentication workflow

First and foremost, have a look at this page if you are not familiar with the Facebook OAuth2 authethication flow: it contains the basic steps to allow applications to make requests to the Facebook Graph API.

When done, go ahead.

Hello world!

Let's start with a classic: the beloved Hello world! page, with a small login link on top of it.

To achieve that, create the main page handler:

class MainHandler(object):
    def GET(self):
        return render.index()

Create a file index.html with the following content (I use Jinja2 as template engine: see this guide if you want to configure webpy application to use it):

<html>
    <body>
        <p style="font-size: small"><a href="/login">Facebook login</a></p>
        <p>Hello, world! </p>
    </body>
</html>

Done!

Login handler

Now, it is time to enter the hot section of the post: write a request handler implementing the OAuth2 authentication flow.

Redirection

The first step of the procedure is to redirect the user to the Facebook Authorization URL (http://www.facebook.com/dialog/oauth), where she will be shown the application authorization screen and she will be able to authorize -- or not -- our application. The Authorization URL should include the id of your application (given by Facebook during the App registration procedure) and the redirect URL used by the auth server to drive the user back to your site after a successfull hand-shake.

At the time being, webpy does not expose a built-in function to get the path_url of the current request, so we create a helper function for that:

def path_url():
    return web.ctx.home + web.ctx.fullpath

Then we start to fill our LoginHandler class with the redirection logic:

args = dict(client_id=FACEBOOK_APP_ID, redirect_uri=path_url())
web.seeother(
        'http://www.facebook.com/dialog/oauth?' + urllib.urlencode(args))

Access token

The user will be redirected to the page specified inside redirect_uri after she has granted permissions to our application. The server doing such redirection will add an addictional code argument to the request: such argument will be used to get the access token to make API requests.

Once inside the handler, we need to read the input code value, add the application secret token, and finally make a request to https://graph.facebook.com/oauth/access_token to get the access token.

If you implement the logic above enhancing the previous snippet, you will end up with something similar to this:

data = web.input(code=None)
args = dict(client_id=FACEBOOK_APP_ID, redirect_uri=path_url())
if data.code is None:
    web.seeother(
            'http://www.facebook.com/dialog/oauth?' + urllib.urlencode(args))
    return

args['code'] = data.code
args['client_secret'] = FACEBOOK_APP_SECRET
response = urlparse.parse_qs(
        urllib.urlopen(
            "https://graph.facebook.com/oauth/access_token?" +
                urllib.urlencode(args)).read())
access_token = response["access_token"][-1]

Note that the presence of the code argument differenciates the first step from the second.

Facebook profile

We have the access_token, then we are allowed to query the Graph API for information about the user's profile:

profile = json.load(
        urllib.urlopen(
            "https://graph.facebook.com/me?" +
            urllib.urlencode(dict(access_token=access_token))))

Session management

The login procedure has come to its end but before redirecting the user back to the home page we need somehow to store login information in a way that it can be accessible while viewing other pages.

We will use cookies for that. In particular, we will store the id and the name of the user so that the home page will be able display the name and the profile picture of the authorized user.

web.setcookie(
        'fb_userid', str(profile['id']), expires=time.time() + 7 * 86400)
web.setcookie(
        'fb_username', str(profile['name']), expires=time.time() + 7 * 86400)
return web.seeother('/')

Logout handler

There ain't a login page without a logout one, so let's create the LogoutHandler class containing the necessary lines to clear cookies set in the previous step.

class LogoutHandler():
    def GET(self):
        web.setcookie('fb_user', '', expires=time.time() - 86400)
        web.seeother('/')

Modified Hello, world!

We are almost done. We just need to apply some changes to the MainHandler class and to the index.html template page to take into account logged-in users.

To do so, we create a helper class (inherited by all request handlers) which exposes the method current_user(). Such method reads the cookies associated with the connection and returns a dictionary containing the id and the name of the logged-in user, if any.

class BaseHandler():
    def current_user(self):
        if not hasattr(self, "_current_user"):
            self._current_user = dict()
            user_id = web.cookies().get('fb_userid')
            user_name = web.cookies().get('fb_username')
            if user_id and user_name:
                self._current_user = {'id': user_id, 'name': user_name}

        return self._current_user

We modify MainHandler to extend the BaseHandler helper class and to pass the current logged-in user to the template engine:

class MainHandler(BaseHandler):
    def GET(self):
        return render.index(user=self.current_user())

Finally, we modify the index.html template page to add support for the input user dictionary:

<html>
    <body>
        {% if user %}
            <p style="font-size: small"><a href="/logout">Logout</a></p>
            <p>Hello, <a href="{{ user.profile_url }}">{{ user['name'] }}</a>!</p>
            <p><img src="https://graph.facebook.com/{{ user['id'] }}/picture" /></p>
        {% else %}
            <p style="font-size: small"><a href="/login">Facebook login</a></p>
            <p>Hello, world! </p>
        {% endif %}
    </body>
</html>

If you open again the index page, click on the Facebook login link and wait for the authentication hand-shake to be over, the page will be automatically reloaded and you should be able to see your Facebook name and profile picture.

Wrapping up

You have just learned how to integrate a Facebook login link to your webpy application. Simple, isn't it?

Where to go next? Use a database to keep track of the users connected with your application (I created a Bitbucket example project implementing that), or try to integrate python-oauth2 within your application.

Threading with Tkinter, done properly

2012-06-03T12:57:00.000+02:00

Lately, I spent my spare time working on logfilter, a mix of tail -f and grep -e PATTERN alike commands, with a rocky GUI to display results.

In a nutshell, logfilter starts to poll a file waiting for new content; when a new line is added, configured filters are applied to it and then passed to the GUI, which in turn colors matching part of that line and displays the result inside a text widget.

It was my first time programming with Tkinter, so before starting to code, I dug the documentation page looking for tips and advices. The most important I come up with was:

Never ever update your widgets outside the mainloop thread!

I made a bit of GTK programming before, so I was already used to this: do your CPU intensive processing in a separate thread/process, and when done, schedule a GUI update (e.g. gobect.idle_add(func, args...)). After some Googling, I found this page, and found out that tk.after_idle was the function I was looking for:

Register an idle callback which is called when the system is idle (that is, when there are no more events to process in the mainloop). The callback is only called once for each call to after_idle.

With all that knowledge gathered so far, I finally started to code, and in a couple of days the first alpha version of logfilter was there, ready to be tested. As you could imagine, problems were waiting me, just behind the corner. Indeed, as soon as I started to stress the application, I immediately noticed strange behaviours, a clear sign that something has been done wrong.

First and foremost, the application was crashing over and over again if run on Windows. In particular, if the file analyzed contained a lot of matching lines, the GUI was able to execute a handful of text widget updates before crashing (obviously, without leaving any useful error messages). Given that on Ubuntu everything was working flawlessly, I tried to update Python and Tkinter libraries to a more recent version. I also tried to implement some coalescing-alike mechanism to limit the amount of GUI updates done per unit of time, but unfortunately, none of those were helpful to solve the issue.

Second problem, a systematic deadlock while quitting the application during updates of the text widget. No matter how much effort did I put to implement the graceful shutdown of the application; every time I pressed that window's X button while updating the text widget, one of the active threads (the one in charge of schedule GUI updates using tk.after_idle), did not stop correctly. I tried to debug the application using both log messages and also the Eclipse debugger, but again, I was still stuck at the same point.

As my last call, I decided to write a couple of emails to the tkinter-discuss and python-list mailing lists, and luckily, after some investigation, it turned out that my use of tk.after_idle was wrong, and that it was by no means thread-safe as I thought it to be (the function, is indeed expected to be invoked inside the mainloop thread).

The solution was simple (you can find the implementation here): threads in need of GUI updates, will push function calls into a synchronized queue shared with the GUI; the GUI, periodically, will pop items from such a queue, and safely invoke the method tk.after_idle to actually schedule the update. That's it!

It goes without saying that the above implementation solved both the issues presented before!

Prompt user for input while processing a file, line by line, in Bash

2012-01-15T11:02:00.000+01:00

Have you ever had the need in Bash of looping on a file (or the output of a piped command), line by line, and then in the middle of such process, ask user for inputs (e.g. a confirmation of an action to be taken)? Well solving this has not been as easy as I expected.

The task

I come into this problem while writing a Bash script for semi-automatic merge of two subversion repositories (henceforth development and release). What I had in mind was very simple:

Run a recursive diff between the two repositories;
Parse the command output so that we are able to distinguish between files to be merged and files to be imported from development to release;
Ask user for confirmation, and then act consequently;

Let's start with the function which actually does the merge:

function do_merge
{
    local src=$1
    local dst=$2

    if [ -f ${dst} ]; then
        read -p "Merge ${dst}? [N/y] "
        if [ "${REPLY}" = 'y' ]; then
            # prefer the destination file to be on the left
            vimdiff ${dst} ${src}
        fi
    else
        read -p "Import ${src}? [N/y] "
        if [ "${REPLY}" = 'y' ]; then
            cp -R ${src} ${dst}
        fi
    fi
}

Its logic is very simple: if the destination file (${dst}) exists, we are asked to merge two files, consequently ask the user for confirmation and then fire up vimdiff. On the other hand, if the file does not exist and the user answered y, copy the file to the new location.

Place the above code inside a temporary file (~/tmp/foo.sh) and test it:

> cd ~/tmp
> mkdir a b
> fortune > a/hello
> (source ~/tmp/foo.sh; do_merge a/hello b/hello)
Import a/hello? [N/y] y
> fortune > a/hello
> (source ~/tmp/foo.sh; do_merge a/hello b/hello)
Merge b/hello? [N/y]

As expected, on the first run file a/hello is imported into b/hello; on the second run, the two files are merged.

Have a look now, at the code of the function which analyzes the two repositories and calls do_merge (when needed) to synchronize them:

function merge
{
    local line src dst
    local development=$1
    local release=$2

    diff -urq ${development} ${release} | \
    grep -v '.svn' | \
    while read line; do
        set -- ${line}
        # For files exising only inside one of the two directories, ``diff``
        # output a line such as 'Only in directory: file'.
        # In this case $3 is equal to the directory containing the new file $4;
        # such file should be imported only if it belongs to the ${development}
        # directory
        if [[ "${line}" =~ 'Only in' ]]; then
            if [[ $3 =~ ${development} ]]; then
                # remove trailing ':' from directory and append filename
                src=${3%%:}/$4
                dst=${release}/${src##${development}/}
                do_merge ${src} ${dst}
            fi
        # For existing files which differ, ``diff`` output a message like 'Files
        # /path/file1 and /path/file2 differ.'
        # Files to be merged are inside $2 and $4 respectively.
        elif [[ "${line}" =~ 'Files ' ]]; then
            src=$2
            dst=$4
            do_merge ${src} ${dst}
        else
            echo 'Aw shit!'
        fi
    done
}

Even if comments should be self-explanatory (I hope so), here we run diff between the two development and release repositories, exluding .svn entries, preparing do_merge arguments based on the output of previous commands, and finally invoke the function.

Let's see if it works as we expect:

> (source ~/tmp/foo.sh; merge a b)
>

Nothing happens!

However, echoing do_merge calls produces the right output:

> (source ~/tmp/foo.sh; merge a b)
do_merge a/hello b/hello

What is happening here? Why aren't we prompted for confirmation?

The problem

Have you ever heard about subshells? If not, please spend a couple of minutes reading the previous link: it is surely worth it.

Inside the while loop (and consequently inside the body of the function do_merge) we are actually inside a subshell which is using diff piped output as input; consequently, there is no hope for the aforementioned function to wait for user key-press actions because kind of detached from the parent shell (the one running the function merge, and the one enable to intercept user input).

The solution

The solution is pretty simple, once you have found this.

We need to to force all the read -p operations inside the subshell to be fed directly by the keyboard; on Unix this is achieved appending </dev/tty to those guilty lines. Another option (which I prefer most), is to append redirects to function calls and leave the function do_call as is (as presented below):

...
dst=${release}/${src##${development}/}
do_merge ${src} ${dst} </dev/tty
...
dst=$4
do_merge ${src} ${dst} </dev/tty
...

Now run the script again:

$ (. ~/tmp/foo.sh; merge a b)
Import a/hello? [N/y]

and be joyful! ;-)

About CakePHP and Vagrant!

2012-01-08T10:36:00.000+01:00

Weeks ago I found this, an article describing how to test your CakePHP application using a Vagrant-powered virtual environment; unfortunately the author makes use of a custom box already configured with PHP, MySQL, etc, which is a solution I don't find quite reliable (and the fact that the author decided to create a backup torrent with that box, is an action that speaks for itself).

In this post I would like to describe how to create a CakePHP testing environment (on top of Vagrant), from scratch, starting from a standard Vagrant box and leverage the power of the Puppet provisioning system to install binaries and configure daemons automatically.

Here follows the agenda of the article:

Repository layout
Vagrant

Simple configuration file
Puppet
Fire up the machine

Develop ...
... Test!
Conclusions

Repository layout

Before starting with the technical details about the setup of a Vagrant virtual environment, let me spend a couple of words describing how I usually arrange the files inside a repository containing a Vagrant-aware CakePHP application:

`-- puppet/
|   `-- base.pp
`-- src/
|   `-- cakephp/
|   `-- app/
`-- README
`-- Vagrantfile

As you can see from the ASCII art above, I have a puppet directory containing various configuration files for the Vagrant provisioning system, a src directory with a CakePHP subrepository and the directory containing our app, and finally the README and the Vagrant configuration file.

You could complain that adding testing environment stuff inside your application repository is not the right way to do things, but what if you are working in a team? How to share all those configuration files with all the other members of the team? Using emails? Using a wiki page? I think the repository containing the application is the best place for those files!

In the following section I will talk briefly about Vagrant, focusing on its configuration files and how to modify them to create your own CakePHP testing environment.

Vagrant

At a first glance, Vagrant could be considered as a simple VirtualBox wrapper, letting its users easily create and manage virtual machines. Vagrant is more than this: it is shipped with different provisioning systems enabling automatic configuration of services and binaries inside virtual machines, it automatically configures shared folders between the host and the guest environments, it automatically setup port forwarding to enable the host to ssh into the guest machine, etc.

From the readme:

Vagrant is a tool for building and distributing virtualized development environments.

By providing automated creation and provisioning of virtual machines using Oracle's VirtualBox, Vagrant provides the tools to create and configure lightweight, reproducible, and portable virtual environments.

As you could imagine, this tool comes in handy for our purposes. Imagine you are carrying on your CakePHP application together with 3 or 4 other developers; to work on it, each member of the team is in need of a machine running MySQL, Apache and (obviously) with the PHP interpreter installed. At this point you have to choose between two different options:

Leave each member of the team responsible of setting up its development machine, and hope for the versions of the installed binaries and libraries to be the same.
Ship a Vagrant image (the configuration file, not the image itself!) inside the repository containing your app so that each developer is guaranteed working with the same tools as the other members of the team.

If you chose the former, then stop reading: you will end up being very annoyed! If you chose the latter, please go on, you are in the right place!

Simple configuration file

The Vagrant configuration file is written in Ruby, but don't be scared if you (like me) don't know its syntax: edit the file is really straightforward! Here is how my VagrantFile looks like:

Vagrant::Config.run do |config|
    config.vm.host_name = "app"
    config.vm.box = "lucid32"
    config.vm.box_url = "http://files.vagrantup.com/lucid32.box"
    #config.vm.boot_mode = :gui

    #                              guest <-- host
    config.vm.forward_port "http", 80,       8080
    config.vm.network("33.33.33.10")

    config.vm.provision :puppet do |puppet|
        puppet.manifests_path = "puppet"
        puppet.manifest_file  = "base.pp"
    end
end

Let me explain the meaning of the most interesting lines:

config.vm.host_name: set the name of the virtual machine, and its internal hostname;
config.vm.box: set the name of the box to use for this Vagrant image; all Vagrant images use so called boxes as their base OS setup, so that multiple Vagrant images can share the same base operating system, and specialize it for example, configuring and tweaking the provisioning system;
config.vm.box_url: url from which to fetch the chosen box, if not yet downloaded;
config.vm.boot_mode: start the virtual machine with or without the GUI (this is something I learned since I started using Vagrant: VirtualBox images can be fired up also in headless mode!);
config.vm.forward_port: enable port forwarding for HTTP requests, given that we are going to install Apache inside our Vagrant machine and send to it HTTP requests and display received results, and that we want to avoid possible conflicts with other web-servers running on the host machine;
config.vm.network(): assign a static IP the interface of the virtual machine, so that we can for example log into such machine using the given IP without using Vagrant CLI interface;
puppet.manifests_path: choose the provisioning system;
puppet.manifest_file: set the location of provisioning system entry point configuration file;

Now that our Vagrant configuration file has been completely filled, it is time to move on and learn how to configure the chosen provisioning system.

Puppet

Puppet is very simple and at the same time powerful provisioning system. What is its aim? Simplify the life of system administrators, automating the whole process which, starting from a default operating system installation, leads to 100% operating system containing all the tools and services already configured and ready to be used.

From the project home page:

Puppet is an open source data center automation and configuration management framework. Puppet provides system administrators with a simplified platform that allows for consistent, transparent, and flexible systems management.

Nice, but how can we use it to configure our Vagrant virtual machines automatically? Fortunately, as explained before, Vagrant is shipped with the support for the Puppet provisioning system, which means our effort to configure everything is reduced to the minimum.

In the following sections I will show you the minimal configuration file enabling Puppet to install most of the binaries needed by CakePHP to serve requests and run the tests. Unfortunately, I will not spend any time explaining Puppet syntax, so please have a look at the official documentation before going on.

Update everything

The first thing to do, once you are ready with your Vagrant image, is to update the list of available packages; if you skip this step, you are likely to end up receiving lot of error messages saying that some of requested packages cannot be downloaded: a simple sudo apt-get update should fix the problem but obviously, if you want to make a step further and automate this operation with Puppet, you will need put some lines inside the Puppet configuration file (puppet/base.pp).

How could I tell Puppet to update the list of packages only once? Well, the simplest solution I come up with, is presented in the following snippet of code:

exec {"apt-get-update":
    creates => "/updated",
    command => "apt-get update && touch /updated",
}

This basically tell Puppet that such exec { ... } command creates the file /updated and that it won't be run again once that file has been created. The second line, is nothing more than the command Puppet will execute in order to create the aforementioned file.

Now put that snippet inside a Puppet class (similar to a function used to wrap multiple operations), together with other OS standard operations (e.g. user permissions, or default directory creation), as presented below:

class ubuntu {
    group { "puppet": ensure => "present"; } ->
    group { "vagrant": ensure => "present"; } ->
    user { "vagrant": ensure => "present"; } ->
    file { "/home/vagrant/bin":
        ensure  => "directory",
        owner   => "vagrant",
        group   => "vagrant",
        mode    => "755",
    }

    # update once
    exec {"apt-get-update":
        creates => "/updated",
        command => "apt-get update && touch /updated",
    }
}

The use of -> between Puppet actions, makes possible to synchronize actions one another (some sort of dependency): in the snippet, we are telling Puppet to create the user vagrant only after the group vagrant, and again, after the group puppet, and so forth.

On the next reboot of your Vagrant machine, the list of packages will be automatically updated and the file updated will be created inside your root folder.

MySQL

I know it, CakePHP can be configured to run with SQLite databases, but for this article I will assume your application has been thought to work with a MySQL database. Moreover, for the sake of simpleness and readability, I will wrap all the instructions needed by Puppet to install MySQL (and later on Apache, PHP, etc), inside a Puppet class:

class mysql {
}

At this point we need to tell Puppet to install MySQL, which is provided by the mysql-server package; to do so, add the following line inside your Puppet class:

package {"mysql-server": ensure => installed}

Now, ensure that the MySQL daemon gets loaded each time the virtual machine is booted:

service { "mysql":
    ensure => running,
    hasstatus => true,
    hasrestart => true,
    require => Package["mysql-server"],
}

The line containing require => ..., tells Puppet that the current service { ... } operation needs the package mysql-server installed to be be executed correctly.

Configure now, the Administrator's password:

$password = "vagrant"
exec { "mysql-set-password":
    subscribe => Service["mysql"],
    unless => "mysqladmin -uroot -p${password} status",
    command => "mysqladmin -uroot password ${password}",
}

With these lines, we are telling Puppet that the command mysqladmin -uroot password ${password} should be executed after the service mysql has been properly started and only if mysqladmin -uroot -p${password} status exits with a non-zero value.

Finally create the tables of our application and populate them with some dummy record, if any (replace the path of the SQL script containing tables definitions):

exec { "mysql-db-init":
    command => "mysql -uroot -p${password} < /vagrant/src/app/Config/Schema/schema.sql",
}

Done! Here is the final mysql Puppet class (with some explicit synchronization rules):

class mysql {
    package { "mysql-server": ensure => installed }

    $password = "setup"
    service { "mysql":
        ensure => running,
        hasstatus => true,
        hasrestart => true,
        require => Package["mysql-server"],
    } ->
    exec { "mysql-set-password":
        subscribe => Service["mysql"],
        unless => "mysqladmin -uroot -p${password} status",
        command => "mysqladmin -uroot password ${password}",
    } ->
    exec { "mysql-db-init":
        command => "mysql -uroot -p${password} < /vagrant/src/app/Config/Schema/schema.sql",
    }
}

Apache

Now that we learned how to configure MySQL, it would be easy to adapt the previous steps and tell Puppet to install and configure an Apache server inside our development virtual machine. Let's start with an empty Puppet class:

class apache2 {
}

Then place inside inside of it the code to automatically download and install all the needed binaries:

package { "apache2": ensure => installed }

At this point, it's time to configure the daemon; depending of the installed version, we likely are going to receive warnings or errors messages if a ServerName is not specified inside the Apache configuration file; to avoid that, add the following (change app with whatever you want) to the Puppet class:

exec { "servername":
    subscribe => Package["apache2"],
    unless  => "grep '^ServerName' /etc/apache2/apache2.conf",
    command => "echo ServerName app >> /etc/apache2/apache2.conf",
    notify  => Service["apache2"],
}

In the snippet above we are telling Puppet to add Servername app to /etc/apache2/apache2.conf (if not already present) and notify the service apache2 about possible configuration changes (a.k.a. restart the service!).

CakePHP needs mod_rewrite enabled; achieve that with the following lines:

exec { "mod-rewrite":
    unless => "ls /etc/apache2/mods-enabled/rewrite*",
    command => "a2enmod rewrite",
    notify  => Service["apache2"],
}

Moreover, we need to set AllowOverride equal to All for the default Apache virtual host; the following is a bit tricky because the configuration file contains three lines with those strings, but we need to take into account the only one referring the document root which in my case, is found at line 11; consequently:

$config = '/etc/apache2/sites-available/default'
exec { "allow-override":
    unless => "sed -n '11 p' $config | grep 'AllowOverride All'",
    command => "sed -i '11 s/None/All/' $config",
    notify  => Service["apache2"],
}

When Apache gets installed, the user www-data is created and used to serve HTTP requests; to make sure we don't get permissions errors running our CakePHP application, we ensure that such user is part of the group vagrant:

user { "www-data":
    groups => ["vagrant"],
    notify  => Service["apache2"],
}

Create a symbolic link inside the server document root pointing to the source directory of our application which (if have followed the tutorial so far) is inside /vagrant/src/app:

file { "/var/www/app":
    ensure => link,
    target => "/vagrant/src/app",
}

Finally, make sure the server starts automatically at boot:

service { "apache2":
    ensure => running,
    hasstatus => true,
    hasrestart => true,
    require => Package["apache2"],
}

This is how the apache Puppet class looks like:

class apache2 {
    package { "apache2": ensure => installed }

    $config = '/etc/apache2/sites-available/default'
    exec { "servername":
        subscribe => Package["apache2"],
        unless  => "grep '^ServerName' /etc/apache2/apache2.conf",
        command => "echo ServerName app >> /etc/apache2/apache2.conf",
        notify  => Service["apache2"],
    } ->
    exec { "mod-rewrite":
        unless => "ls /etc/apache2/mods-enabled/rewrite*",
        command => "a2enmod rewrite",
        notify  => Service["apache2"],
    } ->
    exec { "allow-override":
        unless => "sed -n '11 p' $config | grep 'AllowOverride All'",
        command => "sed -i '11 s/None/All/' $config",
        notify  => Service["apache2"],
    } ->
    user { "www-data":
        groups => ["vagrant"],
        notify  => Service["apache2"],
    } ->
    file { "/var/www/app":
        ensure => link,
        target => "/vagrant/src/app",
    } ->
    service { "apache2":
        ensure => running,
        hasstatus => true,
        hasrestart => true,
        require => Package["apache2"],
    }
}

PHP

Let's start with the standard boilerplate for required binaries:

$php = ["php5-cli", "php5-mysql", "libapache2-mod-php5", "php-pear"]
package { $php: ensure => "installed" }

Once the interpreter has been installed, we need to modify the PHP's include_path variable to automatically include the directory containing CakePHP core libraries; PHP looks for different configuration files depending on how the interpreter has been fired up: if it is being issued from the command line, it will look inside /etc/php5/cli/cli.php, otherwise, if it was Apache who invoked the interpreter, the file /etc/php5/apache2/php.ini will be used; consequently we need to modify both of them to cover all possible scenarios.

By default the configuration line defining include_path is commented out, so it is easy to understand if we need to tweak php.ini files or not:

$include = '.:/usr/share/php:/vagrant/src/cakephp/lib/'
$cliini = '/etc/php5/cli/php.ini'
$apacheini = '/etc/php5/apache2/php.ini'
exec { "cli-include-path":
    subscribe => Package["php5-cli"],
    unless => "grep -q ^include_path $cliini",
    command => "echo include_path = $include >> $cliini",
    notify  => Service["apache2"],
} ->
exec { "apache-include-path":
    unless => "grep -q ^include_path $apacheini",
    command => "echo include_path = $include >> $apacheini",
    notify  => Service["apache2"],
} ->

Kinda hackish! However, it is now the turn to enable the PHP extension for Apache; do it with:

exec { "mod-php":
    command => "a2enmod php5",
    unless => "ls /etc/apache2/mods-enabled/php5*",
}

Finally, in order to be able to run CakePHP tests, we need PHPUnit properly configured on the machine; unfortunately, a bare apt-get install phpunit will not be sufficient to install needed binaries because there are a couple of dependencies which are not made available through apt; consequently we are going to install everything using pear and its external (and more updated) repositories:

exec { "phpunit":
    command => "pear upgrade pear && \
                pear channel-discover pear.phpunit.de && \
                pear channel-discover components.ez.no && \
                pear channel-discover pear.symfony-project.com && \
                pear install --alldeps phpunit/PHPUnit",
    creates => "/usr/bin/phpunit",
    require => Package["php-pear"],
}

The php Puppet class, inclusive of all the synchronization rules, will look like this:

class php {
    $php = ["php5-cli", "php5-mysql", "libapache2-mod-php5", "php-pear"]
    package { $php: ensure => "installed" }

    $include = '.:/usr/share/php:/vagrant/src/cakephp/lib/'
    $cliini = '/etc/php5/cli/php.ini'
    $apacheini = '/etc/php5/apache2/php.ini'
    exec { "cli-include-path":
        subscribe => Package["php5-cli"],
        unless => "grep -q ^include_path $cliini",
        command => "echo include_path = $include >> $cliini",
        notify  => Service["apache2"],
    } ->
    exec { "apache-include-path":
        unless => "grep -q ^include_path $apacheini",
        command => "echo include_path = $include >> $apacheini",
        notify  => Service["apache2"],
    } ->
    exec { "mod-php":
        unless => "ls /etc/apache2/mods-enabled/php5*",
        command => "a2enmod php5",
        notify  => Service["apache2"],
    } ->
    exec { "phpunit":
        creates => "/usr/bin/phpunit",
        command => "pear upgrade pear && \
                    pear channel-discover pear.phpunit.de && \
                    pear channel-discover components.ez.no && \
                    pear channel-discover pear.symfony-project.com && \
                    pear install --alldeps phpunit/PHPUnit",
        require => Package["php-pear"],
    }
}

Miscellaneous

Create now a new Puppet class for all the utilities which are not strictly related to CakePHP, but still handy. Modify the niceties array with whatever package you want:

class misc {
    $niceties = ["htop", "sudo", "vim", "curl", "cowsay", "figlet",
            "inotify-tools"]
    package { $niceties: ensure => "installed" }

    file { "/usr/local/bin/cowsay":
        ensure => link,
        target => "/usr/games/cowsay",
        require => Package['cowsay'],
    }
}

Synchronize all

Well, so far we learned how to create Puppet classes containing commands to install packages, configure daemons, create files, etc; now we need to put everything together and possibly create some further synchronization between those operations.

If you remember, we created a command to update the list of available packages once but never told the php or apache class to wait for the update process to be done. To do so, create a new class executing all the class objects defined so far (mysql, apache, etc) and implementing the synchronization logic described before (update first, install later!):

class cakephpbox {
    class { "ubuntu": }
    class { "apache2": require => Exec['apt-get-update'], }
    class { "mysql":  require => Exec['apt-get-update'], }
    class { "php": require => Exec['apt-get-update'], }
    class { "misc": require => Exec['apt-get-update'], }
}

Finally tell Puppet to run the cakephpbox class at boot time::

class { "cakephpbox": }

Fire up the machine

It's time to fire up our carefully configured Vagrant image. Issue the following command from the repository containing your application and cross your fingers:

vagrant up

If everything went fine, the terminal from which you issued the previous command would look like this:

Moreover, you should be able to ping the virtual machine ...

> ping 33.33.33.10
PING 33.33.33.10 (33.33.33.10) 56(84) bytes of data.
64 bytes from 33.33.33.10: icmp_req=1 ttl=64 time=2.73 ms
64 bytes from 33.33.33.10: icmp_req=2 ttl=64 time=0.210 ms
64 bytes from 33.33.33.10: icmp_req=3 ttl=64 time=0.508 ms
64 bytes from 33.33.33.10: icmp_req=4 ttl=64 time=0.493 ms
^C
--- 33.33.33.10 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.210/0.985/2.732/1.015 ms

... and issue HTTP requests against the Apache web-server running inside virtual machine (http://localhost:8080/app):

Develop ...

As mentioned above, Vagrant automatically setup shared folders between host and guest machines: in particular it shares the host folder containing VagrantFile and makes it available inside the guest environment under the directory /vagrant. The obvious consequence of this, is that you can still develop your applications inside the host environment, with your preferred gui-enabled editor, and make use of all the niceties your operating system is offering to you.

... Test!

CakePHP let developers run application test suites in two different ways: on the one hand you can run tests directly from the browser; on the other, you can use your beloved console. If you followed all the steps described so far, both methods should work flawlessly.

Do you feel more gui-friendly? Then open your favorite browser and point to http://localhost:8080/app/test; follow one of the links, and you should be presented with a page similar to the following:

Do you prefer CLI over GUI? Then spawn a new terminal, ssh into the virtual machine (ssh vagrant@33.33.33.10 or move into your repository and run vagrant ssh), go inside the directory containing your application sources (cd /vagrant/src/app) and finally run tests with Console/cake testsuite app:

Conclusions

That's all folks, we finally arrived to the conclusions section.

I apologize for all the inaccuracies I left in the article but as I said on top, Internet is full of well-written documentation analyzing with much more details all the different arguments touched in the article: what Internet was missing in my opinion, was an article putting all that information together and shading most of the technical details to better focus on how to combine all those tools, in a very practical way.

Thanks for reading.

Bluetooth on FreeBSD

2011-11-28T11:13:00.000+01:00

I found the holy Grail! Add these lines inside your kernel config:

device          netgraph
options         NETGRAPH
options         NETGRAPH_SOCKET
options         NETGRAPH_BLUETOOTH              # ng_bluetooth(4)
options         NETGRAPH_BLUETOOTH_UBT          # ng_ubt(4)
options         NETGRAPH_BLUETOOTH_HCI          # ng_hci(4)
options         NETGRAPH_BLUETOOTH_SOCKET       # ng_btsocket(4)
options         NETGRAPH_BLUETOOTH_L2CAP        # ng_l2cap(4)

For additional information: sys/conf/NOTES.

About CakePHP and sqlite

2011-10-15T09:40:00.001+02:00

I would like to shed some light on how to make CakePHP 2.0RC2 correctly work with a sqlite database (really useful for development and debugging purposes).

First of all, CakePHP 2.0 DOES support Sqlite3 by default; so please, stop wasting your time trying to install stale 3rd party plugins as explained here and here: everything you need is already present in the distribution package.
However, if you are still getting troubles trying to connect to the database, keep in mind that the new CakePHP release uses the PDO library to talk to databases, hence double check that your PHP interpreter has been built with such library:

$ php - i  | grep 'PDO support'
PDO support => enbled

and that you have the specific driver to connect to sqlite databases:

$ php -i | grep 'PDO drivers'
PDO drivers => mysql, sqlite

At this point, it is time to configure how your CakePHP application is be able to communicate with the database.
Edit app/Config/database.php (if not present, copy it from app/Config/database.php.default), and change your $default settings for the keys datasource and database as presented below:

class DATABASE_CONFIG {
    public $default = array(
        ..
        'datasource' => 'Database/Sqlite',
        ..
        'database' => 'yourdatabasename.sqlite',
        ..
    );
}

Finally, remember that when the datasource setting is set to Database/Sqlite, CakePHP's database engine expects to find inside database the path of your database file; moreover, bear in mind that if you specify a relative path, that should be relative to the app/webroot directory.

After all these steps, (hopefully) all the bars will be green!

Curriculum Vitae

2011-10-03T21:49:00.000+02:00

I finally managed to finish the first draft of my curriculum vitae.

What do you think about that? Have you got any advices to make it better? I would really appreciate it!

Someone told me to stick to the Europass Curriculum Vitae format, but I don't really understand why.

I see the curriculum vitae as something really personal, so why should I have to make mine equal to other ones? Do recruiters really care about that? Is my CV likely to be discarded just because it does not follow that AWFUL template?

I think what really matters is the content: your education, your job experience and your personal interests. Everything else is just noise. That's why I went for a very plain template.

Hope I'm not the only one thinking like that.

Windows, Cygwin and PHP

2011-10-02T22:40:00.000+02:00

Those of you who constantly try to use Windows as a standard Linux/Unix OS, will agree with me that cygwin represents the only solution to make Microsoft operating systems friendlier for users coming from different OSs.

Some days ago I needed to install PHP and mysql on my machine; I fired cygwin setup.exe application, updated the list of available packages, and finally looked for aforementioned binaries.

With my surprise, none of them was available for download.

Do I need to install a them by myself as standalone binaries?

Fortunately, not! Thanks to Google, I come to this page and followed the steps listed inside; in particular, I added a new mirror to the list of cygwin mirrors, updated the list of available packages, and in the end, on the package selection page I was able to find both PHP and mysql binaries.

Bingo!

FreeBSD hacking!

2011-09-14T11:34:00.000+02:00

Time has come to spend some time describing the workflow I followed for about 9 months when working on my master's degree thesis; in that period I implemented a new character device for the FreeBSD Kernel, hence I have been busy all the time modifying the Kernel, compiling it, and finally testing it.

In the first section of this post I will show you how to compile a custom FreeBSD Kernel, then how to speed-up the debugging and testing phases of the developing process making use of a very powerful tool called picobsd.

Build a Kernel

Download the source tree
Compile the toolchain
Configure it, build it, and install it

picobsd

Where can I find the script?
Bootstrap
Configure an image and build it
Run

Further tweaks

Conclusions

Build a Kernel

Extracted from the official handbook:

Building a custom kernel is one of the most important rites of passage for advanced BSD users. This process, while time consuming, will provide many benefits to your FreeBSD system.

And again:

This has a number of benefits, such as: - Faster boot time. Since the kernel will only probe the hardware you have on your system, the time it takes your system to boot can decrease dramatically. - Lower memory usage. A custom kernel often uses less memory than the GENERIC kernel by omitting unused features and device drivers. This is important because the kernel code remains resident in physical memory at all times, preventing that memory from being used by applications. For this reason, a custom kernel is especially useful on a system with a small amount of RAM. - Additional hardware support. A custom kernel allows you to add in support for devices which are not present in the GENERIC kernel, such as sound cards.

In my specific case, I was essentially moved by the need of creating a new Kernel module; but let's cut this sort of introduction, and move to the next section.

Download the source tree

The first thing needed is obviously a Kernel source tree; depending on the steps you followed while installing your OS, you are likely to find that directly inside /usr/src/sys; otherwise, proceed to download the source tree and move it somewhere on your hard disk. I usually prefer to checkout a OS snapshot directly from the official subversion repository and dump it in my home folder, inside the directory ~/workspace/FreeBSD/head:

cd ~/workspace/FreeBSD
svn co http://svn.freebsd.org/base/head

Usually this is a very long operation, so I advise you go out for a walk: on your come back, everything will be there waiting for you.

Compile the toolchain

Once you are ready with the source tree, you are in need of building all the necessary tools you are going to use later on; indeed, this step will prevent you from trying to compile new KERNELS (or even worst a WHOLE NEW WORLD) using old compilers or linkers and eventually get strange errors like this. Then issue:

cd head
make kernel-toolchain

After that, you will be able to compile your first hacked FreeBSD Kernel.

Configure it, build it, and install it

Slowly, we are getting at the heart of the matter, indeed it's now time to take a Kernel configuration file from the downloaded source tree and start to tweak it in order to match our needs. Kernel configs are usually placed inside the directory sys/$arch/conf of the source tree, hence to start from the default configuration file for a i386 machine, issue:

cp sys/i386/conf/GENERIC sys/i386/conf/MYKERNEL

One important thing to take into account while tweaking a FreeBSD Kernel config is that if you want to remove the support for a specific device (e.g. Realtek network adapters, or Sound Blaster audio cards), it is not sufficient to remove the specific lines (e.g device re, or device snd_emu10k1) from the aforementioned configuration file; by default, everything contained inside the Kernel config is indeed built statically inside the Kernel image, while everything commented out is compiled as an external module. Hence, to force the Kernel tool-chain not to compile external modules, edit (or create) the file /etc/make.conf, and add the following line:

NO_MODULES=1

Committed this to memory, enjoy tweaking your own new Kernel. Once you feel enough confident of the new Kernel config, compile it (and possibly fix dependencies errors):

make buildkernel KERNCONF=MYKERNEL -j8

(the -j specifies the number of parallel processes involved in the building process. As a rule of thumb, that value should be set equal to two times the number of cores, hence in this case I'm assuming you are running a quad-core processor.) At this point, if the previous command finished without any error, then install the new Kernel image, reboot the machine, and cross your fingers:

sudo make installkernel KERNCONF=MYKERNEL

If something goes wrong (i.e. unable to boot the machine), please have a look at this: there is a solution for everything! Well, the first section of this post has come to its end; let's move on the next one, and talk about a useful tool which is going to speed-up your Kernel debug and testing phases.

picobsd

Originally, PicoBSD was a one floppy version of FreeBSD 3.0-current, which in its different variations allows you to have secure dialup access, small diskless router or even a dial-in server. Nowadays, picobsd is THE script which produces such minimal implementations of FreeBSD. From the man page:

The picobsd utility was originally created to build simple standalone systems such as firewalls or bridges, but because of the ability to cross-build images with different source trees than the one in the server, it can be extremely useful to developers to test their code with- out having to reinstall the system.

As you can see from the quoted paragraph, this turned out to be the perfect tool for my needs: instead of installing a new Kernel each time I touched a module, reboot, and hope modifications did not break anything, I could create a new PicoBSD image (the Kernel would indeed be automatically built) and either dump it on a USB stick and boot a test machine or fire a new virtual machine with that image as hard disk.

Where can I find the script?

If you are reading this paragraph and you followed all the steps described so far, you will be able to find the picobsd utility inside the source tree folder, in particular inside the directory $tree/release/picobsd. Otherwise, checkout a copy of the utility folder from the official subversion repository.

Bootstrap

It is now time to setup the PicoBSD environment. This part is very similar to the one in which we built the kernel-toolchain, with the difference that now we need to build a NEW OPERATING SYSTEM. Luckily, the new OS is very minimal, and consequently the building process is not supposed to take too long to complete; in addiction, the following is a one-shot procedure, which means you need to do it only once, before to build the first PicoBSD image. The utility, in order to start to build all the necessary stuff need to be instructed about the position of a FreeBSD source tree; this is done using by the option --src (i.e. --src ~/workspace/FreeBSD/usr); hence, assuming you checked out the picobsd utility together with the FreeBSD source tree, to prepare the PicoBSD environment issue a simple:

cd ~/workspace/FreeBSD/head/release/picobsd/build
./picobsd --src ~/workspace/FreeBSD/head -n --init

(The option -n disables interactive mode.) If earlier I advised you to exit for a walk while downloading a fresh new OS tree, this time you are better off going off the records for a nap! Once you find the following message on your prompt, then you are ready to move on.

*** Type  NOT FOUND
=== Press enter to continue
---> fail: Error <1> error code  in <>
Error: unknown floppy type
---> Aborting ./picobsd

Configure an image and build it

picobsd is shipped with a couple of pre-configured images you can use as starting points. Those can be found inside the picobsd root folder (e.g.bridge and qemu directories):

cd ~/workspace/FreeBSD/head/release/picobsd
cp -R bridge mypimage

For the moment let's keep the image as is: just try to compile it and fix possible errors (the utility is indeed very oldish and barely updated):

cd ~/workspace/FreeBSD/head/release/picobsd/build
./picobsd --src ~/workspace/FreeBSD/head -n mypimage

If the previous operations exits with the following error don't worry, there is an easy fix.

(.text+0xd4d): undefined reference to `ulog_login'

The error tell us that the compiler has not been able to find the library ulog, hence we need to modify the configuration file of the image to instruct the compiler to include such library. To do so issue:

echo 'libs_so -lulog' >> ../mypimage/crunch.conf

and re-try to build the image; this time everything should be OK and inside the directory build_dir-mypimage you will be able to find the PicoBSD binary image:

ls -al build_dir-mypimage/*bin
-rw-r--r--  1 matteo  matteo  8388608 Sep 13 14:19 build_dir-netmap/picobsd.bin

Run

We are now ready to test the new image. To do so, you can choose either to run it using a Virtual Machine like QEMU:

qemu build_dir-netmap/picobsd.bin -net nic,model=e1000

or even better, dump the binary on a usb-stick:

dd if=build_dir-mypimage/picobsd.bin of=/dev/da0

plug it into a real machine and finally turn it on. In both cases, in a few seconds you will be presented a screen like this:

Further tweaks

Let me finish with a couple tips for those who want to build a fully customized PicoBSD image.

How is it possible to include a certain application (i.e. vi) inside the binary image?

picobsd

crunchgen(1)

progs

PicoBSD

cd ~/workspace/FreeBSD/head/release/picobsd
echo 'progs vi' >> mypimage/crunch.conf

How is it possible to include a third-party application not included in the operating system source tree?

crunchgen

special

foo

mv /path/to/foo mypimage/

cruchgen

progs foo
special foo srcdir @__CWD__@/bar

foo

bar

PicoBSD

~/workspace/FreeBSD/head/release/picobsd/mypimage

Conclusions

That's all folks; I hope you enjoyed the ride.
For more details about any specific utilities and procedures, please read the links spread all around the post or leave a comment.

Cheers.

More about mercurial notify extension..

2011-09-03T12:02:00.001+02:00

This morning, working on a personal mercurial repository hosted on my home server, I found out that the email commit notification system (described on this previous post) was not working anymore; on each commit I got back the following message:

changegroup.notify hook raised an exception: [Errno 2] No such file or directory: '../subscription.conf'

I checked the root folder of the repo, and the subscription file was there.

I checked the configuration file of the repo, and it was correctly pointing to the right subscription file.

What has happened to my mercurial repository? I didn't touch anything!

The solution!

After a couple of shots in the dark, I eventually managed to solve the issue. As described in the previous post, I used a relative path to make the repository configuration file aware of the list of subscribers, and for some strange reasons (honestly, I did not investigate it too much...), this path was not followed anymore. Consequently I tried with an absolute path, and commits started to be notified again. Yay!

Setting up a Mercurial server.

2011-07-24T22:45:00.001+02:00

[Update: if you get error messages from the email notification extension, read this]

Today I'm going to write down a list of notes and tips I've collected these days while trying to setup a Mercurial server. Even if Internet is full of well-written guides and tutorials describing how to approach the task, I chose to keep gathered information in a single place (i.e. this post) so that to simplify the life of others pursuing the same goal.

Introduction

My needs were very simple: share Mercurial repositories by mean of my humble EEEBox (which is actually hosting matteolandi.net), make committers log in with the same system account (using public-keys), send email notifications on each repositories' push and finally make it possible to browse aforementioned repositories via web (in particular using Apache).

Create a new user and configure access keys

As stated above, my intention was to map all the committers on a single system user, hg, and use different public-keys to access the repository.

To do so, create the new system user issuing the command:

$ sudo useradd hg

and follow the instructions printed on screen:

Adding user `hg' ...
Adding new group `hg' (1003) ...
Adding new user `hg' (1003) with group `hg' ...
Creating home directory `/home/hg' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for hgg
Enter the new value, or press ENTER for the default
        Full Name []: 
        Room Number []: 
        Work Phone []: 
        Home Phone []: 
        Other []: 
Is the information correct? [Y/n] Y

After that, edit the file authorized_keys contained inside the directory .ssh:

$ sudo su hg
$ vim ~/.ssh/authorized_keys

and add as many public-keys as needed, associated with a command telling the ssh daemon to execute the program hg-ssh on incoming connections:

# Matteo Landi, laptop
command="hg-ssh /home/hg/*" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3yL4G9dOPZ30koZ6v7UKzA2LnfbvH1orel9Gyj7+P0WtYGj1TXhUPU6znREoQ7CcHlABQ5sNfC1tg3rZ2RimmCL4XjK9XNQdHx67TqxAOfcfChnetKb0MMWdrd8ZRMKPC1m2rVdrvXtFO19BUrc+M3JFSY5ZNXZCz3bznC+fNNLqrhE3hdkpCyUNhFClMpLrQMu9i+LvcclurCdQlcn39UvBh8RyIrPIB4QWd43R536TUjYZ+1rRBIyilYIeQ4naaxrQxsxfnmcuaKjrycgz2tWunGC2oRx6odMR2RBsW9BaIH7rMAfK4EC7/tibRxJsmRWtM26qlSAbUlZNsJQBEQ== matteo@COMet

At this point, trying to reach the remote machine hosting the Mercurial server with the command:

$ ssh hg@code.matteolandi.net

you should get back the following error:

Illegal command '?'
Connection to code.matteolandi.net closed.

which is nothing but the evidence that (till now) everything has been done correctly.

Get Mercurial

Now that the system is almost ready to host the Mercurial server (later we are going to configure Apache), we need to get Mercurial installed on the system (note: Mercurial client version is enough, for it includes all the extensions and scripts necessary for our purposes).

On Debian-like systems (for other ones, please refer to the official site), to install such a package you simply have to issue the command:

$ sudo apt-get install mercurial

and everything should be ready in a couple of minutes:

$ hg
Mercurial Distributed SCM

basic commands:

 add        add the specified files on the next commit
 annotate   show changeset information by line for each file
 clone      make a copy of an existing repository
 commit     commit the specified files or all outstanding changes
 diff       diff repository (or selected files)
 export     dump the header and diffs for one or more changesets
 forget     forget the specified files on the next commit
 init       create a new repository in the given directory
 log        show revision history of entire repository or files
 merge      merge working directory with another revision
 pull       pull changes from the specified source
 push       push changes to the specified destination
 remove     remove the specified files on the next commit
 serve      start stand-alone webserver
 status     show changed files in the working directory
 summary    summarize working directory state
 update     update working directory (or switch revisions)

use "hg help" for the full list of commands or "hg -v" for details

Create and configure repository

As a rule, we want to host all Mercurial repositories inside /home/hg, hence let's start by creating a new foo (duh) repository (expected user: hg; expected directory: /home/hg):

$ hg init foo

and then enable email notifications for each push action. To do so, edit the file .hg/hgrc contained inside the repository folder, and add the text below:

[web]
# the baseurl is used for constructing hgweb-URLs for the mails 
baseurl = http://code.matteolandi.net/hg

[extensions]
hgext.notify =

[hooks]
# send one email for each push
changegroup.notify = python:hgext.notify.hook

[email]
# address sending notifications
from = Code | matteolandi.net <noreply@code.matteolandi.net>

[smtp]
# host of the smtp server; use localhost, if local
host = smtp.tiscali.it

[notify]
# Presumably an email will be sent if an action corresponding to any of
# the listed keywords is performed.
sources = serve push pull bundle

# if True, print email content instead of actually sending it
test = False

# subscription information: which user is interested in changes of which
# part of the repository
config = ../subscription.conf

# changeset template
template = Subject: {webroot}: changeset {rev}\n\ndetails: {baseurl}/{webroot|basename}/rev/{node|short}\nchangeset: {rev}:{node|short}\nuser: {author}\ndate: {date|date}\ndescription:\n{desc}\n

# max lines of diffs to include (0=none, -1=all)
maxdiff = 300

Moreover, create the subscription file inside the root of the repository with the following content:

[reposubs]
# key is a global matching part of a repo, value is a comma separated
# list of subscribers
* = Matteo Landi <you@myourdomain.net>

This tells Mercurial that the user Matteo Landi (a.k.a. me) is interested in receiving notifications about all the changes committed to the repository (for additional information about the notify extension, have a look at its documentation).

Configure Apache

The amount of effort needed to configure Apache or other web-servers to display Mercurial repositories is very limited. The following is a configuration based on cgi-scripts, however there are plenty alternatives lying around.

hgweb, our HTTP repository viewer, consists of a cgi-script invoked by the web-server, and a configuration file which tells the application where to find repositories.

Let's start by copying the script (shipped with Mercurial) inside the home directory of the user hg:

$ cp /usr/share/doc/mercurial-common/examples/hgweb.cgi ~

and modify its content to match our needs; in particular edit the config variable to make it point to our configuration file (we are going to create it later):

...
config = "/home/hg/hgweb.conf"
...

Create then, such hgweb.conf file and fill it with:

[collections]
repos = /home/hg

Those lines tell hgweb that each folder contained inside /home/hg corresponds to a Mercurial repository.

Finally modify your Apache virtual host to make it fire the cgi-script on /hg/... URL requests:

ScriptAlias /hg /home/hg/hgweb.cgi

At this point, after a reboot of the web-server:

sudo apache2ctl graceful

you should be able to browse your Mercurial repositories with no problems, as shown in the figure below.

In addition, if you want hgweb to disable web-pushes (default configuration enables them only for SSL connections), edit the file .hg/hgrc with the repository, and add:

[web]
push_ssl = False

Test: clone, edit, commit and push

The time has come to verify the correctness of all operations executed so far; on your local machine or laptop, move to a temporary directory:

$ cd /tmp

and clone the foo repository:

$ hg clone ssh://hg@code.matteolandi.net/foo

Move inside the cloned repo and create a new file with random content:

$ cd foo
$ fortune >> bar

Commit changes to the local repository with:

$ hg add bar
$ hg ci bar -m "`fortune`"

and finally propagate changes to the remote repository:

$ hg push

If everything is ok (and I hope so), you should get back a message similar to this:

pushing to ssh://hg@code.matteolandi.net/foo
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: notify: sending 1 subscribers 1 changes

and your inbox should have a new message waiting for your attention:

I'm still alive .. (cont.)

2011-02-20T19:44:00.005+01:00

Sorry for the silence of the last month, but I've been busy carrying on my master's degree thesis. When you are really focused on something for the 70% of the day, it's hard to find the time to keep you up to date: all you need is a couch and some silence.
Yesterday, I decided to break the silence with an image; today I'm going to talk about a couple of things.

Books

I can't believe it! At the tender age of 24, I started enjoying reading: reading in the train, reading before to sleep, reading everywhere every time you are passively waiting for something. I began with some tech books about programming languages, editors and databases, and then I switched to something different like the one I'm currently reading: Gomorra.
Finally, a personal advice: Rework from 37signals.

Music

After 5 years of inactivity, I decided to put my hands on my electric-bass again: I did the dusting, bought a new pick and finally switched on the amp! What a joy!

Events

2011 seems to be an year full of tech events.
Two weeks ago I moved to Milan for the 1st Hackitaly day. 10 hours of pure programming with one single goal: create mashups making use of the APIs offered by a list of registered companies. I worked in team with @kinglizard86 and @lawrenceoluyede, but unfortunately we didn't succeeded in presenting our creations. However, I do really enjoyed the ride: here is a slide-show of the event.

Upcoming events:
- Codemotion, held in Rome the 5th of March.
- EuroPython, held in Florance from 20th to 26th of June.
- BetterSoftware, held in Florance from 27th to 28th of June.

Projects

As I told you before, it's hard to find time for projects other than my thesis. However, in a couple of days I created a simple application which helped me a lot in keeping focus on the things I'm working on: it's based on the pomorodo technique.
Another funny script (thought with @kinglizard86) I would like to share is s, a svn wrapper that tries to enhance the behavior of a couple of subversion commands: in particular the wrapper was created to make it possible to display the changes of the current commit while writing the log message.

Thesis

Dulcis in fundo, a couple of updates about the ongoing work of my master's degree thesis.
After more than month spent in researching the state of art of packet filtering and kernel interaction from userspace, I finally started to hack the kernel. In particular I learned how to create character devices, how to modify the Intel lem driver to make it aware of my device for userland communication, and finally how to implement a very simple and inefficient memory allocator.
Even though there are a lot of things that need to be done such as asynchronous I/O and integration with other drivers such as em and igb (unfortunately not yet supported by qemu), I'm very happy with the work done!

That's all folks for tonight. Thanks for reading!

I'm still alive..

2011-02-19T17:19:00.002+01:00

..stay tuned!

Hitsmash: conclusion (part 7)

2010-12-30T12:28:00.001+01:00

It's been a month since the first blog entry about Hitsmash. The time has come give a brief summary of this funny game.

Analytics

Have my readers enjoyed Hitsmash?

They do ... on the first day.
This is perfectly aligned with my expectations; indeed I created anything innovative or attractive. How is it supposed to capture the attention of the reader and make him come back a second time and vote a couple of videos? It does not.

Difficulties

Difficulties arise when you work with something you are not used to; PHP in my case. I'm not used to PHP, and the quality of the code is the evidence of this. However, it seems to do the trick, isn't it?

Google it!

Please, stop reinventing the wheel (Matteo, I'm also talking to you!), especially if you are interested in presenting a working prototype of your creation as soon as possible.
I needed to parse an HTML table and I was not the first one facing up to this problem. If you were me, what would you have done? Knowing my not so strong skills with regexps, I decided to use someone else's wheel.

Source code

Finally, you can find all the scripts of the previous posts, and the ones I have not published yet (like the chart page), inside the hacks repository (hitsmash sub-folder).
Feel free to find any bugs (Locks? MySQL injection? ..) so I can try fix them, eventually.

FreeBSD: kernel networking for dummies

2010-12-13T16:15:00.002+01:00

Today I would like to share with you some notes I collected in the last month during my research study for the final work of my master's degree. At the time of writing, the title of my thesis is something like "Design and Development of an API for Packet Filtering in Userland", hence I will try to hack the actual FreeBSD network stack in order to improve, hopefully, the performance of operations such as packet filtering and packet routing.

Introduction

In order to keep things as simple as possible, let's start with a list of events that describe what happens when a frame is received by the network adapter.
First of all, the received frame is queued inside a FIFO; when the FIFO is full, new packets will be dropped. Afterward, one frame is extracted from the FIFO and placed inside the memory of the OS. At this point an interrupt is raised and the OS is notified about the new frame (in reality it's not true that an interrupt is raised per received frame: indeed, a mechanism called "Interrupt Mitigation", is used to prevent the CPU from spending too much time handling such interrupts instead of doing "useful" work). In the end the kernel serves this interrupt by scheduling a task which is going to collect these frames.

Network Device

Let's try to analyze now, how network devices work, what kind of data structures they need and how they are supposed to move data to and from the OS memory.
We could think of a network device as a collection of registers with no memory at all. We only need to know how to modify those registers, the shape of wrappers created around the received frames and obviously how to register notifications for different events.
Network devices are usually designed to work with ring-buffers in order to simplify things: instead of allocate a new buffer each time a frame is received, we pre-allocate a bunch of them, and use pointers to mark the available ones. As a result we only need to tell the device where the ring is located, where to put a new upcoming frame and where is the oldest received frame not yet processed by the kernel.

In the example above, we have pre-allocated nine slots, each one wide enough to contain a wrapper for the frame. Furthermore, five of nine slots contain useful data for the OS. Once the kernel collects these frames, it will increment the tail register, so that the network device is notified about the availability of new slots (when the kernel modifies these pointers, it actually sets the registers of the network device).
I spoke about wrappers because once a network device receives a new frame, it needs some extra-space to store information about received data such as length, status and so forth. Moreover, the chunk of memory used to store incoming data is not contained inside the wrapper, but is reachable by a pointer contained into the slot. A more realistic view of the ring-buffer is shown in the figure below.

Kernel's point of view

At this point, an interrupt is raised and a task is scheduled to collect these frames.
From the kernel's point of view, packets are wrapped inside mbufs, structures that contain both information about the header and about the data of a packet. Moreover, mbufs are arranged into lists, so that it is possible to have packets larger than a single network frame. The structure that handles all these mbufs is close to the one shown in the figure above: we have a ring of slots, each with a pointer to an mbuf containing the data of the frame, and we have a pointer to the next slot to collect.

In the figure above, I intentionally used two different colors to mark that the ring holds two different packets: the first contained inside SLOT[1:3] and the second inside SLOT[4:6].
The task, once scheduled, will join together the frames of a given packet, detach the mbufs from the ring, send the chain to the upper level of the network stack and increment the pointer.

In reality, a new buffer is allocated and linked to the freed slot before next_to_check is incremented.

NIC & Kernel

So far we have seen how network devices are supposed to work, and how the kernel reacts to inputs from such devices. Let's try now to join these two views together and show how everything actually works inside the FreeBSD kernel.
We have two rings of data: does this means that we have two copies of the same frame stored in different buffers? No! In order to explain why, you have to know that we have two ways to store data inside mbufs: the first one is put data inside the structure itself and the second one is to allocate an external buffer and point to it.

The latter could be used for our purposes: we can make the slots reserved for the network device, point to the external memory associated to the mbuf. This way, there is no need to copy data between the device-reserved memory and the kernel one: network devices will automatically dump frames inside kernel memory.

Reality

As you can imagine, things are not as easy as I pictured above. Network devices don't know anything about virtual memory: they write directly inside physical memory (Direct Memory Access). For this reason we have to make a couple of changes to the pictures above: the ring buffer used by the network device and the buffers pointed by both the network slots and the mbufs, have to be DMA-mapped.

Transmit

While transmitting data, things are not that different from the ones shown above.
First of all, when the kernel needs to send some data, it creates a chain of mbufs and it queues them inside a FIFO. Then, a chain is extracted from the FIFO, broken down into single mbufs, and attached to a ring buffer (more or less equivalent to the structure rx_buffers shown in the picture above). At this point, we notify the network device about the presence of a new frame so that it can send it.

Author's Note

For more information about the subject, checkout the kernel, take a deep breath, and start diving.

Thanks for reading

P.S. Feel free to comment for questions and/or errors.

Hitsmash: deployment (part 6)

2010-11-23T14:56:00.002+01:00

Are you ready to take the plunge? I hope so!

Clear the developement database, and populate it with new data.
Create a new Google Analytics profile, and paste the activation code inside the PHP scripts.
Register a new 3rd level domain pointing to your homemade server.
Configure Apache in order to accept requests addressed to the new registered domain.
Restart Apache.
Cross your fingers.

http://hitsmash.matteolandi.net

Hitsmash: ui (part 5)

2010-11-23T13:25:00.003+01:00

I finally finished the user interface for Hitsmash.
It's very simple: a title, two boxes , two videos, two buttons and one footer. The hard part runs in the background.

When you visit the homepage, two random hits are extracted from the database and new fight entry is added to the fights table; finally the page described above is shown to the user.

Once you make your decision and click one of the two Vote buttons, a bunch of operations are executed on the server side:
* check if the fight identifier is present in the database;
* update the rank values of the challenging hits using the Elo rating system [1];
* remove both the current and old fights from the database in order to discard fake votes;

The code I wrote is kind of a mess: I avoided either templating systems or object-oriented database management systems in order to keep things as simple as possible.
I preferred to concentrate on the logic rather than on code aesthetics.

Here is a preview of the site: SLAM!

[1] http://en.wikipedia.org/wiki/Elo_rating_system