Emanuele Rocca

systemd is your friend

2015-10-07T14:45:00+02:00

Today I want to talk a bit about some cool features of systemd, the default Debian init system since the release of Jessie. Ubuntu has also adopted systemd in 15.04, meaning that you are going to find it literally everywhere.

Logging

The component responsible for logging in systemd is called journal. It collects and stores logs in a structured, indexed journal (hence the name). The journal can replace traditional syslog daemons such as rsyslog and syslog-ng, or work together with them. By default Debian keeps on using rsyslog, but if you don't need to ship logs to a centralized server (or do other fancy things) it is possible to stop using rsyslog right now and rely on systemd-journal instead.

The obvious question is: why would anybody use a binary format for logs instead of a bunch of tried and true plain-text files? As it turns out, there are quite a lot of good reasons to do so.

The killer features of systemd-journald for me are:

Index tons of logs while being able to search with good performance: O(log(n)) instead of O(n) which is what you get with text files
No need to worry about log rotation anymore, in any shape or form

The last point in particular is really critical in my opinion. Traditional log rotation implementations rely on cron jobs to check how much disk space is used by logs, compressing/removing old files. Log rotation is usually: 1) annoying to configure; 2) hard to get right; 3) prone to DoS attacks. With journald, there is pretty much nothing to configure. Log rotation is built into the daemon disk space allocation logic itself. This also allows to avoid vulnerability windows due to time-based rotation, which is what you get with logrotate and friends.

Enough high-level discussions though, here is how to use the journal!

Check if you already have the directory /var/log/journal, otherwise create it (as root). Then restart systemd-journald as follows:

sudo systemctl restart systemd-journald

You can get all messages produced since the last boot with journalctl -b. All messages produced today can get extracted using journalctl --since=today. Want to get all logs related to ssh? Try with journalctl _SYSTEMD_UNIT=ssh.service.

There are many more filtering options available, you can read all about them with man journalctl.

journald's configuration file is /etc/systemd/journald.conf. Two of the most interesting options are SystemMaxUse and SystemKeepFree, which can be used to change the amount of disk space dedicated to logging. They default to 10% and 15% of the /var filesystem respectively.

Here is a little cheatsheet:

journalctl -b                # Show all messages since last boot
journalctl -f                # Tail your logs
journalctl --since=yesterday # Show all messages produced since yesterday
journalctl -pcrit            # Filter messages by priority
journalctl /bin/su           # Filter messages by program
journalctl --disk-usage      # The amount of space in use for journaling

Containers

A relatively little known component of systemd is systemd-nspawn. It is a small, straightforward container manager.

If you don't already have a chroot somewhere, here is how to create a basic Debian Jessie chroot under /srv/chroots/jessie:

$ debootstrap jessie /srv/chroots/jessie http://http.debian.net/debian/

With systemd-nspawn you can easily run a shell inside the chroot:

$ sudo systemd-nspawn -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
root@jessie:~#

Done. Everything works out of the box: no need for you to mount /dev, /run and friends, systemd-nspawn took care of that. Networking also works.

If you want to actually boot the system, just add the -b switch to the previous command:

$ sudo systemd-nspawn -b -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.

Welcome to Debian GNU/Linux jessie/sid!

Set hostname to <orion>.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Syslog Socket.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
         Mounting FUSE Control File System...
         Starting Copy rules generated while the root was ro...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Reached target Slices.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Started Copy rules generated while the root was ro.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started Load/Save Random Seed.
         Starting LSB: Raise network interfaces....
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started LSB: Raise network interfaces..
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting /etc/rc.local Compatibility...
         Starting Login Service...
         Starting LSB: Regular background program processing daemon...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting System Logging Service...
[  OK  ] Started System Logging Service.
         Starting Permit User Sessions...
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started LSB: Regular background program processing daemon.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
         Starting Console Getty...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.

Debian GNU/Linux jessie/sid orion console

orion login:

That's it! Just one command to start a shell in your chroot or boot the container, again zero configuration needed.

Finally, systemd provides a command called machinectl that allows you to introspect and control your container:

$ sudo machinectl status jessie
jessie
           Since: Wed 2015-10-07 11:22:56 CEST; 55min ago
          Leader: 32468 (systemd)
         Service: nspawn; class container
            Root: /srv/chroots/jessie
         Address: fe80::8e70:5aff:fe81:2290
                  192.168.122.1
                  192.168.1.13
              OS: Debian GNU/Linux jessie/sid
            Unit: machine-jessie.scope
                  ├─32468 /lib/systemd/systemd
                  └─system.slice
                    ├─dbus.service
                    │ └─32534 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile -...
                    ├─cron.service
                    │ └─32539 /usr/sbin/cron
                    ├─systemd-journald.service
                    │ └─32487 /lib/systemd/systemd-journald
                    ├─systemd-logind.service
                    │ └─32532 /lib/systemd/systemd-logind
                    ├─console-getty.service
                    │ └─32544 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
                    └─rsyslog.service
                      └─32540 /usr/sbin/rsyslogd -n

With machinectl you can also reboot, poweroff, terminate your containers and more. There are so many things to learn about systemd and containers! Here are some references.

systemd-nspawn(1)
machinectl(1)
http://0pointer.net/blog/systemd-for-administrators-part-xxi.html

This stuff is pretty exciting. Now that all major distributions use systemd by default, we can expect to have access to tools like journalctl and systemd-nspawn everywhere!

A (very) brief history of Australia

2014-05-23T15:35:00+02:00

This post is mostly a sum-up of the Wikipedia page History of Australia, with some content taken from History of the British Empire. Both texts are available under the Creative Commons Attribution-ShareAlike License.

I do not seem to be able to learn about a new topic without taking notes: in this case I have decided to publish my work, hoping that someone will find it useful. Some very important themes such as the Gold Rush and Australian History during the World Wars have been impudently ignored.

Indigenous Australians

The ancestors of Indigenous Australians are believed to have arrived in Australia 40,000 to 60,000 years ago, and possibly as early as 70,000 years ago.

By 1788, the population of Australia existed as 250 individual nations, many of which were in alliance with one another, and within each nation there existed several clans, from as few as five or six to as many as 30 or 40. Each nation had its own language and a few had multiple, thus over 250 languages existed, around 200 of which are now extinct.

Permanent European settlers arrived at Sydney in 1788 and came to control most of the continent by end of the 19th century. Bastions of largely unaltered Aboriginal societies survived, particularly in Northern and Western Australia into the 20th century, until finally, a group of Pintupi people of the Gibson Desert became the last people to be contacted by outsider ways in 1984.

European explorers

Terra Australis (Latin for South Land) is one of the names given to a hypothetical continent which appeared on European maps between the 15th and 18th centuries. Although the landmass was drawn onto maps, Terra Australis was not based on any actual surveying of such a landmass but rather based on the hypothesis that continents in the Northern Hemisphere should be balanced by land in the south.

The first documented European landing in Australia was made in 1606 by a Dutch ship led by Willem Janszoon. Hence the ancient name "Nova Hollandia". The same year, a Spanish expedition had landed in the New Hebrides and, believing them to be the fabled southern continent, named the land: "Terra Austral del Espiritu Santo". Hence the current name "Australia".

Although various proposals for colonisation were made, notably by Pierre Purry from 1717 to 1744, none was officially attempted. Indigenous Australians were less able to trade with Europeans than were the peoples of India, the East Indies, China, and Japan. The Dutch East India Company concluded that there was "no good to be done there".

In 1769, Lieutenant James Cook tried to locate the supposed Southern Continent. This continent was not found, and Cook decided to survey the east coast of New Holland, the only major part of that continent that had not been charted by Dutch navigators.

Cook charted and took possession of the east coast of New Holland. He noted the following in his journal:

"I can land no more upon this Eastern coast of New Holland, and
 on the Western side I can make no new discovery the honour of
 which belongs to the Dutch Navigators and as such they may lay
 Claim to it as their property, but the Eastern Coast from the
 Latitude of 38 South down to this place I am confident was never
 seen or viseted by any European before us and therefore by the
 same Rule belongs to great Brittan."

Colonisation

The American Revolutionary War (1775-1783) saw Great Britain lose most of its North American colonies and consider establishing replacement territories.

The British colony of New South Wales was established with the arrival of the First Fleet of 11 vessels in January 1788. It consisted of over a thousand settlers, including 778 convicts (192 women and 586 men). A few days after arrival at Botany Bay the fleet moved to the more suitable Port Jackson where a settlement was established at Sydney Cove on 26 January 1788. This date later became Australia's national day, Australia Day.

Between 1788 and 1868, approximately 161,700 convicts (of whom 25,000 were women) were transported to the Australian colonies of New South Wales, Van Diemen's land and Western Australia. Early colonial administrations were anxious to address the gender imbalance in the population brought about by the importation of large numbers of convict men.

In 1835, the British Colonial Office issued the Proclamation of Governor Bourke, implementing the legal doctrine of terra nullius upon which British settlement was based, reinforcing the notion that the land belonged to no one prior to the British Crown taking possession of it and quashing any likelihood of treaties with Aboriginal peoples, including that signed by John Batman. Its publication meant that from then, all people found occupying land without the authority of the government would be considered illegal trespassers.

A group in Britain led by Edward Gibbon Wakefield sought to start a colony based on free settlement and political and religious freedoms, rather than convict labour. The South Australia Act [1834], passed by the British Government which established the colony reflected these desires and included a promise of representative government when the population reached 50,000 people. Significantly, the Letters Patent enabling the South Australia Act 1834 included a guarantee of the rights of 'any Aboriginal Natives' and their descendants to lands they 'now actually occupied or enjoyed'.

In 1836, two ships of the South Australia Land Company left to establish the first settlement on Kangaroo Island. The foundation of South Australia is now generally commemorated as Governor John Hindmarsh's Proclamation of the new Province at Glenelg, on the mainland, on 28 December 1836. By 1851 the colony was experimenting with a partially elected council.

Development of Australian democracy

Traditional Aboriginal society had been governed by councils of elders and a corporate decision making process, but the first European-style governments established after 1788 were autocratic and run by appointed governors.

The reformist attorney general, John Plunkett, sought to apply Enlightenment principles to governance in the colony, pursuing the establishment of equality before the law.

Plunkett twice charged the colonist perpetrators of the Myall Creek massacre of Aborigines with murder, resulting in a conviction and his landmark Church Act of 1836 disestablished the Church of England and established legal equality between Anglicans, Catholics, Presbyterians and later Methodists.

In 1840, the Adelaide City Council and the Sydney City Council were established. Men who possessed 1,000 pounds worth of property were able to stand for election and wealthy landowners were permitted up to four votes each in elections. Australia's first parliamentary elections were conducted for the New South Wales Legislative Council in 1843, again with voting rights (for males only) tied to property ownership or financial capacity. Voter rights were extended further in New South Wales in 1850 and elections for legislative councils were held in the colonies of Victoria, South Australia and Tasmania.

Women became eligible to vote for the Parliament of South Australia in 1895. This was the first legislation in the world permitting women also to stand for election to political office and, in 1897, Catherine Helen Spence became the first female political candidate for political office, unsuccessfully standing for election as a delegate to the Federal Convention on Australian Federation. Western Australia granted voting rights to women in 1899. Early federal parliamentary reform and judicial interpretation sought to limit Aboriginal voting in practice, a situation which endured until rights activists began campaigning in the 1940s.

Road to independence

Despite suspicion from some sections of the colonial community (especially in smaller colonies) about the value of nationhood, improvements in inter-colonial transport and communication, including the linking of Perth to the south eastern cities by telegraph in 1877, helped break down inter-colonial rivalries.

New South Wales Premier Henry Parkes addressed a rural audience in his 1889 Tenterfield Oration, stating that the time had come to form a national executive government:

"Australia [now has] a population of three and a half millions,
 and the American people numbered only between three and four
 millions when they formed the great commonwealth of the United
 States. The numbers were about the same, and surely what the
 Americans had done by war, the Australians could bring about in
 peace, without breaking the ties that held them to the mother
 country."

Though Parkes would not live to see it, his vision would be achieved within a little over a decade, and he is remembered as the "father of federation".

The Commonwealth of Australia came into being when the Federal Constitution was proclaimed by the Governor-General, Lord Hopetoun, on 1 January 1901.

Australia took part in WWI. The contributions of Australian and New Zealand troops during the 1915 Gallipoli Campaign against the Ottoman Empire had a great impact on the national consciousness at home, and marked a watershed in the transition of Australia and New Zealand from colonies to nations in their own right. The countries continue to commemorate this occasion on ANZAC Day.

Australia achieved independent Sovereign Nation status after World War I, under the Statute of Westminster, which defined Dominions of the British empire in the following way:

"They are autonomous Communities within the British Empire, equal
 in status, in no way subordinate one to another in any aspect of
 their domestic or external affairs, though united by a common
 allegiance to the Crown, and freely associated as members of the
 British Commonwealth of Nations."

The parliaments of Canada, Australia, New Zealand, the Union of South Africa, the Irish Free State and Newfoundland (currently part of Canada) were now independent of British legislative control, they could nullify British laws and Britain could no longer pass laws for them without their consent.

The Australia Act 1986 removed any remaining links between the British Parliament and the Australian states.

Antifeatures and Antibugs

2014-01-29T16:02:00+01:00

Software Engineering distinguishes between software features and software bugs.

It is usually understood that features are positive, expected characteristics of a computer program. Features make users happy by allowing them to do something useful, interesting, or fun. Something good, anyways. Bugs are instead undesirable and annoying. You're sitting there at your computer writing a long email and the software crashes right before your email is sent. Bad stuff.

Features are generally implemented by programmers on purpose, whereas bugs are purely unintentional. They are mistakes. You don't make a mistake on purpose.

We might at this point be inclined to think that: i) what is good for users is done on purpose by software manufacturers; ii) what is bad for users was not meant to be. It happened by mistake.

Here is a handy table to visualize this idea:

	On purpose	By mistake
Good	Feature
Bad		Bug

It seems to make a lot of sense. But you might have noticed that two cells of the table are empty. Right!

In a great talk titled When Free Software isn't better, Benjamin Mako Hill mentions the concept of antifeatures, and how they relate to Free Software.

Antifeatures are features that make the software do something users will hate. Something they will hate so much they would pay to have those features removed, if that's an option. Microsoft Windows 7 is used in the talk to provide some examples of software antifeatures: the Starter Edition does not allow users to change their background image. Also, it limits the amount of usable memory on the computer to 2GBs, regardless of how much memory the system actually has. Two antifeatures engineered to afflict users to the point that they will purchase a more expensive version of the software, if they have the means to do that.

I have another nice example. The Spotify music streaming service plays advertisements between songs every now and then. To make sure users are annoyed as much as possible, Spotify automatically pauses an advertisement if it detects that the volume is being lowered. A poor Spotify user even tried to report the bug on The Spotify Community forum, only to find out that what she naively considered as a software error was "intentional behavior". A spectacular antifeature indeed.

Whenever a piece of technology does something you most definitely do not want it to do, such as allowing the NSA to take complete control of your Apple iPhone, including turning on its microphone and camera against your will, that's an antifeature.

	On purpose	By mistake
Good	Feature
Bad	Antifeature	Bug

Both bugs and antifeatures are bad for users. The difference between them is that antifeatures are engineered. Time and money are spent to make sure the goal is reached. A testing methodology is followed. "Are we really sure customers cannot change their wallpaper even if they try very very hard?"

Engineering processes, of course, can fail. If the poor devils at Microsoft who implemented those harassments would have made a mistake that allows users to somehow change their wallpaper on Windows Starter... Well, I would call that a glorious antibug.

	On purpose	By mistake
Good	Feature	Antibug
Bad	Antifeature	Bug

There is no place for antifeatures in Free and Open Source Software. Free Software gives users control over what their software does. Imagine Mozilla adding a feature to Firefox that sets your speakers volume to 11 and starts playing a random song from the black metal artist Burzum every time you add a bookmark, unless you pay for Mozilla Firefox Premium Edition. The source code for Firefox is available under a free license. People who are not into Burzum's music would immediately remove this neat antifeature.

I have spent many years of my life advocating Free and Open Source Software, perhaps using the wrong arguments. Mako's talk made me think about all this (thanks mate!). All these years I've been preaching about the technical superiority of Free Software, despite evidence of thousands of bugs and usability issues in the very programs I am using, and contributing to develop.

Free Software is not better than Proprietary Software per se. Sometimes it is, sometimes it's not. But it gives you control, and freedom. When it annoys you, when it doesn't do what you expect and want, you can be sure it's not on purpose. And we can fix it together.

Useful tools for Python developers

2013-07-04T18:56:00+02:00

Python is a great language with an impressive number of tools designed to make developers' life easier. Sometimes, however, the problem is getting to know that these tools exist in the first place. By contributing to projects like OpenStack's Nova client and Falcon, I have recently come across some useful tools that can seriously improve the quality of your code.

The first one is called pyflakes, a passive checker of Python programs developed by Phil Frost. What it does is parsing your source files and checking for possible errors such as undefined names and unused imports. Let's consider the following example:

import urllib

print "pyflakes example"
urlib.urlopen('http://www.linux.it')

The code above contains a typo, we have misspelled urllib. Here is what pyflakes thinks about our program:

$ pyflakes example.py
example.py:1: 'urllib' imported but unused
example.py:4: undefined name 'urlib'

On line 4 we try to use urlib which is not defined. Also, we import urllib on line 1 and we do nothing with it. Our typo has been spotted! Notice that, even though our program contains a print statement, 'pyflakes example' has not been printed. That is because pyflakes parses the source files it checks, without importing them, making it safe to use on modules with side effects.

pyflakes can be installed with pip or apt-get.

The second tool I want to talk about is Ned Batchelder's coverage.py.

No doubt you write unit tests for your programs. Right? Good. coverage.py is out there to help you checking how much of your program is actually covered.

Let's use as an example codicefiscale, a Python project of mine.

First we install coverage:

pip install coverage

Then we run our unit tests:

$ coverage run --source=codicefiscale tests.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s

We pass the module we want to test with --source=codicefiscale so that coverage will only report information about that specific module.

Now that our tests have been performed successfully it is time to check how much of our code is covered by unit tests:

$ coverage report -m
Name            Stmts   Miss  Cover   Missing
---------------------------------------------
codicefiscale      73      4    95%   61, 67, 95, 100

Not bad, 95% of our module is covered! Still, coverage let us know that 4 lines have not been touched by the unit tests. With this information, we can go write some meaningful test cases that will also cover the missing lines.

Story of a bug in Ubuntu

2012-09-02T00:00:00+02:00

Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff Desktop-as-a-Service.

The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well.

Now, as I said that was in May. Ubuntu 12.04 LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default PDF viewer. Everybody happy.

In the meantime, obviously, I was interested in this issue so I’ve tried to run run evince from a terminal, getting the following output:

(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.

Funny. On another test system running Debian Sid (unstable) everything was working smoothly.

The diff between Ubuntu’s version of evince and Debian’s is a 6MB monster. Among other changes, I noticed that Ubuntu’s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I filed a bug on launchpad.

How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a “we won’t hide problems” mindset. For an example of how bad is the proprietary approach, just check a random bug in Adobe’s bug tracking system.

But let’s go back to the evince bug.

Other users reported that their VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately it did not actually address the issue. Somebody else reported that RDP sessions were broken too. At the beginning of June Precise was still affected. Finally, on August the 6th a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis accepted into precise-updates, case closed.

Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu’s main target audience.

This bug never affected Debian, simply because utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.

First steps with Sinatra on Heroku

2011-12-14T18:21:00+01:00

Lately I have been playing a little bit with different Platform as a Service providers. So far, the most developer-friendly one seems to be Heroku. In particular, Heroku provides an incredibly straightforward way to get started. After creating an account on their website, you just have to install the appropriate gem:

sudo gem install heroku

The command heroku help will then show you all the possible invocations of the command line client, just to give you an idea of what you can do with it (everything).

Alright, that’s not particularly interesting without an application to play with, is it? So let’s create a new Sinatra application. In order to do that, we need a new directory containing a file like the following one:

# webapp.rb
require 'rubygems'
require 'sinatra'
# Add other gems you might need here

get '/' do
  "Your Sinatra app is working!"
end

Also add a Gemfile with all the required dependencies, in our trivial example sinatra is actually the only gem you need:

source :rubygems
gem "sinatra"

Lastly, you need a config.ru file where you can require our newly created Ruby program and specify that it is a Sinatra application. In this example, the Ruby file is called webapp.rb.

require './webapp'

run Sinatra::Application

A very useful program when developing a Sinatra application is shotgun. gem install shotgun and you will be able to run your new shiny app by executing the command shotgun. While developing on your machine, shotgun will take care of reloading the application for every request, so that you don’t have to restart the server after every modification to the source code.

Run bundle install to easily install all the required dependencies in your development environment, and shotgun to start the app locally. Pointing your browser to http://127.0.0.1:9393/ you can check if everything works as expected. If nothing went wrong, you can create a git repository to track changes to the source code:

git init
git add . 
git commit -m "Initial commit"

Perfect! Now, how do you create and deploy your application on Heroku?

heroku create
git push heroku master

Heroku will automatically find out that you deployed a Sinatra application, it will install all the required dependencies and start serving requests. A few interesting commands you might want to run to see what’s going on are heroku info, heroku ps and heroku logs. The client also provides a command, heroku open, that opens your browser at the right production URL.

So, you now have a working application under revision control. Dependencies are handled by specifying them into the Gemfile, and the required gems can be installed locally using bundle. Shotgun is our friend when it comes to use the application for development purposes. A git push heroku master is all we need to deploy a new version of the code.

Of course there is way much more about Sinatra and Heroku than what I’ve covered in this mini-intro. Please refer to the Sinatra Book and the Heroku Dev Center for more information!

A neat VLC extension to download subtitles

2011-10-10T00:00:00+02:00

As somebody who often watches movies and TV shows I have the recurring problem of finding the right English subtitles for the video I am about to enjoy. Now, that is really a tedious thing to do. You go on a website such as opensubtitles.org and type the title of what you are looking for. Then you usually have to choose from a plethora of results, download the archive, unzip it, fire up VLC, go to Video → Subtitles Track → Open File, and finally open your subtitles file. And what you typically find out at that point is that the subtitles are completely out of sync with the audio, but rather than doing the whole process again you prefer to just sync them with a bit of VLC-fu. Which always takes longer than you thought, but hey.

Now, what if there was an extremely simple way to just choose the subtitles for a given movie directly from VLC?

The good news is that somebody wrote a beautiful VLC extension that does exactly that. He also wrote an introductory article about VLC extensions. Thank you, Jean-Philippe André.

Installing the thing is as simple as downloading the subtitles extension and putting it in the VLC scripts folder. On Linux machines, the extensions folder can be found at ~/.local/share/vlc/lua/extensions/, on Windows APPDATA\VLC\lua\extensions. But seriously, don’t use Windows.

Once the extension is installed you can start watching a movie and then click View → Subtitles to open the subtitles search interface.

This is how it looks like:

Now that the practical problem is solved, please take some time to look at the source code. VLC uses Lua as the language for writing extensions. I certainly cannot say that the web is full of resources about how to extend VLC using Lua, but the piece of code I wrote about in this post is certainly a good way to start hacking, and maybe scratch yet another itch.

Gospel according to Tux

2011-10-05T00:00:00+02:00

Some years ago I came across a really peculiar newsgroup post. It was not about technicalities of any sort. It was about history. A beautifully written history of computers. From the Turing machine to the Free Software world, the original author managed to capture all the important events of the computer revolution with a great deal of humor. Re-posting it here, it is just brilliant.

The Gospel of Tux (v1.0)

In the beginning Turing created the Machine.

And the Machine was crufty and bodacious, existing in theory only. And von Neumann looked upon the Machine, and saw that it was crufty. He divided the Machine into two Abstractions, the Data and the Code, and yet the two were one Architecture. This is a great Mystery, and the beginning of wisdom.

And von Neumann spoke unto the Architecture, and blessed it, saying, “Go forth and replicate, freely exchanging data and code, and bring forth all manner of devices unto the earth.” And it was so, and it was cool. The Architecture prospered and was implemented in hardware and software. And it brought forth many Systems unto the earth.

The first Systems were mighty giants; many great works of renown did they accomplish. Among them were Colossus, the codebreaker; ENIAC, the targeter; EDSAC and MULTIVAC and all manner of froody creatures ending in AC, the experimenters; and SAGE, the defender of the sky and father of all networks. These were the mighty giants of old, the first children of Turing, and their works are written in the Books of the Ancients. This was the First Age, the age of Lore.

Now the sons of Marketing looked upon the children of Turing, and saw that they were swift of mind and terse of name and had many great and baleful attributes. And they said unto themselves, “Let us go now and make us Corporations, to bind the Systems to our own use that they may bring us great fortune.” With sweet words did they lure their customers, and with many chains did they bind the Systems, to fashion them after their own image. And the sons of Marketing fashioned themselves Suits to wear, the better to lure their customers, and wrote grave and perilous Licenses, the better to bind the Systems. And the sons of Marketing thus became known as Suits, despising and being despised by the true Engineers, the children of von Neumann.

And the Systems and their Corporations replicated and grew numerous upon the earth. In those days there were IBM and Digital, Burroughs and Honeywell, Unisys and Rand, and many others. And they each kept to their own System, hardware and software, and did not interchange, for their Licences forbade it. This was the Second Age, the age of Mainframes.

Now it came to pass that the spirits of Turing and von Neumann looked upon the earth and were displeased. The Systems and their Corporations had grown large and bulky, and Suits ruled over true Engineers. And the Customers groaned and cried loudly unto heaven, saying, “Oh that there would be created a System mighty in power, yet small in size, able to reach into the very home!” And the Engineers groaned and cried likewise, saying, “Oh, that a deliverer would arise to grant us freedom from these oppressing Suits and their grave and perilous Licences, and send us a System of our own, that we may hack therein!” And the spirits of Turing and von Neumann heard the cries and were moved, and said unto each other, “Let us go down and fabricate a Breakthrough, that these cries may be stilled.”

And that day the spirits of Turing and von Neumann spake unto Moore of Intel, granting him insight and wisdom to understand the future. And Moore was with chip, and he brought forth the chip and named it 4004. And Moore did bless the Chip, saying, “Thou art a Breakthrough; with my own Corporation have I fabricated thee. Thou thou art yet as small as a dust mote, yet shall thou grow and replicate unto the size of a mountain, and conquer all before thee. This blessing I give unto thee: every eighteen months shall thou double in capacity, until the end of the age.” This is Moore’s Law, which endures unto this day.

And the birth of 4004 was the beginning of the Third Age, the age of Microchips. And as the Mainframes and their Systems and Corporations had flourished, so did the Microchips and their Systems and Corporations. And their lineage was on this wise:

Moore begat Intel. Intel begat Mostech, Zilog and Atari. Mostech begat 6502, and Zilog begat Z80. Intel also begat 8800, who begat Altair; and 8086, mother of all PCs. 6502 begat Commodore, who begat PET and 64; and Apple, who begat 2. (Apple is the great Mystery, the Fruit that was devoured, yet bloomed again.) Atari begat 800 and 1200, masters of the game, who were destroyed by Sega and Nintendo. Xerox begat PARC. Commodore and PARC begat Amiga, creator of fine arts; Apple and PARC begat Lisa, who begat Macintosh, who begat iMac. Atari and PARC begat ST, the music maker, who died and was no more. Z80 begat Sinclair the dwarf, TRS-80 and CP/M, who begat many machines, but soon passed from this world. Altair, Apple and Commodore together begat Microsoft, the Great Darkness which is called Abomination, Destroyer of the Earth, the Gates of Hell.

Now it came to pass in the Age of Microchips that IBM, the greatest of the Mainframe Corporations, looked upon the young Microchip Systems and was greatly vexed. And in their vexation and wrath they smote the earth and created the IBM PC. The PC was without sound and colour, crufty and bodacious in great measure, and its likeness was a tramp, yet the Customers were greatly moved and did purchase the PC in great numbers. And IBM sought about for an Operating System Provider, for in their haste they had not created one, nor had they forged a suitably grave and perilous License, saying, “First we will build the market, then we will create a new System, one in our own image, and bound by our Licence.” But they reasoned thus out of pride and not wisdom, not forseeing the wrath which was to come.

And IBM came unto Microsoft, who licensed unto them QDOS, the child of CP/M and 8086. (8086 was the daughter of Intel, the child of Moore). And QDOS grew, and was named MS-DOS. And MS-DOS and the PC together waxed mighty, and conquered all markets, replicating and taking possession thereof, in accordance with Moore’s Law. And Intel grew terrible and devoured all her children, such that no chip could stand before her. And Microsoft grew proud and devoured IBM, and this was a great marvel in the land. All these things are written in the Books of the Deeds of Microsoft.

In the fullness of time MS-DOS begat Windows. And this is the lineage of Windows: CP/M begat QDOS. QDOS begat DOS 1.0. DOS 1.0 begat DOS 2.0 by way of Unix. DOS 2.0 begat Windows 3.11 by way of PARC and Macintosh. IBM and Microsoft begat OS/2, who begat Windows NT and Warp, the lost OS of lore. Windows 3.11 begat Windows 95 after triumphing over Macintosh in a mighty Battle of Licences. Windows NT begat NT 4.0 by way of Windows 95. NT 4.0 begat NT 5.0, the OS also called Windows 2000, The Millenium Bug, Doomsday, Armageddon, The End Of All Things.

Now it came to pass that Microsoft had waxed great and mighty among the Microchip Corporations; mighter than any of the Mainframe Corporations before it had it waxed. And Gates’ heart was hardened, and he swore unto his Customers and their Engineers the words of this curse:

“Children of von Neumann, hear me. IBM and the Mainframe Corporations bound thy forefathers with grave and perilous Licences, such that ye cried unto the spirits of Turing and von Neumann for deliverance. Now I say unto ye: I am greater than any Corporation before me. Will I loosen your Licences? Nay, I will bind thee with Licences twice as grave and ten times more perilous than my forefathers. I will engrave my Licence on thy heart and write my Serial Number upon thy frontal lobes. I will bind thee to the Windows Platform with cunning artifices and with devious schemes. I will bind thee to the Intel Chipset with crufty code and with gnarly APIs. I will capture and enslave thee as no generation has been enslaved before. And wherefore will ye cry then unto the spirits of Turing, and von Neumann, and Moore? They cannot hear ye. I am become a greater Power than they. Ye shall cry only unto me, and shall live by my mercy and my wrath. I am the Gates of Hell; I hold the portal to MSNBC and the keys to the Blue Screen of Death. Be ye afraid; be ye greatly afraid; serve only me, and live.”

And the people were cowed in terror and gave homage to Microsoft, and endured the many grave and perilous trials which the Windows platform and its greatly bodacious Licence forced upon them. And once again did they cry to Turing and von Neumann and Moore for a deliverer, but none was found equal to the task until the birth of Linux.

These are the generations of Linux:

SAGE begat ARPA, which begat TCP/IP, and Aloha, which begat Ethernet. Bell begat Multics, which begat C, which begat Unix. Unix and TCP/IP begat Internet, which begat the World Wide Web. Unix begat RMS, father of the great GNU, which begat the Libraries and Emacs, chief of the Utilities. In the days of the Web, Internet and Ethernet begat the Intranet LAN, which rose to renown among all Corporations and prepared the way for the Penguin.

And Linus and the Web begat the Kernel through Unix. The Kernel, the Libraries and the Utilities together are the Distribution, the one Penguin in many forms, forever and ever praised.

Now in those days there was in the land of Helsinki a young scholar named Linus the Torvald. Linus was a devout man, a disciple of RMS and mighty in the spirit of Turing, von Neumann and Moore. One day as he was meditating on the Architecture, Linus fell into a trance and was granted a vision. And in the vision he saw a great Penguin, serene and well-favoured, sitting upon an ice floe eating fish. And at the sight of the Penguin Linus was deeply afraid, and he cried unto the spirits of Turing, von Neumann and Moore for an interpretation of the dream.

And in the dream the spirits of Turing, von Neumann and Moore answered and spoke unto him, saying, “Fear not, Linus, most beloved hacker. You are exceedingly cool and froody. The great Penguin which you see is an Operating System which you shall create and deploy unto the earth. The ice-floe is the earth and all the systems thereof, upon which the Penguin shall rest and rejoice at the completion of its task. And the fish on which the Penguin feeds are the crufty Licensed codebases which swim beneath all the earth’s systems. The Penguin shall hunt and devour all that is crufty, gnarly and bodacious; all code which wriggles like spaghetti, or is infested with blighting creatures, or is bound by grave and perilous Licences shall it capture. And in capturing shall it replicate, and in replicating shall it document, and in documentation shall it bring freedom, serenity and most cool froodiness to the earth and all who code therein.”

Linus rose from meditation and created a tiny Operating System Kernel as the dream had foreshewn him; in the manner of RMS, he released the Kernel unto the World Wide Web for all to take and behold. And in the fulness of Internet Time the Kernel grew and replicated, becoming most cool and exceedingly froody, until at last it was recognised as indeed a great and mighty Penguin, whose name was Tux. And the followers of Linus took refuge in the Kernel, the Libraries and the Utilities; they installed Distribution after Distribution, and made sacrifice unto the GNU and the Penguin, and gave thanks to the spirits of Turing, von Neumann and Moore, for their deliverance from the hand of Microsoft. And this was the beginning of the Fourth Age, the age of Open Source.

Now there is much more to be said about the exceeding strange and wonderful events of those days; how some Suits of Microsoft plotted war upon the Penguin, but were discovered on a Halloween Eve; how Gates fell among lawyers and was betrayed and crucified by his former friends, the apostles of Media; how the mercenary Knights of the Red Hat brought the gospel of the Penguin into the halls of the Corporations; and even of the dispute between the brethren of Gnome and KDE over a trollish Licence. But all these things are recorded elsewhere, in the Books of the Deeds of the Penguin and the Chronicles of the Fourth Age, and I suppose if they were all narrated they would fill a stack of DVDs as deep and perilous as a Usenet Newsgroup.

Now may you code in the power of the Source; may the Kernel, the Libraries and the Utilities be with you, throughout all Distributions, until the end of the Epoch. Amen.