Noufal Ibrahim : constrained creativity

Sports Karate

2022-06-17T00:04:33+05:30

I am a practising student of Karate and have been doing so for a long time. I first studied Shotokan for around 7 months when I was a kid and then after a long gap, studied kalaripayattu for 2 years. After another long gap, I studied WTF Taekwondo for a few years and got a first dan black belt. These days, I train in Okinawan Goju ryu.

I moved from Taekwondo which I saw purely as a sport rather than martial art. Both of these contribute to fitness, a sense of self worth, and offer a way to find a circle of like minded friends. However, only the latter provides a genuine way to learn how to fight and provides a vehicle for an almost spiritual change. Those are my motivations for going into martial arts and it’s the reason why I moved away from Taekwondo even after I spent quite a bit of time earning a black belt.

One avenue for this is combat sport oriented martial arts like BJJ and muay thai. However, I like the technicality and softer aspects of Karate and wanted to find a good master to teach me that. I found such a person and over the past few years have been practising Okinawan goju ryu. More on that in a later post.

Karate is traditionally taught by studying basics (kihon) like punches, kicks, blocks. These are combined and stylised into a fixed series of moves called a kata. A kata is usually a series of moves (e.g. block, counter punch, block, counter kick). They are traditionally taught with a breakdown or interpretation (bunkai). A dedicated and thoughtful practitioner will, over time, be able to understand how the various things fit in together and figure out how to make the series of moves practical. As an example, this is the first black belt kata in goju ryu called Seyunchin

The corresponding bunkai is

With a good partner, this is an excellent pedagogical technique and one that has been tested over the years. It also embodies the aphorism “karate ni sente nashi” (there is no first strike in Karate). Almost all kata start with a block or another defensive move and then moves forward to end the fight quickly and efficiently. It also allows constraints inside which one can get creative.

Essentially, you learn the alphabet (kihon), the grammar (kata) and then learn how to apply these to handle real world situations (bunkai). Over time, you develop instincts to move forward by yourself. All good.

But now, comes the question of sports. There are two kinds of karate contests. The first is the kata contest. This is a solo or group performance of a kata. Judges award points based on various criteria like technical skill, athletic performance etc. It’s a visual treat to see high quality kata performance especially at the world level. The first example that comes to mind is Rika Usami performing the kata called Chatanyara Kushanku here

However, thanks to the judging criteria, this is more or less like a dance. If you compare the way in which Higaonna (who’s not a fan of sports competition) above does seyunchin and how a sports contestant does the same kata, the differences become clear. Now, this is not say that the athletes are, in some way, worse than the master. No. My point is that performance kata and kata for teaching techniques are not completely aligned and have diverged quite a bit.

The second is the kumite contest which is a bout or a fight. Sports karate contests are semi contact so you don’t actually hit the opponent. You hold back but if you didn’t, the blow would be fatal and it’s for this that you’re awarded points. This allows a “traditional” sports style timed bout with multiple rounds to be balanced with the idea of karate as a defensive martial art. Opponent attacks, you dodge or block and then counter attack. Referee interrupts, awards you points for your manoeuvre and the first starts again. The fights get extremely technical but if you’re an MMA or a boxing fan, these look rather childish

Now, if you’re looking to study karate as a fighting system, the sports line is not going to serve your purpose. A more traditional approach would be better but unfortunately, most places emphasise the sports angle a lot since that’s what young students are looking for. Older adults who want to get harder gravitate towards muay thai and bjj so karate is left with children who treat it a like a sport and masters who need to satisfy them.

On the overall, I think the seemingly weird kumite that’s part of the contests are an ingenious way of converting traditional karate into a modern sport but there should be no illusion that it’s just a sport with very rigid artificial rules.

As for traditional karate, it’s hard and very demanding. I can’t think of a better way to summarise this than to post a clip here of how a traditional 80 year old master trains.

Train hard and good luck!

Creating a presentation by hand using calligraphy

2017-11-04T22:44:39+05:30

I’ve been involved with PyCon India since the first edition of the conference in 2009. This year, I was invited to give the keynote at the 2017 edition of the event in Delhi. This post is about some personal experiences preparing the speech and making the slides.

I think it’s worth writing about because it was very different from what I usually do and also because, I think, it was a genuine improvement to my process of doing this. Some of the changes which I made are, in my opinion, generally beneficial. A few people like Anand mentioned that the preparation was visible in the talk so this is not just my own opinion.

The content

I decided to speak on Mentoring. It was something that I’d been doing for a while after I started The Lycaeum. I mentioned this on the mailing list and it seemed like it would click.

Ordinarily, I make slides and then use them as talking points while ad-libbing the talk on stage. This time, I decided to do it differently. I’ve been talking at Toastmasters gatherings in Kozhikode and recently went through Mortimer Adlers excellent book on speaking. It has a very old school approach to speaking where you write out summaries and things like that before you give the talk. I highly recommend it.

Getting the content

The first task was to figure out what exactly to talk about.

I spent a week or so brainstorming the talk itself on small pieces of paper. I’ve been making a habit of keeping a binder of loose sheets which are numbered using a system similar to the zettelkasten system. I’ve found this better than a computer screen because I can scribble, draw arrows and do things that are hard to do on a screen and I’ve found it better than a notebook because you can write on multiple sheets at the same time as well as rearrange them as per need and logic. Bound paper sheets might not be sufficient for a topic and might be wasteful for some.

These are not filtered at all. I just jot down everything that occurs to me on the topic (at any time of the day). There were repeated points, quotes, ideas for certain slides, turns of phrase that I thought were cool, jokes, examples, techniques which I could use for effect etc. A smorgasbord of raw material if you will. It felt like squeezing all the stuff I had relevant to the topic out of me. A worthwhile technique. I didn’t have to focus on organisation, presentation or anything else. Just make sure that everything was out on paper. This general thing about having a brain storming stage for any project is a good take away from this exercise.

Organisation of the content

The talk was split into 4 parts - Intro, What, Why and How. This split was one of the ideas that occurred to be during brainstorming. I got a ton of points from the above process. Here are two examples - Wisdom is contextualising knowledge, “Those who can do, those who can’t teach”. The next step was to structure them. I filtered out things that were very similar and then grouped the rest under the four main headings. Here’s a picture of what that looked like.

Now I more or less had what I was going to say and when. I needed to beat this into form. I clustered these into cogent subgroups. For example, “difference between teaching and mentoring”, “criticism” etc. and these became individual slides. I had roughly 20 or so such slides. The themes they fell under became slide titles. I had a vauge idea by now that I’d make each slide a heading plus a relevant quote. I started searching for quotes to fit each sub topic and time boxed this exercise for 2 days. Once I got sufficient quotes and refined them, I made a beamer deck which I used to refine my own ideas and make sure things were in proper form.

At this point, I had what I wanted to speak about roughly in shape. The specifics of the speech were not concrete yet. That would be the last stage. The next step was to make the actual slides used for the presentation which was a much bigger affair.

The slides

I had a plan to do the slides by hand. I’ve been practising calligraphy for a long time now and this seemed like a good “serious” project to actually try. My first thought was to make each slide a custom poster similar to this but I soon decided that that would take a huge amount of time. Unlike my regular practice sheets, this has a time limit and a clear deliverable.

A small digression. I’ve enjoyed watching the BBC series Mastercrafts. In it, traditional craftsmen take aspiring artisans under their wing and coach them through a six week apprenticeship. The masters often find the newcomers lacking in discipline and one comment I remember is when one tells an aspirant who was too lost in expressing herself that she should be “more of a craftsman and less of an artist”. The former was one who respected deadlines, client requirements and things like that. The latter was just someone pursuing their own creative interests. I wanted to be a craftsman here.

I scrapped the idea of doing individual posters for each slide and after searching a few of my books on the calligraphy for some inspiration, decided on a simple layout of title on the left and text on the right. I had met a friend of mine over coffee a few weeks before this whole exercise and wrote something for her on a sheet paper which was then ruined by a wet glass on top of it. However, this produced a certain grungy effect and I tried to replicate that a few times to see if I could make a “theme” for the slides. Here are some attempts

This kind of thing where I don’t have rules to follow or targets to meet is very relaxing and gets the creative juices flowing and that’s exactly what I did.

I then set about making the first and last slides. I thought it would be nice to give it a background and thought it would be nice to give it a palimpsest feel. With this in mind, I diluted some of the red ink I had and used a flexible speedball nib to write out a paragraph from a book onto a sheet of white paper using copperplate script like so

I had to make repeated samples of this to make the opacity of the ink just right. To measure the amount of extra water added, I was using a syringe. With that done, I took a few colour photocopies of the sheet so that I could experiment without fear. Didn’t want to destroy the original.

You can see how I wrote “colour” a few times on a separate sheet of paper to get the red light enough. I liked the cream coloured paper rather than white so I decided to use that for the deck. After a few attempts, I decided to do it like this

Similarly, the final page. This specific one had some problems so I decided to discard it and redo it but the layout was confirmed.

The background was done with a cheap ink but the actual text was done with Winsor and Newton calligraphic inks which is a much better type of ink. I used speedball C-series nibs of varying sizes to do the actual text. Here is a picture of some of them after cleaning.

The next step was to make “thumbnails”. I got a 4”x3” notepad and made rough “slides” with them to decide on the general layout, order as well as content. This was the deck.

This was quite useful. I was able to make quick and dirty outlines, reorder slides and do other things like that very easily.

Once I had the title slide and the list of slides to make ready, I went straight ahead to making the same layout for all the slides. The title would be on the left half of the sheet somewhat centered. The hand would be Trajan Majuscules with a scarlet colour. The text, well, that had to be decided. Here are the finished slides without any of the text. This took me about 6 hours to do. I attempted to do proper brush capital serifs but apparently, that’s something even master calligraphers have trouble. I couldn’t manage to get them consistently decent so I ditched the idea and went with slightly less formal, less elegant and easier letter forms.

There were some slides that required some artwork (e.g. a timeline for “my journey”). I kept this for later. I also had some plans to make a poster for the oath of the mentor by Harry Percival. I wanted to read this out sentence by sentence so that the audience could repeat after me. This was a nascent plan but if I decided to go ahead with it, I’d need a different layout for this so I kept that aside as well.

Once that was done, I decided to start the actual work on the slides. Each one was different. I eschewed some of the techniques I like (e.g. mixing colours inside the nib) in the interest of time. I didn’t want everything to look the same though. I decided to mix hands, colours, and sizes for some variety and to play a little with the layout. I have a small library of books on the topic and looked through the galleries in each one for ideas.

As for special effects, I decided to use resist for some of the slides and a little spray painting with a toothbrush for some of the others. Each slide took about 40 minutes and within a few weeks, I was done. There were several errors in the slides which meant that many had to be discarded.

Here are some of the good slides that came out

Some of the slides had errors which I decided were too tiny to be worth redoing. I touched up the mistakes using a sharp blade on the actual sheets and using the GIMP after scanning. Here’s one

Some mistakes though were catastrophic and couldn’t be salvaged. They had to be thrown away. Here are two examples of that

Some were easy to do since the layout wasn’t very complicated. Others needed some work and those took longer e.g.

Not as good as I hoped it would be but time was a factor. It was about this time that I tweeted this out.

All the slides had quotes. One was my own and I decided to do it in an uncial hand with huge letters. I made a mistake with the spelling on the first attempt

When this happens, you have to throw away the whole sheet including the heading. So that’s about an hour of work wasted. I started to really appreciate the little things like C-z and Backspace which are there to help you when you do digital slides.

This went on day after day and finally. My nibs really stepped up and delivered

Some slides were particularly stubborn and I had to redo them multiple times. Here was one particularly tough one.

Finally, I was done with all the slides except for the “My journey” one and the “oath of the mentor”. I decided to drop the first in the interest of time and put aside the other after for a week.

I had a small business trip about 10 days before the conference and I took the slides with me there to remove the pencil marks and rulings. I did that at the hotel and then, decided to work on the actual speech contents.

The speech

Mortimer Adler suggests making a detailed outline of the speech and then using those as notes. I wasn’t familiar with how to do this so I gave the speech like I would usually do and recorded it on my phone. It was excruciating to listen to myself but I did it. Took around 45 minutes. I transcribed everything I said onto paper and that gave me my first draft.

Once this was done, I cut out several gaudy points, refined my language a little, put in better analogies and examples. I also used the notes I had initially created for the slide brainstorming to supplement slides that needed more content. I marked some of the points as “optional” which meant that I could kick them out if I wanted to speed things up. I also put some as good lead ins. There were also small points between sections which allowed me to glue individual slides into a continuous whole. I could say half a sentence as a lead-in, then change the slide and continue. It was a wonderful feeling rather than first switching slides and then trying to remember what I planned to say.

This up front preparation worked much better than ad-libbing the speech on stage. You get to correct all the details you don’t like. The only thing you need to master is the ability to deliver the speech with freshness and finesse so that it appears to be done on the fly. I highly recommend it. It was without doubt, my most valuable take away from this exercise.

Once I had that, I gave the speech in front of a mirror using the notes as an aid. This was a really nice experience since I knew in advance before each slide came onto the screen and I could give the whole talk in one smooth stream rather than a series of broken pieces. I also timed myself when I gave the speech and wrote down at what times I should reach certain slides. This allowed me to course correct if I ran ahead and to insert some ad-hoc bits if I was going too slow. When I actually talked, I didn’t have to do this and it ran on time but nevertheless, it seemed useful. Here’s what my notes looked like.

I kept some time aside for the oath at the end and clocked myself at around 40 minutes. 15 minutes for questions and then 5 minutes for the oath. Seemed right.

The oath

The oath was a bit risky. I imagined that it would be well received and that it would start the conference on a high note. On the other hand, if it bombed, all my carefully prepared notes and slides would be ruined. I was in a bit of a fix. Then I decided to make slides for the oath but leave the decision whether to present it or not for later.

I couldn’t do the oath in the same way as I did the rest of the slides so I decided to do a more conventional “poster”. I had almost not experience with a blackletter hand but decided to do it anyway. It was, without doubt, the worst slide from a technical standpoint but by this time, I just wanted to be done with.

Scanning and touch ups

After this, I had a day left before my plane ticket. I spent that day scanning in all the slides.

Once that was done, I scaled, rotated and cleaned them up a little using GIMP and imagemagick. The final result was a single PDF with the slides. Here’s a montage of all the slides

The oath slide, I cut up into individual sentences and then put them all together as a separate PDF so that I could have everyone read off the screen.

Giving the talk

I was a little worried about whether the slides would be visible and the resolutions which would be appropriate. I went to the venue the night before my talk and checked it out on the projector. It looked okay. They of course, changed the resolution the next day so my preparations didn’t really help. I ended up giving the talk with a blank laptop screen. Didn’t bug me much since I had my notes and the timings were spot on.

I was extremely nervous about giving the talk since I had prepared so much and wanted every detail to be right. The interesting part was that it did go quite well. All the small details were noticed and appreciated by many people, the hand made slides were well received, the message resonated with many of the people in the audience and the oath really had the effect that I hoped it would. People took the whole thing quite enthusiastically and my final “thank you” was drowned in applause.

I decided to donate the slides for the pyladies auction at the next PyCon in the States. Most of them are just headings and a quote so the good ones might be serviceable as posters.

Conclusions

Some of the things I learned from this whole affair

Detailed planning and work pays really good dividends. Our brash overconfidence in our abilities might be justified but even then, detailed work and preparation can really help fix problems that even you’re not aware of. This is, by far, my biggest take away. I’m going to give up “winging it” completely if I can and outright refuse to do stuff if I don’t have enough time to prepare.
Deadlines make things serious. The slides would have been hobbies lost in a fog of perfectionism if not for the deadline. I had to cut corners, make compromises, live with mistakes and do anything necessary to get the whole thing done by the trip. Exposed me to how terrible my calligraphy really is.
My time with the Toastmasters club in Kozhikode taught me subtle skills that I wasn’t aware I learned. Giving speeches is serious business and I have a lot to learn in that department.

If you want to follow my practise with calligraphy, I put up my stuff on http://calligraffiti.in/ and tweet about it at @incalligraffiti.

Hamon

2016-03-24T23:23:17+05:30

I’ve been freelancing for a while now it’s been a learning experience. I had to confront parts of me that I was not aware of many of which I’m still struggling to change. However, after my last project, I’ve decided to try a different route and establish a small company offering software services.

The Lycaeum under which I do all my student trainings and mentorship has been getting a decent stream of students since I started it and I’m planning to hire interested students into my new company and grow it slowly and organically.

The company is called Hamon and we’re currently picking up python automation and infrastructure projects. If you’re looking for someone to help you with your project, drop us a line at contact@hamon.in

For the curious, the Hamon is a line that appears on a traditional Japanese sword blade due to a differential tempering technique. The idea is to create a strong and ductile backbone (which gives reliability) and a hard and sharp cutting edge (which gives effectiveness). The joining of these two seemingly antagonistic properties into a single blade produces an artifact called a hamon (literally “blade pattern”). I’ve always been a proponent of tried, tested and battle hardened over new, shiny and fancy and this has informed most of the projects I’ve worked on. At the same time, I pick up new technologies that, in my judgment, look promising and use them. This approach is similar to the way the sword blade is and hence the name. Also, because of my own interest and background in the martial arts, the name speaks to my soul so there you have it. If you’re interested in the details of how the blades are made, check this out.

Ledger CLI

2015-11-07T12:10:09+05:30

This is a post describing my experiences with the command line accounting tool ledger. My general preference for text based tools that “fit” into the UNIX ecosystem and which can work well with Emacs led me to this. The fact that it’s written by someone I admire a lot was an additional reason. The post that follows is a description of how I use it and what I’ve learnt over the past year. There will be gaps in my usage and outright mistakes due to my lack of understanding of accounting concepts and ledger features so be warned.

The tool is fairly simple. It requires that you maintain a list of your transactions in a double entry system. Double entry book keeping means that all transactions should have a debit account (one which receives money) and a credit account (from which money is deducted). At any point in time, the accounts should balance to zero. This took some getting used to for me but not too much and I’ve started thinking of my accounts in this way now.

Basics

A typical input file (which I’ll call the ledger file from now) will be a list of entries that look like this

2014/10/21  Acme groceries
    Expenses:Groceries     500 Rs
    Assets:Cash            -500 Rs

This records that on the 21 of October 2014, I spent 500 Rs cash to buy groceries. It moves 500 Rs from the Assets:Cash account to the Expenses:Groceries account. It’s also possible to have more than two accounts as long as the amounts balance. For example

2015/01/15 Acme restaurant
   Expenses:Eating out     1000 Rs
   Assets:Cash             -500 Rs
   Assets:Savings          -500 Rs

This says that I spent 1000 Rs on a meal at Acme Restaurant and paid 500 Rs in cash and the remaining 500 directly from my savings account (via. my debit card).

Accounts are hierarchical and the hierarchies are split using the :. So, you’d have things like Expenses:Groceries, Expenses:Internet which are two accounts all under the higher level Expenses account. I use Assets for accounts that hold my assets (like my bank accounts etc.), Expenses for expenses, Income to track where my money is coming in from, Liabilities to track money I owe and Loans to track money owed to me. I also have an Equity account to “withdraw” my initial assets from. More on this later. I use an Emacs mode that comes along with ledger to add entries to the file and use git to version control it. I have a few simple shell aliases that make certain queries available with a small number of keystrokes so that I can run things easily. More on this too, later.

Some notes on the non tech. aspects of the whole “system”. While there are many good applications out there on all possible platforms, people usually fail to keep their accounts due to a lack of discipline. They start off in full gusto, like I’ve done before with Gnucash and other tools, and then they run out of steam, falter and, also like I did with Gnucash and other tools, give up. Many try again but finally just let it go for good. The problem is, in summary, lack of discipline. There are two ways, as I see it, to fix this problem. The first is to construct tools that work even if one doesn’t have discipline. So there are apps that allow you, with a smart phone, to add an accounting entry to a database with just a few taps. Others which track SMSs from your bank to maintain a running balance of your financial situation. While not especially disciplined in my daily life, my approach is to try to develop it rather than adjust my lifestyle and tools to not need it. So, I make it a point to note down cash transactions during the day and then finally at the end of the day, as part of a daily ritual, update my ledger file with the expenses of the day. Once a week, as part of a weekly ritual, I cross check my balances against my actual accounts in my bank and wallet to make sure that everything is “in sync”. This took some trouble to get ready but after a month or so, the whole thing has become automatic. I’ve resisted the temptation to try and “remember” things to add into the file since I usually end up forgetting. Everything is either written down or in the file.

The first entry in my ledger is for Jul 1 2014 and I’m still going strong. That’s over a year of information and there are around 1800 transactions in there so it seems to be working.

I have a directory called ~/notes/accounts/ which has several ledger files. There’s one called accounts.dat which is the most heavily used. I also keep a separate one called lycaeum.dat which tracks expenses and income for my mentoring institute. This is version controlled using git and pushed to multiple locations as backup. I have a shell function called l defined like so

l () {
        ledger -w -f ${LEDGER_FILE} -V --date-format='%d-%b-%y' $*
}

This inovkes ledger with a few preset options like -w which prints output in a wide (132 column) format, -f to specify which ledger file to use. The LEDGER_FILE variable is set to ~/notes/accounts/accounts.dat. The -V displays market values for commodities (something which I’ll describe later) and finally, the --date-format to print dates in a way that I can read them quickly. Getting how much I have in my savings accounts would then be done like so

$ l balance Assets:Savings

The top like of the ledger file is ;-*-ledger-*- which means that Emacs opens it up in ledger-mode. This has a few useful keybindings that I use. e.g. C-c C-a to quickly add an entry and C-c C-q to fix alignment. It also has a lot more but I’m not really that heavy a user.

Using ledger

Let’s take a little scenario and describe how things might work. It’s artificial but still representative of how I use it.

I wake up in the morning and look at my TODO list. I have to pay electricity bills today. I pay them online using my credit card and then add an entry like so

2015/11/07 Acme power company
    Expenses:Electricity    1000 Rs
    Liabilities:CC

One of the accounts in an entry can be left blank and ledger will automatically fill it in. In this case, the entry is as if we entered -1000 Rs for the Liabilities:CC account.

I get some work done and receive a text message which says that one of my service providers has charged my credit card for a VPS that I have. So,

2015/11/07 Acme Cloud 
    Expenses:Hosting    1500 Rs
    Liabilities:CC

After a while, I have to run out to get some groceries and stuff. I first withdraw some cash from the bank and then spend some of it at a few stores. I come back with a few bills. I decide to update the journal right away.

2015/11/07 Withdrawal
    Assets:Cash     3000 Rs
    Assets:Savings

2015/11/07 Acme Groceries
    Expenses:Groceries    1500 Rs
    Assets:Cash

2015/11/07 Acme Fisheries
    Expenses:Groceries:Meat    500 Rs
    Assets:Cash

This says that I withdrew 3000 Rs. from the savings bank account and then spent 2000 from that on some groceries and some fish (which I usually track separately). Back at home, I continue with my day and have to go out again. I refuel my motorcycle and keep the amount noted down. I also stop for a quick bite to eat from a small bakery. The resulting ledger entries are like so

2015/11/07 Acme fuel
    Expenses:Petrol:Bike    500 Rs
    Assets:Cash

2015/11/07 Acme bakery
    Expenses:Food          50 Rs
    Assets:Cash

After entering this, just to see if things are in sync, I run

l b Assets:Cash

Commands to ledger can be abbreviated so b means balance. It says

     1,250.00 Rs  Assets:Cash

I check my wallet and see that there’s just 1,150 Rs in there. I spent 100 Rs somewhere during the day which I’m not sure of and I quickly make an entry to balance to discrepancy

2015/11/07 Acme fuel
    Assets:Cash    =1150 Rs
    Expenses:Unknown

The = in the amount here sets the amount in that account to the specified number and will add or deduct from the Expenses:Unknown account to make the Assets:Cash correct. This over time, accumulates accounting errors and ideally, it should be 0.

I get a text message indicating a bank transfer. A student from The Lycaeum has transferred a part of the fees for a mentoring course that I’m conducting. Okay

2015/11/07 Mentoring course Fees
    Assets:Business      5000 Rs
    Income:Lycaeum:Fees

I keep a separate bank account for business transactions so this amount goes there and it’s credited from the Income:Lycaeum:Fees account. Now, I need to book a few tickets for an upcoming training which I’m going to do. This means

 2015/11/07 Acme Bus service
     Expenses:Travel      1000 Rs
     Liabilities:CC

To make things more realistic, let’s add a section to the top of our file like so

2015/11/01 Initial balance
    Assets:Business    50000 Rs
    Assets:Savings    25000 Rs
    Assets:Cash        5000 Rs
    Equity

These are the initial amounts taken out of a special Equity account. This is how initial balances are handled.

I will also add similar entries for a few other days in the month so that the file is more interesting. I’ve annotated it with comments (the lines that start with ;) for clarity. The final file looks like this

And now, let’s run some queries on it.

How much do I have in my various accounts? My assets.

$ l b Assets
            27100 Rs  Assets
             7500 Rs    Business
             2400 Rs    Cash
            17200 Rs    Savings
--------------------
            27100 Rs

The ledger tool takes a query type as its first argument. There are several query types. The one we’ve used above is balance (abbreviated to b). You can then pass a query expression which will limit the entries that are displayed. In this case, I’ve asked for the balance query for the Assets accounts. It shows me a list and a total. That’s nice. Now, What were my expenses this month? (sorted)

$ l r -M --period-sort total Expenses
01-Nov-15 - 30-Nov-15                        Expenses:Food                                          50 Rs                50 Rs
                                             Expenses:Unknown                                      100 Rs               150 Rs
                                             Expenses:Maintenance                                  200 Rs               350 Rs
                                             Expenses:Medicine                                     250 Rs               600 Rs
                                             Expenses:Petrol:Bike                                  500 Rs              1100 Rs
                                             Expenses:Entertainment                                595 Rs              1695 Rs
                                             Expenses:Groceries:Meat                               700 Rs              2395 Rs
                                             Expenses:Fine                                         900 Rs              3295 Rs
                                             Expenses:Fitness                                     1000 Rs              4295 Rs
                                             Expenses:Travel                                      1000 Rs              5295 Rs
                                             Expenses:Entertainment:Restaurant                    1500 Rs              6795 Rs
                                             Expenses:Hosting                                     1500 Rs              8295 Rs
                                             Expenses:Calligraphy                                 2500 Rs             10795 Rs
                                             Expenses:Groceries                                   2700 Rs             13495 Rs
                                             Expenses:Domestic                                    3500 Rs             16995 Rs
                                             Expenses:Maintenance:Washing machine                 4000 Rs             20995 Rs
                                             Expenses:Charity                                     5000 Rs             25995 Rs
                                             Expenses:Rent                                        5000 Rs             30995 Rs

This time, we use the register (abbreviated to r) query. It’s probably the most heavily used one. Here, we pass -M to group transactions by month. By default, there’s no grouping and all transactions are displayed. Here, we want it to group so that we can further process it. We then provide the --period-sort option which will sort the entries in the group by the given parameter. We sort by “total” (which means total amount). After this, I use “Expenses” since that’s what I’m interested in and find out that I spent the most on charity and rent.

How much do I owe?

$ l b Liabilities
            -5595 Rs  Liabilities:CC

A simple balance query on Liabilities. How much do I withdraw on average per week?

$ l reg -A  -W Assets:Cash and @Withdraw
01-Nov-15 - 07-Nov-15                        Assets:Cash                                          5400 Rs              5400 Rs
08-Nov-15 - 14-Nov-15                        <Adjustment>                                        -3900 Rs                    0
                                             Assets:Cash                                          2400 Rs              3900 Rs
15-Nov-15 - 21-Nov-15                        <Adjustment>                                        -6967 Rs                    0
                                             Assets:Cash                                          8500 Rs              5433 Rs

This is another register query. The first column in a register query is the transaction amount. The second is usually a running total. The -A flag converts that to show a running average rather than a running total. The -W asks ledger to group things weekly (we used -M for montly earlier). The query expression is a little more complicated. It says “all transactions in touching the Assets:Cash account and which have Withdraw in the description”. The @ matches against descriptions rather than account names. The final 5344 Rs. tells us that we withdraw that much every week.

Automated transactions.

Ledger can also automatically perform transactions based on criteria. I have an entry like so

= /^Income/
    (Liabilities:Service tax)                 0.14

Which means, for all transactions that start with “Income”, put 14% into a Liabilities:Service tax amount. This means that if I have a transaction like so

2015/10/10
    Assets:Business   1000 Rs
    Income:Consulting

where the amount against Income:Consulting is -1000 Rs, ledger would automatically credit 140 Rs from the Liabilities:Service tax account. I can then quickly see how much I owe as service tax.

Budgeting

This is a feature that I don’t use enough yet but which I’ve started with. The basic idea is to create a budget for each account up front and then check against that. Let’s try that with. A version of the accounts file with added budgets is shown below

The added section is

~ Monthly
    Expenses:Rent                               5000 Rs
    Expenses:Groceries                          3500 Rs
    Expenses:Domestic                           4000 Rs
    Expenses:Electricity                        1500 Rs
    Expenses:Entertainment                      2000 Rs
    Expenses:Food                               1000 Rs
    Expenses:Fitness                             500 Rs
    Expenses:Hosting                            2500 Rs
    Expenses:Medicines                          1000 Rs
    Expenses:Petrol                             5000 Rs
    Assets

This tells ledger that every month, it should allocate this much money for each of the accounts specified and all of it comes from the Assets account. We could (and usually also do) have a ~ Yearly budget.

A register query for the current month for expenses like so. The -p allows us to narrow the date range to the current month.

$ l r -p "this month" Expenses
01-Nov-15 Acme groceries                     Expenses:Groceries                                    500 Rs               500 Rs
02-Nov-15 Acme Electricals                   Expenses:Maintenance                                  200 Rs               700 Rs
02-Nov-15 Acme stationeries                  Expenses:Calligraphy                                 2500 Rs              3200 Rs
04-Nov-15 Acme fine dining                   Expenses:Entertainment:Restaurant                    1500 Rs              4700 Rs
06-Nov-15 Traffic fine                       Expenses:Fine                                         900 Rs              5600 Rs
07-Nov-15 Acme Cloud                         Expenses:Hosting                                     1500 Rs              7100 Rs
07-Nov-15 Acme Groceries                     Expenses:Groceries                                   1500 Rs              8600 Rs
07-Nov-15 Acme Fisheries                     Expenses:Groceries:Meat                               500 Rs              9100 Rs
07-Nov-15 Acme fuel                          Expenses:Petrol:Bike                                  500 Rs              9600 Rs
07-Nov-15 Acme bakery                        Expenses:Food                                          50 Rs              9650 Rs
07-Nov-15 Acme fuel                          Expenses:Unknown                                      100 Rs              9750 Rs
07-Nov-15 Acme Bus service                   Expenses:Travel                                      1000 Rs             10750 Rs
08-Nov-15 Acme Groceries                     Expenses:Groceries                                    700 Rs             11450 Rs
08-Nov-15 Acme meat products                 Expenses:Groceries:Meat                               200 Rs             11650 Rs
10-Nov-15 Acme save the whales fund          Expenses:Charity                                     5000 Rs             16650 Rs
12-Nov-15 Gog                                Expenses:Entertainment                                595 Rs             17245 Rs
15-Nov-15 Domestic help                      Expenses:Domestic                                    3500 Rs             20745 Rs
17-Nov-15 Acme medicals                      Expenses:Medicine                                     250 Rs             20995 Rs
18-Nov-15 Acme repair company                Expenses:Maintenance:Washing machine                 4000 Rs             24995 Rs
21-Nov-15 Acme Housing                       Expenses:Rent                                        5000 Rs             29995 Rs
25-Nov-15 Acme Gym                           Expenses:Fitness                                     1000 Rs             30995 Rs

Now, let’s see how this balances against our budgets. This is as simple as adding the --budget flag to the query

$ l r --budget Expenses
01-Nov-15 Budget transaction                 Expenses:Rent:House                                 -5000 Rs             -5000 Rs
01-Nov-15 Budget transaction                 Expenses:Groceries                                  -3500 Rs             -8500 Rs
01-Nov-15 Budget transaction                 Expenses:Domestic                                   -4000 Rs            -12500 Rs
01-Nov-15 Budget transaction                 Expenses:Electricity                                -1500 Rs            -14000 Rs
01-Nov-15 Budget transaction                 Expenses:Entertainment                              -2000 Rs            -16000 Rs
01-Nov-15 Budget transaction                 Expenses:Food                                       -1000 Rs            -17000 Rs
01-Nov-15 Budget transaction                 Expenses:Fitness                                     -500 Rs            -17500 Rs
01-Nov-15 Budget transaction                 Expenses:Hosting                                    -2500 Rs            -20000 Rs
01-Nov-15 Budget transaction                 Expenses:Medicines                                  -1000 Rs            -21000 Rs
01-Nov-15 Budget transaction                 Expenses:Petrol                                     -5000 Rs            -26000 Rs
01-Nov-15 Acme groceries                     Expenses:Groceries                                    500 Rs            -25500 Rs
04-Nov-15 Acme fine dining                   Expenses:Entertainment                               1500 Rs            -24000 Rs
07-Nov-15 Acme Cloud                         Expenses:Hosting                                     1500 Rs            -22500 Rs
07-Nov-15 Acme Groceries                     Expenses:Groceries                                   1500 Rs            -21000 Rs
07-Nov-15 Acme Fisheries                     Expenses:Groceries                                    500 Rs            -20500 Rs
07-Nov-15 Acme fuel                          Expenses:Petrol                                       500 Rs            -20000 Rs
07-Nov-15 Acme bakery                        Expenses:Food                                          50 Rs            -19950 Rs
08-Nov-15 Acme Groceries                     Expenses:Groceries                                    700 Rs            -19250 Rs
08-Nov-15 Acme meat products                 Expenses:Groceries                                    200 Rs            -19050 Rs
12-Nov-15 Gog                                Expenses:Entertainment                                595 Rs            -18455 Rs
15-Nov-15 Domestic help                      Expenses:Domestic                                    3500 Rs            -14955 Rs
25-Nov-15 Acme Gym                           Expenses:Fitness                                     1000 Rs            -13955 Rs

Now the numbers look different. What ledger does is that it deducts each budget item from the appropriate account thereby reducing it and then actual expenses increase it again. If the final total is negative, it means that you’re below budget. If it’s above, it means that you’ve crossed your budget. To clarify, let’s look at just one account

$ l r --budget Expenses:Rent
01-Nov-15 Budget transaction                 Expenses:Rent                                       -5000 Rs             -5000 Rs
21-Nov-15 Acme Housing                       Expenses:Rent                                        5000 Rs                    0

Tells us that we had 5000 Rs budgeted for the rent and it was fully spent. Not so for entertainment

$ l r --budget Expenses:Entertainment
01-Nov-15 Budget transaction                 Expenses:Entertainment                              -2000 Rs             -2000 Rs
04-Nov-15 Acme fine dining                   Expenses:Entertainment                               1500 Rs              -500 Rs
12-Nov-15 Gog                                Expenses:Entertainment                                595 Rs                95 Rs

where we overshot our budget by 95 Rs. However, for Domestic expenses, we are under budget

$ l r --budget Expenses:Domestic
01-Nov-15 Budget transaction                 Expenses:Domestic                                   -4000 Rs             -4000 Rs
15-Nov-15 Domestic help                      Expenses:Domestic                                    3500 Rs              -500 Rs

Viewing everything like we first did gives us the information that we’re within our budget which is good.

Concluding remarks

Like I said above, I’ve been using the tool for a year now. The whole thing has worked for me and as of today, I have around 30k in my Unknown account. That’s not very good but since this is the first time I’m keeping accounts and this is over a whole year, it’s not that bad either.

It’s an ideal tool for UNIX aficionados. It’s text based, command line driven, super fast, has a somewhat steep learning curve with great rewards at the end, extremely flexible, transparent about what it does and a few other things which characterise the kind of software I like but that’s for another post. The official site is ledger-cli.org. The manual is worth reading just for itself and is fairly complete.

SQLAlchemy and full text searching in postgresql

2013-11-29T11:28:48+05:30

Introduction

Postgresql has support for full text search. The basic idea is to create a column of type tsvector and then you can run full text queries (represented as tsquery strings) using the @@ operator. This is different from the LIKE queries using the %string% since this is language aware and can provide things like ranking etc. As an example,

SELECT 'I am satisfied with postgresql' LIKE '%satisfied%' as found;

will return

found 
-------
t

However, if we use "%satisfy%" as the query string, it will fail since it’s text based.

SELECT 'I am satisfied with postgresql' LIKE '%satisfy%' as found;
 found 
-------
 f

If we use a full text query, we will get it right though.

SELECT to_tsvector('I am satisfied with postgresql') @@ to_tsquery('satisfy') as found;
 found 
-------
 t

SQLAlchemy is the Python database toolkit of choice. It supports most of the popular databases out there and has dialect specific features for mysql, postgresql etc. The tsvector type is not natively supported. It’s not very hard to add support for this but I couldn’t find a single reference that helps me do it. After some trial and error, I managed to get this to work (atleast for my purposes) so I’m going to write out what I did. I’ll put links to the articles, posts and other materials on the net which helped me get this to work.

Feedback is welcome as are suggestions on how to get this fully feature complete. If sufficiently done, I’ll contribute this as a patch back to SQLAlchemy.

The table we want

First, I create a table like so

CREATE TABLE example (
    name VARCHAR(10),
    details TEXT
);

And then, I insert 1000 rows into this using

COPY example FROM '/home/noufal/projects/scratch/sa/items.csv' (FORMAT csv);

Now, I have a database to play with. I can do full text searches like so

SELECT * FROM example where to_tsvector('english', details) @@ to_tsquery('life') limit 3;
  name   |                                                                     details                                                                     
---------+-------------------------------------------------------------------------------------------------------------------------------------------------
 item-20 |  Life is a grand adventure -- or it is nothing.                 -- Helen Keller 
 item-46 |  Life is a gamble at terrible odds; if it was a bet you wouldn't take it.               -- Tom Stoppard; Rosencrantz and Guildenstern are Dead 
 item-63 |  Life is like a 10 speed bicycle.  Most of us have gears we never use.          -- C. Schultz

The 'english' in the to_tsvector is optional. If I skip it, it’ll use the default.

Full text items that can be searched are referred to as documents in postgresql. Now, I create an extra column that holds the document to be searched like so.

ALTER TABLE example ADD COLUMN details_tsvector TSVECTOR;

Then, I run an update on the table that creates the tsvector documents and puts them into this column like so

UPDATE example SET details_tsvector = to_tsvector(details);

Now, if I do a full query, it’ll return the actual tsvector documents.

SELECT name, details_tsvector FROM example limit 3;
  name  |                    details_tsvector                     
--------+---------------------------------------------------------
 item-1 | 'bathroom':7 'left':2 'wallet':4
 item-2 | 'bit':7 'could':11 'difficult':3 'effort':9 'imposs':13
 item-3 | 'avoid':3 'hedg':4 'least':6 'think':11

I can search the table like so

SELECT name, details FROM example WHERE details_tsvector @@ to_tsquery('life') limit 3;
  name   |                                                                     details                                                                     
---------+-------------------------------------------------------------------------------------------------------------------------------------------------
 item-20 |  Life is a grand adventure -- or it is nothing.                 -- Helen Keller 
 item-46 |  Life is a gamble at terrible odds; if it was a bet you wouldn't take it.               -- Tom Stoppard; Rosencrantz and Guildenstern are Dead 
 item-63 |  Life is like a 10 speed bicycle.  Most of us have gears we never use.          -- C. Schultz

This, of course, is faster than the other approach since the documents have already been generated. However, running an EXPLAIN shows us how the query is working.

EXPLAIN SELECT name, details FROM example WHERE details_tsvector @@ to_tsquery('life');
                        QUERY PLAN                        
----------------------------------------------------------
 Seq Scan on example  (cost=0.00..53.30 rows=26 width=92)
   Filter: (details_tsvector @@ to_tsquery('life'::text))

One advantage of tsvector columns (over LIKE queries) is that they can be indexed. If I add an index like so,

CREATE INDEX details_idx ON example USING gin(details_tsvector);

The output of the EXPLAIN command changes.

EXPLAIN SELECT name, details FROM example WHERE details_tsvector @@ to_tsquery('life');
                                 QUERY PLAN                                 
----------------------------------------------------------------------------
 Bitmap Heap Scan on example  (cost=12.20..50.17 rows=26 width=92)
   Recheck Cond: (details_tsvector @@ to_tsquery('life'::text))
   ->  Bitmap Index Scan on details_idx  (cost=0.00..12.20 rows=26 width=0)
         Index Cond: (details_tsvector @@ to_tsquery('life'::text))

which will be quicker.

One problem with this is that the details_tsvector column is not automatically updated when new rows are inserted. I can fix this by creating a trigger that will automatically compute and add the values when a row is UPDATEd or INSERTed into.

CREATE TRIGGER details_tsvector_update BEFORE INSERT OR UPDATE
ON example
FOR EACH ROW EXECUTE PROCEDURE
tsvector_update_trigger('details_tsvector', 'pg_catalog.english', 'details');

Now, if I run an INSERT statement, the details_tsvector will get updated.

The table definition looks like this

                Table "public.example"
      Column      |         Type          | Modifiers 
------------------+-----------------------+-----------
 name             | character varying(10) | 
 details          | text                  | 
 details_tsvector | tsvector              | 
Indexes:
    "details_idx" gin (details_tsvector)
Triggers:
    details_tsvector_update BEFORE INSERT OR UPDATE ON example FOR EACH ROW EXECUTE PROCEDURE tsvector_update_trigger('details_tsvector', 'pg_catalog.english', 'details')

This is what I need for my application but I need to do it using SQLAlchemy inside my application rather than in SQL. The rest of the article will cover this.

Scaffold

To play with the whole thing, I have a tiny program that takes command line arguments to run various database operations. It’s what I use to manually test the code. Here it is. There are a few imports which are not necessary at this point but which we’ll use later.

import subprocess

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, VARCHAR, create_engine, func, MetaData, Table, Index, event, DDL
from sqlalchemy.orm import sessionmaker

engine = create_engine('postgresql://noufal:abcdef@localhost/test', echo = True)
Base = declarative_base()
Session = sessionmaker(bind = engine)
session = Session()

class Example(Base):
    __tablename__ = 'example'

    name = Column(VARCHAR(10), primary_key = True)
    details = Column(String)


def create_tables():
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)


if __name__ == '__main__':
    import sys
    for i in sys.argv[1:]:
        print "\n","=================",i,"==================="
        dict(create = create_tables)[i]()

This should be familiar to you if you’ve used SQLAlchemy before. It simply defines a table in the new declarative format. One difference between this and our original setup is that the name field is now a primary key and therefore has a uniqueness constraints and an auto increment. The file is called sample.py If we run

python sample.py create

We’ll get the example table. The schema is very plain now and doesn’t have the tsvector type.

Creating the type

For this, we’ll need to derive from the sqlalchemy.types.UserDefinedType. This is mostly based on the postgis example in the SQLAlchemy source tree. We simply create a new type derived from UserDefinedType and then give it a name. We have just one method inside it which is get_col_spec.

class TsVector(UserDefinedType):
    "Holds a TsVector column"

    name = "TSVECTOR"

    def get_col_spec(self):
        return self.name

The get_col_spec function is used by the expression compiler to decide what the name of the type will be in the DDL. Since it’s called TSVECTOR, that’s what we should return here. The core types, as far as I know have their types coded directly into the compiler (e.g. For the SQLAlchemy provided Boolean type translates to the BOOLEAN type in the DDL). For Use defined types, the compiler will explicitly call the get_col_spec function to get the type name. This is the bare minimum to create the table.

Creating the tables.

First, we add the above snippet to our code and then we add a column of type TsVector to our Example class and run the script again. This time, we’ll get the table with the tsvector column. Our code looks like this now.

import subprocess

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, VARCHAR, create_engine, func, MetaData, Table, Index, event, DDL
from sqlalchemy.orm import sessionmaker
from sqlalchemy.types import UserDefinedType

engine = create_engine('postgresql://noufal:abcdef@localhost/test', echo = True)
Base = declarative_base()
Session = sessionmaker(bind = engine)
session = Session()

class TsVector(UserDefinedType):
    "Holds a TsVector column"

    name = "TSVECTOR"

    def get_col_spec(self):
        return self.name

class Example(Base):
    __tablename__ = 'example'

    name = Column(VARCHAR(10), primary_key = True)
    details = Column(String)
    details_tsvector = Column(TsVector)


def create_tables():
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)


if __name__ == '__main__':
    import sys
    for i in sys.argv[1:]:
        print "\n","=================",i,"==================="
        dict(create = create_tables)[i]()

Add a few helper functions

I’m adding a new function called insert_data like so

def insert_data():
    for i in range(1, 20):
        u = Example(name = "name-{}".format(i),
                    details = subprocess.Popen("/usr/games/fortune", stdout = subprocess.PIPE).stdout.read())
        session.add(u)
        print ".",
    session.commit()

And another called dump_data like so

def dump_data():
    for i in session.query(Example):
        print "name: ", i.name
        print "details: \n------------\n",i.details
        print "details_tsvector: \n--------\n",i.details_tsvector
        print "=================================================="

and making them available from the command line using

 dict(create = create_tables,
      insert = insert_data,
      dump = dump_data)[i]()

Now, I can run

 python sample.py create insert

and I get a database with 20 rows in it. Running

 python sample.py dump

will show me what’s in there. The details_tsvector row is always empty.

Adding an index for this column

As it stands, the details_tsvector row doesn’t have it’s own index. We can alter the table definition to add an index by adding

__table_args__ = (Index('details_tsvector_idx', 'details_tsvector', postgresql_using = 'gin'),)

to the Example table.

__table_args__ allows you to add extra stuff like constraints and things to the table. It expects this to be a tuple or a dictionary so we wrap it in the ( ) (along with the notoriously ugly , at the end for single valued tuples). The Index('details_tsvector_idx', 'details_tsvector', postgresql_using = 'gin') line creates a gin index called details_tsvector_idx on the details_tsvector column. The postgresql dialect module interprets the postgresl_using parameter to create a gin index. Different indices have different tradeoffs and you should select one that works for you.

Reference.

Automatically updating the column using a trigger.

I need to automatically update this column when a new row is inserted into the table. We can accomplish this inside postgresql itself using a trigger. This is done like so.

trigger_snippet = DDL("""
CREATE TRIGGER details_tsvector_update BEFORE INSERT OR UPDATE
ON example
FOR EACH ROW EXECUTE PROCEDURE
tsvector_update_trigger(details_tsvector,'pg_catalog.english', 'details')
""")

event.listen(Example.__table__, 'after_create', trigger_snippet.execute_if(dialect = 'postgresql'))

This automatically updates the rows that were modified by an insert or update operation. The tsvector_update_trigger is a function provided by postgresql that takes 3 arguments - the name of the column that needs to be updated, the configuration to use for the conversion and then a list of columns that will be included in the document. It will take care of NULL columns.

I think it’s also possible to do this using the sqlalchemy event system but I’m not sufficiently familiar with it.

Adding elements now will automatically populate the index column.

Reference

The code now, looks like this.

import subprocess

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, VARCHAR, create_engine, func, MetaData, Table, Index, event, DDL
from sqlalchemy.orm import sessionmaker
from sqlalchemy.types import UserDefinedType

engine = create_engine('postgresql://noufal:abcdef@localhost/test', echo = True)
Base = declarative_base()
Session = sessionmaker(bind = engine)
session = Session()

class TsVector(UserDefinedType):
    "Holds a TsVector column"

    name = "TSVECTOR"

    def get_col_spec(self):
        return self.name

class Example(Base):
    __tablename__ = 'example'

    name = Column(VARCHAR(10), primary_key = True)
    details = Column(String)
    details_tsvector = Column(TsVector)

    __table_args__ = (Index('details_tsvector_idx', 'details_tsvector', postgresql_using = 'gin'),)

trigger_snippet = DDL("""
CREATE TRIGGER details_tsvector_update BEFORE INSERT OR UPDATE
ON example
FOR EACH ROW EXECUTE PROCEDURE
tsvector_update_trigger(details_tsvector,'pg_catalog.english', 'details')
""")

event.listen(Example.__table__, 'after_create', trigger_snippet.execute_if(dialect = 'postgresql'))
    

def create_tables():
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)

def insert_data():
    for i in range(1, 20):
        u = Example(name = "name-{}".format(i),
                    details = subprocess.Popen("/usr/games/fortune", stdout = subprocess.PIPE).stdout.read())
        session.add(u)
        print ".",
    session.commit()


def dump_data():
    for i in session.query(Example):
        print "name: ", i.name
        print "details: \n------------\n",i.details
        print "details_tsvector: \n--------\n",i.details_tsvector
        print "=================================================="


if __name__ == '__main__':
    import sys
    for i in sys.argv[1:]:
        print "\n","=================",i,"==================="
        dict(create = create_tables,
             insert = insert_data,
             dump = dump_data)[i]()

Now, you can run python sample.py create and get the whole thing done. The defined table looks like this.

               Table "public.example"
      Column      |         Type          | Modifiers 
------------------+-----------------------+-----------
 name             | character varying(10) | not null
 details          | character varying     | 
 details_tsvector | tsvector              | 
Indexes:
    "example_pkey" PRIMARY KEY, btree (name)
    "details_tsvector_idx" gin (details_tsvector)
Triggers:
    details_tsvector_update BEFORE INSERT OR UPDATE ON example FOR EACH ROW EXECUTE PROCEDURE tsvector_update_trigger('details_tsvector', 'pg_catalog.english', 'details')

So, we have what we need. The next thing is querying.

Querying

In SQL, it is possible now to run queries like so.

select * from example where details_tsvector @@ to_tsquery('life') limit 1;

which finds rows with the lexeme life in our full text search column.

I’d like to do this in Python using my custom type. Something like this.

session.query(Example).filter(Example.details_tsvector == "life")

And this should translate into

SELECT * FROM example WHERE details_tsvector @@ to_tsquery('life');

This might not be optimal since the == is distinct from the @@ operator but it’ll illustrate the comparator factory and allow us to use a native Python comparison operator in our expressions.

We can accomplish this by adding a

class comparator_factory(UserDefinedType.Comparator):
    """Defines custom types for tsvectors. 
    
    Specifically, the ability to search for ts_query strings using
    the @@ operator.

    On the Python side, this is implemented simply as a `==` operation.

    So, you can do 
      Table.tsvector_column == "string" 
    to get the same effect as
      tsvector_column @@ to_tsquery('string')
    in SQL

    """
    
    def __eq__(self, other):
        return self.op('@@')(func.to_tsquery(other))

to our TsVector class. This basically defines a way for the __eq__ operator to be converted into the @@ operator in SQL world.

We also add a query_data function to our harness so that we can try to run a query. It looks like this

def query_data():
    vals = session.query(Example.name, Example.details).filter(Example.details_tsvector == "life")
    for i in vals:
        print "name: ", i.name
        print "details: ", i.details
        print "=================================================="

And update the commands dictionary with a query key so that I can run python sample.py query. The generated query looks like this

SELECT example.name AS example_name, example.details AS example_details 
FROM example 
WHERE example.details_tsvector @@ to_tsquery('life')

which is exactly what I want.

Full code

The entire program looks like this now

#!/usr/bin/env python

import subprocess

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, VARCHAR, create_engine, func, MetaData, Table, Index, event, DDL
from sqlalchemy.orm import sessionmaker
from sqlalchemy.types import UserDefinedType

engine = create_engine('postgresql://noufal:abcdef@localhost/test', echo = True)
Base = declarative_base()
Session = sessionmaker(bind = engine)
session = Session()

class TsVector(UserDefinedType):
    "Holds a TsVector column"

    name = "TSVECTOR"

    def get_col_spec(self):
        return self.name


    class comparator_factory(UserDefinedType.Comparator):
        """Defines custom types for tsvectors.

        Specifically, the ability to search for ts_query strings using
        the @@ operator.

        On the Python side, this is implemented simply as a `==` operation.

        So, you can do
          Table.tsvector_column == "string"
        to get the same effect as
          tsvector_column @@ to_tsquery('string')
        in SQL

        """

        def __eq__(self, other):
            return self.op('@@')(func.to_tsquery(other))


class Example(Base):
    __tablename__ = 'example'

    name = Column(VARCHAR(10), primary_key = True)
    details = Column(String)
    details_tsvector = Column(TsVector)

    __table_args__ = (Index('details_tsvector_idx', 'details_tsvector', postgresql_using = 'gin'),)

trigger_snippet = DDL("""
CREATE TRIGGER details_tsvector_update BEFORE INSERT OR UPDATE
ON example
FOR EACH ROW EXECUTE PROCEDURE
tsvector_update_trigger(details_tsvector,'pg_catalog.english', 'details')
""")

event.listen(Example.__table__, 'after_create', trigger_snippet.execute_if(dialect = 'postgresql'))


def create_tables():
    Base.metadata.drop_all(engine)
    Base.metadata.create_all(engine)

def insert_data():
    for i in range(1, 20):
        u = Example(name = "name-{}".format(i),
                    details = subprocess.Popen("/usr/games/fortune", stdout = subprocess.PIPE).stdout.read())
        session.add(u)
        print ".",
    session.commit()


def dump_data():
    for i in session.query(Example):
        print "name: ", i.name
        print "details: \n------------\n",i.details
        print "details_tsvector: \n--------\n",i.details_tsvector
        print "=================================================="


def query_data():
    vals = session.query(Example.name, Example.details).filter(Example.details_tsvector == "divide")
    for i in vals:
        print "name: ", i.name
        print "details: ", i.details
        print "=================================================="


if __name__ == '__main__':
    import sys
    for i in sys.argv[1:]:
        print "\n","=================",i,"==================="
        dict(create = create_tables,
             insert = insert_data,
             dump = dump_data,
             query = query_data)[i]()

Things to implement

Table reflection doesn’t work (I don’t need it now). I haven’t put this into production so I don’t know what problems I’ll face then.

Feedback is welcome.

Update (10 Dec 2013)

This was accepted as a patch into SQLAlchemy so you don’t need to do this anymore. The == overloading was not used. Instead, the Python .match operator implements the @@ operator on the sql side. I’ll leave this article up here though as a tutorial on adding custom types to SQLAlchemy.

Lines

2013-10-08T14:27:14+05:30

Very often during my work, I’ve needed to treat a list of things (usually stored in a file one per line) as a set. This means that we should be able to to find the union of two files, intersection of two files etc. For example, if one file A looks like this

item1
item2
item3
item4

and file B looks like

item3
item4
item5
item6

Their union should be a file that has item1, item2, item3, item4, item5 and item6. The intersection would be a file that has item3 and item4. A - B would be the set of all elements in A which are not in B (in this case item1 and item2). The symmetric difference of A and B would be the set of elements in A or B but not in both. In other words (item1, item2, item5 and item6). It’s possible to script my way through this tangle with the usual UNIX tools and a bunch of temporary files but I needed these often enough to justify a new tool.

This was born the unimaginatively named lines (PyPI link). It gives you a command line program to do things like this. Here are a few examples.

(x)% cat examples/f1
a
b
c
d
(x)% cat examples/f2
c
d
e
f
(x)% python lines.py -u examples/f1 examples/f2 #Union
a
c
b
e
d
f
(x)% python lines.py -i examples/f1 examples/f2 # Intersection
c
d
(x)% python lines.py -d examples/f1 examples/f2 # difference (f1 - f2)
a
b
(x)% python lines.py -s examples/f1 examples/f2 # symmetric difference (xor)
a
b
e
f
(x)%

I’ve found this useful when I need to deal with lists of file names or lists of tests.

I added two more features which are more flaky but still useful. The first one squeezes empty lines from a file.

(x)% cat examples/f3
a
b

c

d

e
f
(x)% python lines.py --squeeze examples/f3
a
b
c
d
e
f
(x)%

The other is more interesting which I first wrote while working for a an Internet Archive project. With a very long set of elements that are almost similar, it’s useful to be able to figure out anomalies. In my case, I had a list of item name. Each item name looked something like this cuil-domainshard-corpus5-large-merge-rev1.00478-of-25000 The 00478 would change. There would be a few items in this list that didn’t “fit the pattern” usually caused by an interrupted transfer or something similar. It’s useful to be able to weed these out quickly. The --patterns option looks through the entire list and partitions the set into subsets. The elements of a subset will all have an upper bound on their Levenshtein distance from one another. This means that for a file like this

Archive.001-of-020.part
Archive.002-of-020.part
Archive.003-of-020.part
Archive.004-of-020.part
Archive.005-of-020.part
Archive.006-of-020.part
Archive.007-of-020.part
.Archive.008-of-020.part.zbnrw
Archive.009-of-020.part
Archive.010-of-020.part
Archive.011-of-020.part
Archive.012-of-020.part
Archive.013-of-020.part
Archive.014-of-020.part
Archive.015-of-020.part
Archive.016-of-020.part
Archive.017-of-020.part
Archive.018-of-020.part
Archive.019-of-020.part
Archive.020-of-020.part

Running python lines.py --patterns examples/f6 -l 5, would give me

19 elements
1 elements - {'.Archive.008-of-020.part.zbnrw'}

This means that there are 19 elements in one subset and 1 in another. The -l option allows us to tune the upper bound on the levenshtein distance and the -p parameter allows us to configure a “outlier percentage”. If the number of elements in a subset is less than this percentage, the elements will be printed out in the output.

This is an alpha release so it’s probably buggy. Comments and feedback are welcome.

UNIX command line course

2013-08-03T19:04:06+05:30

I grew up on DOS and then jumped straight to UNIX (Solaris and then Linux) without really using Windows. I was introduced to “The UNIX programming environment” by Kernighan and Pike by a friend and studied the basics of using the shell from it. Most of the tricks stuck with me since then and while I’ve never used shell for a production program, it’s always my go to tool for throw away scripts and glue programs. A lot of what I’ve picked up over the years has evaporated but the general feeling that anything is possible once you have shell access has stayed and has been proven true repeatedly.

I have, however, seen ace programmers who aren’t comfortable with the command line and repeatedly resort to using the mouse or larger GUI based programs (often less powerful that their CLI counterparts) to get simple things done. The sharper ones write scripts in their favourite languages to get small things done while the slower ones do them manually by hand. Recent posts like this which was popular on hacker news point to an education problem and since I’m fond of teaching, I’ve decided to put together a course on command line UNIX and shell scripting. I hope to do it in association with HasGeek in Bangalore just like I do my git workshops.

The plan, atleast for now, is to cover the following

The command line - Mostly editing, navigating, working with the history, prompts etc.
Shell as a language - Text manipulation commands (uniq, cut, paste, sed, awk etc.), redirection, pipes, substitutions.
Shell scripting proper - Writing scripts that do little tasks. Useful in places like cron jobs, monitoring scripts, log analysers and such.
Power tools - Tools like netpbm, the imagemagick suite, netcat, ffmpeg, make etc. all of which perform tasks in their own domain amazingly well but which are unexploited due to their interfaces being unfamiliar.

I don’t know how long it will be. 2 or 3 days maybe. We’ll see. The primary references I have with me are the book I mentioned above, The advanced bash scripting guide, The UNIX power tools book, The Unix guru universe website and the commandlinefu website.

I’ll start working on this in mid August. It should be ready in a month - that’s mid September. I’ll make the course available hopefully in October. Suggestions for books and other materials are welcome as are any other bits of feedback or criticism.

Git tree and repo diagrams

2013-07-31T12:33:09+05:30

Update: The original version used the grit gem which is now deprecated. I’ve updated the scripts to use rugged which are the offical ruby bindings for libgit2.

Many people have been asking me for the scripts I use to generate diagrams of git repositories which I use during my trainings.

It’s a bunch of quick and dirty ruby scripts (my first ruby programs I think) which walk the repo and output dot programs that are then converted into png files and displayed using qiv.

There are two of them. There’s a showdag.rb which tries to draw the DAG of the current repo. It will draw the branch markers as well as the commits, trees and blobs. I’ve found it useful to describe how the DAG grows over time. There’s a second called showrepo.rb which is really just a graphical git log --graph kind of thing. I often use it to show people the difference between merging and rebasing and how these operations affect the DAG. There are also two shell functions which call the ruby scripts and then use dot to generate the images and then call qiv to display them. They’re run by executing showdag . or showrepo . at the top level of the repository you’re in.

Here’s the code.

Git workshop in July

2013-06-19T11:05:18+05:30

I’m conducting my git workshop in association with hasgeek. The planned dates are 27 and 28 of July 2013. If you missed the last one and want to be a part of this one, do sign up. The website is http://gitworkshop.hasgeek.com/.

See you all there!

New version of Hyde

2013-06-18T12:58:05+05:30

I’ve just put out a new version of my Emacs mode to edit Jekyll/Octopress blogs.

This has some simple code to embed images into posts. It doesn’t work perfectly but well enough for me to manage Calligraffiti.in. Fixes are welcome.

Bye bye Bangalore

2012-12-02T13:53:38+05:30

After 11 years of living in Bangalore, I’m finally ready to leave.

I came here in 2001 as an employee of Cisco and then moved through a 2 jobs before starting to freelance. I got involved in the local Python scene, helped kickstart Pycon India and met a lot of wonderful people in the process.

On a personal side, I got married and had two kids all while I was in Bangalore.

I met some great people here and have a lot of experiences to cherish. I never really imbibed any of the indigenous culture or language but I quite like the communities I was part of and don’t want to break any ties. I’m also going to make regular trips back to the city for at least the HasGeek events and maybe an occasional training. I will also still be alive on the mailing lists and websites that I’m already active on.

I’m moving back to Kerala and going to be working out of Calicut for a while. I’m hoping to do some work with the student communities at my alma mater and maybe try to kick start a tech. scene in Calicut. I also have some other projects in mind but don’t want to talk about them yet.

I think life’s going to change quite a bit. I’ve rented out a rather spacious house for approximately the same rent as the tiny flat in which I currently stay. Calicut is not as “happening” as Bangalore. It does have a coastline though and nice restaurants that serve seafood. I don’t think any tech. culture exists there and even if it did, it wouldn’t hold a candle to Bangalore. I hope to do more things non tech especially on the religious side - a part of me that I’ve been neglecting too much.

Let’s see how things turn out. As it stands, I’ll be out of here in the last week of December. Somewhere around the 25th.

So long and thanks for all the fish!

Calligraphy

2012-04-28T00:00:00+05:30

Everyone’s got to have a hobby and mine is Calligraphy. Like many people, I enjoy doodling but I’m not very gifted when it comes to drawing so I channel my energy into writing beautifully. I’m trying to develop this into a serious endeavour and this is a post describing how I do it.

My “gear”

I don’t remember how I got interested in this but the earliest calligraphy pen which I got was an Artline calligraphy marker. I must have had that before 1991 so that’s my “starting point”. They’re cheap and fun to use. I then bought a Sheaffer calligraphy fountain pen set which I still have but hardly use. I used to practise on and off but for the past few years, I’ve been seriously at it writing a page a day on the average. In 2010, I bought myself a set of Speedball nibs and a holder. I started writing with a dip pen and haven’t really looked back on fountain pens since then. The nib I use the most is a 2mm flat nib. In Speedball land, this is called a “C3” nib. It’s used for all the hands I can manage. Of late, I’ve been trying with wider nibs for larger work. I can’t handle the smaller ones yet withoug damaging the paper. I also have a number of flex nibs but lack the skill to write using a Spencerian or Copperplate hands. Someday perhaps. I managed to get my hands on one automatic pen which lets me mess with really large letters. It’s fun but I don’t use it very often. I have tried making bamboo pens but haven’t been very successful so I don’t use them. I also have a cup full of brushes which I practise with somewhat irregularly. A few flat ones to use with resist and to charge my pen when I write longer pieces and a few pointed ones including one rigger - this is particularly awesome when you’re trying to do fancy flourishes. I have a few cheap chisel edged felt tipped markers for quick and dirty stuff. I have 2 brush pens which I sometimes play with but they really require more skill to effectively use.

I carry around a Waterman expert fountain pen for daily use and a Platignum calligraphic fountain pen for doodling. My regular hand, like that of most “digital natives” is horrible but I still enjoy using the Waterman and don’t miss a chance to whip it out and scribble. I used to use Sheaffer Skrip ink but after my last bottle switched to Private reserve velvet black ink. It’s way darker and feels much better. For other colours, I make do with plain old poster colour or water colour since calligraphic gouache is next to impossible to find in India.

For calligraphy, I usually use 90gsm copier paper. It can handle some amount of soaking and is transparent enough to see rulings through. It’s not nice looking enough to do “finished” work but is great for practice. I use cartridge paper when I have to do stuff with paints and colour since it can handle water better. However, it’s rather thick and I don’t have a light table yet so ruling becomes a problem. I have a few other coloured papers which I use when I’m feeling fancy but the 90gsm bond is what I most use. I used to use a lighter 75gsm notebook to practice on and while they handle fountain pens reasonably well, they disintegrate with dip pens and clog the nibs. I bought some real calligraphic paper from an art store during my last trip to San Francisco but I’m too scared to waste it on real writing till I’m satisfied that I’m good enough to do so.

I’ve found Bhaskar art center in Chickpete and William Penn in Koramangala to have most of what I need especially the former. If you’re looking to find stuff, you should visit these places.

My setup

I don’t have a studio like real calligraphers do. I make do with a regular table and a chair with arms on which I rest a drawing board to get the angle right. I keep a sheet of paper stuck to the board on the right on which I drain off excess ink in my pen. I wrote a little Python script that generates PDFs with rulings of dimensions. This works better for me than manually ruling the page. It’s a tiresome job and one that really kills my interest. I’m sure that this violates some ancient rule but I’ve not been formally trained as calligrapher.

I use a wad of older paper taped together as a writing pad. It’s more comfortable than a real writing pad since it “gives” just a little under the nib and produces a nice effect. I hold the sheet on which I write to the ruling sheet using a bunch of paper clips.

My fonts

I’ve practised most with Edward Johnston’s foundational hand. It’s deliciously beautiful and not too hard to write. I also practise a standard uncial script. Of late, I’ve diversified a little into a little gothic. Italics and anything with a pointed nib still eludes me. I also don’t do flourishing very well.

My material

I enjoy reading poetry and most of the stuff I practise with are classical poems. I’m particularly partial to William Blake, S.T. Coleridge, Alfred Lord Tennyson and of course Shakespeare (especially the Sonnets).

My plans

I’m completely self taught never having met a real calligrapher in my life. This also means that I’ve never been judged by someone who knew what they were talking about. I’m not a big believer in autodidacticism especially in areas that are more subtle. I think a good teacher can really help a student along and imbues a certain deeper quality to the experience that can’t be had from a book. The “dead” in dead tree takes on a special meaning in this context. The cyberscribes page has a mailing list that runs “card exchange” events etc. which are nice to get a third party opinion for your work. I also managed to find a university that offers courses in calligraphy one of which I plan to take later this year.

I also want to actually make something and sell it this year or maybe early next year. I don’t plan to make this a career but when someone is willing to pay for the privilege of owning your work, it does say something about all the effort put into it.

Letters are symbols that turn matter into spirit. - Alphonse de Lamartine

Emacsmovies.org

2012-03-24T00:00:00+05:30

I came across a stack overflow question asking for good Emacs screencasts and couldn’t find any on the web. There is a website for Vim screencasts so I thought I’d try my hand at making one for Emacs.

Thus was born emacsmovies.org. I registered the domain at the end of last year in October and have been spending time sporadically developing the tools that I thought I needed to do this.

I spoke to Zed Shaw about his learn code the hard way series of books and screencasts and he suggested that I write a book to go along with the screencasts.

I’ve finally launched the website and the first screencast so if you’re interested in trying out the venerable old editor or an expert who wants to help out with the screencasts, check out http://emacsmovies.org and send me some feedback at noufal@emacsmovies.org.

Moving topic branches in git

2012-01-09T00:00:00+05:30

I hit an interesting situation recent which I’ll try to recreate here in the hopes that others might benefit.

Imagine that we cut a branch called tp1 from our master and did some work on that branch. At the same time, some other work was done and pushed onto master. Our current DAG looks ike this.

![Initial scenario] (/images/git/rebase-0.png)

Now, while on the topic branch, we do some work and commit them to create this. I’ve used tb2 in the commit messages so that you can identify them easily.

![Two commits on topic branch 1] (/images/git/rebase-1.png)

Now, we realise that these two commits should be on a different topic branch which should be based off master. We look at the logs (while on the topic branch) and see this

(ol)[1] noufal@sanitarium% git log --pretty=oneline 
32abaeefed66ee5165b7e7edddb5604ecdb2fa64 tb2 5
4541e86113cb4ceaf654ed74abe85d477950ed2f tb2 4
84d1f43da8b6d63592627aca6ecd4c7e3df29600 tb1 3
7956bc382d183674208f1b04c3e62d12806c517c tb1 2
a6d74c2cb1e0b7ba456626898fd63a3c0d1fc863 tb1 1
c60c4c22f2eb8d3c7465ad3b138a7d4a3aee3c5f master 3
0c99224a32fcc61e123f94837d971456a2659bb0 master 2
c3522d06410e2a37444ed5759f7a6f4c7b67ad17 master 1

We want 8a94e9cf00aa2df580d7beed8efb52557bdfce16 to be based off the current tip of the master branch

(ol)[1] noufal@sanitarium% git log --pretty=oneline master
755b7823acfa30f60ade5ad6c5c575116c38fe92 master 5
95da533bb52e60497ba5312dcd0540c1bd32db96 master 4
c60c4c22f2eb8d3c7465ad3b138a7d4a3aee3c5f master 3
0c99224a32fcc61e123f94837d971456a2659bb0 master 2
c3522d06410e2a37444ed5759f7a6f4c7b67ad17 master 1

In other words, off commit 755b7823acfa30f60ade5ad6c5c575116c38fe92. How do we do this?

We first create a the new topic branch (lets call it tp2). Our DAG now looks like this

![New topic branch created] (/images/git/rebase-2.png)

And then we rebase this onto the master branch using the 84d1f43da8b6d63592627aca6ecd4c7e3df29600 as the branching point like so git rebase --onto master 84d1f43da8b6d63592627aca6ecd4c7e3df29600 tp2.

(ol)[1] noufal@sanitarium% git rebase --onto master 84d1f43da8b6d63592627aca6ecd4c7e3df29600 tp2
First, rewinding head to replay your work on top of it...
Applying: tb2 4
Applying: tb2 5

Once this is done, we get

![Rebasing is done] (/images/git/rebase-3.png)

And then we switch back to the tp1 branch and reset it to discard the two extra commits that are now tracked by tp2.

noufal@sanitarium% git co tp1
Switched to branch 'tp1'
noufal@sanitarium% git reset --hard 84d1f43da8b6d63592627aca6ecd4c7e3df29600
HEAD is now at 84d1f43 tb1 3

We get this

![Final product] (/images/git/rebase-4.png)

Homeschooling in India (part 3)

2012-01-02T00:00:00+05:30

I’ve talked about the my problems with traditional schooling and some alternate methods in previous posts. In this final part, I’ll talk about my implementation plans. My daughter is not yet four and I’m factoring that into my plans here. Be warned though that these are my plans and might not be suitable for everyone. Also, this is more of a statement rather than an opening point for discussions and this will reflect in my replies to comments in any.

Overall plans

No education system is “unbiassed”. As far as I can tell, everyone has an agenda and this makes sense. Every teacher has a world view, opinions, facts and other things which they try to pass on to a child.

I bemoan the lack of emphasis these days on the humanities, on language and on the arts. Skills like writing, talking etc. are being pushed back in schools with increasing emphasis on technology and the need to get into a “good college”. So, this is something I want to pay some attention to. I want to emphasise things like poetry, history, literature and languages. I foresee some trouble but there are some groups in Bangalore that might be able to help. I’ll talk about these later.

Dorothy Sayers’ essay “The lost tools of learning” is a must read for anyone interested in education these days. It’s my starting point. In it, she talks about the trivium which consists of the three “subjects” of grammar, logic and rhetoric. Primary education consists of teaching kids these skills and that’s my plan as well. This is medieval and there is a modern version of the system in the book The Well trained mind which is my handbook for this trip. It’s very detailed and goes into the ages at which different subjects are to be taught and how. It recommends books and other resources which can be used etc. I’ve not year read the whole thing but I’ve browsed through a copy and am quite satisfied with the overall content.

The second thing which I want to emphasise is religion. My own world view is religious and conservative (although I don’t like pigeonholing ideas in this fashion) and I don’t want to compromise religious education. This is not a fashionable thing to say in “modern” liberal circles but it’s my plan anyway. I’m not going to “justify” this here. That’s not a path I want to go down. Operational religious virtues are best taught by example so the household is the first thing here. Children learn by imitation so that covers most of this. Your kids are going to be as religious as you are. More formal religious theology, requires a specialist to teach so I plan to get a home tutor who will teach these subjects a few times a week when my daughter gets older. This is the way I was taught. There are also a few http://en.wikipedia.org/wiki/Madrasahs near my house so maybe that’s an option too. I’m undecided about the details but there’s still time for this.

It’s too early to have a clear view of the entire path but these are my overall aims.

Timeline

I don’t really plan to do any “formal” schooling till about 5 years of age. I’m not a big fan of the preschool, nursery, kindergarten and then first grade trend in cities these days. I do spend time playing around with my daughter though. She can count a little and recite the alphabet. She can handle a pair of scissors to cut out stuff and can generally keep herself busy with a few pieces of paper, a stick of glue, some colours and such things.

I also spend quite a bit of time reading stories to her and discussing them. She picks up vocabulary quite quickly and any stories which have a sort of rhyme (e.g. some of the Dr. Seuss books), she memorises automatically.

One thing I’m deadly against is the television (and by extension the computer). It has a way of eclipsing everything else that the world can throw at you. This is perhaps a tad controversial but I’m generally satisfied with not having a TV in the house and not putting my daughter in front of youtube to watch standard Disney cartoons. We tried this earlier (when she was around 2.5) and that’s all she wanted. The books, the toys, the outside, the pretend games all stopped. Not worth and it The Well Trained Mind is in agreement with me about this.

At 4.5 or so, I plan to start the work with the book mentioned above and to go on with that. It will probably not be possible to do it entirely according to their prescriptions but it is going to be my guidebook.

At about 6, I plan to start formal religious education too.

I expect the subjects to get more and more complex as my children grow older and so after a period of time, I’ll probably take the help of private tutors for specialised subjects and to prepare them for examinations (which are sadly compulsory these days if you want to do anything).

It’s too early to have a crystal clear plan right now but I’ll probably blog about this as time goes by.

Socialisation

This is often the first question that people ask when it comes to home schooling. How will she learn to socialise? How will she get friends? The question is not really a question. These are two separate issues.

The first has been answered repeatedly before.

The basic point is that have a child sit without any real guidance in a class of 35 to 40 children of like age is not going to help them socialise at all. They going to imitate their peers and one rotten kid is enough to spoil the rest. The “free for all” jungle atmosphere of the school teaches a certain kind of socialisation that might not always be the best kind.

Real socialisation comes from letting kids interact with a diverse group of people. As a simple example, my own daughter had a “conversation” with a random lady she met in a local lending library about which books were bad and which were good. It’s not that hard and she definitely didn’t need school to teach her this.

It’s important to get them into such groups on a regular basis. To let them speak to different kids and older people. It’s also good to curate their company a little. While it’s important to experience “the real world”, no one wants their kids to learn behaviour from some ill mannered ruffian. You expose them to people with good personalities, their natural ability to imitate will do the rest. This of course, puts a lot of responsibility on the parent but no one said that raising kids was easy.

The second question about friends is more serious. The parents need to find places where the children can play with peers. Local parks, meetups with other homeschoolers, gated communities, neighbours, evening classes for things like art, martial arts, music etc. There are lots of avenues. I myself plan to spend one or two weeks every two months in my native place where my children can actuall spend time with their grandparents and their cousins.

Another point which can rub people the wrong way is that I think it’s good idea to have more than one child. It’s unfashionable these days but I’ve met homeschooling families with a large number of kids and they were delightful to talk to. The younger ones were mature in a certain nice way and not spoilt brats. The older ones too. This is decision that requires some consideration by parents but I think it’s something worth thinking about.

Support groups

Bangalore seems to be, atleast in my research, the ideal place in India to homeschool kids. The Alternative Education in India website contains a lot of information on groups and answers a lot of FAQs. It also covers the question of certifications which I’m not going to repeat here. There is an community of home schoolers on http://indiahomeschoolers.ning.com/ which is a social networking site. Announcements are usually made over there regarding events and meetups. A mailing list called alt-ed-india is also quite active with a large number of homeschoolers on it. Great place to ask questions.

Most home schooling families I’ve lean towards the liberal and secular. However, there are conservative and more religious groups out there too. The muslimhsers group has been very helpful to me. The Kinza Academy too is a useful resource although a tad expensive if you’re in India.

There are also tons of other resources on the web. Like most things on the net, this is a “drinking from a firehose” problem. The above are the sites and lists which I found useful personally. The book I’ve mentioned is an excellent buy too if you’re planning this.

Why not to homeschool?

Finally, if you are planning to do this, here are some reasons not to just to get a proper perspective.

Competition. The dog eat dog kind of competitiveness you need to get far in some areas comes I think better from the conventional system.
Time. Doing this requires a massive time investment. One parent pretty much needs to do it full time and the other part time. If you don’t have that kind of time and energy available, it might not be wise.
Risk. Doing anything outside the mainstream is risky. Quitting your job and starting your own company is an example that many of the people reading this blog will be more familiar with. This is one such activity and has risks.

Finally

That’s my plan. My daughter is a little under four years old now and I’m spending time in the evenings making plans for when she’s older. I also take the time to visit people who I meet on the mailing lists and go to the meetups. Let’s see where this goes.

Tomobi

2011-12-18T00:00:00+05:30

I own an Amazon Kindle and while I don’t buy ebooks online, I do use the device as a replacement for paper. Before I had it, I used to take hard copies of longish articles so that I could read them while offline. The Kindle is a great replacement. You can covert almost any format into something it can support using calibre and then read it at your pleasure.

There are many articles on the net which I’d like to read but don’t have the time to do so. It would be nice to have a Chrome extension that would allow me to convert the web page I was viewing into a .mobi file which I could then drop onto the Kindle and read it. There is an extension that does exactly this called “send to kindle”. I’ve used it and it works nicely. Due to Javascript restrictions on executing commands on the local machine, this uses a service hosted on Google AppEngine. As far as I can tell, it receives the page URL, fetches it, does some processing and then sends the processed document to youremail@kindle.com which will convert it into the Amazon proprietary .azw format and sync it to your Kindle when you get online.

This works fine and has been in the market for quite a while now which makes it quite robust. If you know me personally, you know that I’m a little paranoid about giving out stuff to third parties. The approach above requires that Amazon and the klip website (which is the one where the app resides) know about every page I visit and convert. Amazon even has a history of documents you’ve converted. Not nice.

So, I rolled out my own. Tomobi is a chrome extension to send the current page URL via HTTP to a web app that converts it into a .mobi file and keeps it in a directory. It currently just works for me and is not tinker free to get up and running. Details are on the github page.

Let me know if you like it.

This project was called mobify but I renamed it since that’s the name of a company.

Showkeys

2011-11-30T00:00:00+05:30

As part of a larger plan to do some screencasts, I wrote a simple program to display keystrokes called showkeys. It is similar to key-mon but has less features and a different design that I prefer.

key-mon is more modern looking and tracks mouse as well as keyboard input. It’s prettier, is themable and on the overall, something that fits into a modern desktop better than showkeys. Showkeys has it’s own advantages. These are

It doesn’t rely on gtk.
It outputs stuff to the screen directly using libXosd and so doesn’t require a separate window. This makes is work better with tiling window managers like Xmonad (which I use).
It displays a history of keystrokes which is useful while doing a screencast. Key-mon does this but in an ugly way and doesn’t remember modifiers.

It’s been a while since I coded in C so it’s probably rusty. The program uses the X record extension to capture keystrokes from the X server and then puts them on the screen using libXosd.

Licensed under the GPLv3. Source at https://github.com/nibrahim/showkeys. Feedback welcome.

Git training post mortem

2011-10-25T00:00:00+05:30

These are some thoughts on the recent git training which I conducted here in Bangalore.

I’ve done several corporate trainings before and spoken at local events but I’ve never conducted something like this before.

Organisation

I’ll first talk about the organisation. The dates were bad. I didn’t consider Diwali. I had other plans for the last week of October and early November and after that is DroidCon so I just dropped it on October 22rd and 23rd. A lot of people went back home for the holidays and so the attendance dropped. Quite a few people told me explicitly that this was a problem.

The venue was overpriced for something like this. I don’t want to give out actual numbers but this place is designed for more “corporate” style events and has “corporate” prices which someone like me can’t afford if I want to keep doing this on a regular basis. However, no one really complained about my prices. I explicitly asked about this in the feedback forms and no one really complained about the prices. I asked some third parties and they generally said that the price was low. A typical open source style conference here charges between 1000 and 2000 INR. I charged 5500 for the training and with the low number of people and personal attention, I guess it was worth it for the participants.

I created the slides using landslide and added a macro to display command output on the slides. I kept paper notes for the complex parts like discussions of the merge algorithms and some more involved examples. This was useful. I tried to use the whiteboard which the venue manager provided but it was too unstable and I couldn’t really use it.

Content

The morning of the first day was mostly introductory stuff. While I generally feel that git is easier understood from the inside out, you’ve got to start somewhere. Create a few repositories, commit files, view logs, diffs etc. The usual version control stuff. After that, I spent some time talking about the index which is something that git users have to be aware of.

Day 1 morning

After this we dived into git objects. Blobs, trees and commits. I went over these trying to drive home the idea that git is internally organised like a file system. We used the cat-file command to dig into the various objects and then went into the .git directory to actually see what’s happening. We tried running some commands to see how the repository was changing so that people actually understand what’s going on when they do stuff like git add etc. I used a few scripts using grit which outputs files that can be understood by dot. These draw the commits over time and the various heads which help people understand how commits are made. I also had some scripts that actually draw the DAG as it grows over time and pictures like this help people understand how this takes place. We went on with this for quite some time and the morning session of the first day got over.

Day 1 afternoon

Then we started the business of branching and merging. I tried to emphasise the lightweigt nature of git’s branches and the fact that they’re private unless published. I tried to spend an equal amount of time on the actual implementation of graphs as well as their uses. We discussed merging, fast forward and true. We supplemented all this with examples and exercises but I’ll come to that later. We mentioned rebasing and then stopped.

Day 2 morning

We spent about 45 minutes on a recap of what happened the previous day. After that, we dived into the business of rebasing and after that spent time talking about merging. We sidetracked a little talking about the diffing and merging algorithms and I tried to explain why merging works better in git than it does in subversion. We then talked a little about how git does merging (including the internal implementation). After this, we started with collaboration. We just mentioned remotes, bare repositories, and tracking branches and then stopped for lunch.

Day 2 afternoon

Here, we paired up and spent time actually pushing and pulling changes to a remote. I set up a bunch of bare repos on my machine and created an ad hoc wireless lan which everyone used to collaborate. I tried to cover details of different workflow styles, refspecs, an important gotcha of pushing discussed here and when everyone was satisfied, I wrapped up by discussing a bunch of useful leftover commands like bisect, archive, cherry-pick etc.

Things to be improved

I didn’t like the way I created the exercises. I think they needed to be highly stuctured (no ad hoc stuff at all). Also, I need to seriously rethink them. I didn’t have a “list of commands” section. I didn’t think it was a good idea and introduced commands as we needed them but I wonder now whether that was the right way to do it. I didn’t give any homework. One of the participants suggested that I should have. I didn’t cover any of the bridges (git-svn etc.).

I should have advertised a little more and given more gap between the announcement and the day of the event. I should have booked a cheaper venue. I should have chosen the dates more carefully.

All things considered, I think it went well. If anyone here is interested in a repeat, I’d be happy to do one in December or so. Please holler. :)

Git training

2011-10-07T00:00:00+05:30

I am offering a training on the git version control system on the 22nd and 23rd of October 2011 at The Millenia. For details, please visit the training page at http://nibrahim.net.in/trainings/git.html.

Please spread the word.

Homeschooling in India (part 2)

2011-10-03T00:00:00+05:30

Part 1 of this article discussed my problems with the current education system. Why I don’t like it and why I don’t think it makes sense to put ones children through the ordeal.

In this part, I hope to discuss alternatives in the Indian context. In the third part, I hope to talk about strategies and my own plans on how to implement them for my children.

Alternative education

No one really wants anything bad for their kids. All parents will attest to this. They put their kids in reputed schools, they buy them the best of things and generally try to provide something good for their children. These days where families are smaller and more nuclear, children are being spent on even more than in the old days of joint families. One of the most important things that parents invest in for the sake of their children is their education. They want their children to be equipped with mental and academic skills that will ensure a well paying job and success (for some definition of the term). This is all well and good.

The problem is that unlike maybe half a century ago, sending your kids to a “good school” is neither sufficient nor necessary for these things. I can speak more authoritatively about the IT profession than about others since that’s my own area. Smaller companies have been started up locally, new markets have opened up and the internet and other technological advances have empowered people to do things by themselves. This article does a good job of explaining the job market and makes some predictions as well. On the contrary, like I discussed in the last article, there are downsides which might disadvantage them.

What are the alternatives? Well, the first thing is to accept that there are other ways to educate our children. The methods available and approaches are varied and diverse as the kids themselves and just like it’s impossible to “outsource” parenting, it’s impossible to disconnect ourselves completely from the education of our children. If we take an active interest and role in this, new options open up which I’ll try to discuss below.

A couple of provisos before I start. Firstly, “Mainstream education” is clearly defined. There are classes, grades, syllabi etc. “Alternate education” on the other hand is hard to neatly pigeonhole so this list is not authoritatively categorised. Secondly, some of these are mainstream but I’ve chosen to list them here as alternative simply because they can contain some of the damage of the regular system and because they’re fast becoming alternative approaches. Thirdly, these categories are not mutually exclusive. People can do more than one at a time. The divisions are just for didactic convenience. I’ve tried to list them in increasing of order of “away from mainstream education”. Finally, I don’t want to be misconstrued as blaming other parents who have chosen something different from what I plan for their kids. Like I said earlier, no parent wants something bad for their children.

Being involved with a school

Many parents “outsource” the education of their children to a school. They don’t check what’s happening and have no idea what’s going on there. Children are incredibly adept learners and a conventional school atmosphere is a place where they can pick up a lot of things, not all necessarily healthy. A parent sending her child to a school should be actively involved with the school. She should know the childs teachers and what he’s learning on a day to day basis. School is mostly a machine for disseminating government approved factoids. It is the responsibility of the parent to make sure that the larger framework of education is taken care of.

Children (especially younger ones) learn by imitation and in a large classroom where the teacher is a remote person in one corner of the room, children learn by imitating their peers all of whom might not be very good role models. Other issues like abusive or unqualified teachers who actually harass and hurt children should be dealt with. Harsh punishments for simple “offences” like, for example, eating during class hours shouldn’t be taken sitting down by any serious parent. Parents should stand up for their kids. At their ages, the only person they can truly depend on are their parents and they should feel confident in that. The “teacher” of today is not the “guru” of matha pitha guru devam. They’re paid employees who work for an institution that charges a truck load of money to educate children. Parents should take them to task for things which they find are harmful to the development of their children.

On the more positive side, parents should understand that the syllabus as prescribed is not sacrosanct and if the child shows interest in other subjects, that should be encouraged. They shouldn’t be horrified at the idea of (for example) taking a few days, a month or even a year off school to do something else constructive. Many of the schools rules are arbitrary and while they may be enforced in a draconian way inside the campus, the parent and child should understand that it’s not really an ironclad law and treat it that way.

The child should find an older mentor who shows them the wonders of the world in their parents. This requires a considerable amount of effort but no one said that parenting was easy.

The basic idea here is that the parent should engage with the school and child and work daily to offset the harmful effects of mainstream schooling.

Alternate schools

A lot of schools have come up which range from ones that employ a different system of education (e.g. the Montessori method) to ones that are much more radical in what and how they teach (e.g. the Centre for learning in Bangalore).

Like I said earlier, “alternate school” doesn’t capture the myriad of options here. The basic idea is to find a community or a group of like minded students and parents and to travel along with them. Ofttimes parents are worried about external factors like examinations. Being in a group helps to answer such problems. Options like Montessori schools allow parents to liberate their children from some of the harmful effects of conventional schooling but still remain somewhat in the mainstream.

This is not a “one size fits all” solution and it’s best for parents to actually check the schools they’re considering to see whether they align with their philosophies of education before they send their children to them.

Home schooling

A third option which many parents adopt is to “school” their children by themselves at home. Children are taught by the parents and often write standardised exams as private candidates. They then “integrate” back into the mainstream at some later period or simply stay outside it all their lives.

The methodology varies from parent to parent. Some prefer a regular “study time” system where their kids study subjects in a manner similar to a conventional school. Others prefer a more relaxed approach. The basic idea is that the parent takes over the responsibility of educating their child. Some parents prefer to follow an existing syllabus and textbooks. Others prefer to create their own. This is often done by religious people to make sure that their religious teachings are properly taught to their children. Contrary to what Richard Dawkins says in the God delusion, I don’t consider teaching religion to children a form of child abuse but that’s a topic for another day. Sometimes, especially with older children, parents take help from a professional tutor to teach a subject that they’re not familiar with. Most times, a small group of parents collaborate on something like this and teach the children together creating the environment of a small one room school.

This requires an enormous amount of effort and at least one parent who dedicates a fixed amount of his or her time to teach the children. The approach has disadvantages and advantages which I’ll discuss in the third part.

Grading, exams etc. are all upto the discretion of the parent and vary from person to person.

Unschooling

Possibly a more radical approach, unschooling is an “approach” (for want of a better term) which recognises the innate learning ability in a child and lets them learn based on real life experiences. There are no formal “classes” or “study hours” but the parents expose the child to various situations (e.g. household responsibilities, play etc.) which gives them opportunities to learn skills and acquire knowledge in whatever way and at whatever pace they are comfortable with.

Understandably, this scares a lot of parents. What about exams? What about joining college? Will they learn what’s necessary to handle themselves in real life? The people who often go for this are those who don’t particularly care for walking on the trodden path. They often don’t look for accolades like “certificates” and “awards” but try to lead their lives by a personal code.

My own personal preference lies somewhere in between the last two items mentioned above. I have, based on my own background and experiences, a list of subjects that I think the children should learn and learn well. I have others that are desirable but not that important. My plan is to semi structure the important ones on my list and prioritise those and to provide enough space for them to explore. I will elaborate on these ideas and how I plan to implement them in the next part.

Homeschooling in India (Part 1)

2011-09-06T00:00:00+05:30

This is a dump of all my thoughts and resources I’ve been able to gather about homeschooling/unschooling children in India. I’ll try to make this structured but no promises. Also, much of this is opinion based on my worldview e.g. I’m a religiously orthodox Muslim and this was considerably affected my views on the whole matter. I welcome comments on points where I’m factually incorrect and also on opinions which others disagree with. None of this is meant to be judgemental but I do have strong negative feelings about the attitudes some parents have towards their kids and the current system of education which our children are put through and so it’s possible that it will be misconstrued as finger pointing.

This is going to be a multi part article. This is the first part and talks about my own educational background and what prompted me to think about this whole business.

Background

I first put my daughter (who is now 3.5 years old) into a small school in Bangalore a stones throw away from where I stay when she was 2. This was for two to three hours a day mostly to play with kids of the same age. Given the fact that we live in Bangalore, it’s hard to get kids to play with unlike in a village. As far as I can tell, this helped her overcome her aversion to strangers and excessive attachment to her parents. I had seen parents running around like chickens trying to get application forms from “top schools” and didn’t want to join the race. I actually bought application forms for a school near where I stay (Presidency, Kasturi Nagar) and then decided to drop it and let my daughter continue going to her current school when she was 3 (which is when kids “graduate” from play school to nursery). This is the situation right now and at the end of this academic year, I have to “enroll” my daughter into kindergarten somewhere.

A little background about my own inclinations might be relevant here. I was brought up in Oman. I studied at Indian School, Muscat from kingerdarten to ninth grade after which I had to come back to India. I finished off my school at Ansar English School, Perumpilavu and then did my engineering at NIT, Calicut. The first school where I studied was quite good. There was just one school for Indians in the country and so it was well funded and could offer good facilities to students. The second school was my first brush with the disaster than education is in India. My college was decent since it was extremely relaxed with rules and I had the freedom and resources to explore whatever I wanted.

My parents were never too hung up on us being the topper in class or getting high grades or getting into the best colleges. They were never strict with homework and attendance either. I fondly remember a couple of episodes when my father stood up for me against some of my “teachers” when they went overboard with their insistence on completing homework etc. They were also non insistent on a particular career path for me. Academic inflation wasn’t was bad as it was then as it is now and a professional degree was a reasonable guarantee that you’d get a well paying job. This was something I got from them but there was no “my son has to be an engineer” attitude which is all to prevalent these days. Another thing was that my parents used to spend a lot of time with me and my work. I remember studying about the internal combustion engine the first time and my father promptly got an old unused generator from his workplace along with a cool spanner set and we took it apart to study and see the innards. Much more 3D than a textbook. There was also an undercurrent in the house which was anti “marks for marks sake”. All of these things are part of my world view and the subject of this post.

There were downsides too. All my hindi teachers from the fourth to the ninth grade were from the northern parts of India. They all had a certain contempt towards the south Indians, none of them could speak english coherently (which was supposed to be the medium of instruction) and I have, on multiple occasions, been on the receiving end of nasty remarks from them that were pointed at my faith. I had a natural inclination towards science (especially chemistry) and maths and had a very low opinion of the humanities and the need to study them. Something I’ve come to regret and am still trying to remedy.

Views on education

I have a dim view of the quality of education that’s available to our children and in this section, I’ll try to summarise why.

Content

I think I can summarise my views on education saying that over time we’re slowly replacing “education” with “vocational training”.

I define the former as something which is designed to take the rough uncut gem that is a child and polish her into a diamond that the family, society and mankind can be proud of regardless of what she applies herself to. The idea is to teach the child skills which are necessary to interact with the people she’s going to meet in her daily life, to teach her to think clearly and express her ideas well, to teach her to write and most importantly, to teach her to learn. In a world where the pace of changes in increasing rapidly, the ability to assimiliate new concepts quickly is becoming supremely important. An educated child might not be able to work out complex mathematical sums in her head. She might not be many of the things that “smart kids” are these days but she’d have a certain grip on reality that others don’t. She’d have an area of interest that she pursues with a passion that most others don’t.

The latter is “training” on specific skills that are designed to make you employable. This is not a bad thing in itself but it’s not a substitute for education. Training is a quick thing you can get at a course if you need it for a specific purpose. If the education is the ability to write, then training is the pen with which you do it.

This general trend has cost us a lot. Things like public speaking, writing, art, history, poetry don’t really count for much when the aim of a childs life (and the life which most parents want their children to live) is to get into IIT. Schools and the board have responded to this by altering the syllabi to emphasise maths and science much more than the arts and other softer skills producing people who are little more than cogs in the system. The problem with such people is that they have no idea of who they are in the larger society around them. They have no idea how to respond to the scams and opportunities that the world is throwing at them. They’re afraid of anything except the beaten path (which is sadly no longer a guarantee of a financial success). Their self worths are often derived from external criteria like the opinions of people in authority, grades, certificates etc. They’ve been taught that “learning” is over once they’ve left college and now it’s “time” to “work”.

Quality of teachers/teaching

Structured formal education has become a huge business and like many things that have been tainted by the promise of great profits, the original aims are lost. A school near my house charges 93000 INR as annual fees for kindergarten. The silliness of the situation would make me laugh if I didn’t know how parents are eager to pay this kind of money to give their children “good education”.

Schools often advertise their marble lined floors and air conditioned classrooms rather than the quality of their teachers. Many schools have designer uniforms that cost a bomb. I know a kindergartener who wears Adidas shoes as part of his uniform.

The quote “those who can, do. Those who can’t, teach” is very true these days. Many English teachers can’t spell properly and teach Shakespeare even though they haven’t read a single one of his works. This is something I know personally and not just hearsay. This is especially a malaise when it comes to teaching humanities. They’re supremely important and it’s not possible to impart love of poetry to a child if one doesn’t have it already. The teacher in question is genuinely incapable of teaching the subject to the child and no one seems to care.

Parent involvement

“You are the bows from which your children as living arrows are sent forth.” said Khalil Gibran to parents. School is a form of outsourcing education as I see it and I think it’s harmful. It’s even more painful in households where both parents work till late in the evening and most of the childrearing is done by a maid. Parents are not a part of the childs development in many of these places and this really damages them. Given how high living costs are, this is a sad reality that many parents have to deal with but the long term costs are, in my opnion, considerable.

I think parents should be deeply involved with their childrens education. It should be a priority in their lives. They should have an active say in everything from how to what and when.

Schools, of course, state in their prospectuses that they believe that parents are a part of the childs educational process. I couldn’t agree more. However, this is actually translated into “our teachers are too lazy to spend time teaching the hard stuff and will send it all home as homework”. Children sit late into the night to get their homework done and don’t have much time for anything else.

The system

I don’t like the classroom. Think about it. There’s a teacher standing far away saying something and you’re in room of 35 to 40 people all of the same age. You hear what’s said, take down some things in a book and switch contexts every hour or so. It’s a criminal offense to talk, eat, go to the toilet or do anything of the sort. There are right answers and wrong answers and nothing in between. I don’t like the idea and I think it does a lot of damage to children.

It’s also very rigid. There are a few subjects which are compulsorily taught. There’s very limited freedom to explore different avenues and even if you have it, you have to do so in one hour intervals.

Agendas

A state run education system where parents send their children to can be dangerous. Incidents like this where a minister edited the books to paint him and his party in a more positive light have far reaching consequences that our narrow minded politicians cannot see. Another problem is the trend of saffronisation of education. All of these are because of the disconnect between the people who decide what needs to be taught and the people who are educated.

Conclusion

So, I’m more or less convinced that the current educational system is not really an advantage for children. I think responsible parents should understand the tradeoffs that are made when they send their kids to one of these schools and make decisions. In the next part, I’ll ty to outline what I plan to do and the resources which I’m planning to use.

My org mode PIM setup

2011-07-17T00:00:00+05:30

This post describes the way in which I’ve set up org-mode to track my work and life in general. It’s broken down into a couple of “sections” and I’ve linked to the relevant parts of my config on my github account. I’ve tried to be a little verbose to explain how things work so that you can try this out if you’re inclined. If you hit any snags, mail me and I’ll try to help you out.

All this is stuff that I use on a daily basis. I’m not as organised as I like to be but I mostly practice what I’m about to preach in this post. This covers only the PIM part of my setup. I also use org for some authoring which I haven’t discussed here.

Also, my emacs configuration files are in need of cleanup. They’re functional as far as config files go thinking of them as “programs” makes me retch.

History

I’ve messed with quite a few ways of keeping myself “organised”. I took a course on the Franklin Covey system and for a while, kept my weekly and monthly sheets up to date. I tried a bunch of programs that helped me track time, keep notes and organise myself in general. None of them really worked and in retrospect, there were a few reasons

They were hard to “Get back into” - Everyone makes mistakes and stumbles when trying out a new system. If it’s hard to reset and get back into the flow when you mess up, the inertia is hard to beat and you end up ditching the system altogether.
They were hard to tie up into other spheres of my life - Like most people, I get todo items, information etc. from various source (e.g. email, phone calls, personal conversations etc.). While your system needn’t be omniscient enough to track it all in a single place, it should be powerful enough to do most of what you need so that your mental strain is reduced. A smaller variant of this problem is the inability of most software systems to integrate with the rest of your work.
I’m lazy, disorganised and a disgrace to humanity - No point denying this. But once I realised and accepted it, it’s given me a set of constraints to work under.

I tried GTD but the “process” doesn’t work for me (although I will admit however that it was better than the Franklin Covey strategies). It did however solidify my ideas of what I wanted out of a task tracking system. I’ll try to summarise below

One place for to look for things. I don’t want to have to wade through sheets of paper and jump through programs just to figure out what I have to do. Did I pay this months rent? Have I paid my utility bills? What work do I have to do today? What was that website with the colour pickers that I used last week? All should be in a single place.
Should be quick. If someone tells me something that’s I have to work on (later or immediately), I need a way of grabbing it immediately before my attention deficit addled brain discards it. If software, I want the kind of thing where I can develop muscle memory and forget about the mechanics of doing things. I especially value this given my email reading experiences with mutt which is still in my opinion the fastest way to read and reply to emails.
Should never get lost. I don’t want to end up doing the “Did you ask me to do that? Damn. I was sure that I had starred that email” dance.
Should be “fun”. Being a tinkerer, I like stuff that’s fun to do and with a few knobs and wheels. If they’re missing, I’ll probably get bored and give up.
Flexible. If I want to, from tomorrow say, start tracking habits, my system should be flexible enough to accommodate that.

So I flirted with the idea of writing some command line scripts to do this kind of thing and found todo.txt which “worked” but didn’t really cut it for me.

My suite of “productivity tools” consisted of something called “gtimer” (I think) which was a simply app to keep track of time spent on projects, EmacsWiki (the precursor to Emacs muse) and TomBoy to keep notes (thereby violating the one place only rule), a notebook and pen to track todo items and a couple of ad hoc thing like starred emails etc. to keep track of “projects”. No, it didn’t work. Yes, I wanted something better.

I had used the Emacs outline mode and hated it but then came across Carsten Dominiks excellent Google tech talk on org mode. I tried it and it was love at first sight. Never looked back since.

Now, let’s get onto the meat of the matter.

Current setup

My org mode files are stored in a git repository that I keep on my private account. I don’t put it up on github since it contains a lot of personal information and I don’t want anyone to clone it and send me a pull request.

The layout of the files is as follows

.
|-- non-org
|   |-- bank-statements
|   `-- invoices
|
|-- official
|
|-- pycon
|   |-- 2009
|   |-- 2010
|   |   |-- IPSS
|   |   |-- logos
|   |   |-- presentations
|   |   `-- pycon docs
|   `-- 2011
|-- specs
|-- training
|   |-- business
|   |-- materials
|    `-- outlines
|
|-- donotdo.org
|-- habits.org
|-- ideas.org
|-- projects.org
|-- projects.org_archive
|-- reference.org
|-- refile.org
|-- scratch.org
`-- toread.org

There are a few parts that are not heavily accessed or modified and they have accumulated cruft.

The non-org directory contains stuff like exported PDFs, .odt files and stuff which contain invoices etc. Mostly binaries. The official directory contains agreements and other formal materials. It should probably be a sub directory of the non-org one. pycon is a top level directory mainly because it was one of my “big” projects and I was experimenting with org-mode features. specs contain org-mode files which are outlines of projects I’m thinking about. It serves as my notebook and doesn’t have anything in that I track. A digital pocketbook of sorts. training contains all the stuff that I use while taking and planning trainings. Course materials (e.g. this was written as an org file), outlines which businesses want, different versions of my training profile etc. donotdo.org is a file that contains a list of harsh lessons that I’ve learnt over time. It might seem silly to put them in a text file and forget them but I find the act of writing it down useful. habits.org is a file I’m trying to use to track a few habits I’m working on. It’s not yet “production” ready so you can ignore it. ideas.org is a bucket where I dump semi-detailed ideas which occur to me. Sort of a cold freezer to search for things to do when I have time. Not necessarily computer related. projects.org is my main file around which my life revolves. Most of this article will be discussing that. There’s an _archive version of the same file which I use to keep older subtrees. reference.org contains phone numbers, links, addresses, things which I would forget if I didn’t write them down. refile.org is my default capture target (I’ll discuss this later). scratch.org is a sandbox file I use to pimp features to people when I’m in front of them and for writing out small things which I can export to different formats. toread.org is a file that stores URLs, names of books, links to videos, movies etc. which I plan to look at. It should really become a section in projects but like I said, the layout needs work.

The first thing I’ll discuss is the refile.org file which is where all tasks land up before they are moved off into the right places. The process of putting them there is something worth mentioning and it addresses point 2 in my list of requirements.

Data sources

Things come into my life via. email, real life (phone + meeting people), the web and while coding (Ah! I should do this, fix this later). This might seem like an oversimplification but it works.

The intention here is to capture things that come in via. these channels into our PIM. Once that’s done, it becomes a regular “item” that can be managed using our regular workflow. The capturing process has to be quick and non-intrusive.

Capturing

Org has a somewhat new system for “capturing” things into org-files. The idea here is to get things into your system quickly rather than think about where to file stuff.

The idea is quite simple. You define a bunch of templates and then globally assign a key which you can hit anywhere to “capture” something.

Let’s take email as an example. Suppose I get an email that asks me to “do something”, I hit C-c r and my org-capture buffer pops up

This allows me to capture the email using any of these templates. If (for example), it’s not a TODO but just something I need to keep track of (e.g. a phone number or an address), I’d hit r. It would allow me to fill some stuff extra information in, annotate it with a link back to the email and add an entry to my reference.org file. Now, it’s safe.

If it was an email asking me to do something (e.g. fix this bug), I’d hit t and it would open a buffer asking for information. Usually, at this time, if I can quickly think of when I want to get to that, I schedule it using C-c C-s and put a date in there so that I don’t have to worry about that later. This would get dropped into refile.org.

If I’m working on the openlibrary code base and see a little bit of code that I need to fix but don’t want to interrupt my current flow, hit C-c r and capture it using o. It captures it as a TODO and drops it into refile.org. This works as a trackable /*TBD*/ in your code. I keep this separate from the regular TODO because it’s work related and large enough to be treated separately.

The j is used when I want to write a diary entry. I’m probably going to get rid of this since I can’t really track or mine journal entries and it’s more fun writing them on paper anyway.

So, at the end of the day, the refile.org file might look something like this

* TODO Fix default assigneed for support cases    
  SCHEDULED: <2011-07-19 Tue>
  Email from George Oates: Fwd: Re: Default assignee
* TODO This task needs more logging
  SCHEDULED: <2011-07-19 Tue>
  [2011-07-17 Sun 14:33]
  file:~/github/nibrahim/openlibrary/openlibrary/tasks.py::@oltask

For real life things which occur while I’m at the computer (e.g. phone calls etc.), I just fire a capture and record it as a TODO. For things that happen while I’m away from my computer, I use a stack of paper pieces I carry with me where ever I go and write down what I need to do. One task per piece of paper. I have a little ritual of putting these tasks into my system every night before I hit the sack which I’ll come to later.

At the end of this process, everything is either in the refile.org file or the references.org file. i.e. within my control. The next thing to manage them properly.

Refiling

At the end of every day, before I crash, I have a little refiling ritual. The bottom line is to empty the refile.org file. If this file has entries in it that are more than a day old, you know I’ve been procrastinating.

org-refile is bound to C-c C-w by default and can be configured to present you with a list of possible trees into which you want to move an entry. My projects.org file used to be a directory of multiple files each dedicated to a different area of my life (e.g. personal.org, work.org etc.) but I moved them all into a single file under different headings.

You can see my refile configuration here. It uses ido to make the refiling process smoother and more “Emacs like”. Here’s what a refiling looks like

It’s asking me where I want to move this item to and offering me the four headings in my projects.org file. I select one and it gets “refiled”.

Scheduling and agendas

Org-mode has an extremely powerful scheduling and agenda system. The basic idea is that you schedule your tasks for a specific day and/or set deadlines for them and then ask org mode to generate a view that prints out what you should be doing today, this week, this month or this year. The scheduling system is flexible enough to handle repeating tasks (e.g. pay rent on 10th of each month), effort estimates, clocking etc.

I personally use just the basic scheduling capability.

So, when I refile, I try to schedule stuff for certain dates. There are some tasks which need to be done “at some time” for which I randomly schedule them a few weeks from now and forget about them till they pop up on the agenda again. Some things are already scheduled while capturing so that’s cool. Some, I know when I will have time to do them so I schedule them appropriately. In any case, the process of refiling makes sure that scheduling is done. The dates needn’t be accurate but tasks should have some date when you’ll look at them.

Once that’s done, C-a a lists out my agenda for the current week (focussing on the current day) and I can get to work. Here’s a sample of my agenda from last month (the capture templates were different back then so it’s slightly different from what you’d see currently).

Monday     20 June 2011 W25
Tuesday    21 June 2011
Wednesday  22 June 2011
  projects:   Scheduled:  DONE Email from George Oates: Support System - bugs/tweaks l [5/5] :tbd:
  projects:   Scheduled:  DONE Email from George Oates: Re: Email setup    :tbd:
  projects:   Scheduled:  DONE Implement email system for support [5/5]
  projects:   Deadline:   DONE Implement email system for support [5/5]
Thursday   23 June 2011
Friday     24 June 2011
  projects:   Scheduled:  DONE Email from Anand Chitipothu: Account validation glitches :tbd:
  projects:   Scheduled:  DONE Fix account validation problems 
  projects:   Scheduled:  DONE Mail Vikas about Org-mode information he wanted
  projects:   Deadline:   DONE Email from George Oates: Support System - bugs/tweaks l [5/5] :tbd:
  projects:   Deadline:   DONE Email from George Oates: Re: Email setup    :tbd:
  projects:   Deadline:   DONE Vitalise Reassign/close case in support system
  projects:   Deadline:   DONE Change =/task= to =tasks=
Saturday   25 June 2011
  projects:   Scheduled:  DONE Integrate habits with capture
Sunday     26 June 2011

Now, you just pick out things to do, do them and hit C-c t to mark them as done so that they move off your agenda.

Things I don’t do

I’m not a clocking junkie who has to know exactly how much time I spend on tasks. I tried using arbtt to do that for a while and it works very well but I don’t really need or use it.
Custom agenda views. The default view looks fine to me. I haven’t really felt the need to change it yet.

Things to do

Other things which I want to include in my system are

Capturing directly from my browser using org-protocol and maybe a custome extension. Bookmark and schedule for reading rather than dive into that hacker news article in the middle of work.
Some tiny xmonad customisations to capture while outside Emacs. A popup Emacs capture buffer using emacsclient to take something down and close it.
Integrate skype chats with Emacs using bitlbee and erc so that I can “capture” skype conversations for reference or as tasks.
Try some effort estimation to practice my estimation skills.

Summary

So, the bottom line is that you simply capture stuff from various parts of your life into org, refile them appropriately once a day, schedule them and get them done. No more missing tasks or items.

I welcome your comments and suggestions. My Config files are on github and you’re more than welcome to cannibalise them.

If there’s sufficient interest in a formal Emacs lisp tutorial, I can take a conventional class sometime later this year in Bangalore. Let me know.

At PyCon 2011

I’ve submitted a talk for PyCon India 2011 which will discuss my Python development settings for Emacs. It will discuss mostly programming tools but I do plan to spend a little time discussing tracking stuff with org etc. so if you’re interested, you should sign up.

Git training course

2011-01-24T00:00:00+05:30

I will be conducting a public course on Git during FOSSMeet 2011 at my alma mater, NIT, Calicut.

The course will probably be held on Sunday, 6th February 2011 but the official schedule is a more reliable source of information.

The details of the course including slides are available at the training page.

Python extension training cancelled

2011-01-04T00:00:00+05:30

The extension training which I had planned for Jan 8,9th is cancelled due to a lack of participation.

On the other hand, a shortened version of the tutorial has been accepted for PyCon 2011. I will be presenting it over there and if there’s sufficient interest, will put up the slides/material on my site here.

Calligraphic rulings

2010-12-30T00:00:00+05:30

Calligraphy is a hobby of mine and I spend a few hours every week on it. I’ve tried arabic but it’s too hard to do without precise instruments and a teacher. English is slightly easier and the results look somewhat pleasing even if one is not very skilful.

Calligraphic letter proportions are specified using multiples of the width of the nib of the pen being used. Following is an example of the proportions of various parts of a letter in the “Foundational hand”.

![Foundational hand (from thecalligraphypen.com)] (/images/foundational-hand.JPG)

(from http://thecalligraphypen.com)

As you can see, the body is 4 nibs high and the ascenders and descenders are 3 nibs high each.

The only calligraphic pens I’ve seen in India are the ones from Sheaffer. They’re not bad and usually come as a set of 3 nibs. The fine nib is 1mm wide, the medium one is 1.5mm wide and the broad one is 2mm wide. Here is a picture of my broad nibbed pen

![Sheaffer calligraphy pen (broad nib - 2mm)] (/images/calligraphy_pen.jpg)

One of the things that are necessary to write properly is a ruling sheet. These are similar to the “cursive writing” practice books you must have seen when in kindergarten. The sheets specify the height of the main body, the height of the ascenders and descenders and sometimes the inclination of the letters. They serve as guides when one is writing.

To manually create such a ruling is quite hard and error prone. It’s tedious to actually measure multiples of nib and making sure that all the lines are straight necessitates some good drawing equipment. To save myself some trouble, I wrote a little script which takes a bunch of parameters and creates an A4 PDF with the rulings. The only “documentation” right now is the help message which is pretty sufficient really.

    Usage: ruling.py [options] output_file
    
    Options:
      -h, --help            show this help message and exit
      -n NIB_WIDTH, --nib-width=NIB_WIDTH
                            Width of the nib specified in millimeters. All other
                            measurements are multiples of this.
      -p PARTITIONS, --partitions=PARTITIONS
                            Comma separated list of partitions in each line
                            (specified in nib widths)
      -g GAP, --gap=GAP     gap between lines (specified in nib widths)
      --top-margin=TOP_MARGIN
                            Top margin (specified in nib widths). Default is 2
      -r RULINGS, --rulings=RULINGS
                            How many rulings to draw. Default is 10
      -a ANGLES, --angle=ANGLES
                            Comma separated list of angles (in degrees) for which
                            to draw lines on the page (for pen angle, serifs etc.)
      -t TITLE, --title=TITLE
                            A title for this ruling (usually the font name)

Invoking it like so python ruling.py -g2 -p3,4,3 -n2 -r25 -a30 -t"Foundational Hand" foundational_ruling_sheet.pdf produces this which works pretty decently.

I hope you find it useful.

Update

I’ve repackaged this as a web application and you can access it directly from http://noufalibrahim.name/rulings.

Employment status

2010-12-23T00:00:00+05:30

Its been around a year since I took the plunge. I worked for a few organisations, did a few trainings and generally had an educational and profitable year. I attended two PyCons abroad, conducted one here, met a lot of interesting people, got myself a new laptop, got exposed to a whole gamut of things which were completely new to me (like managing my finances by myself) and generally had a really wild ride.

I thoroughly enjoyed the entire experience and am the richer for it. I won’t deny that there were times when I felt that it was a mistake and almost regretted it but those were transient feelings.

It’s hard to make a “list of things I learnt” since most of them can’t really be verbalised properly. However, if you have some risk taking ability and a desire to stretch your limits, I highly recommend it.

Earlier this month, I got a job offer from the Open Library project (a sub project of the Internet Archive) via. Anand. I accepted it and starting Jan, I’ll be working for them remotely. Lots of challenges, great people, all code is free software, decent pay, and work that has a positive social impact - What more can one ask for?

We’ll probably be setting up a small office of sorts somewhere in Benson town and over time are planning to convert it into a hacker space.

My trainings will still be there. I will make announcements over time. It’s not something I want to give up but I don’t think I’ll be taking any large projects anytime soon.

Python extension training

2010-11-22T00:00:00+05:30

I’m conducting my “Extending Python using C” training on 8 and 9 January 2011.

The training will be conducted at Regus Business Centre, Level 9, Raheja Towers, 26-27 Mahatma Gandhi Road, Bangalore - 560001

You can view details of the course at the info page. To register for the event, please click on the link below

For details on my trainings in general and for schedules, please refer to the trainings page.

Public trainings

2010-11-18T00:00:00+05:30

I’ve always been fond of teaching and have decided to hold public courses on various technologies that I research and use on a semi-daily basis.

I have put up a separate Trainings page which describes my motivation and desires and also mentions way to stay upto date if you want to attend any of the courses that are being offered.

Hyde : An Emacs mode for Jekyll blogs

2010-11-11T00:00:00+05:30

I’ve been working on a simple mode to manage my blog entries using Emacs and Jekyll.

The mode is available at https://github.com/nibrahim/Hyde.

Following is a brief users guide

Usage

Download the all the hyde-*.el files and put them somewhere. Once you do that, add the directory where you put it to your load path like so and require it.

1
2
    (add-to-list 'load-path "/path/to/hyde*.el")
    (require 'hyde)

Operation

This mode is a simply a wrapped for a number of shell commands that are used to create and deploy the site. It doesn’t maintain any local state (in the form of status files etc.) so if you change your repository manually outside it, just refreshing the buffer will bring it upto date.

It’s tailored to the way I work. I keep my posts in a git repository (although I do have a crude DVCS abstraction layer if you’re using hg or any other such system). I make changes, commit them and push the repository to github (you can, for example, see this files source at github). After that, I make the site using jekyll manually and then copy it over to my webspace using rsync. I don’t use any of the git hooks (yet).

Customisation

The variables that you might care to customise are

Location parameters
- hyde-home : The root directory of your blog
- hyde-deploy-dir : The directory where jekyll will generate the site for you to deploy
- hyde-posts-dir : The directory that will contain the actual posts (this is relative to hyde-home and is _posts by default).
- hyde/deploy-command : The command used to deploy the site. scp, rsync or whatever else you might please.
VC parameters
- hyde/vc-uncommittedp : Predicate to check whether the file is uncommitted
- hyde/vc-unpushedp : Predicate to check whether the file is not yet pushed
- hyde/vc-pushedp : Predicate to check if the file has been pushed (inverse of the above)
- hyde/vc-add : Command to add the file
- hyde/vc-commit : Command to commit a file
- hyde/vc-push : Command to push the local changes to the remote end.

Interface

The main interface looks like the following screenshot

The list of posts are presented on top along with a key of what the letters before the post names mean. The post names are also colourised accordingly

The keys you can use at this time are

n : Create a new post (it will be a draft)
c : Commit the current post
P : Push all pending commits (this is only a VC push. Not deployment).
j : Run jekyll and create the new version of the site
d : Deploy the site.
g : Refresh posts (useful if you’ve done something by hand earlier)
p : Promote a post from a draft to a a published post
q : Quit hyde.
RET : Open the current post for editing.

The markdown mode in which the buffers open up for editing is slightly modified. It has a few extra covenience bindings

C-c C-c : Save file and finish editing
C-c C-v : Preview file (this is a markdown preview so extra liquid tags will not work).

It seems to work reasonably well. This post was created, committed and deployed using hyde. There are of course lots of bugs to fix but it’s usable. Here are a few things I can think of

Post drafts
Proper previews (using a local jekyll server)
Status of deployment for the posts to be displayed

Please give it a shot. I welcome your feedback.

First Jekyll post

2010-10-09T00:00:00+05:30

This is a test post to verify the jekyll based system which I just got setup to create blog entries. These are disjointed files and will be assembled together by Jekyll.

The next step is to create an Emacs mode that automates most of these things properly.

Let’s see.

PyCon India 2010 : A report

2010-10-01T00:00:00+05:30

PyCon India 2010 was the second installment of our very own PyCon. A lot of people worked really hard to make this happen and we tried to learn from the experiences of last year and do a better job. The conference was organised by the Indian Python Software Society and was held at the M.S. Ramaiah institute of technology, Bangalore on the 25tr and 26th of September 2010.

We had around 650 people attending on the first day. On the second day, it dropped to around 300.

There were approximately 60 talks spread across 3 tracks and a special track targeted at scientists and engineers conducted by FOSSEE which contained half a day of lectures and one and a half days of workshops.

Our main sponsors this time were ZeOmega and FOSSEE. We also had support from the Python Software Foundation and had smaller sponsorships from TANDBERG and ProjectPlace. Mahiti helped us out with a lot of the logistical issues and we partnered with Vodex technologies to provide offline copies of the presentations. We partnered with DoAttend to help us with the ticketing so that participants could register online beforehand and shorten the queues.

We were able to get David Goodger to come and be our keynote speaker for the conference thanks to help from our sponsors and the PSF.

Like last time, I was the semi-official leader of the entire project. I’m glad to say that it went reasonably well.

Unlike last time, I am listing below the list of people who actively helped out with the event so that it goes on the record (if I’ve missed your name, please drop me an email).

Anand Pillai - Handled Talks and scheduling
Anand C. And Abhishek - Webmasters
Vijay - Sponsor coordinator
Arvind Dixit - Recording and A/V.
Kenneth - Handled creation of society and bank account.
Santhosh - Handled Records (treasurer)
Sree - Swag, Infrastructure, food.
Kunal - Banners, id cards, coupons
Anil and other student volunteers from MSRIT - Ground support and running around
Kunal, Ramki and Vijay - Session chairs

Following is an account of the weekend of the conference.

David arrived on the 22nd. It was his first trip to India and we tried to show him around the place as much as we could.

On the day before the event, there was the looming “threat” of a verdict for a 6 decade old civil case. The local government had initially declared a preemptive curfew on the weekend which would have forced us to cancel/postpone the event. The fear we had is reflected in this blog post. However, the verdict was pushed forward by a week and we were able to conduct the event as scheduled.

Due to work pressure, I was not able to visit the venue on the night before the conference. Sree looked over the place, made sure that everything was working fine. It rained quite heavily the night before the conference. I called all the volunteers who had taken charge of the various aspects and made sure things were in place. The even we were working towards for 6 months was finally going to take place.

The following are my descriptions of the two days. Unlike last year, I wasn’t able to attend too many talks and so it’s mostly just my experiences as a “manager” rather than a delegate.

25 September 2010 - Saturday

I met Anand Pillai at my house at around 600 hours and went to pick David up. We got to the college at around 700 hours and started to get things ready for the keynote. Sree was already there and Anil and his band of student volunteers got things moving quite well. The A/V guys arrived and Arvind took care of doing what was necessary to get them rolling. It started on time without any problems.

Things suffered a little after this. The crowds started pouring in. While Sree, Santhosh and the other volunteers did a great job at the registration desk, a queue did form. People hadn’t taken printouts, they opted for size L and took size XL etc. Some people wanted to register on the spot and started bargaining on the price. The lack of civil sense was… characteristic. However, the queue cleared by around 1100 and things went on.

Wifi was a problem that was reported over and over again. Lack of IP addresses, lack of signal strength and numerous other problems prevented people from using it effectively. This was uncool and we need to figure out a way to do this properly next year. Anand C. had a wireless router which he brought from his house which we set up to help. It did help but wasn’t enough.

Another problem was that one of our halls (the “hitech hall”) was away from the other two halls which were easily accessible. We didn’t stick up maps and the signs were not sufficient. People were wandering around without direction and support at this hall was less than satisfactory. This led to delays and other problems because of which a lot of speakers whose talks were scheduled in this hall suffered. We need to take care of this kind of situation next year. The volunteers Anil had arranged couldn’t all make it since they had placement tests to write and so we were short of people at the venue to help with directions etc.

The caterers messed up with the lunch setup. I don’t like the idea of barking at volunteers since they’re doing the work they are out of their own good will but if we’re paying for something, I expect some amount of professionalism. I was relaxing when Anand told me about the lunch delay. There wasn’t much to be done but they got the tables set up 30 minutes late and the queues grew long and uncomfortable. However, there was still enough time and the afternoon talks didn’t get delayed.

The afternoon sessions went reasonably well. However, there were a few speakers who cancelled their talks without informing us. This messed up our schedules.

I wanted to attend the Zen of Web talk by Anand C. but couldn’t. I missed the celery and twisted talks as well. I missed llvm-py too. I attended part of the ZIO talk by Vijay Kumar and Asim Mittal’s Wii related talk which were both nice. I stayed for Arun R.’s blender talk but was too exhausted and stepped out for a cup of tea. The hardware talks (ZIO and Wii) were well received and I liked the preparedness of the presenters.

We left a little late. I dropped off David and got back home.

26 September 2010 - Sunday

I picked up David in the morning a little late. I had been burning the candle at both ends for longer than is healthy and I was starting to buckle. I reached the venue and saw a significantly lower turnout. My morning talk was postponed to the afternoon since Baiju couldn’t make it due to bad health. I sat in David’s talk for a while in the morning but was too distracted to pay attention. My own talk went okay. Things went mostly well. A few talks were cancelled but we filled in the spots as much as we could.

We ended the day with the first annual general meeting of the Indian Python Software Society. Kenneth chaired it and we had some discussions and motions passed. After that, all of the volunteers headed to a nearby restaurant for a treat and we got back home.

Thoughts

The talk quality was slightly better than last time but it’s still the major sore point (apart from the wifi). We need to be more active with talk screening and give more time to the selected talks so that people can talk better. A separate tutorial day would be good too although that would spill into the weekdays.

I delegated work a lot more than last time. There were large aspects of the conference that I was simply unaware of and completely willing to entrust to the volunteers. This improved the cohesion of our core group of volunteers. That’s a good thing.

During the initial discussions on the mailing list, I put my foot down a few times and that led to some rather bad vibes in the community. In retrospect, I really do sound annoying in that email. This was a bit of learning experience for me. Made me appreciate the kind of balance that real project leaders like Guido and Linus must maintain.

There were quite a few people on the mailing list who brought up completely irrelevant topics during discussions and wasted valuable volunteer time and energy discussing useless topics. This was annoying and it’s taking all the strength I have to stay quiet.

The conference was bigger and better than last years in all respects in my opinion. We didn’t take any steps back but have a lot of steps to take forward. The road is clearer now.

This was a humbling experience for me. To pull something like this off. A foreign keynote speaker, over 600 participants, so many interactions, so many volunteers. I think it went well.

Next years event is slated to be held in Chennai. I will be taking a subordinate role during the event. People are probably fed up of my attitude anyway. :)

See you all there!