orestis.gr

This blog is no more.

Mon, 07 Nov 2016 13:53:54 +0200

I have rebooted my blog, and you should visit it at orestis.gr.

WSGI, Twisted and Server Sent Events

Mon, 03 Feb 2014 18:49:43 +0200

Old-school web applications were easy to create. Big powerful frameworks like Django give you a lot of tools you can leverage. One weak point of all those WSGI framework is that they didn't integrate well with anything that broke outside the usual request-response cycle.

The usual approach nowadays is to use WebSockets for real-time communication between browser clients and web servers. The usual way to do that would be to use a server capable of handling many concurrent connections, and use a message bus from the WSGI app to communicate to that service. That is a lot of moving parts.

In my use case where I build a lot of intranet applications, deploying and maintaining all this infrastructure is a very big burden, so the result is usually to not even explore this kind of functionality.

However, given that I deploy on Twisted, I wanted to explore what kind of cool things I could build on it.

Enter SSE

Server-Sent Events aren't that new - they have just been shadowed by WebSockets. They are a simple data format that is send from the server to the client via a plain HTTP connection. The Javascript API is quite simple, and it even handles retries for you. It's compatible with a lot of recent browsers, but I haven't really done a lot of research on it.

Sample Code

Here is a very simple WSGI app (using bottle.py). It just has a form and a form POST handler.

And a basic twisted server to run it:

And a SSE-savvy twisted.web resource:

And a very simple index.html

How it works

The WSGI app just calls some Python code. Through crochet we ensure that it gets back a useful result (though in this case, we just throw it away). We use a plain POST to send data to the server. Converting that to an AJAX request is left as an exercise to the reader. The SSE handler is a singleton that keeps track of all the listeners that are connected to it, and broadcasts messages to it.

Does it scale?

It should! I have no experience running Twisted Web under heavy load but it's more than enough for intranet-style apps (even when I have 50 machines hitting some API endpoints quite frequently). If someone wants to run some testing, please get in touch.

What's next?

I would like to make this a bit more reusable, with some better discovery than the current "inject a global function into the namespace". Also, Django integration is something I'd like to investigate. And why not try if the same approach can be extended to web sockets as well?

Comments

Restoring a Time Machine backup to a smaller drive

Thu, 13 Dec 2012 09:39:46 +0200

Recently, my SSD started failing. That's a usual fact with SSD and I was prepared for it, or I thought I was. I got a replacement Intel SSD from amazon which was advertised as 240GB rather than 256GB. I originally thought that this was due to formatting (GiB vs GB) but it turns out it is actually smaller.

I put in the Macbook Pro, launched Internet Recovery (boot with cmd-R) and selected "restore from time machine". Unfortunately, the system complained that my destination drive didn't have enough space.

I embarked upon a day-long quest to start to remove things that bloated the Time Machine backup, for example, backups of Google Chrome (which I can download), and my iPhoto Library (which I have backed-up elsewhere too). I was too distracted by this painful procedure to realise that nothing I did had any effect - the restore system always complained about the target disk being smaller.

Given that it didn't spend a huge amount of time calculating the actual size of the backups, I felt that the size information was stored somewhere and wasn't updated. In now obvious retrospect, I used the xattr utility to examine the metadata of various things on the TimeMachine backups. Lo and behold:

orestis-edm-mini:swarthy sor$ xattr -l 2012-11-19-004303/M4_256/
com.apple.backupd.SnapshotVolumeFSEventStoreUUID:
00000000  45 41 39 39 45 39 38 44 2D 39 34 44 41 2D 34 34  |EA99E98D-94DA-44|
00000010  38 46 2D 42 34 37 42 2D 44 33 31 38 30 36 36 33  |8F-B47B-D3180663|
00000020  45 33 46 38 00                                   |E3F8.|
00000025
com.apple.backupd.SnapshotVolumeLastFSEventID:
00000000  35 33 31 30 36 34 35 39 00                       |53106459.|
00000009
com.apple.backupd.SnapshotVolumeUUID:
00000000  33 37 33 31 35 37 46 43 2D 33 45 43 39 2D 33 38  |373157FC-3EC9-38|
00000010  42 32 2D 38 43 35 41 2D 45 33 45 42 31 43 33 32  |B2-8C5A-E3EB1C32|
00000020  42 38 34 35 00                                   |B845.|
00000025
com.apple.backupd.VolumeBytesUsed: 232715759616
com.apple.backupd.VolumeIsCaseSensitive: 0
com.apple.metadata:_kTimeMachineNewestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 B6 59 AA 07 00 00  |bplist003A.Y....|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032
com.apple.metadata:_kTimeMachineOldestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 B6 59 A9 3B 00 00  |bplist003A.Y.;..|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032

Among other things, there's an obvious "VolumeBytesUsed", that obviously isn't being updated when I trimmed down various files from inside the TimeMachine interface. After issuing xattr -w com.apple.backupd.VolumeBytesUsed 219637221376 2012-12-10-142137/M4_256/, the restore interface happily accepted my latest backup. It's now restoring the backup as we speak, and I will update this post once I see some results.

In conclusion, I really need to have a bootable SuperDuper backup to streamline this kind of restore. On the other hand, if an SSD drive starts failing, and it silently corrupts data, can we even trust backups?

Comments

EuroPython 2011 impressions

Sat, 02 Jul 2011 15:51:21 +0200

So, EuroPython 2011 came and went, and it was great fun. I'll try to write down some impressions before I forget them entirely. UPDATE: I finally got to publish this more than a year after it has been written. I won't even bother to read it much.

The location

Florence is just beautiful. It was very tempting to leave the conference and just go walk around. I only did it once :) Jokingly we argued that conferences should be held in duller towns to avoid conflicts of interest. In reality though, having such a pleasant backdrop is hugely satisfying. I'm definitely looking forward to visiting Florence next year again, and I'll definitely visit without any conference to actually see the sights properly. I only had time to do a 3h guided tour before the conference started, which was great fun though I was surprised at how few people showed up.

The venue

The 'Grand Hotel Mediterraneo' was very good. Spacious, air-conditioned rooms, very good facilities, brilliant coffee breaks and lunches, very pleasant overall. Although, before the conference started, they tried to charge me 1 euro for a glass of water, which I found absurd.

The organizers

I've found that the organizing team did a great job putting everything together. Before the conference started they put up a great site with all the details, they communicated very efficiently with the speakers and they had lots of equipment available. I've asked for cabled network for my training, whiteboards and plenty of water and they delivered in all fronts. Excellent job. The only huge problem was the wifi and internet connection. Due to legislation, they could not provide anonymous wireless internet, so we had to go through an arduous log-in process via ComCom. While they had lots of ComCom guys around trying to fix stuff, the interent and wireless was basically unacceptable. Which is a shame because the conference never got the online coverage (twitter, blogs) that it deserved.

The talks

I'm afraid I was a bit let-down by the available talks. There was a 'community voting' process where people that bought a ticket could vote on interesting talks, but I think this is a flawed approach. There are different levels of expertise that a conference should seek to satisfy, and I've found that most talks were at a very basic level. I think that the concept of 'tracks' (eg web track, systems track, language track) would help make better decisions both in talk selection and in planning, and also an expert track would be required. I didn't have any serious conflicts, though I've found that scheduling the gEvent talk at the same time as my Twisted training was a bit unfortunate, as people interested in async stuff would probably want to do both.

On the other hand, I've found that having the trainings side-by-side the actual talks was a very good idea. It made the conference longer, but it gave much more people the opportunity to decide to skip a talk in favor of a training. If the trainings were a separate event I feel much less people would have chosen them.

My training

The Twisted training was a blast. I was fortunate enough to do it on the first day, which meant my nervousness only lasted a day. I missed the morning keynotes, though since the audio was piped in the room I've locked myself into, I could still get the gist of them. I was horrified to hear the Spotify people discuss how they've built a lot of stuff with Twisted, since this meant I would get lots of questions I wouldn't know how to answer.

Fortunately two Twisted core devs, Laurens van Houtven (lvh) and Stephen Thorne (jerub) were right there at the front row, taking notes and fielding the difficult questions. I thank them again for this, I didn't even know they'd be there.

The room was packed and we had to send people away, which led the organizers to ask me politely to re-deliver the training. At first I declined (4 hours standing up is a long time and tiring) but then we made a deal, if there was enough interest I'd do it. Turns out we still had to turn people away but in the end a couple just sat on the floor with their laptops on their knees. The second time I was much more relaxed but I still didn't finish on time. I think that 3:15 hours is just not enough to cover all the Twisted basics. Next year I'll ask for a whole day, so that people can actually have time to do the coding exercises, and so that the concepts can sink in before we move on.

So, all in all, 50ish people attended, which is roughly the number of new twitter followers I got since the conference. I think it went well. I have put the slides on the conference page and I've started converting the training into a written tutorial

Comments

Twisted Training, Europython 2011

Fri, 17 Jun 2011 01:31:26 +0200

I'll be giving a 4 hour long Twisted training in 3 days. Now is the time to give me input on what things you'd think it's interesting to show.

Teaser: I have already 150 slides, covering servers, deferreds, clients, and a bit of web. The training is mostly aimed at people that haven't done any kind of async programming so it spends lots of time explaining the basics.

I'm working madly on adding new stuff (learning a ton on the way) so if you:

Know Twisted and feel there's something I should definitely showcase or
Tried Twisted and couldn't wrap your head around something

Leave a comment here, or at the talk page, or send me an email at orestis@orestis.gr or message my on twitter @orestis.

Comments

Python Greece users meetup

Wed, 01 Jun 2011 10:30:01 +0200

The Python Greece community is shaping up nicely - a meetup will be held on Wednesday, 8th June, at 20:00, at the Nixon. State your interest at the Lanyrd page!

We are looking for a couple of extra speakers to give a 15-20 minute presentation on interesting things, please volunteer!

Did I mention we also have a website for the Python Greece community?

Comments

Job opening - Cocoa developer

Wed, 11 May 2011 11:00:13 +0200

If you are a Cocoa developer and familiar with Core Animation, Quartz and/or graphics technologies in general, and are interested in participating in a project developing interactive exhibits for a major new museum, drop me a line at orestis@edmstudio.com. Please include a resume (with emphasis on your Cocoa experience), a rough idea of your desired hourly wage, and your availability.

This is fixed-term opening, with long-term cooperation on upcoming projects possible. Telecommuting required!

Comments

The Voyage Limpid Sound are Happy

Fri, 21 Jan 2011 11:10:49 +0200

I don't usually put personal stuff here, but now that the cat is really out of the bag, I urge every music-loving person to have a look at a band called The Voyage Limpid Sound and buy their album on Amazon UK, Amazon.com or iTunes.

You can have a listen at their BandPage: The Voyage Limpid Sound BandPage. (BTW if anyone knows of better solutions to put up preview tracks for the world to listen, please tell).

Here's a silly video-clip:

Disclaimer: All the band members are good friends and I've been singing in the choir for the past couple of concerts.

Comments

Epson Perfection V33 Scanner review

Wed, 19 Jan 2011 11:54:36 +0200

I recently bought a new scanner, the Epson Perfection V33 Scanner , to replace my previous, decade old HP scanner that had no driver support for Mac OS X. I'm very pleased with it, so here's a small review.

The object

The scanner itself isn't as small and nimble as some other Canon scanners I've seen. It's roughly as thick as 2/3 of a standard 500 A4 paper stack, and as wide as a 20'' monitor. Or, 28 x 43 x 4 cm (W D H). It has roughly the same footprint as my trusty Canon ip4200 printer. Certainly small enough to be unobtrusive.

It has a hinged cover that can open up to 180 degrees to facilitate scanning books. 4 buttons of which I only use the on/off, a USB cable and power adapter. Can scan an area slightly larger than a A4 sheet.

The scan quality

I've no idea. It's good enough. No really, I'm using this mostly for office documents etc. If you want to scan film negatives look at something at 3x the price. Look below for actual scans.

The speed

This thing is fast. It features an LED lamp which takes literally no time to warm up. I can just switch it on and have a scan within a few seconds, which is awesome. My previous HP scanner would take over a minute (perhaps more) to get ready to even show a preview.

The software

Epson bundles a bunch of horrible looking (on Mac OS X, at least) drivers. I installed the bare minimum to get the scanner to appear under the native Image Capture utility. I've no idea how that works (whether the manufacturer can provide utilities there) but it's a pleasure to use. Here's a screenshot:

(The books above are Tenor: History of a Voice and Surely You're Joking Mr Feynman. Download the scans here and here.)

The best features are:

You can scan multiple items at the same time (you get multiple files)
It does a very good job at detecting angles and such so you always get a nice upright image. (Note, in that screenshot those two books could not be detected, possibly because of light spillage. You can always override the selected items though).

Conclusion

Scanners that cheap have become commodity hardware nowadays. The most one can say about such hardware is that it's a pleasure to use, no pain points at all. So I'm pleased.

Comments

Billing hourly: A followup

Wed, 10 Nov 2010 10:47:17 +0200

When I penned down some simple thoughts on the way I work with a single client, I didn't expect the flurry of activity that followed it. 150,000 visitors from all over the world, more than 350 comments here, on reddit and on Hacker News. I probably hit a very sensitive nerve of the developer community. I'll try in this post to sum up the various comments and perhaps clarify some of my original points.

YMMV

The obvious point - I've described a way that I work that found works for me, at my current situation, with my current experience and my current client. I'm not advocating this as the solution for all software business issues. In fact, next year or even with the next client I might do things differently. Right now, however, this way Works For Me.

The response

A lot of very good points were raised, and I'm certainly wiser from them. A big thanks to everyone who replied (including guys belittling me or my work experience, being grounded now and then is valuable, too).

This works for only a specific kind of project

Certainly! This particular project is small and also a bit open-ended, the client has an idea of where he wants the application to go and is looking for someone to help him implement it. It's not something that has a clearly-defined boundary. Billing hourly in this case means the client can spend his limited budget on the things that give him the best bang for the buck.

Larger projects/organisations need fixed quotes

Of course, the moment you have to get an actual department to give you money, billing hourly will not work. They will need to see a fixed bid, or if you're lucky a non-binding, best-effort estimate. Being somewhat inexperienced in those things, I shy away for fear of getting burned. People with more experience might prefer those because if you play your cards right you will have a higher margin.

If you were more experienced, you'd be able to give accurate estimates

Definitely, if you have created the same thing 5 times already, coming up with a reasonable estimate is easy. In fact, after I saw the code I had to work with, I was able to provide very accurate estimates myself, never once going over. But would I commit to an arbitrary estimate before getting the chance to investigate? No. Would I agree to spend 3 days scoping out free of charge? No.

Given a detailed set of requirements, fixed bids are possible

Related with the previous point, if someone has a detailed set of requirements, and they are absolutely certain that's what they want, you may be able to give a fixed bid. However, I maintain that unless they got a software developer to draft up the specs, they're not going to be in the form required to give accurate estimates (and even then the chance of missing parameters is present).

How can the client trust you to be good, fast and honorable?

That's obvious - they can't. However, the potential client has already spent 1 hour on the phone and 2 hours in face-to-face meeting with me, explaining the application and his requirements. He has already had the chance to judge some qualities of my character (What kind of questions do I ask? Do I write things down? Do I understand the business concepts?). Hell, in many cases an organisation will hire an employee with less of an interview than this. Certainly a client can spend a few hundred bucks to see whether I can deliver on my promises. What if I made a fixed bid, got a down payment and produced crap?

Better developers will spend less time. What should they charge?

They should charge more because their hourly rate will be higher. I'll worry about it, when/if I reach this stage.

The client is often clueless

You need to protect yourself from them. The alternative is educating them, but you have to make sure you are compensated for that, otherwise you will be doing spec work for free.

Charge not the cost, but the value

A very good point. If I bill $600 dollars for something that gives the client a $20,000 value, didn't I get the short end of the stick? Probably. Depends on how content I am with even the short end of the stick. Perhaps as I get more experience with projects, I'll be able to pull off higher margins. Until then, I'd rather be safe.

Psychology of hourly rates

A very good point. A lot of time, the hourly rate you'll quote will be higher than someone's salary. Of course it's not a valid comparison, but it's hard to get that point across. It seems that fixed price is easier to swallow, even if the total outcome is the same. I've run into this myself.

Shower time!

Do you bill for time spent thinking in the shower? I'll leave that to my judgement for any given time. Time spent on the couch deliberately thinking, though, is definitely accounted for.

Honorable mentions

http://www.bestbrains.dk/Blog/2009/06/03/CollaborativeAgileContracts.aspx

A very good idea - a nominal hourly rate, less than the usual, to cover your costs, then a set of fixed payments upon certain milestones. I might try that on my next project.

There's more

There's more than 250 comments in reddit I didn't have time to do yet. I'll update the post when I get round to it.

Some more clarifications

To all people asking about what do I use for my automated reports, I'm using Harvest. The reports aren't completely automated, in that I have to click the actual buttons, but I'm getting essentially a report of the uninvoiced billable hours so far in a nice item list than I can send as PDF to my client.

One important aspect here is that the Greek IT market is completely random. The phenomenon of the "nephew that could do that in 2 hours" is rampant, a lot of unqualified people that claim they are web developers because the can setup a Wordpress blog are skewing the prices down. In this environment, and with a lowest-bidder-wins attitude, a fair fixed bid will have a hard time surviving. A high hourly rate sends a different message about personal value that I want to maintain.

Excuse me while I raise a sociological point - a lot of the feedback on hacker news was about how billing hourly will not scale. That is a) absolutely correct and b) somewhat irrelevant to me. I'm at a stage of my career where I'm trying to gather experience and expertise, scaling a business is not a priority. My current system suits my needs, gives me an incentive to put in the hours daily (I work from home and motivation can sometimes be hard to find), and allows me to know my monthly income. It's a very good safety net that will protect me from getting shafted or burnt-out.

Comments

Why I bill hourly

Sat, 06 Nov 2010 14:39:10 +0200

UPDATE: I've posted a followup.

I was recently contacted by my first Greek potential client, wanting to do some new features in a small Django application. I quoted him my hourly rate and he was shocked! He wanted me to quote a fixed price for the features he wanted, and I strongly declined. The outcome?

We ended up working together, he is very pleased with both the results and the money he's paying, and I didn't budge on my initial quote. Here are the arguments I used to support that an hourly rate is better for everyone. Hopefully they will help somebody else as well.

A comparison

I like to link software development to erecting a building, starting only from a plot of land. The process usually goes like so:

Client:

Here's the patch of land we want to build on.
Here's some more-or-less vague sketches of what we want to build.
Can you quote us an exact price for it?

Any engineer would immediately recoil from such a suggestion, however many times software developers, probably anxious to close the deal, will do a wild estimate, double it and add 30% and hope for the best.

Back to harsh reality

Here's my response:

My hourly rate is XX/hour.
I can spend 2-3 hours free of charge, trying to define the scope of what you want.
After that, I will start charging, even for a 5 minute telephone conversation.
The rule of thumb is: If I am doing something which I wouldn't do if it weren't for your project, you get charged for it.

My client was terrified. His first response was that XX/hour is way too much. He mentioned other developers that have wasted his time and money and left him with unusable crap. His fear was valid - how do I know I can trust you?

A valid question

I then presented him with the following facts:

XX/hour may seem too much, but a) I'm good and b) I'm fast.
I use a time meter (much like a taxi driver) to accurately track my time.
I am honorable. I will not overcharge you.
Before embarking on any task, I will give you reasonable estimates.
Those estimates will not exceed 4 hours.
If at any point I realise something is going to exceed my estimate, I will present you with alternatives and you will decide the course of action.

Let's visit those one by one.

I'm good and fast.

Really, if I can get something done in 1 hour where someone else would need 3, do you end up paying less or more? Enough said.

Track time.

I mean it - I press "start" when the editor launch, "stop" once I've checked in. Any interruption longer than 30 seconds and I will pause the meter. I give daily reports (automated, of course) so the client can track the cost and can budget accordingly.

Honor

I could easily overcharge you. I hold all the domain knowledge, and I can claim I spent 2 hours where I spent half. I would probably be able to slip in a dozen extra hours per month. It's not worth it. My reputation is far more valuable than any amount of short-term money. I plan on getting references and leads from a happy client, and any suspicion of being dishonest will ruin that chance.

Communication

You won't have to worry about hidden costs. I will let you know, to my best efforts, what the potential costs are going to be, and you get to decide and prioritise. I will not disappear for a month then come back with a finished product.

Estimation

This one is tricky. I use this handy rule - if I feel a feature is going to take more than 4 hours, I write down "UNKNOWN". Think of it as asking someone to paint a room of unknown dimensions. Any figure he'll be giving you is going to be random. In the case of software, I will need to spend some time planning and thinking about the feature before coming up with an estimate. And you get billed for that time, as well.

This particular client understood this immediately, and he is now giving me very detailed specifications, data schemas, even mockups of the interface. This way he keeps his bill low and he also ensures feature creep is not going to destroy his project.

That last point is of particular value. Any client will have grand dreams of his product. He will constantly ask for new things, and think those were naturally implied from his vague sketches. If you have quoted a fixed price, and you were smart enough to have a detailed contract, you will need to fight him on every step, saying "we didn't spec that". Whereas, with an hourly rate, you can say "sure, let me estimate how much will this be". The client can then decide if it's worth the money.

Uncertainty and unknowns

In software, it is quite expected that unforeseen problems will arise and budgets will slip. There are two ways to deal with this:

Try to factor the risk into your price or
be honest with the client and let him decide what the outcome will be.

With the first way, the client may overpay for something that wasn't really necessary, or the developer must live with the anxiety of overshooting your budget and working for free.

With the second way, the client will need to pay more only if necessary, and he is part of the decision making process.

Being agile

This whole essay is about Agile Project Management. I'm the expert on managing software projects, my client is not, hence I need to educate him. I don't really care about the whole array of tricks available to manage software teams, only about some core topics:

Daily results
Constant communication
Trust

Keeping the tasks small enough to be completed within a day means I can show my client daily results. Compare that to the previous developer, who vanished for 6 months and delivered a piece of crap.

Daily results mean the he can steer the project to where he wants, because I allow him to see if what he thought he wanted is what he really wanted. We communicate daily - once before I start work, having a brief chat about the feature I'm about to implement, and once after I finish, where I ask him to have a look and see if that's what he wanted. This puts the client in power, because he's the one making the decisions, and his decisions doesn't cost a ridiculous amount of money.

Those two items help build trust. The client, seeing daily results and interacting in a professional way, is convinced that indeed my hourly rate is worth it and he is getting exactly the things he wants.

I firmly believe that this way of working gives both the client the best result for the least amount of money, and provides me with the peace of mind that I will not work for peanuts because of a bad estimation made on shaky evidence.

Comments

Παντού Έλληνες!

Thu, 09 Sep 2010 14:34:01 +0200

Πόσο μα πόσο εκνευρίζομαι με την μαγκιά της φυλής! Βλέπω λοιπόν στο in.gr άρθρο με τίτλο "Ένας Έλληνας στον τελικό διαγωνισμού φωτογραφίας του National Geographic" και μπαίνω να δω αν ο αδερφός μου διακρίθηκε παγκόσμια...

Και βέβαια κατευθείαν αποδεικνύεται πως ο "Έλληνας" ονομάζεται Φρανκ Φακίνος, και όπως ο ίδιος γράφει στη σελίδα της εταιρείας του είναι "Native Californian", με ούτε καν αναφορά στην καταγωγή του.

Αλλά οοοοοχι, εμείς θέλουμε να είναι Έλληνας... για να πείθουμε στους εαυτούς μας ότι μπορεί όλα γύρω μας να είναι σκατά, αλλά επειδή είμαστε ανώτερη φυλή τα προσπερνάμε!

Κοιμόμαστε τον ύπνο του δικαίου. Όλοι οι άνθρωποι του κόσμου που βλέπουν ότι στην πατρίδα τους δεν έχουν τις ευκαιρίες που θέλουν, τα μαζεύουν και φεύγουν, γίνονται μετανάστες. Πάνε σε ένα τόπο που τους δίνει τις ευκαιρίες και την υποστήριξη που χρειάζονται, και δοκιμάζονται. Κάποιοι διακρίνονται, κάποιοι τα κουτσοκαταφέρνουν, κάποιοι ζητάνε δανεικά για το εισιτήριο της επιστροφής.

Βέβαια, τα media, με εξαίρετο selective reporting αναφέρουν κάθε τόσο και από έναν θαυμαστό επιστήμονα που ανακαλύπτει θεραπείες για καρκίνους και άλλα τέτοια όμορφα. Αντί να πούνε, πότε θα βγεί ένας θαυμαστός επιστήμονας που μεγάλωσε, εκπαιδεύτηκε, ζει και εργάζεται στην Ελλάδα;

Είναι σαν ένας κηπουρός να φυτεύει από τους ίδιους σπόρους, λίγους σε εύφορο έδαφος και τους υπόλοιπους στην ξεραΐλα, να κοιτάει αυτούς που ανθίζουν στο καλό χωράφι και να λέει: "Καλά ε, φοβεροί σπόροι". Για το ξερό χωράφι, μούγκα.

Α σιχτήρ συγχύστηκα μεσημεριάτικα.

Comments

Life goes on

Wed, 07 Jul 2010 21:02:47 +0200

As the song goes, "La la how the life goes on". I noticed that I haven't updated this blog since March so I'll the obligatory update on my affairs and apologise for the silence, promise I'll blog more often etc. So here goes:

Personal stuff

I've bought a flat! I can now do all kinds of crazy things, like dance around while code is compiling, really laugh out loud at internet jokes and really not giving a damn about what I wear or look like :)

I've also joined the Kifissia Youth Choir as a tenor and already took part in three concerts, I'm having a great time and met some wonderful people. I should've done this ages ago!

I still spend time doing voluntary work at the Greek Guide Association, with teenagers this time.

Work stuff

I'm still happily working with EDM Studio, creating a personalised visitor experience for schoolchildren visiting the Museum of Australian Democracy using RFID cards and crazy mesh technology. I got to pick and use cutting edge technologies like CouchDB and Cappuccino, but I don't think that was a very wise decision. Perhaps I'll find time to blog about it some day. There's also another huge project that we are consulting for, but I can't discuss details yet.

I'm still using Python, Twisted, PyObjC and Cocoa daily. I have various thoughts about how Python (and probably high-level languages in general) take away the pain of implementing what you want, while exposing the far deeper issue of actually finding out what you want. I may be a bit of a Python expert, but I still don't know a lot about graphics, async stuff and robust systems architecture. I am lucky enough to have a very experienced colleague who doesn't mind explaining the fundamentals again and again, so I'm getting better. Thanks Petr!

I find freelancing and working with a distributed team a very good fit for my lifestyle. I hope I can go on working like this!

I'm going to miss this year's EuroPython, but I hope next year I can go both to PyCon in San Francisco and to EuroPython in Firenze!

Misc

I've been slowly dropping out of Twitter - I've caught myself interrupting my workflow constantly, so I'm now checking it only the iPhone and via the web.

Till next time!

Comments

The promise pattern

Thu, 04 Mar 2010 10:14:35 +0200

After reading my William Reade's post about a small path library he wrote to tame a build script, I thought I'd share a cool trick that has helped me greatly in the current project - the promise pattern.

The code itself is greatly simple:

class FilePromise(object):

    def __init__(self, filename, basepath=None):
        self.basepath = basepath
        self.filename = filename

    def fetch(self):
        filepath = self.filename
        if self.basepath is not None:
            filepath = os.path.join(self.basepath, filepath)
        return open(filepath)

The cool thing about this is not the code itself, but the amazing flexibility it offers your code when instead of working with paths and files, you work with promises. All you need to do to support arbitrary data sources is create a class with a filename attribute (which might not make sense in some cases, but is useful for debugging) and a fetch method that will return a file-like object.

I have so far implemented additional promises that:

Fetch files from a CouchDB database
Decode embedded images in SVG
Crop and resize images (well not yet really, but plan to)

Why not use a path element like William does? In my case, I really do not care about the file system, I just care about contents. A promise also can easily be replaced for testing reasons - just create a small MemoryPromise that will return data passed in the constructor as a StringIO.

Why not pass around a file-like object? Two reasons - it's hard to debug (how do you print that?) but mainly, the fetch method might be expensive, and should be run near the point the contents will be used, not at construction time. Also, the intent of the code becomes clearer this way, I think.

The idea is stolen from Cocoa's drag-and-drop system, where a drag source can either supply the drag data directly to the drop target, or provide a promise the target will follow-up, if it's expensive to provide the data up-front. Hopefully Apple will not sue me. In drag-and-drop a lot of times a drag is initiated but not completed (user changes his mind) so this way an expensive calculation is avoided.

Something I haven't yet considered is how to embed this in an async program. Presumably an AsyncPromise fetch will return a deferred that will fire when the data is ready. The client code needs to be expecting this though. I've seen twisted's maybeDeferred for this kind of pattern, I'll follow up if I can do this.

Comments

Async needs syntax support

Fri, 19 Feb 2010 22:51:06 +0200

OH: "async won't take hold until it sucks less." It cannot suck less because we are writing async code in languages designed to be procedural, given that you read them top-to-bottom.

I am starting to get more familiar with Twisted and it is indeed very powerful. Even if I don't like it, I will never do any network or async stuff using threads any more. Other libraries may be nicer for web but when you want a variety of protocols Twisted is the only game in town.

The thing is, Python is a procedural language. You write commands top-to-bottom, and they are executed as such. When you want to encapsulate things running in parallel, or at a later time, you put them in a function and register that. There are more things you want to do - block until all results have come in, or fail early if one result fails and so on.

The best way to visualise this is to use some kind of visual support - think arrows flowing around and whatnot - but it will be clunky and hard to implement, and mind you us Vim users will hate it (of course there's going to be an emacs plugin soon ;).

So I don't think async will start to suck less soon, because the basic way we get to interact with it is just the wrong approach. I've no idea what a better approach would be though.

Comments

Innovative interfaces

Fri, 05 Feb 2010 16:55:04 +0200

Everyone talks about the new iPad and the revolution in interfaces that Apple brings upon us and I'm getting bit sick of it, even though I hope the iPad will be good enough so I can get my mom one (tech support sucks :). However, the specs page doesn't list Greek as a supported language, which got me thinking...

Mac OS X does not have Greek UI support at all. The local importer provides a patch that adds it but I never bothered with that, having literally grown up amidst English interfaces (I learned the correct pronunciation of many tech words when I moved to London for a year - until then, I've never heard anyone saying words like 'archive' out loud). You can install an aspell dictionary to get spell checking, and can write in Greek (thanks to Unicode) and that's pretty much it. Most of the nice fonts (Gill Sans, Futura) do not even have Greek glyphs.

However, the iPhone 3.1 OS, coinciding with Apple making deals with 2 of the big 3 telecom carriers in Greece supports a lot in Greek: UI, fonts, input dictionary and 3GS also has voiceover support (Hear a voice over of this post ). If the iPad supports a similar level of Greek, it's a sure thing my mom gets one.

What does this have to do with 'innovative interfaces'? I'll get to that, but first let me introduce you to the concept of accents in written Greek. An accent, called "tonos", is a diacritical mark that is positioned on a Greek words to denote on which syllable the stress goes to. It's only omitted for single-syllable words, for obvious reasons. Here is an example: "τόνος", the cute little ΄ there is the "tonos".

So far in my computing experience, DOS + Windows + Mac there's been a single way to enter those: Switch the keyboard to the Greek layout (a slightly modified US layout), hit the ';' button, then type the vowel to be accented. Mac OS X has a nice feature where it shows you the accent-to-be-inserted:

This means that if you forget to put an accent in, you need to delete the letter, then try again. Not exactly intuitive, but it's something you learn really quickly when typing Greek.

iPhone introduced a different mechanism: You type the vowel first, then you hit a dedicated accent button, and the accent is set:

I was very pleasantly surprised the first time I saw that. The reason? It's the way Greek children are taught to put accents when learning to write (the alternative would be absurd to think about, really). So someone at Apple, which as a company does not have a good track record of Greek localisation, sat down and thought about how entering accents in Greek could be better, and implemented it. I don't have to show my mom how to put accents on an iPhone, because it's in plain sight, and the first thing she'll try to do is going to work.

This is something us geeks how are really entrenched in the current way of doing things should really, really think about.

So why the iPad doesn't list Greek as supported? Probably because it takes time to do proper localisation, and Greece is a small market. If Apple doesn't sell the iPad through a carrier deal, it may never get Greek support. However, having seen this example of detail-oriented design, I hope they are just taking their time to do things properly. We'll see when the damn thing is actually released, so I promise this is the first and last time I mention it on this blog.

Comments

PyAthens 2nd meeting - results!

Wed, 11 Nov 2009 11:39:50 +0200

The results of the 2nd PyAthens meeting: 17 participants, 8 presentations, souvlaki & beer ending, what's not to like?

Αντιγράφω κατευθείαν το μήνυμα μου από τη λίστα:

Φαίνεται ότι ενθουσιαστήκαμε όλοι τόσο πολύ που έπεσε νέκρα στη λίστα :)

Ο Κώστας Αραβάνης από την "TasPython" έγραψε μια ωραία περίληψη την οποία αντιγράφω και συμπληρώνω. Συγχωρήστε με αν ξεχνάω κάτι!

17 συμμετέχοντες

Ξεκινήσαμε με 1 rant/flame του Αντώνη Χριστοφίδη σχετικά με το πως αλλάζει η συμπεριφορά του default encoding όταν γράφεις στην κονσόλα σε σχέση με όταν γράφεις σε ένα αρχείο μέσω redirection. Τελικώς βρήκαμε μια λύση μέσω του locale module.

Στη συνέχεια ο Βαγγέλης Μπαλάσκας μας μίλησε για το inotify/pyinotify και πως το χρησιμοποίησε για να συγχρονίζει δεδομένα στους server που διαχειρίζεται. Η παρουσίαση

Ο Θανάσης Πρίφτης και ο Ξενοφώντας Παπαδόπουλος μας μίλησαν για την προσπάθεια που γίνεται από το re-public με τα OLPC στα σχολεία, και πως ετοιμάζεται μια πλατφόρμα που να επιτρέπει τη γρήγορη ανάπτυξη εκπαιδευτικών εφαρμογών.

Ο Λευτέρης Κρητικός και ο Μιχάλης Πιτίρης μας μίλησαν για το foss.ntua και πως η ομάδα τους κάνει "πρακτική" δημιουργώντας ένα social μουσικό site, το MusicPool.

O Ορέστης Μάρκου μίλησε για τις διαφορετικές υλοποιήσεις της Python (CPython, Jython, IronPython, PyPy, Unladen Swallow), τι στόχους έχουν και σε ποιο στάδιο βρίσκονται. Η παρουσίαση.

Ο Παναγιώτης Κρανιδιώτης μίλησε για την προσπάθεια που γίνεται να εξελληνιστεί το OpenERP για να μπορεί να χρησιμοποιηθεί από ελληνικές επιχειρήσεις αντί του κτηνώδους SAP.

Ο Χρήστος Γεωργίου έκανε μια μικρή παρουσίαση του unicode και πως χρησιμοποιείται σε διάφορες γλώσσες προγραμματισμού και στην Python ειδικώτερα. Χωρίς διαφάνειες!

Την βραδυά έκλεισε πάλι ο Αντώνης Χριστοφίδης, μιλώντας μας για τις πατέντες λογισμικού και την προπαγάνδα που γίνεται στο να πειστούμε ότι είναι καλές και άγιες (δεν είναι!).

Στη συνέχεια μεταφερθήκαμε σε κοντινό σουβλατζίδικο, φάγαμε σουβλάκια και ήπιαμε μπύρες, και η κουβέντα περιστράφηκε σε διάφορα θέματα, όπως πως πρέπει να διδάσκεται ο προγραμματισμός σε μαθητές/φοιτητές, η εικόνα της αγοράς εργασίας στην Ελλάδα και το εξωτερικό και σίγουρα και άλλα τα οποία δεν έπιασα.

Εγώ προσωπικά το χάρηκα πάρα πολύ. Καιρός να προγραμματίσουμε το επόμενο, τι λέτε;

Ορέστης

Comments

Surprised by Python lambdas

Mon, 09 Nov 2009 12:32:36 +0200

The surprise

I haven't been surprised by Python in a long time, but today I got surprised by lambdas and how do they capture their surrounding state.

>>> def set(i):
...  print 'SET', i
... 
>>> l = []
>>> for i in range(3):
...  l.append(lambda: set(i))
... 
>>> l[0]
<function <lambda> at 0x7b0b0>
>>> l[0]()
SET 2

The explanation

I would have expected the value of i to be captured there and then, but it turns out that lambda only stores the name i. This is easily demonstrated by the following:

>>> i = 123
>>> l[0]()
SET 123

However, if this is done inside a function:

>>> def aaaa():
...  i = 321
...  l[0]()
... 
>>> aaaa()
SET 123

So lambdas capture the scope dictionary as well.

The solution

I found two workarounds. Both amount to storing the value of the required scope somewhere else:

def f(i):
...  return lambda: set(i)

from functools import partial
partial(set, i)

I think I'll go for the partial solution, just because I don't want to declare a new function just for this.

Comments

PyAthens 2nd meeting: 21 Oct 2009

Thu, 08 Oct 2009 14:20:14 +0200

Announcing the second meeting of the Athens Python User Group: Wednesday 21 October 2009, 19:00.

Με χαρά ανακοινώνω τη δεύτερη συνάντηση του Athens Python User Group. Η συνάντηση θα πραγματοποιηθεί την Τετάρτη, 21 Οκτωβρίου 2009, ώρα 19:00. Θα συναντηθούμε στην Πολυτεχνειούπολη Ζωγράφου, Κτίριο Υδραυλικής, 2ος όροφος, αίθουσα 206. Χάρτης.

Ο χώρος έχει όλες τις ανέσεις (πίνακας, μαρκαδόροι, projector, wireless). Προγραμματίζεται επίσης ΠΙΤΣΑ ΚΑΙ ΜΠΥΡΑ (αλλά πρέπει να βάλουμε ρεφενέ ;). Θα ακολουθήσει βραδυνή έξοδος σε μαγαζί της επιλογής μας, για όσους το επιθυμούν.

Όσοι σχεδιάζουν να κάνουν παρουσίαση, ας αφήσουν εδώ ένα σχόλιο για να προγραμματίσουμε το πρόγραμμα. Για παραπάνω συζήτηση, παρακαλώ γραφτείτε στη λίστα.

Με πυθωνικούς χαιρετισμούς, Ορέστης

Comments

Hiring: Python developers

Wed, 07 Oct 2009 17:05:07 +0200

At my current gig we need to expand the team. We're looking for good all-round Python developers, who are comfortable with learning new technologies. Send your resume and a few words about you at orestis@orestis.gr.

Why apply?

You will have the chance to work on an exciting project, based on an existing installation.
You will be able to see actual people using the software you write in real time.
You will be a member of a relaxed and fun team.
You will be working for a company that loves open source and prefers funding open-source projects instead of buying proprietary tools (though practicality beats purity).
You will be working for a company that scores at least 10/12 on the Joel test.

There are two positions we want to fill:

Lots of experience

We are developing user-facing applications that run 24/7/365 and we need someone who is at ease with complex systems, network communications, robustness and with attention to detail. Experience with Twisted will be considered a bonus.

Less experience

We also want someone who is not necessarily very experienced but who is smart and wants to learn as he goes along, as part of the team. Familiarity with Python is required, and experience with sys-admin stuff will be considered a bonus.

Other requirements

Access to a Mac for development. We use Cocoa/PyObjC a lot, and you should either be familiar with it, or be able to easily learn.
Ability to work from home. This is a globally distributed team.
Testing mindset. We strive to do Test Driven Development, although with practicality in mind.
Results oriented. Architect astronauts need not apply - we have a client to satisfy.
Friendly and communicative. Most of our communication is via chat, with the occasional Skype call.
Can learn fast. We want to use the best tool for the job, so we continuously evaluate and use new technologies.

This is going to be a full-time contracting job, lasting at least until February. If the conditions are right, it may evolve to something more long-term.

Give it a go!

If all this sounds interesting, send your resume and a few words about you at orestis@orestis.gr. I promise to reply to all candidates. If the team likes you, we'll have a Skype chat, where you would be requested to write some code snippets.

Comments

How to send an email with Twisted

Sat, 26 Sep 2009 13:32:23 +0200

After a long IRC chat with the kind Twisted folks on IRC, I am writing here a post that could easily serve as an overview page of twisted.mail.smtp. In my opinion, Twisted sorely lacks overview pages, with real words and not just links to examples. It also needs some Google love, so I hope this page will help others looking how to send emails with Twisted.

twisted.mail.smtp overview

Twisted mail contains implementations of many email-related protocols, such as POP3, IMAP, SMTP and others. It allows writing email clients and servers.

Twisted mail is not meant to be used by people who do not want to deal with protocol details. For example, it has nothing to do with MIME handling and relevant RFC2822 issues. There's a perfectly fine email library in the stdlib for that.

twisted.mail.smtp is an implementation of the SMTP protocol, client and server. However, here I will only deal with SMTP clients (ie, how to send emails).

How to send an email with SMTP

If you're using an open SMTP server, there's a convenience function you can use:

from twisted.mail.smtp import sendmail
from email.mime.text import MIMEText

host = 'smtp.example.com'
from_addr = 'sender@example.com'
to_addrs = ['recipient@example.com']

msg = MIMEText("This is the message body")
msg['Subject'] = 'This is the message subject'
msg['From'] = from_addr
msg['To'] = ', '.join(to_addrs)

dfr = sendmail(host, from_addr, to_addrs, msg.as_string())
def success(r):
    print r
    reactor.stop()
def error(e):
    print e
    reactor.stop()
dfr.addCallback(success)
dfr.addErrback(error)

reactor.run()

See the API Docs for sendmail for more information.

See the stdlib examples for sending more advanced messages with attachments, alternative versions and so on.

Authentication

Unfortunately, sendmail does not support authentication yet. However, it's very easy to send emails via a server requiring authentication, by using the ESMTPSenderFactory:

dfr = Deferred()
factory = ESMTPSenderFactory(username, password, from_addr, to_addrs,
    msg, dfr)
reactor.connectTCP(smtphost, port, factory)

# attach callbacks to dfr as before

There's a more complete example in the Twisted docs, completely ignored by Google so far, for some reason.

Advanced usage

If you are not covered by the simple sendmail function, or it's secure replacement, you probably know enough about email that you can get by using SMTPSenderFactory and ESMTPSenderFactory directly.

Visit the API Docs for twisted.mail.smtp

Mistakes? Omissions?

I'm not a Twisted dev so perhaps I've misunderstood something. Please send feedback via comments.

License

I don't have an explicit copyright policy for my blog posts, but since I want this to end up in the Twisted docs, I am releasing this post under a Creative Commons Attribution-Share Alike License.

As far as I understand it, it means that I only want an attribution when this documentation is used. If that's not the case, please educate me on licensing in the comments.

Comments

Twisted reactor in 60 seconds

Fri, 25 Sep 2009 10:59:59 +0200

In the spirit of the wonderful "Twisted web in 60 seconds" posts, I'm going to contribute here a small post about the Twisted reactor and Deferreds. I know this can be found on the Twisted examples page, but writing it from scratch was helpful for my understanding as well :)

A mainloop

The most usual problem when trying to understand the reactor is context. In my case, the penny dropped when I saw the reactor in the context of a plain desktop application - not a script, not a web server.

Desktop applications are built around and event loop - the user moves the mouse, presses keys, the windowserver captures those events and calls various event handlers on your application.

Twisted reactor is a similar thing:

from twisted.internet import reactor

reactor.run() # this blocks!

After you call reactor.run, your program has an event loop, and the reactor takes over. When embedding Twisted in desktop applications, you need to install a reactor compatible with your main loop (gtk, cocoa and so on).

Deferreds

Event handlers in twisted parlance are called "deferreds". When you ask twisted to do something asynchronously, be it fetching a web page, sending an email, resolving a DNS name, connecting to a server, you need a way to know when that action succeeded (or failed, of course.)

Twisted deferreds can support chaining the results. More on this on a later example, though.

An example

Think about the canonical example when doing desktop application programming. You want to fetch some information from a webserver (perhaps to check for updates). Normally you would spawn a thread, fetch the page, process it, then update your user interface (of course taking care to update the UI from the GUI thread and other similar arcane issues).

In twisted there is only a single thread, and everything is set up using deferreds and the reactor. Let's write some simple code to do that:

We'll need to import the reactor and a getPage function:

from twisted.internet import reactor
from twisted.web.client import getPage

This is a desktop application, so here's a small function to update the UI:

def updateUI(message):
    print "UI:", message

Let's write our processing function:

def processPage(pageContent):
    print 'got the page'
    # process the page here
    # ...
    # update the UI
    updateUI('SUCCESS')

Remember, this will get called when twisted has fetched the page. Its return value will be passed on to the next callback down the chain. Here, we just return the length of the page.

Let's also write an error handler:

def handleError(error):
    print 'got error'
    # update the UI
    updateUI('Whoops! %s' % error)

Having written all the pieces, let's bring them together:

pageFetchedDeferred = getPage("http://orestis.gr")
pageFetchedDeferred.addCallback(processPage)
pageFetchedDeferred.addErrback(handleError)

Finally, we need to start the reactor. In a desktop app, you would've done that at the final stage of initialisation, but this is an example, right?

reactor.run()

Here's the whole program:

from twisted.internet import reactor
from twisted.web.client import getPage

def updateUI(message):
    print "UI:", message

def processPage(pageContent):
    print 'got the page'
    # process the page here
    # ...
    # update the UI
    updateUI('Received %d bytes' % len(pageContent))

def handleError(error):
    print 'got error'
    # update the UI
    updateUI('Whoops! %s' % error)

pageFetchedDeferred = getPage("http://orestis.gr")
pageFetchedDeferred.addCallback(processPage)
pageFetchedDeferred.addErrback(handleError)

reactor.run()

Remember, you need to kill your program with Ctrl-C because the reactor.run call blocks!

Conclusion

If you understand how deferreds and the reactor works, you've made an important step on using Twisted. As I use and understand Twisted more, I'll keep writing!

Comments

Athens Python UG - 1st meeting results

Mon, 21 Sep 2009 14:31:47 +0200

The Athens Python User Group had its first meeting on 9/9, at the Eleftheroudakis bookstore Café. This is a post describing what happened and what are the next steps.

Μετά την ανακοίνωση για τη δημιουργία ενός Python User Group (PUG, για συντομία), και τα 20 σχόλια που ακολούθησαν, έμενε να φανεί η ανταπόκριση των Πυθωνιστών εν Αθήναις.

Προσωπικά ο στόχος μου ήταν να μαζευτούν 5 άτομα, να πιούμε ένα καφέ και να κουβεντιάσουμε, για να δούμε πως θα κινηθούμε από εκεί και πέρα. Ο στόχος αυτός επετεύχθη, καθώς τελικά στο τραπέζι βρεθήκαμε έξι! Ο Θάνος Λευτέρης, ο Χρήστος Γεωργίου, ο Τάσος Κουτσοβασίλης, ο Γιώργος Παγκλές, ο Αντώνης Χριστοφίδης και εγώ περάσαμε 2 ώρες περίπου συζητώντας πως έγινε και καταπιαστήκαμε με την Python και διάφορα θέματα όπως το unicode, test-driven development, και γενικώς τη χαρά του να προγραμματίζεις σε Python.

Τελειώσαμε τη συνάντηση με την επιθυμία να ξαναβρεθούμε, να είμαστε περισσότεροι και να γίνουν και κάποιες παρουσιάσεις. Επίσης, ΠΙΤΣΑ ΚΑΙ ΜΠΥΡΑ. Ο Αντώνης προσφέρθηκε να κλείσει κάποιο χώρο στην Πολυτεχνειούπολη που θα μπορεί να συμβεί αυτό. Συζητήσαμε αν πρέπει να συμβεί καθημερινή ή σαββατοκύριακο, πόση ώρα να είναι και πως θα οργανωθεί. Έπεσε και μια πιθανή ημερομηνία, 18 Οκτωβρίου (δυστυχώς κατόπιν διαπίστωσα ότι δεν μπορώ τότε!).

Για να γίνει περαιτέρω συνεννόηση, ζήτησα από τον postmaster του python.org να δημιουργηθεί μια λίστα για το APUG, πράγμα που είναι πλέον γεγονός: Η λίστα του PyAthens πλέον λειτουργεί! Όποιος ενδιαφέρεται ας γραφτεί. Ανακοινώσεις για συναντήσεις θα ανεβαίνουν σε αυτό το blog, και θα μπαίνουν και στη wiki σελίδα του APUG.

Γραφτείτε στη λίστα, και διαδόστε το!

Comments

Backwards compatibility is a straw man

Wed, 16 Sep 2009 16:50:20 +0200

I recently signed up to stdlib-sig so I could just nod in agreement to the people that suggested that the stdlib needs to evolve. In the discussions that ensued, the backwards compatibility argument came up often. I think it's not a valid argument for the specific discussion, though. Here are my thoughts.

What is stdlib?

The standard library is a set of packages and modules that get shipped by default with the Python interpreter. Despite what I thought when I was starting with Python, it doesn't represent the so-called "best-practices" modules. There is no overarching design that dictates why things are the way they are. Instead, it just represents a set of modules that have been picked up at some point in time. Some of them are close to the language (such as collections, itertools and others) and some are domain-specific tools. The difference is significant.

Here is an attempt to divide the ones visible at the docs. Of course, I'm choosing here based on personal preference, YMMV. Here we go:

Language

String Services:

string, re, struct, StringIO, cStringIO, codecs, unicodedata

Data types:

datetime, collections, heapq, bisect, array, sets, sched, mutex, queue, weakref, UserDict, UserList, UserString, types, new, copy, pprint, repr

Numeric and Mathematical Modules:

numbers, math, cmath, decimal, fractions, random, itertools, functools, operator

(why itertools, functools and operator are here is beyond me)

File and Directory Access:

os.path, stat, statvfs, filecmp, tempfile, glob, fnmatch, shutil

Data Persistence:

pickle, cPickle, copy_reg, shelve, marshal

Data Compression and Archiving:

zlib, gzip, bz2, zipfile, tarfile

this category can argued to be both domain-speficic and close to language.

Cryptographic Services:

hashlib, hmac, md5, sha

Generic Operating System Services:

os, io, time, getpass, platform, errno, ctypes

Optional Operating System Services:

select, threading, thread, dummy_threading, dummy_thread, multiprocessing, mmap

Interprocess Communication and Networking:

subprocess, socket, ssl, signal, popen2, asyncore, asynchat

asyncore and asynchat can be said to be domain specific, but async io is fundamental IMO

Internet Data Handling:

base64, binhex, binascii, uu

these should probably live alongside codecs

Internet Protocols and Support:

wsgiref, uuid

wsgi is meant as an interop protocol, so I put it close to the language.

Internationalization:

gettext, locale

Development Tools:

pydoc, doctest, unittest, 2to2, test

Debugging and Profiling

bdb, pdb, hotshot, timeit, trace

Python Runtime Services

sys, __builtin__, future_builtins, __main__, warnings, contextlib, abc, atexit, traceback, __future__, gc, inspect, site, user, fpectl

Importing Modules:

imp, imputil, zipimport, pkgutil, modulefinder, runpy

Python Language Services:

parser, ast, symtable, symbol, token, keyword, tokenize, tabnanny, py_compile, compileall, dis, pickletools, distutils, pyclbr, compiler

a lot of these arguably are domain-speficic, but given the domain is Python...
the compiler package is included here as well

Domain specific

String Services:

difflib, textwrap, stringprep, fpformat

Data types:

calendar

File and Directory Access:

fileinput, linecache, dircache, macpath

Data Persistence:

anydbm, whichdbm, dbm, gdbm, dbhash, bsddb, dubmdbm, sqlite3

File Formats:

csv, ConfigParser, robotparser, nterc, xdrlib, plistlib

Generic Operating System Services:

optparse, getopt, logging, curses, curses.*

Optional Operating System Services:

readline, rlcompleter

Internet Data Handling:

email, json, mailcap, mailbox, mhlib, mimetools, mimetypes, MimeWriter, mimify, multifile, rfc822, quopri

Structured Markup Processing Tools:

HTMLParser, sgmllib, htmllib, htmlentitydefs, xml.*

Internet Protocols and Support:

webbrowser, cgi, cgitb, urllib, urllib2, httplib, ftplib, poplib, imaplib, nntplib, smtplib, smtpd, telnetlib, urlparse, SocketServer, BaseHTTPServer, SimpleHTTPServer, CGIHTTPServer, cookielib, Cookie, xmlrpclib, SimpleXMLRPCServer, DocXMLRPCServer

Multimedia Services:

audioop, imageop, aifc, sunau, wave, chunk, colorsys, imghdr, sndhdr, ossaudiodev

Program Frameworks

cmd, shlex

GUI with Tk:

Tkinter, Tix, ScrolledText, turtle, IDLE, Others

Custom Python Interpreters:

code, codeop

Restricted Execution:

rexec, Bastion

Both have been removed from Python 3.0

Miscellaneous Services:

formatter

MS Windows Specific Services:

msilib, msvcrt, _winreg, winsound

Unix Specific Services:

posix, pwd, spwd, grp, crypt, dl, termios, tty, pty, fcntl, pipes, posixfile, resource, nis, syslog, commands

Mac OS X specific services:

ic, MacOS, macostools, findertools, EasyDialogs, Framework, autoGIL, ColorPicker

MacPython OSA Modules:

gensuitemodule, aetools, aepack, aetypes, MiniAEFrame

SGI IRIX Specific Services:

al, AL, cd, dl, DL, flp, fm, gl, DEVICE, GL, imgfile, jpeg

SunOS Specific Services:

sunaudiodev, SUNAUDIODEV

GRAND TOTAL

130 language-related

151 domain-specific

Damn isn't that a lot of packages. For reference, PyPI currently hosts ~7500 of them. Truly, Python has a lot of batteries.

A platform, or a framework?

I can easily see a neat split there - the first half is Python, the platform. The second half are the batteries. However nowadays the batteries are not enough. While you may be able to write a quick and dirty script with them, if you're doing web stuff you're probably using another framework, if you're doing desktop stuff you're probably using another toolkit as well. Of course, there are other uses I probably don't know nothing about, and for them Python becomes the framework.

I know that many frameworks built on top of Python-the-platform start with the batteries, and then they start writing their own implementations to fix bugs or add features. Django-the-framework runs on Python-the-platform 2.3-2.6 so it can't rely on features being present or bugs fixed in the batteries - it has its own.

Backwards compatible

My issue with the backwards compatibility argument is this: No one forces anyone to update to any version of Python. Developers make a conscious decision - to develop software for a specific (or a range of) version of Python, and specific versions for all the other libraries they depend on. Any change to the dependencies of a piece of software may lead to breakage. I see no reason why Python should be different for that purpose.

I can't see backwards compatibility as an argument against upgrading Python, adding features, deprecating and removing modules, and of course fixing bugs. (Aside: Microsoft is so backwards compatible so as to emulate bugs if important programs need it. We don't want to do that!). Instead, I see backwards compatibility as an argument for better isolation of Python-the-framework. If a program needs specific versions of Python and libraries, it should be trivial to guard them against change. If an operating system depends on a specific version of Python, it should hide it away and not allow modifications.

On the other hand, I would argue against radical changes to Python-the-platform. Of course, this has been the case so far, with one exception in Python 3.0 to fix issues that needed to be fixed. In fact, there's a nice forwards-compatible feature for changes to the platform - __future__. People have been upgrading to new Python features with minor complaints, so I don't see why changing the batteries part of stdlib is tickling people so much.

Best of breed

When I started with Python, I only used modules from stdlib - I had no idea about PyPI, and I assumed that things from python-core would be more high-quality. However, this is only true for the language modules, not the domain specific modules. The reason is simple - python-core are experts on Python and language design, but not experts on the numerous domains the batteries cover. There are now replacements for most, if not all (os-specific stuff probably excluded) domain-specific modules. People trying to get a GUI running with Tk and not knowing about wx, Qt, Gtk, or the platform-specific choices is bad. People trying to do image manipulation and not knowing about PIL is bad.

I would argue that domain-specific parts should be spun off the stdlib and be released as separate PyPI modules. We can keep Python-the-framework going by having a download with the kitchen sink provided (as Jesse Noller proposed), and cooperate with packagers/distributions so that they can fortify their installations against change.

Conclusion

The argument on stdlib-sig is huge, and thankfully it seems that something is getting done in the end. I expect a some people to agree with me, and some to disagree. Writing my thoughts makes me think, so please keep in mind that I am willing to be persuaded otherwise, with the correct arguments.

As far as my day to day use is concerned, 99% of the batteries could disappear from my site-packages, and I would not care. Of course, packages I actually use and import (twisted, pyobjc) would care (actually, both of those will most likely use their own batteries). I wonder for how many people is this situation familiar. Find your imports, and see what the results are.

Comments

Athens Python User Group Venue

Fri, 04 Sep 2009 19:16:22 +0200

We have a venue for the first meeting of the Athens Python User Group: See you at the Eleftheroudakis Bookstore café, Panepistimiou 17, 6th floor, on Wednesday 9 September, 7.00pm.

Η συνάντηση για καφέ μεγάλωσε, και απέκτησε και τόπο! Ραντεβού στην καφετέρια του βιβλιοπωλείου "Ελευθερουδάκης", Πανεπιστημίου 17, 6ος όροφος. Ημέρα Τετάρτη, 9 Σεπτεμβρίου και ώρα 7 το απόγευμα.

Θα έχουμε ένα τραπέζι στη διάθεση μας, κλεισμένο στο όνομα Ορέστης Μάρκου. Διαδώστε το!

Comments

Homebrew: New Mac OS X package manager

Fri, 04 Sep 2009 15:12:43 +0200

Homebrew is a new packager for Mac OS X. I installed it, and hey, it works! I tried it on a fresh (clean install) Snow Leopard machine.

The installation instructions on git are a bit convoluted, so here's the abridged version:

1. Install XCode
2. sudo mkdir /usr/local
3. Drop the contents of the download in /usr/local 
4. sudo chown -R `whoami`:staff `brew --prefix`

I installed git this way: brew install git. Here's the output (notice how I didn't chown the first time round):

macbook:~ orestis$ brew install git
==> Downloading http://kernel.org/pub/software/scm/git/git-1.6.4.2.tar.bz2
######################################################################## 100.0%
==> Error: Permission denied - /usr/local/Cellar
macbook:~ orestis$ brew --prefix
/usr/local
macbook:~ orestis$ sudo chown -R `whoami`:staff `brew --prefix`
macbook:~ orestis$ brew install git
==> Downloading http://kernel.org/pub/software/scm/git/git-1.6.4.2.tar.bz2
File already downloaded and cached
==> ./configure --prefix=/usr/local/Cellar/git/1.6.4.2
==> make install
==> Downloading http://kernel.org/pub/software/scm/git/git-manpages-1.6.4.2.tar.
######################################################################## 100.0%
==> Finishing up
/usr/local/Cellar/git/1.6.4.2: 379 files, 15M, built in 70 seconds

Neat. Next I will try grabbing software directly from source (I like to custom build macvim and git, for example). I'll update this post with news.

Comments

Athens Python User Group

Tue, 01 Sep 2009 21:19:44 +0200

UPDATE: We now have a venue!

This is an announcement for the just-created Athens Python User Group. The 1st meeting will take place on Wednesday 9 September, 7:00 pm. Venue: Eleftheroudakis Bookstore café, Panepistimiou 17, 6th floor!

Σε προηγούμενο ποστ ανέφερα την ιδέα για ένα Python user group στην Αθήνα. Ο ΤΖΩΤΖΙΟΥ ενδιαφέρθηκε, και τώρα που μαζεύτηκα από τις διακοπές, έκλεισα ραντεβού για καφέ.

Αποφάσισα λοιπόν να βαφτίσω αυτόν τον καφέ την "1η συνάντηση του Athens Python User Group" και να καλέσω όσους φίλους της Python βρίσκονται στην Αθήνα (ή μπορούν να πεταχτούν) να γνωριστούμε και να συζητήσουμε ό,τι μας κατέβει στο κεφάλι σχετικά με την Python και όχι μόνο.

Το ραντεβού ορίστηκε για την Τετάρτη, 9 Σεπτεμβρίου και ώρα 7 το απόγευμα. Θα βρεθούμε στην καφετέρια του βιβλιοπωλείου "Ελευθερουδάκης", Πανεπιστημίου 17, 6ος όροφος

Όποιος ενδιαφέρεται, ας αφήσει ένα σχόλιο εδώ να ξέρουμε πόσοι είμαστε.

Με πυθωνικούς χαιρετισμούς, Ορέστης :)

Comments

EuroPython 2009

Fri, 03 Jul 2009 17:58:43 +0200

EuroPython is now over, so it's time for me to write down my thoughts and impressions!

The venue

I have been at the same venue, the Birmingham Conservatory/Music school last year at PyCon UK '08, so the place was familiar. This year however the weather was unusually hot and humid and it was obvious that the building wasn't designed to handle that kind of conditions. And of course geeks are not famous for their love of hygiene! (And Greeks are famous for whining about any other kind of weather :)

The crowd

This year there were around 450 delegates from 20something countries - I can't be bothered to look it up. The venue felt a bit too small for that number of people, and some rooms overflowed. Still, nothing that put me off - I have no trouble sitting on the floor for an interesting talk.

Notably missing were the US delegates - at least that's the feeling I have from looking at name tags. This had the effect that there was an absence of "famous" people. Perhaps my expectations were set too high by last year's PyCon UK, where there were at least 3 core python devs around. It may be that months after PyCon '09 it would be difficult for people to go to another conference. There was a plan to bring Guido virtually via a video conference, but two attempts that were made failed - perhaps for the better given that some questions that managed to squeeze through were not that interesting.

The community

A big concern of mine this period is the community. After moving back to Athens, I really miss the interactions that my colleagues and the various user groups provided. I have been always thinking about starting a group in Athens, and I was really pleased by both the unconference/open spaces discussion that was held, and by realising that the PSF is willing to fund such efforts. Once work-related things relax a bit, and the famous Greek summer is over I will try to get the ball rolling.

My talk

My introduction to PyObjC talk went well, I think. About 25 people showed up at what had to be the hottest room in the whole Birmingham. We had two little fans that were pushing hot air around. We even had some questions by people who did not know me, which is good! I think a lot of people were disappointed to find out that you can't develop iPhone apps with Python though.

This and that

I don't know if I'm spending too much time reading blogs and following people on Twitter, but I couldn't find enough talks that really jumped out for me. From the ones I attended, Simon Willison's talk about Crowdsourcing with Django stood out, first because Simon is a really good talker (it'd be interesting to find out how much he prepares for his talks), and also because the first project I ever made with Django was a crowdsourcing app for gathering Greeklish word transliterations. (Aside: A top 5 is a great idea, and I wish I had come up with a progress bar back then). Also, a big thanks to Greg Holling for his Ctypes talk!

I feel that there were not enough talks that covered things like that - very useful, basic libraries that you can't easily approach on your own, or that have to be demonstrated to you in order to make sense. I would love for example a pdb talk or an introduction to Twisted talk. I think that next year I will propose a 'Python Dev Environments' talk. There are a lot of people out there that don't know about virtualenv, pyflakes, tabnanny and a lot of other things that make your life easier.

I was drawn at the closing prize draw, and picked the much-coveted (and sold out) Natural Language Processing with Python book. Last year I won the 'Best Lightning Talk' award (an Xbox 360 ), so it seems Birmingham is turning out to be quite a profitable venue :)

I also bought The Geek Atlas and Head First Web Design, from the O'Reilly stand. I also bought another 4-5 books from amazon, which I hope to read and review during the summer.

I met some wonderful people that unfortunately did not have any cards made, so I hope they will email me instead - I has some nice little MOO cards made and I gave out a lot of them. I also learned a fantastic card game called "Durac" (or something like that) that is apparently the official PyPy game. Or as Carl put it "the only game that is consistently played at PyPy sprints". Trust a German PHD student to be as accurate as possible in his definitions :)

I tried to participate on the sprints on Friday morning but my brain doesn't really function after drunken nights, so my attempt to refactor (almost rewrite) PySmell failed miserably.

Closing thoughts

My biggest gain from Europython '09 is a renewed vigor to help organise the Greek Python community. It's a shame that there were only two Greek developers there, one being Nikolaos Papagrigoriou who I met by accident in a talk, the other being me. (Incidentally, we were not marked as Greek in the delegates list, because we used German and UK addresses when booking). I hope to remedy this for the next one :)

I think we need more invited speakers for the next EuroPython. Self-proposed talks can only get you so far - perhaps some way for the participants to highlight their interests when signing up? Or just drop all talks and organise an unconference? Though for 450 people it might be tricky.

A big thank you to the organisers for pulling this off in their spare time, and I hope I'll see you again next year in Birmingham!

Comments

Git is user friendly!

Wed, 17 Jun 2009 17:22:45 +0200

I'm always amazed at the amount of people that don't know that git comes with two very useful graphical utilities bundled. Here are two screenshots.

Git Gui

Git gui allows you to review the current diff, interactively add and remove hunks/lines/files from stage, commit your changes and amend the last commit. It can also be run whenever you have merge conflicts, however you still need to use an external merge tool to resolve them.

Gitk

Gitk is a bit more complicated, as it has a lot of features:

browse history (view every commit of every branch, including diffs)
manage branches (checkout, create, delete, reset branches)
search history (a bit hard to use though)

For free!

Both of these tools are included and work in both the msysgit installation for Windows and when building from source on Mac OS X (and presumably linux). On Mac OS X, you get nicer results in gitk if you have built tk with quartz support (else X11 will be launched everytime). Macports has a working variant that works out of the box.

There are also other tools, mainly for Mac OS X, such as GitX and GitNub that are a lot prettier, but I've found that both git gui and gitk are working for me so I have no desire to switch yet.

Tips

I have defined two functions for git gui and gitk so that they are launched without blocking:

function gitg
{
  command git gui &
}
function gitk
{
  command gitk --all &
}

The --all switch to gitk makes it display all the branches, instead of just the current one.

Hopefully people will stop claiming git lacks graphical tools. It might be more difficult to initially grok than say, Mercurial, but it doesn't lack the tools!

Comments

Looking for a top Pythonista (contracting)

Sun, 14 Jun 2009 18:03:22 +0200

My employer is looking for an excellent Pythonista to fill a contracting position, part-time or full-time.

The working environment is relaxed, we have automated build and deployment scripts, a buildbot instance integrating, we use SVN and Trac for issue tracking, Basecamp and Campfire for communication. Our applications are experienced by thousands of people weekly, visiting the Museum of Australian Democracy.

Teaser pic:

Things we're looking for:

experience in Python
experience in developing desktop applications in general
ability to work from home as part of a small distributed team
access to a Mac
ability to be able to learn new technologies fast
a love for testing

Things that will be a bonus:

experience with PyObjC
experience with Twisted
TDD experience

In short, we look for an experienced, laid-back, cool all-round Pythonista! If you're interested, send a brief bio, availability and sought-after hourly rate to darran@edmstudio.com.

While this is a short-term position, if you like us and we like you there is the option of extending it to a long-term one.

Note: only potential fitting candidates will be contacted.

Comments

Lies, bloody lies and statistics

Mon, 08 Jun 2009 11:41:08 +0200

Having read The Tiger That Isn't, I never trust statistics of any kind.

So let's examine the voter turnout statistic being paraded around from yesterday evening, from the moment the polls closed. Major news outlets put it at 48% percent, which has huge political implications - nearly half the voters didn't bother to go to the polling station to cast their vote. Of course, this will be interpreted according to the needs of the parties/news station affiliations.

How did this number appear? The official source, Ministry of Interior gives (greek link):

Registered 9,994,849
Voted 52.63 %

Doing the math, 5,260,289 people voted.

Immediately, one thing jumps out: almost 10 million registered voters! The ballpark figure for the population of Greece is "around 10 million", so it would appear there are no under-18s in Greece any more. Let's get a more accurate number.

Directly from the National Statistical Service, the 2001 census puts the population of Greece at 10,964,020. However, this includes non-citizens, who can't cast a ballot. Drilling down a bit more, this pdf (in greek) gives us the breakdown of people by nationality and age.

So, Greek citizens of all ages: 10,171,906. We also have 490,965 children aged 0-4, 502,291 aged 5-9, 540,553 aged 10-14 and 666,539 aged 15-19. Given that a citizen has the right to vote when he is 18 years old, and from this pdf (in greek) we see that the breakdown of ages in the 15-19 group is pretty much constant, we can say that the 15-17 age group is roughly 333,000. This gives us a total of roughly 1,850,000 children that can't vote, putting the voting population at 8,320,000.

How does that change the turnout? From 52.63 % it now becomes 63.22 %, meaning that the absent percentage is at 36.78%, much less impressive. In absolute numbers, 6.3 million voters turned up in the 2004 EUP elections, versus 5.2 million in the 2009 ones. So indeed the absentees increased, but from that fact to huge headlines about how 1 in 2 doesn't care/trust the politicians and the political system is a huge leap of statistical blundering. I recommend reading The Tiger That Isn't, it provides a very useful guide about how to challenge statistics that are usually an attempt to misguide.

So to sum up, the actual voter turnout was 63.22%, two thirds of the voting population do care enough to go to vote on a very hot day instead of lying down on a beach recliner. May I ask again why we don't vote by mail/absentee ballot? Lots of other countries do it, so presumably there's a way to solve the logistic problems. In addition, why is it a tradition to hold elections on Sundays in Greece? Weekends are for resting, not for dragging yourself to a voting station. Give people some time off on a Wednesday, make them vote on their place of residence (instead of the municipality they are registered at, which for some idiotic IMO reason can be different). Remove the obstacles.

Of course, after I finish this post, I find this article (greek) that pretty much states the same thing. So there you go.

Comments

Complicated laws don't work (Greek)

Sat, 06 Jun 2009 14:24:52 +0200

This is a post about the soon-to-be-introduced in Greece smoking ban and why its complexity will make it fail. Here's a Google translation

Μετά από πολύ καιρό, ένα ποστ στα Ελληνικά, για την απαγόρευση του καπνίσματος και πως η πολυπλοκότητα του νόμου θα δυσκολέψει (αν όχι αναβάλλει) την εφαρμογή του.

Διάβασα στην Ελευθεροτυπία σχετικά με την απαγόρευση του καπνίσματος και τις πολλές διατάξεις του νόμου, αρκετές από τις οποίες δεν έχουν ακόμα διευκρινιστεί, ένα μήνα πριν την υποτιθέμενη εφαρμογή του νόμου.

Προσωπικά συμφωνώ απόλυτα με την γενική απαγόρευση του καπνίσματος σε κλειστούς χώρους, ακόμα και αν ο ίδιος καπνίζω. Η εμπειρία μου στην Αγγλία δείχνει ότι αρκετός κόσμος (εμού συμπεριλαμβανομένου) κόβει ή μειώνει το τσιγάρο, καθώς η απαγόρευση μετατρέπει μια κοινωνική συνήθεια σε μια αγγαρεία εξάρτησης. Αυτός νομίζω πρέπει να είναι και ο σκοπός κάθε τέτοιου νόμου - όχι στην "τιμωρία" των καπνιστών αλλά στην εξώθηση τους να μειώσουν το τσιγάρο, προστατεύοντας βεβαίως την υγεία όλου του πληθυσμού.

Δυστυχώς όμως, αυτός ο νόμος είναι καταδικασμένος στην αποτυχία, όχι λόγω του 'Ελληναρά' που δηλώνει ότι θα καπνίζει όπου γουστάρει. Μπορεί να μην είμαστε 'Εγγλέζοι' στην υπακοή στους νόμους, αλλά δεν είμαστε και τόσο τραγικοί όσο φερόμαστε. Αν σήμερα ανάψει κάποιος τσιγάρο σε νοσοκομείο ή σε δημόσια υπηρεσία με κόσμο, σίγουρα θα ενοχλήσει. Σύμφωνα με τους κοινωνιολόγους [citation needed], το τι είναι αποδεκτό σε μια κοινωνία φαίνεται από το πόσο ενοχλεί, όχι με τι συχνότητα γίνεται - πάντα υπάρχουν παραβάτες.

Πολυπλοκότητα και διαφθορά

Αυτός ο νόμος είναι καταδικασμένος στην αποτυχία γιατί είναι πολύπλοκος. Ακόμα χειρότερα, αφήνει τεράστια παράθυρα και ευκαιρίες διαφθοράς. Παραθέτω:

Αν όμως το μαγαζί (εστιατόριο, καφέ, μπαρ ή κέντρο διασκέδασης) είναι εμβαδού μέχρι 70 τ. μ. τότε ο ιδιοκτήτης μπορεί να το κάνει καπνιστήριο.

Ωθεί τους καταστηματάρχες να δηλώνουν λιγότερα τετραγωνικά και να κανονίζουν με τις υπηρεσίες του δήμου να παίρνουν την άδεια. Χάνει λοιπόν και ο δήμος έσοδα από τα δημοτικά τέλη.

Αν είναι μεγαλύτερο θα πρέπει να φτιάξει μέσα σε αυτό καπνιστήριο αλλά ακόμα δεν έχει τις προδιαγραφές στα χέρια του.

Προλαβαίνουν οι καταστηματάρχες όλης της Ελλάδας να το κάνουν αυτό σε ένα μόνο μήνα; Έχουν κεφάλαιο; Είναι διατεθειμένοι να το επενδύσουν σε έργο που δεν θα τους επιφέρει κανένα κέρδος; Μάλλον όχι. Συνεπώς αρκετοί θα περιμένουν το πρόστιμο, και την ευκαιρία να το αποφύγουν καταλλήλως.

Στους χώρους εργασίας τώρα - ιδιωτικούς και δημοσιους - ο εργοδότης μπορεί επίσης να φτιάξει αδιευκρίνιστους χώρους για τους καπνιστές καθώς το υπουργείο δεν έχει ακόμα προκαθορίσει πώς πρέπει να είναι αυτοί.

Επίσης. Ακόμα χειρότερα, ο εργαζόμενος δύσκολα μπορεί να διαμαρτυρηθεί, γιατί φοβάται τις συνέπειες. Τουλάχιστον στα μαγαζιά οι πελάτες ψηφίζουν με το πορτοφόλι τους και μπορούν να καταγγείλουν παραβάσεις.

Οπως αναφέρει ο νόμος σε "νηπιαγωγεία, παιδικούς σταθμούς, βρεφονηπιακούς σταθμούς, παιδότοπους, σχολεία πρωτοβάθμιας και δευτεροβάθμιας εκπαίδευσης, φροντηστήρια, στα κτίρια των δημοσιων υπηρεσιών, νομικών προσώπων δημοσίου δικαίου, νομικών προσώπων ιδιωτικούς δικαίου που εποπτεύονται και επιχορηγούνται από το κράτος, οργανισμών και άλλων ιδιρυμάτων (Ο.Τ.Ε, Δ.Ε.Η, ΕΛ.ΤΑ κλπ.) στα νοσηλευτικά ιδρύματα, ιδιωτικές κλινικές, κέντρα υγείας, ιατρεία νομικών προσώπων δημοσίου ή ιδιωτικού δικαίου (ΙΚΑ κλπ) ορίζονται ειδικοί χώροι (καπνιστήρια) στους οποίους θα υπάρχει ισχυρή εγκατάσταση συστήματος εξαερισμού για τους εργαζόμενους καπνιστές".

Δηλαδή ο δάσκαλος, η γιατρός, ο νοσοκόμος, ο υπάλληλος κοκ θα φεύγει απ' τη δουλειά του να πηγαίνει σε ένα χώρο κλειστό μαζί με άλλους 5, θα ντουμανιάζει και θα βρωμοκοπάει τσιγαρίλα και μετά θα έρθει να αγκαλιάζει παιδιά, να φροντίζει ασθενείς κλπ. Επίσης, πως ορίζεται η 'ισχυρή εγκατάσταση εξαερισμού', ποιος θα την ελέγχει, και ποιος θα την πληρώσει;

Συνεπώς, ο νόμος θα ευνοήσει:

Τους ελέγχοντες υπαλλήλους και τους εισπράττοντες τα πρόστιμα
Τους εμπόρους/εγκαταστάτες συστημάτων εξαερισμού
Τους τζαμάδες, που λέει και ο πιτσιρίκος

Και θα ζημιώσει:

Τους καταστηματάρχες, που θα πληρώσουν για να μετατρέψουν το μαγαζί τους
Τον Έλληνα φορολογούμενο, που θα κληθεί να πληρώσει τους μισθούς 'ελεγκτικών κλιμακίων' και 'μικτών επιτροπών', οι οποίοι θα έχουν και την ευχέρεια να 'κανονίζουν' τα πρόστιμα.

Απλότητα και διαφάνεια

Σε αντιδιαστολή, ο νόμος περί απαγόρευσης του καπνίσματος στην Μ. Βρετανία έχει ως εξής (κρίνοντας από την εφαρμογή του, δεν έχω διαβάσει το νόμο):

Απαγορεύεται το κάπνισμα σε όλους τους δημόσιους κλειστούς χώρους.

Συνεπώς, το κάπνισμα επιτρέπεται:

Στους ιδιωτικούς χώρους (οικίες)
Στους ανοιχτούς χώρους

Τελεία και παύλα. Συνεπώς, ως καπνιστής ξέρεις ότι εφόσον δεν είσαι σε ανοιχτό χώρο ή σπίτι σου, δεν μπορείς να καπνίσεις. Ως πολίτης, ξέρεις ότι αν κάποιος καπνίζει δίπλα σου σε κλειστό χώρο, είναι παραβάτης. Ως καταστηματάρχης, ξέρεις ότι το κάπνισμα στο μαγαζί σου απαγορεύεται. Ως ελεγκτική αρχή, δεν έχεις περιθώρια να ρυθμίσεις τίποτα - το πρόστιμο το τρώει και ο καπνιστής και ο καταστηματάρχης. Ως εργοδότης ή καταστηματάρχης, το κόστος σου είναι να βάλεις μία πινακίδα που να λέει 'NO SMOKING'.

Δε θα συζητήσω τις επιπτώσεις της απαγόρευσης του καπνίσματος στην οικονομία, στην κοινωνία κοκ. Κάθε νόμος έχει επιπτώσεις, και υποτίθεται ότι ο νομοθέτης τις περιμένει και τις επιδιώκει. Δυστυχώς όμως ένας δαιδαλώδης και πρόχειρος νόμος δίνει την ευκαιρία σε κάποιος να τον παρακάμψουν και τελικά επιφέρει κόστος μόνο για τον φορολογούμενο και τον νομοταγή.

Προσωπικά θα προτιμούσα μια ολική απαγόρευση του καπνίσματος. Αν οι Άγγλοι αντέχουν να κάθονται έξω στη βροχή και να καπνίζουν, κι εμείς θα αντέχαμε να καθόμαστε έξω στη λιακάδα. Όσο για τα μπουζούκια (άκου επιχείρημα), ας πέσουν και λίγο τα κέρδη της παραλιακής και των τραγουδιστών - σιγά την προσφορά στην κοινωνία, εδώ που τα λέμε.

Comments

Come see my PyObjC introduction at Europython!

Mon, 01 Jun 2009 20:42:06 +0200

Yay!

So, my PyObjC introduction talk was accepted at EuroPython. The conference is held at Birmingham, UK, from 28th June to 4th July. My talk is scheduled for Wednesday, 1st July.

Help!

It's the first time I'm doing a technical presentation like that, so I'd like to get some input before I start. If you plan to attend my talk, what would you like to see?

Code, code, lots of code
Architectural overview
Live demos
Screenshots
Common traps and pitfalls
All of the above
Something completely different?

Even more important is, what should be the goal of the talk? I think that a reasonable, attainable goal is:

Audience members should learn enough PyObjC to write simple Cocoa applications, and know where to look for help and how to learn more.

That probably means that a lot of advanced Cocoa features (like CoreData, bindings and so on) are ruled out, but I should cover how Frameworks are made available to PyObjC.

Previous talks on the subject were given by far more qualified people, and they focused greatly on Cocoa. I'm not an expert on either PyObjC or Cocoa, so at best I will give a "beginner's walkthrough".

I'm a fan of the "annotated code" way of introducing things, so I'll probably use it. Not sure how to handle showcasing the tools though. Live demos would be optional but are a risk, and screenshots are too static. Perhaps I'll record screencasts and embed them in Keynote.

Please let me know in the comments if you think this is the right/wrong way of approaching this!

Comments

What I've been up to

Mon, 25 May 2009 16:56:53 +0200

Well, it's over a month since my last update, so I might as well describe what I've been working on for the past month or so!

As pointed out, I'm now back in Athens, doing freelance contracting. I've hinted at the project I've been busy with, but now it's the time to lift the curtain.

For the past three months, I've been a contractor for EDM Studio, working on the Museum of Australian Democracy at Canberra, Australia.

My main focus was the interactive 'Timeline' exhibit, where visitors interact with 14 52-inch touch screens, each displaying 20 years history of the Australian Democracy (catch a glimpse of it here. I've also been the testing 'champion', promoting Test Driven Development, and creating an integration testing framework for our apps.

All our applications are developed with Python and PyObjC, with a dash of Twisted. We've hit some performance problems that relate to Core Animation and the Python bridge, but the productivity gains realised probably offset these. As we wrap up the exhibit, we'll probably investigate these. This is first time I use Twisted in a production setting, and I have mixed feelings, which I hope I'll capture in a later post. Oh, I will also give a talk about PyObjC at EuroPython!

This is the first time I've worked in a globally distributed team, and it's been a fun challenge. Trac has been invaluable, as was Campfire. Still, a lot of time is spent in communication and coordination, compared to having people in the same room, or even on the same keyboard (yes, I miss pair-programming). Time zone differences were also fun to deal with :)

It's also the first time I've had that hard a deadline. The museum opening was at 8 May, so we had to ensure a smooth visitor experience. 'We'll ship when it's ready' was not an option. In the end, we pulled it off with some mad sprinting in the last couple of days, including the classic bug discovered 6 hours before opening time (and fixed just in time :). We had to cut back some features and keep things simple. KISS and 'simplest thing that might work', together with iterative development helped a lot there.

In other news, I'm slowly adjusting myself to being back in Greece, and also working from home. I'm contemplating starting a Python-Athens user group, so if you're interested, drop me line!

Comments

Book heads-up: iPhone in Action

Fri, 03 Apr 2009 01:24:08 +0200

This is a long overdue post about the book iPhone in Action. It's not a complete review as I haven't finished it yet: Mainly because I don't have enough time, but also due to the fact the kind guys at Manning sent me the PDF version (disclaimer: for free).

The book is split into two parts: Web applications tailored for the iPhone and using the iPhone SDK. I have only read the second bit, and not in-depth, given how I realised I hate reading PDFs, and the lack of any decent PDF reader for the iPhone (if you happen to know one, please tell!).

Overall, the book is very well written. The Manning copy editors do a very good job, there was no place where I had trouble understanding, everything was clear and nice. The content is incremental, and you get nice pointers to previous chapters in case (like me) you're cherry picking chapters you're interested about. At the end of every chapter you get links for further reading, especially useful where they're touching complex issues . I would say it's very well suited to beginners and people who don't know much about programming on a Mac.

What it covers

Here's a quick rundown of the contents:

Chapters 1 and 2 are an introduction the the iPhone as a device, hardware capabilities and tech-specs, and a comparison of the two ways of developing applications for it (web and SDK).

The web

Chapters 3 to 9 cover developing web applications on the iPhone. I've only read chapter 3 which covers how to optimise existing sites for the iPhone. Very well written, explains all the concepts you need to understand like viewports, events, CSS optimisations and user agent detection. Chapter 4 covers advanced WebKit functionality like CSS transforms and gradients, the database and gestures. Chapter 5 covers the iUI library, chapter 6 using Canvas and chapter 7 building web applications using Dashcode.

The SDK

Chapters 10 to 20 cover developing applications using the iPhone SDK. I've read most of them, and enjoyed it.

Chapter 10 introduces you to Objective-C and the iPhone OS. Chapters 11 and 12 introduce you to Xcode and Interface Builder, and you create your first "Hello, World!" project. Chapter 13 covers view controllers, a cornerstone of building iPhone apps. Chapter 14 covers events and actions: the responder chain, buttons, textfields and notifications. Chapter 15 shows some more view controllers, like the tab bar controller and the navigation controller, both major tools for applications.

Chapter 16 covers writing and reading to files, SQLite, preferences and access to the Address Book. Chapter 17 shows how to use the accelerometers and Core Location. Chapter 18 introduces media: displaying images, accessing photos, the Media Player framework and other audio cases. Chapter 19 gives a taste of Quartz, Apple's drawing framework, and Core Animation. This is the chapter I plan to read again, as for my current project I use both heavily and a good overview would be nice, if only to confirm my understanding.

Lastly, chapter 20 covers interacting with the web - a very well thought addition since most applications will want to follow URLs, show web pages, parse XML responses and POST to a server, all of which is covered, including an introduction to low-level networking.

Conclusion

Overall, I like this book. I enjoy the style and the bite-size approach. My experience with Apple's documentation is that it's thorough but sometimes fails to give you a coherent overview on how things should be used or how everything fits together. I think this book fills that role very well. If you already know Cocoa and understand its architecture, everything should slide into place neatly and you can skip most introductions. If it's your first time with Cocoa and Objective-C, you should get enough help to get started and pointers to expand your knowledge.

Overall, I'd suggest it!

Comments

Key-Value Coding in PyObjC

Tue, 31 Mar 2009 21:59:47 +0200

Key Value Coding or KVC is a very important part of Cocoa. Accessing these properties can be a kludge though - you have to use setValue_forKey_(value, key) and valueForKey_(key) which is quite verbose. Luckily, PyObjC has a nice shortcut: a single underscore.

from Quartz import CALayer 
# CALayer accepts arbitrary properties, perfect for testing!
l = CALayer.layer()
l.setValue_forKey_('value1', 'key1')
v = l.valueForKey_('key1')
assert v == 'value1'

l._.key1 = 'value2'
v = l._.key1
assert v == 'value2'
v2 = l.valueForKey_('key1')
assert v2 == 'value2'

This is not very discoverable so I thought I'd put it up here in case anyone is wondering if there's a shortcut for key-value coding in PyObjC.

Note: CALayer accepts arbitrary properties but I've found that things like bounds, frame and position work best if you use directly setBounds_, setFrame_ and setPosition_. I haven't looked deeply into it though.

Comments

Python programmers needed

Thu, 26 Mar 2009 15:48:54 +0200

As I mentioned some time ago, I'm now doing contracting. At my current project we need to get some more people on board. So if you're a Pythonista and want to work on exciting and challenging stuff for a couple of months (or more), read on!

The project is a large museum in Australia. On the digital side, the 10,000 square foot gallery features:

a 14-screen touch "table" interactive
a wraparound wall of community voices/faces
touch interactives that solicit visiter opinions on museum content
RFID-enabled passes that, among other things, provide content for non-English speakers and accessibility to vision impaired visitors
roughly 45 short films spread over 20 different viewing areas

We are using all the usual development tools such as Subversion, Trac, branches and friendly code reviews. We are committed to testing and quality and we'd rather drop features than drop quality. We are collaborating with Basecamp and Campfire.

If this sounds interesting and you:

are a savvy Python programmer with experience in some the following: Twisted, PyObjC, Cocoa or OpenGL. If you're a mostly Cocoa person with some experience in Python, you're welcome as well.
have a Mac and know how to use it
can work as part of globally-distributed team (museum is set up with network cameras for remote access, and you can work from anywhere in the world).
are committed to testing (ideally test-driven-development).
are available to work between half and full-time

Then contact my client directly at darran@edmstudio.com. Please provide an indication of your experience, your availability, and your sought-after hourly rate.

We're doing really cool stuff, so apply!

Comments

On my own

Sun, 01 Mar 2009 12:52:04 +0200

Last Friday was my last day at Resolver Systems. I am now doing contracting work (mostly Python) for various clients, and I plan to move back to Greece soon.

While I enjoyed working at Resolver Systems, and I am grateful for the chance of working with such a smart and knowledgeable team, a lot of small reasons quickly added up to this decision.

The main reason I guess is the London weather and lifestyle. When I moved to London 1 year ago I was brimming with excitement (new city, new experiences etc). After 1 year of rain, clouds and cold, the excitement wore off. I have to confess though that London in spring and summer is beautiful. I'll be back for EuroPython in June, and hopefully I'll stay a few days.

Another problem was the lack of close friends. I guess I gravitate to a "loner" lifestyle if left to my own devices, which isn't good. When you have a lot of friends around, you never have the chance to be alone.

I never regret my choice of coming here - it has made me a stronger person, and of course if I ever get bored in Athens I can always move back. It's easier when you've done it once already :)

Alea iacta est, as the Romans would say. I will be doing some exciting PyObjC stuff for the next couple of months, which hopefully I'll be able to blog about. Here's to an exciting future!

My infinite gratitude again to the wonderful team at Resolver Systems, and of course they are welcome to visit lovely Greece any time!

Comments

iPhone!!1

Sun, 01 Feb 2009 19:46:46 +0200

I finally caved in a sent Steve Jobs a £400 get-well-soon note. I bought a 16GB 3G iPhone.

Jailbreaking and unlocking worked perfectly, Virgin mobile PAYG is happily powering the phone.

The App Store is brilliant, although unusable. There's so much crap in there, I have to rely on recommendations and 3rd-party reviews, which is annoying. So perhaps someone might find my choices useful:

Oh my home screen I currently have, other than the built-in apps:

Remote
Byline (read and sync with Google Reader, brilliant application!)
Tweetie (Nice twitter client)
London Tube (Maps and route planning)
Fring (IM and calls for Skype, among others)

From these, only Fring is free - I was hesitant at first to pay for random apps, but after I've seen recommendation for most of them, I bit the bullet. If Apple had a try-before-you-buy scheme, I'm sure that a lot of developers would earn more money, and if the downloads that don't turn into sales are a good metric, a lot of crap could be filtered out.

I use Byline daily, loading app feeds before I leave home. It will archive the full post, with pictures, and also the actual item, meaning that I can read sites where they post only snippets of the actual content. It will sync read, starred and shared items when a network connection is available. It has made my tube journeys feel a lot shorter! Highly recommended.

Other random stuff:

Games: Galcon
Facebook
Google & Google earth
Now Playing (brilliant app for movie theatres)
Stanza (haven't used that yet)
Dictionaire (nice free, offline dictionary app)
Translator (uses Google Translate)
Pocket guitar (gimmicky but fun guitar simulator)

Things that have impressed my girlfriend: Apple Remote (control iTunes remotely) - she giggles every time she changes the volume or selects a new song. It's doubly impressive because we use airtunes to pump music to remote speakers, where no computer is in sight. Now Playing, when it showed our two nearest cinemas with movie listings. Pocket guitar, when she shaked the phone and it bended the sound.

I'm also dabbling in jailbroken apps, if only because I have to jailbreak the iPhone to add a Greek keyboard. Thanks to iKeyEx I can create my own custom keyboards (which I'm afraid I'll have to do).

If you have killer iPhone apps, please share in the comments!

Comments

Web Science Conference 2009 - Society On-Line

Mon, 26 Jan 2009 21:55:13 +0200

A good friend of mine asked me to give this conference a bit of publicity, so here goes. Executive overview: Tim Burners-Lee will be one of the keynote speakers, in a very nice-looking place in Athens. Definitely looks interesting.

Web Science Conference 2009 – Society On-Line is the first conference devoted to the scientific study of socio-technical aspects of the Web as a standalone artifact. The conference will be held at the Theatron, at the Hellenic Cosmos of the Foundation of the Hellenic World, in Athens, Greece, March 18th - 20th, 2009.

Web Science focuses on understanding, designing and developing the technologies and applications that make up the World Wide Web. However, the WWW does not exist without the participation of people and organizations. Now that a significant proportion of everyday life is spent on-line in many countries, it makes sense for the first Web Science conference organized by the Web Science Research Initiative (WSRI) and the Foundation of the Hellenic World (FHW) to be dedicated to the presentation of research into society on the Web.

Thematic areas

How do people and organizations behave on-line – what motivates them to shop, date, make friends, learn, participate in political life or manage their health or tax on-line? Which Web-based designs will they trust? To which on-line agents will they delegate? How can the dark side of the Web – such as cybercrime, pornography and terrorist networks – be both understood and held in check without compromising the experience of others? What are the effects of varying characteristics of Web-based technologies – such as security, privacy, and network structure, the linking of data – on on-line behavior, both criminal and non-criminal? And how can the design of the Web of the future ensure that a system on which – as Tim Berners-Lee put it – democracy and commerce depends remains ‘stable and pro-human’?

WWW Forum

The WWW Forum is the opening event of the Web Science Conference 2009 – Society On-Line. During the WWW Forum, Tim Berners-Lee, WWW Inventor and Top Agenda Setter will discuss with the public and famous scientists and politicians the following themes:

Web Science and Research
Web Technology and Practice
Web for Society

How can we learn from people in socially or economically deprived communities? How the Web can better serve them? In which ways can we leverage the Web to empower people, especially in under-served populations, by lowering barriers to life-critical services? Can we ensure the Web is accessible and useful to people, including people with disabilities, from different cultures, and language and literacy skills that span the range of the Earth’s population? Which methods do we need to implement in order to study and understand how the Web works in order to anticipate and ensure its future? Can we understand how to make the Web stable and secure, and to mitigate threats and weaknesses? Is it possible to promote the development of technology and standards that foster creativity, collaboration, communication, and commerce?

Keynote Speakers

Tim Berners-Lee (WWW Inventor)

A graduate of Oxford University, England, Tim Berners-Lee is the 3COM Founders Professor of Engineering in the School of Engineering, with a joint appointment in the Department of Electrical Engineering and Computer Science at the Laboratory for Computer Science and Artificial Intelligence (CSAIL) at the Massachusetts Institute of Technology (MIT) where he also heads the Decentralized Information Group (DIG). He is co-Director of the new Web Science Research Initiative (WSRI) and is a Professor in the Computer Science Department at the University of Southampton, UK. He directs the World Wide Web Consortium, founded in 1994.

In 1989 he invented the World Wide Web, an internet-based hypermedia initiative for global information sharing while at CERN, the European Particle Physics Laboratory. He wrote the first web client and server in 1990. His specifications of URIs, HTTP and HTML were refined as Web technology spread.

In 2001 he became a fellow of the Royal Society. He has been the recipient of several international awards including the Japan Prize, the Prince of Asturias Foundation Prize, the Millennium Technology Prize and Germany’s Die Quadriga award. In 2004 he was knighted by H.M. Queen Elizabeth and in 2007 he was awarded the Order of Merit. He is the author of “Weaving the Web”.

Joseph Sifakis (Turing Award 2007)

Joseph Sifakis is a CNRS researcher and the founder of Verimag Laboratory, in Grenoble, France. He holds the INRIA-Schneider endowed industrial chair since September 1st 2008. He studied Electrical Engineering at the Technical University of Athens and Computer Science at the University of Grenoble.

Verimag is a leading research laboratory in the area of critical embedded systems. It developed the underlying theory and technology for the SCADE tool, used by Airbus for the design and validation of its critical real-time systems, and is becoming a de facto standard for aeronautics. Verimag has a lasting and strategic collaboration with ST Microelectronics, France Telecom R&D, and Airbus, through which numerous results on validation and testing have been transferred.

Joseph Sifakis is recognized for his pioneering work on both theoretical and practical aspects of Concurrent Systems Specification and Verification. He contributed to emergence of the area of model-checking, currently the most widely-used method for the verification of industrial applications. His current research activities include component-based design, modeling, and analysis of real-time systems with focus on correct-by-construction techniques.

Joseph Sifakis has broad experience with industry, notably though joint projects with partners such as Astrium, the European Space Agency, France Telecom, ST Microelectronics and he has also been active for many years in consulting.

Joseph Sifakis is the Scientific Coordinator of the European Network of Excellence ARTIST2 on Embedded Systems Design. This network gathers 35 of the best European teams in the area, and aims to produce innovative results for cost-effective design of dependable embedded systems. It will also promote innovative methods safe and secure systems, notably through cooperation with key European industrial partners such as Thalθs, Airbus, Ericsson, Philips, and ST Microelectronics.

Joseph Sifakis is the director of the CARNOT Institute “Intelligent Software and Systems” in Grenoble.

Joseph Sifakis is a member of the editorial board of several journals, co-founder of the International Conference on Computer Aided Verification (CAV) and a member of the Steering Committee of the EMSOFT (Embedded Software) conference. He is a member of Academia Europea and a member of the French National Academy of Engineering.

Joseph Sifakis has received with Ed Clarke and Allen Emerson for their contribution to Model Checking, the Turing Award for 2007. He is also the recipient of the CNRS Silver Medal in 2001.

Comments

Book review: Expert Python Programming

Fri, 23 Jan 2009 23:25:46 +0200

Disclaimer: Packt Publishing sent me the book for free, asking me to write a review.

Introduction

Expert Python Programming is a book by Tarek Ziadé, Zope contributor and known to me from his excellent proposals about setuptools and distutils.

I have to be frank from the beginning --- this is an unusual book. Its title suggests that it's about advanced Python usage, topics that a non-expert Python user would wish to know. While it does cover some of these (but not to the breadth that I expected), it extends to a lot more things as well. I'm not sure whether I should call that a good or a bad decision --- I'll let the reader do that, based on his personal needs.

Overview

In its 350 pages, the Expert Python Programming covers:

Getting started
Syntax best practices - above and below the class level
Choosing good names
Writing a package
Writing an application
Working with zc.buildout
Managing code
Managing life cycle
Documenting your project
Test-Driven development
Optimization - profiling and solutions
Useful design patterns

As you can see, this is a very wide range of topics, each of them potentially worthy of its own book. As the book itself states, some of them are directed to Python programmers who wish to go further, while others are aimed at project leads.

This is, as I said, an unusual decision, which leads to the book not having "flow". The chapters can be easily read standalone, and sometimes the context switch between them is a bit steep. Given that format, I'll present each chapter separately, since I can't find a way to link them together.

The writing is terse and to the point, accompanied with numerous coding examples. Sometimes this is taken to the extreme, and given that it's a book not actually addressed to novices, I think that some more detailed explanation was needed.

There is a distinct preference on Zope related solutions like eggs and zc.buildout. I haven't used either so I can't comment on it. I was also a bit put off by the use of Paste to generate some files.

Chapters

1. Getting Started

This chapter should be all you need to install and create a development environment for Python on the three major platforms. Includes MinGW, tab completion with history, iPython, setuptools and how to setup your editor of choice. Thorough and useful.

2. Syntax Best Practices --- Below the Class Level

Covers list comprehensions, iterators, generators, descriptors and properties, decorators and the with statement. Here is where things become a bit icky. List comprehensions are nicely covered, as are iterators. Generators are a bit all over the place, without a coherent overview. Coroutines are next on the list (since they can be implemented with generators), but rather than showing how to implement them, a non-standard multitask package is demonstrated instead. Some functions from the itertools module are then presented, unfortunately with mediocre examples.

After that, decorators and the with statement are presented. The explanations are a little better, but the code is again needlessly complicated. Strangely, the functools module isn't mentioned at all. with and contextlib are nicely presented.

It's unfortunate that the second chapter of the book has so many shortcomings. Fortunately, things pick up in the coming chapters, so read on!

3. Syntax Best Practices --- Above the Class Level

This chapter covers multiple inheritance, MRO and super, descriptors, slots and metaclasses. Multiple inheritance and MRO are exceptional, with nice small examples that present the problems and solutions sufficiently without complicating things.

Descriptors are also well-explained (although I think that describing how attribute lookups are done should help even more), but a tendency of over-complicating examples is here again (also note that the first edition had horrible typographical mistakes that made following this chapter difficult. However, these should alredy be fixed in the print edition). Properties should probably be put before descriptors, as they are probably the single most common use of descriptors in Python. Metaclasses aren't given much space, and rightly so IMO.

4. Choosing Good Names

A short and pleasant chapter. Covers PEP8, best practices for naming things, how to design an API, how to deprecate an API. Also covers code-quality tools such as Pylint and CloneDigger.

5. Writing a Package

Here Tarek presents how distutils and setuptools work. He has written much about this topic, and it shows. His explanation is top-notch, with very good examples and a nice summary at the end.

We then have a pleasant surprise - a thorough presentation of Python Paste. I've never used Paste before (it's a tool that for generating code and other text files for templates, used extensively in Pylons), so I appreciated that bit (although I don't have a use case yet).

After a brief mention on how to use version numbers, the chapter ends, giving way to...

6. Writing an Application

This is a case study that presents the designing and implementation of 'Atomisator', a small planet-like utility for aggregating feeds. To appreciate this chapter you have to pretend that it's actually a larger and much more complicated project, or else your YAGNI alarm will often beep :). I suppose this is true with every book that tries to present a concept using basic examples though.

This chapter adds more tools to your arsenal: The excellent virtualenv package, and the handy nose test runner. It's also done using a TDD approach, which is always a good thing. I didn't like the use of Paste to create all the templates, but that's probably me being cranky.

Overall, this is a very meaty chapter. I wish I have read this 6 months ago - it would have saved me many headaches. If the book has stumbled at the first two chapters, it's completely redeemed itself by this point. I would go so far as saying that this chapter should be required reading for any developer that wants to publish a Python utility.

7. Working with zc.buildout

Here my eyes glazed over. While Tarek makes a valiant effort of explaining buildout, it still feels very over-engineered and un-Pythonic to me. I really can't say anything more.

8. Managing Code

This chapter detaches from Python momentarily and discusses version control and continuous integration. The two models of VCS are presented (Centralized vs. Distruted), and a thorough tutorial on how to setup and use Mercurial is given.

The book then presents CI using Buildbot. Unfortunately for me, zc.buildout is used again, and no alternative way of using buildbot is given. It makes sense from the author's perspective, since the sample application is managed using buildout, but it's a bit annoying if you don't subscribe to the Zope mentality.

9. Managing Life-Cycle

This is a very small chapter on different types of development (waterfall, spiral and iterative) and a nice tutorial on how to setup Trac (although, much to my chagrin, buildout recipes are used yet again - I guess I should really make an effort to learn this thing).

The last two chapters are very good, and should serve as a decent starting base for any team that hasn't implemented version control, continuous integration or issue tracking yet.

10. Documenting your Project

An interesting deviation from technical issues, where advice is given on how to write good documentation, including pointers on style, structure, language etc. A bit too fluffy for my taste, but still has good, sensible advice.

Also, a primer on reStructured Text and Sphinx, complete with installation and usage instructions. Made me want to play with Sphinx!

11. Test-Driven Development

As a practicioner of TDD, I highly recommend this chapter. It gives a thorough introduction of what is TDD and why you should use it. It mainly focuses on unit tests, presenting unittest, doctest, nose and py.test, but it also gives more information on writing acceptance tests. I realized I have bookmarked a couple of these pages to revisit at some later stage for my own projects, which is a good thing in any book.

It also covers Fakes and Mocks, including Ian Bicking's minimock library.

12. Optimization: General Principles and Profiling Techniques

A very nice chapter - provides solid advice on when to optimise, how to measure, what to optimise. Introduces all the main tools for profiling CPU, memory and other resources. Goes hand in hand with...

13. Optimization: Solutions

Discuess Big-O complexity, and some common Python tricks to optimise (use a set instead of list if testing for membership etc). Also discusses the deque, defaultdict and namedtuple collections and their use cases. I have to confess that named tuples don't seem particularly relevant to the optimisation story, though.

Then the chapter continues to cover multithreading & multiprocessing in a good and thorough way, with numerous examples and use cases. It ends with caching and how to implement memoization.

14. Useful design patterns

Covers Singleton, Borg, Adapter, Proxy, Facade, Observer, Visitor, Template. Most of them can be implemented in 5 lines of Python, but it's a good refresher.

Conclusion

Would I have bought this book? I'm not sure, but probably not. It definitely has its good pieces, and I did learn new things from it, but I think I'm not in the target audience.

However, if it was available 2 years ago, when I was just starting with Python, this book would be very helpful. It can easily be read cover-to-cover, it gives you the necessary code and a lot of helpful hints and pointers to more information. So I would recommend it to any novice programmer that wants to learn more Python, and some development best practices. You can find more here: http://www.packtpub.com/expert-python-programming/book .

Sidenote

Poor Tarek has already apologised for his Frenglish, and as a non-native speaker myself, I blame the editors. Nevertheless, I don't think that there was any place in the book where I couldn't understand what it was talking about, so no reason to worry.

Comments

PySmell v0.7.3 released

Fri, 16 Jan 2009 18:41:51 +0200

PySmell is an autocompletion library for Python, supporting Vim, Emacs and TextMate.

This is a very minor release, just because I had some commits unreleased for two months.

Get it from PyPI. PySmell's homepage is hosted at Google Code.

Here are the changes:

Better support for windows line endings.
Added some docstrings
Better error handling
Better module attribute lookups

I have some plans for the next release, but also a lot of other stuff to do. Please submit issues on Google Code so I can prioritize.

Comments

Projects I'd like to do in 2009

Mon, 29 Dec 2008 20:53:00 +0200

2008 has been a very good year. I got a job at Resolver Systems where I learned a lot of exciting stuff like writing parsers, .NET and of course all things Python. It is very refreshing to deal with a complex (but not complicated, I hope!) desktop application that doesn't obey the request-response cycle of web apps. A lot of CS theory can be applied to desktop applications.

I also had the chance to spend my days (and I hope I can continue to do so!) amongst very clever people, and also to spend my lunch breaks coding cool tools for fun. Here's what I did and what I learned, in no significant order:

PySmell is the most important, and also an award winning project. It is a library that provides code completion in Vim, Emacs and TextMate, for Python. It is mature enough to use, although still under development, and could use some more polish.
Vim is my current editor of choice. I am not 100% pleased with it, but for want of something better, it's what I use. I've coded a simple plugin to add pairs of quotes etc, although I've turned it off currently.
I tried Emacs before settling on Vim, but I couldn't stand the key bindings. Elisp was also a factor that drove me away, although I learned enough of it to code another quote pairing plugin (can you spot a pattern)?
I now use Git with git-svn daily at work, and standalone at every other occasion. I have a couple of projects on Github.
Cocoa and Objective-C. Nothing to announce here, I'm afraid. I am trying to follow Stanford's iPhone class but there's not enough free time. I also wanted to help PyObjC, but other than a typo fix on the website, I didn't do much.
wxPython. I have a thing about GUI toolkits, having been exposed to the joys of Apple's Interface Builder. After too much whining, I bit the bullet and decided to sit down and actually try to use wxPython, seeing that it tries to look and behave native everywhere. Of course, I couldn't start small so my toy project is wixed, a text editor to end all text editors (it will be more useful when it can load and save files). So far wxPython is looking good, but the documentation is a bit hard to decipher. This will be one of my main projects for 2009.

I am also writing an article for PyMag and I am reviewing two books (Expert Python Programming and iPhone in Action, watch this space for more on this).

The future

So, what do I plan to do in 2009:

Wixed is going to be my major time sink next year. I have lots of ideas, and they seem to be easy to implement. Of course, Richard Stallman has solved most of the design issues for me, so I just have to reimplement Emacs in Python, rather than reinventing text editors. (People who are going to point out Editra and Peppy are missing the point :)
I also want to create something using GeoDjango. Google maps recently added geolocation for addresses in Athens (and possibly other cities in Greece as well), so I'd really like to see what I can do with it. And of course, doing Django is cool in general.
Finish up PySmell. It was a very good learning experience, I've learned a lot of things and I think that I've demonstrated that developer tools for Python aren't hard to write. I don't plan to extend PySmell forever - once I hit the 80% of the cases target, together with duck inferencing, I will call it done and start something new.
Do another talk somewhere, about something. I enjoy giving (and preparing) presentations immensely, and the PySmell lightning talk has whetted my appetite :). EuroPython 2009 is hosted in the UK, so I'm thinking of submitting a talk. I don't know what it's going to be about, but I'd really like to do something.

Of course, interesting things will show up, and I'll drop everything and follow something shiny for sure, but isn't this the joy of programming? Happy new year everybody!

Comments

Python import hooks

Sat, 20 Dec 2008 00:49:43 +0200

Today I worked with William on the promising ironclad project which allows you to use CPython extension such as numpy under IronPython. Ironclad needs to setup some import hooks to allow the loading of .pyd files. Here's some findings:

ihooks is old

One way to do import hooks is to use the ihooks module. However, this is the old way: it replaces the builtin import, and reimplements some of the functionality of the import statement. It's not deprecated, so you can still use it if it fits your needs. But there's probably a better way out there.

Enter PEP 302

PEP 302 defines two new, more granular ways to extend the import mechanism: path_hooks and meta_path.

path_hooks

I'll get path_hooks out of the way, because it seems to be the least useful (at least to my understanding). If you add your own path hook, whenever an import happens, your hook will get called with every directory that is in sys.path. You can state that you will be responsible for loading everything in that directory by returning a loader instance. Unfortunately, this means that your hook replaces the built-in mechanism for that path - so you'll have to handle the loading of the plain python modules and packages as well.

meta_path

Meta path is much more interesting and useful, and it's what we used in ironclad. Basically, your handler gets called once for every import, with the full name of the module to be imported. So for example, when you do: from numpy.core import multiarray you will first get called with numpy, then with numpy.core and finally with numpy.core.multiarray. You also get an extra path argument which is None for top-level modules/packages or the path of the local import in case of subpackages/submodules.

If you can handle the loading of the requested module, you return a loader instance (which defines load_module). If you can't, and this is the interesting bit, you just return None and the normal import mechanism will take place. You can also raise ImportError to completely block the loading of a module.

Example code

This is what we are now doing in ironclad. There used to be an ihooks-based solution before, but since IronPython does some extra trickery for imports to allow importing .NET namespaces, things were a bit broken.

import sys
import os

# _mapper is added by deep black magic - pls ignore

def loader(path):
    class Loader(object):
        def load_module(self, name):
            if name not in sys.modules:
                _mapper.LoadModule(path, name)
                module = _mapper.GetModule(name)
                module.__file__ = path
                sys.modules[name] = module
                if '.' in name:
                    parent_name, child_name = name.rsplit('.', 1)
                    setattr(sys.modules[parent_name], child_name, module)
            return sys.modules[name]
    return Loader()

class MetaImporter(object):
    def find_module(self, fullname, path=None):
        if fullname == 'numpy' or fullname.startswith('numpy.'):
            _mapper.PerpetrateNumpyFixes()
        if fullname in ('_hashlib', 'ctypes'):
            raise ImportError('%s is not available in ironclad yet' % fullname)

        lastname = fullname.rsplit('.', 1)[-1]
        for d in (path or sys.path):
            pyd = os.path.join(d, lastname + '.pyd')
            if os.path.exists(pyd):
                return loader(pyd)

        return None

sys.meta_path = [MetaImporter()]

A minor annoyance is that load_module isn't called with any path, so you would have to walk sys.path again and somehow know if this a relative import. We avoid this trickery by having a class factory that closes over the path to the .pyd file of the module we want to import.

I hope that this post will help someone struggling in adding their own custom import hook to Python. You can find out more details, plus other niceties of the new hooks in PEP 302.

Comments

PySmell v0.7.2 released

Sun, 16 Nov 2008 23:42:56 +0200

After a ton of bug reports for TextMate, and some very constructive feedback and testing from Jesse Noller, I've decided there's no much point in waiting, so I'm happy to announce PySmell v0.7.2.

Grab it from PyPI and follow the instructions in README.markdown.

Notable changes:

TextMate's dialog no longer errors when dealing with a huge number of entries.
Use argparse.py rather than hand-rolled option parsing
New --input allows mutation of existing PYSMELLTAGS file; useful to run after a file is saved
Analyze the current file when detecting completion type; more up-to-date suggestions
TextMate now completes a word rather than re-writing it.
TextMate honours TM_PYTHON - no need to set PATH
Fixes in pysmell.vim by Krzysiek Goj; Thanks!
Fixed issue 18 (tabs)
Bundle ez_setup for people that don't have setuptools.

Comments

The book meme

Thu, 13 Nov 2008 21:01:30 +0200

Too many cool kids do that, so I'll do it myself:

Colourful prose which captures the principle in the fixing of names to colours --- that of association.

From: "The art of looking sideways", Alan Fletcher

The rules for this meme thing are :

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Comments

Control iChat from Python

Fri, 07 Nov 2008 22:12:23 +0200

I often exchange small snippets or links with my girlfriend, and it's been very annoying having to right click on iChat, select the correct account and start a chat. So I've automated it!

Here is a small PyObjC snippet that will allow you to send iChat messages from the command line, using the ScriptingBridge framework and a bit of elbow grease. The next step would be to make a wrapper for it so it can live a an icon on the Desktop, but I'll try that some other time.

#!/System/Library/Frameworks/Python.framework/Versions/2.5/bin/python

from ScriptingBridge import SBApplication

import sys

ichat = SBApplication.applicationWithBundleIdentifier_("com.apple.iChat")
ichat.activate()
service = 'AIM' # a part of the service name
handle = 'handle@aim.com' # the handle of the contact 
buddy = [b for b in ichat.buddies() if b.handle() == handle and service in b.service().name()][0]
recipient = buddy

service = buddy.service()
chats = [c for c in service.chats() if buddy.handle() in [b.handle() for b in c.participants()]]
if chats:
    chat = chats[0]
    if chat.active():
        recipient = chat
words = sys.argv[1:]
close = False
if words and words[-1] == 'X':
    close = True
    words = words[:-1]
ichat.send_to_(' '.join(words), recipient)

for window in ichat.windows():
    if 'Chat with' in window.name():
        window.setVisible_(not close)

Use like that:

./chat.py Hello, world!

Finishing a message with X will hide the chat window. If you know of a better way to lookup buddies than my current stupid list iterating, please share!

Interestingly, this will allow you to have conversations with yourself, if you put your own handle. Messages will be double, and some (null)s appear in the UI.

Comments

PySmell v0.7 is here, supports Emacs, TextMate and Vim

Sun, 02 Nov 2008 22:02:45 +0200

I'm very proud to announce the release of PySmell v0.7. PySmell is an auto-completion library for Python, meant to be plugged in different editors.

The big news for this release is the support for Emacs and TextMate, on top of the already existing Vim support. This isn't tested as rigorously as I'd like, so please report any issues at Google Code.

EDIT: PySmell 0.7.1 fixes a nasty TextMate bug, so please try again if you were put off by it!

Other niceties include module member completion, so doing:

import os
os.

Should give you the members of os. Same applies with any package/module, of course.

Also, a very simple form of duck inferencing has been added:

class A(object):
    pass

a = A()

a is recognised to be an instance of A. The next release will aim to extend this functionality.

Grab PySmell from PyPI, or you can follow the development at GitHub

Please give feedback in the comments below or at Google Code

Comments

Towards headless IDEs

Wed, 29 Oct 2008 20:02:04 +0200

I'm not sure how much I've written about this, but you can probably discern by the titles of my blog posts that I'm looking for a good text editor. I think I've finally nailed my frustrations, and I think I have a glimpse of a solution.

Some Background

Back in February, 2 weeks after my first day at Resolver Systems, I switched to Emacs. Before that, I was using TextMate on my Mac, before that Eclipse, before that JBuilder, before that UltraEdit.

I gave up on Emacs because it would was damn hard to make it do it things that should be simpler (like line numbers). My colleague Kamil Dworakowski had a nice Vim configuration, nicely tuned for exactly the work we were doing, so I switched to Vim. Since then I've written a couple of scripts for Vim.

I'm still not 100% satisfied with Vim - it feels old and clunky, but I can't move away from it because I've customised it a lot and I really like the keybindings. I really like to try out alternatives, so I search around.

Reusable editors

Yesterday I came across Eclim, which is a blend of Eclipse and Vim. Its novel idea? Keep using Vim as your editor, take advantage of Eclipse's power (when you're doing Java, I guess). The way it's done is by starting a headless Eclipse instance that exposes an API, and communicating with it. Your editor is still responsive and you get all the cool refactorings and completions that Eclipse provides.

My own experience in integrating PySmell into Vim and TextMate (and my comments on other people's Emacs work), together with the Eclim idea point me to one conclusion:

IDEs and editors should follow the unix motto: Do one thing, do it well

For example, in my case, PySmell is a code-completion provider for Python. It doesn't care what your editor is - given some inputs, it will give you the same outputs. This I way it's easily pluggable into any editor that supports calling an external process.

Two calls

So I give out a call to everybody who makes tools: Rip them out of the IDE! Make them small and reusable! Let the community help you plug them into any editor they want. If your tool is useful, you'll hear from them.

However, not all editors are equal. Emacs can't present overlay windows to list the completions. Vim does, but can't present tooltips. TextMate does both, but doesn't provide Vim keybindings and it's not available on Windows (to my dismay).

So I give out a call to everybody who makes editors: Forget the cool features - forget the languages. Provide a way for other people to do that. Give them APIs, dialogs, panels, popup lists, tooltips. Focus on your core strengths: Editing text, juggling files, searching in a buffer, a pleasant UI.

I realise that this may sound a bit over-the-top. Indeed, all editors which I've mentioned provide that kind of extensibility. However, only one of them is modern (TextMate), and all three of them don't see much activity. I know that there are a lot of projects like MonoDevelop, IntelliPad and more that are making editors - it would be for the benefit of all if they are as extensible as possible.

Interchangeable

My idea summed up becomes: Make the editor IDE-agnostic, and make the IDEs editor-agnostic. They can progress in parallel. They don't evolve at the same rate, so they should be independent. That way, when a cool new language appears, there won't be a lot of posts in newsgroups titled "is there a nice IDE I can use". They will be "is there a library I can plug into my editor", to which people can reply: "not yet, but you can write your own". Writing IDE helpers is much easier than writing text editors, so people will write them.

Imagine if you would be able to pick an editor only for its personality - the UI, the keystrokes for moving around and so on, and then you could use all the nice IDE functionality you wanted -- for free. I think you'll agree that is sounds good!

I'd be really interested to find out what other people thing about this - please comment!

Comments

The trouble with setuptools

Sun, 26 Oct 2008 23:31:44 +0200

One of the pleasures of open source is that random people contribute back uninvited. I was extremely flattered when the previous week I received an email from Michael Thalmeier with a link to a git branch containing all the necessary work for PySmell to support setuptools.

Indeed, PySmell is now available at PyPI, however with only Python2.5 support, and not tested on Windows yet (but will soon! Windows is a top priority for PySmell).

The trouble I had with setuptools was:

Muddled documentation

The setuptools documentation assumes an intimate knowledge of distutils. Being completely new to this packaging thing, I was completely lost for a while. Then it clicked on me that setuptools is an extension of distutils, adding some functionality on top. Thus, when you want to search for how to do something, you must check both places.

Too many options

This probably continues from the previous point, but there is no quickstart guide for small projects. PySmell is a package of 6 files, with one script and one non-python file (the vim script). I'm sure that larger projects with C-extensions etc. need all these options, but I think that a small quickstart guide that caters to the 80% of the smaller ones would be godsend. Indeed, if such a thing existed, I wouldn't wait for outside contributions (I have to confess that I was sure that at one point or another someone would do this, so I just did nothing).

Not obvious ways to do stuff

Quick, what is the best way to distribute the pysmell.vim helper script? There are dozens of possible places for it to go, guaranteeing that a guess would be wrong. data_files, MANIFEST.in or include_package_data? What happens when using easy_install? Are there post-install hooks? Can I print my custom message when easy_install finishes (as guidance on what to do next)?

These are actually valid questions so please dear LazyWeb, if you have answers I'd be grateful!

Python-specific eggs

Eggs are tied to the Python version that created them (so far as I can tell). That means that I have to install Python 2.4, 2.5 and 2.6 on the same machine just to do egg releases. This is a major overkill - I can see specifying a minimum Python version (as a dependency?), but forwards compatibility should be taken for granted.

PyPI hiccups

While PyPI prompted me to signup right in the console, actually registering PySmell didn't work until I logged in manually. I couldn't overwrite my upload, and I had to delete it and re-upload it every time I changed my setup.py.

Conclusion

I know that I am only going to hit these issues once, and for my next project I'm going to reuse bits and pieces of my current setup. I also know that there's been a lot of discussion about moving setuptools and friends to the next century. I'm going to paste my setup.py now, hoping that a) I'll get some nice comments about improving it and b) it might help someone who is looking for something simple to get things started.

#!/usr/bin/env python

# -*- coding: UTF-8 -*-

from setuptools import setup

setup(
    name='pysmell',
    version ='0.6',
    description = 'An autocompletion library for Python',
    author = 'Orestis Markou',
    author_email = 'orestis@orestis.gr',
    packages = ['pysmell'],
    entry_points = {
        'console_scripts': [ 'pysmell = pysmell.pysmell:main' ]
    },
    data_files = [
        ('vim', ['pysmell.vim'])
    ],
    include_package_data = True,
    keywords = 'vim autocomplete',
    url = 'http://code.google.com/p/pysmell',
    long_description =
"""\
PySmell is a python IDE completion helper.

It tries to statically analyze Python source code, without executing it,
and generates information about a project's structure that IDE tools can
use.

The first target is Vim, because that's what I'm using and because its
completion mechanism is very straightforward, but it's not limited to it.
""",
    classifiers = [
        'Development Status :: 5 - Production/Stable',
        'Environment :: Console',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: BSD License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Topic :: Software Development',
        'Topic :: Utilities',
        'Topic :: Text Editors',
    ]

)

Please point out errors!

Comments

PySmell v0.6 released

Mon, 13 Oct 2008 22:45:18 +0200

I'm happy to announce PySmell v0.6, an autocompletion library for Python.

New features include:

Import statement completion
Support for multiple TAGS files (that means external libraries)
Support for analysing Python 2.4-2.5 stdlib.

Plus many bugfixes and minor improvements.

Grab it here: PySmell v0.6

Copying README:

PySmell

PySmell is a python IDE completion helper.

It tries to statically analyze Python source code, without executing it, and generates information about a project's structure that IDE tools can use.

The first target is Vim, because that's what I'm using and because its completion mechanism is very straightforward.

Download and Installation

PySmell's code is available at GitHub. You can click 'Download' to get it as a zip/tar if you don't have git installed.

Extract and drop the pysmell package somewhere in your PYTHONPATH. Distutils support coming soon - patches welcome!

Usage

To generate a PYSMELLTAGS file, use:

cd /root/of/project /dir/of/pysmell.py .

If you want to specifically include or exclude some files or directories (eg. tests), you can use:

/dir/of/pysmell.py [Package Package File File ...] [-x Excluded Excluded ...]

Check for more options by invoking pysmell.py without any arguments

Using external libraries

PySmell can handle completions of external libraries, like the Standard Library and Django.

To use external libraries, you have to first analyze the libraries you want, eg. for stdlib:

pysmell.py . -x site-packages test -o ~/PYSMELLTAGS.stdlib

This will create PYSMELLTAGS.stdlib in your HOME. Copy that in the root of your project, and repeat for other libraries by changing the extension. Note that you still have to have a root PYSMELLTAGS file with no extension at the very root of your project.

Partial tags

Sometimes it's useful to not pollute global namespaces with tags of sub-projects. For example, assume that there is a Tests package, which has hundreds of tests, together with a few testing-related modules. You only want to see these completions when working on test file.

To accomplish that, you can put PYSMELLTAGS.* files inside subdirectories, and they will be used only when you're working on a file somewhere in that directory or its children.

/dir/of/pysmell.py Tests/FunctionalTest.py Tests/UndoTestCase.py -o Tests/PYSMELLTAGS.Tests

The information in FunctionalTest and UndoTestCase will only be accessible when editing a file inside the Tests package.

Vim

To use PySmell omnicompletion from inside Vim, you have to have:

Python support
The pysmell package in your PYTHONPATH (sometimes Vim is silly about this)
Source pysmell/pysmell.vim
:set omnifunc=pysmell#Complete Note: If you want to always use pysmell for python, do: autocmd FileType python set omnifunc=pysmell#Complete
[OPTIONAL] Select a matcher of your liking - look at pysmell.vim for options. Eg: :let g:pysmell_matcher='camel-case'

You can then use ^X^O to invoke Vim's omnicompletion.

You can generate debugging information by doing:

:let g:pysmell_debug=1
:e PYSMELL_DEBUG

Debug information will be appended in that buffer, copy and paste it into an email.

Reporting issues

Look in the TODO first. Vote up on issues that you feel strongly about!

Send me an email at orestis@orestis.gr. If you can create a unit test that exposes that behaviour, it'd be great!

Comments

Random: GUI toolkits, The web and python

Fri, 19 Sep 2008 22:02:07 +0200

In PyCon UK, the two keynote speakers, Mark Shuttleworth and Ted Leung raised the issue of how Python should "open up" and aim to be the best language at everything (Mark's point) and the easiest language to work with (Ted's point). I agree with both them, I was nodding all the time when they spoke. They focused on the web and on the desktop, respectively.

Here are my thoughts on the state of GUI toolkits and how could Python take a first step to be on the web.

GUI toolkits

They suck

I think it was Ted that pointed out that there are no good cross-platform GUI toolkits. He is a good authority on this matter, seeing that he was part of the Chandler project, that aimed to be cross platform from day one. I remember him saying:

There's something very wrong if the ugliest version of your product is the Mac version.

(Sorry if I misquote - just my memories!)

I couldn't agree more. There is no way right now to create state-of-the-art GUI application that runs on two platforms without writing the GUI twice. By "state-of-the-art" I mean, as Ted said, that this application should integrate so well in your desktop experience that given the choice to switch to a web application, you would deny!

This means that a user, when first launching your application, shouldn't notice anything. It should be seamless, look just right and more importantly, behave right. Of course it is the Mac users that complain about this so loudly, because the Mac platform gives you a wide variety of built-in behaviours that your application can use that when people roll their own, it looks weird. Look at the recent example of Firefox 3, that took great pains to look as a native application, and you still can't look up words in the dictionary!.

No wonder that people are turning to web applications. They can use them from both their home and their office, they look exactly the same and of course they have the added benefit of safekeeping all their data for them.

Cross-platform chimera

There are a couple of cross-platform GUI toolkits with Python bindings, such as PyQt4 and wxPython, but you could hardly call their API Pythonic or even simple. I've looked at tutorials and while at the beginning they look promising, at some point things get so hairy that I recoil in horror (in PyQT it was signals - why couldn't I use just a function?). In all fairness, I'm sure you can build good applications with them if you try hard, but it's not as easy as using the native API (.NET or Cocoa).

Now, I'm a Mac user for the past year, and of course I had to dabble with Cocoa, and I liked what I saw. An excellent visual designer, a dynamic language (even with an alien to me syntax), very elegant architecture (outlets and actions make for excellent encapsulation) and batteries included.

I was pleasantly surprised for the past 6 months about .NET, it has a sane API (although I don't like generated code), a rich set of libraries and of course the killer feature for me was that I didn't have to use C#, since IronPython has a seamless integration with it. I have a few peeves, but they are related more to Windows itself rather than .NET (can we get a standard way of displaying modal dialogs please?)

An idea

It seems like you have to be knowledgeable about your target platform to know how your application can integrate into it, so no framework can give you that. But the rest of the things you want to do, like buttons and windows and all sorts of widgets, are almost the same. I had an idea about how you could translate a GUI definition (in XML or Python) into multiple platforms. Interface Builder generates (as of Leopard) XML output, so you can start from there as a base and try to generate .NET code to build something similar. Unfortunately I had a look at the output of IB and it is a serialized object graph that looks too opaque to be of great use, but I still think that the idea could yield something. My feeling is that approach like this would have to be constrained and not aim to be the last toolkit you'll ever need. Perhaps the other way round (a translation of Python code to native widgets) would be interesting to try out as well.

Translation

The main point of the previous paragraph was to stop trying to reinvent the wheel and try to reuse tools and technologies that are already there. I'm not sure if the specific approach would work, but I think that there is an example of something like that that worked very well indeed.

Cappuccino/Cocoa

The crazy guys at 280North liked Cocoa and Objective-C too much. Their problem was that they were a web startup, not a Mac startup. So they did the obvious thing and implemented something that looks a lot like Cocoa and something that looks a lot like Objective-C on top of Javascript and named them Cappuccino and Objective-J. They have a finished product to showcase and I they made their libraries open source.

I think that these libraries will gain traction when people want to build web applications, but the point I want to make is that they didn't have to design another big API, they took something that was proven (Cocoa) and brought its ideas into the web. The fact that Objective-J can fool the untrained eye into believing it is Objective-C is irrelevant, although I think that even with a different syntax, if the underlying values were the same it would've been the same.

This is why I think that people could start doing the same thing with Python, with a very big drawback: There is no big and proven GUI Python toolkit that you can port to the web, and I think that there will be none unless some very big company invests money into creating one. (Mr Shuttleworth, time to put your money where your mouth is :)

It is a big undertaking, but not an impossible one. I'll stop rambling now...

Comments

PySmell v0.5 released

Mon, 15 Sep 2008 18:45:46 +0200

I've justed tagged version 0.5 of PySmell, the intellisense (or auto-completion, or omni-completion) provider for Python in github.

You can grab it here: http://github.com/orestis/pysmell/tree/v0.5

UPDATE: I've put up my lighting talk here: http://orestis.gr/static/downloads/pysmell-lightningtalk.pdf

PySmell was received with great joy in Pycon UK, I even won the award for the best lightning talk!

As usual in these kinds of situations, I then found out that there's an infinite loop when there is no PYSMELLTAGS file in the root of your project. I guess this may have put some people off. Fortunately the fix was easy (although this code isn't tested at all - I guess this is the reason it has bugs in the first place), so I tagged version 0.5

I'll be putting up my slides, as I think they give a nice overview of PySmell.

Comments

Pycon UK Day 1

Sat, 13 Sep 2008 13:41:54 +0200

Sitting in the lunch refectory, writing about the talks I've witnessed:

The savoury flavors of Python 2.6 and 3.0

Python 3.0 will fix almost all the minor inconsistencies that bit me when I was trying to learn Python (including unicode, Exceptions, sorting and floor division)

I really liked the changes, enough so that I've downloaded Python 3.0b3 and compiled it, though I'm not sure if I'm going to find any actual time this weekend to fiddle around.

Named tuples are oh so cool, we are definitely going to use them right now (you can get the recipe here)

I like the print function, makes me think that file.write will be made obscure, which is probably a good thing. Class decorators? Good. Simplified formatting everywhere? Good.

I can't actually remember much more, but I found myself nodding on mostly everything, so I guess that's a good thing.

PyPy

PyPy is a very interesting project that promises to solve all the minor incompatibilities between various Python implementations by providing a stack for writing interpreters and generating JIT compilers for them, to target different backends.

This means that once you have say, the C, JVM and CLR backends working (from what I could gather, the C one is top priority with the other 2 chugging along nicely), you can tweak your interpreter specification (for example, for Python 3.1) and generate 3 different compilers that are 100% compatible (and fast!). How cool is that?

The second PyPy talk was about the internals of the JIT generation, which was complicated enough for me to not remember many details, but it does affirm that even PyPy seems very ambitious, the people working on it are capable of pulling it off.

py.test

py.test is a part of the more general pylib. It is very good though unfortunately it doesn't run on IronPython since it does some stack frame magic. I'm pondering about moving PySmell to using it, though I don't really like external dependencies.

Stay tuned for more updates about PyCon UK!

Comments

Blog refresh

Tue, 09 Sep 2008 11:16:34 +0200

As you may or may not have noticed, I've refreshed my blog. It now runs on Django 1.0 and has a simpler layout.

In the process I've managed to break my feeds, resulting in giving everyone doubly-escaped content. Sorry about that!

Of course, the layout is horribly broken in IE7 (I can't even imagine what IE6 looks like!). I'll probably fix it at some point, but it's not a priority...

Please let me know if you find broken links or things that don't make sense!

Comments

Announcing PySmell

Sun, 31 Aug 2008 23:37:08 +0200

Following up from my previous post on Python and static typing, I'm proud to announce PySmell v0.1!

UPDATE2: PySmell v0.5 released

UPDATE: PySmell v0.2 released, get it here

What is PySmell?

PySmell is a python IDE completion helper, that covers 80% of the cases, leaving the rest to superior human brains.

It tries to statically analyze Python source code, without executing it, and generates information about a project's structure that IDE tools can use. There is currently support for Vim's omnicompletion, but porting to other editors which provide similar mechanisms should be straightforward.

In its current state it doesn't do any type inferencing, but it's surprisingly useful even without that. I plan to add simple type inferencing in the coming versions.

Having nothing more to add, I'm copying from the README:

Download and Installation

PySmell's code is available at GitHub. You can click 'Download' to get it as a zip/tar if you don't have git installed.

Extract and drop the pysmell package somewhere in your PYTHONPATH. Distutils coming soon!

Usage

To generate a PYSMELLTAGS file, use:

cd /root/of/project
python /dir/of/pysmelltags.py

If you want to specifically include or exclude some files or directories (eg. tests), you can use:

python /dir/of/pysmelltags.py [Package Package File File ...] [-x Excluded Excluded ...]

Partial tags

If there are files that you want to analyze, but have them only be accessible from a specific package, you can create the PYSMELLTAGS file as above, rename it to PYSMELLTAGS.partial, and put it in the directory you want it to be accessible from.

For example, if you have a bunch of custom test case files in your FunctionalTests package, along with a thousand actual functional tests, you can use:

cd FunctionalTests
python /dir/of/pysmelltags.py FunctionalTest.py UndoTestCase.py
mv PYSMELLTAGS PYSMELLTAGS.partial

The information in FunctionalTest and UndoTestCase will only be accessible when editing a file inside the FunctionalTests package.

Vim

To use PySmell omnicompletion from inside Vim, you have to have:

Python support
The pysmell package in your PYTHONPATH (sometimes Vim is silly about this)
Source pysmell/pysmell.vim

You can then use ^X^O to invoke Vim's omnicompletion.

Reporting issues

Send me an email at orestis@orestis.gr. If you can create a unit test that exposes that behaviour, it'd be great!

Comments

A culinary intermission

Thu, 28 Aug 2008 00:44:23 +0200

In an unprecedented move for this blog, I present you with a recipe that combines raw ingredients to make delicious food, to be consumed by human beings.

Today's special: Green peas with artichokes

Ingredients

750g - 1Kg of frozen green peas
500g of chopped tomatoes (pre-chopped in a carton box will do)
1 onion, chopped into itty bitty pieces
4-5 spring onions, chopped in discs 3-4 mm thick
A bunch of dill, chopped
3-4 medium potatoes, peeled and cut into chunky-chip sized pieces
1-2 cans of artichoke hearts in water (if you can find frozen hearts, even better)
1 big cup of the best quality olive oil you can get your hands on (important: this will make or break the recipe)
Salt and pepper (duh)

To serve (not optional):

FAGE feta cheese, straight from the homeland
Loaf of tasty bread, to gather tasty gravy

Preparation (10-15 mins)

Make sure you have all the ingredients listed above, in the correct form.

And now, the recipe

Put half the olive oil into a pot, set the heat to 3/4 and once it's hot stir in the onion, followed by the spring onions. Leave to simmer for 1-2 minutes. Constant supervision required or it'll turn brown and nasty.
Add the peas, artichokes and dill, stir until the peas don't look frozen any more
Add the chopped tomatoes (you can sprinkle a teaspoon of sugar to counter the strong taste) and pour 1-2 cups of water.
Bring to boil (will take a while), then lower heat to 1/2, cover and simmer for 30 minutes, stirring occasionally. Add the rest of the olive oil half way through.
While the pot is simmering away, put the chopped and washed potatoes into a microwave-safe deep plate, with a splash of water, cover with pierced cling film and microwave on full power (750W) for 5-6 minutes. When finished, add potatoes to the simmering mix.
Just before the end, add salt and pepper. Leave the pot on the warm hob for 10 more minutes (this will "tie" everything together)

Serve with a large chunk of feta cheese, and one-two slices of bread. No salad required.

Comments

On Python and static typing

Mon, 25 Aug 2008 23:02:19 +0200

I have read numerous articles about dynamic languages and static typing, the most recent being Steve Yegge's Dynamic Languages Strike Back, where he argues that most of the time, there is enough type information in a program written in a dynamic language (he uses Javascript) to do all kinds of cool refactorings.

I'm building a small proof-of-concept to do something like that in Python, for Python. Here are my thoughts... UPDATE: Read about PySmell, a tool I wrote to provide static auto-completion for Python projects.

Motivation

My motivation for a tool like that is many nasty and silly mistakes I make when coding in IronPython at Resolver Systems, that could be avoided easily. We have various safeguards in place to catch bugs, the most important being test-first development. We have a pylint check phase that has to pass before we check in, and some of us have pyflakes hooked in to run before we run a test.

The problem is that only pyflakes is fast enough to run without it being a distraction. In fact, it's so fast, that emacs has a mode to run it continuously in the background and highlight any errors reported (I wish I could do the same with Vim!). On the other hand pylint is very slow, as it has to execute a python module to make sure everything is covered. Running a test is not slow per se, but IronPython takes something like 4-5 seconds to start, which makes it a bit annoying.

So how pyflakes manages to be instantaneous? It's using the built-in compiler module to generate, and then analyze the AST of your code. It can't catch everything, but surely catching something is better than cathing nothing!

So my idea is, how far can we stretch the approach taken by pyflakes?

The compiler module

Unfortunately, the documentation for the compiler module is not very good. Thankfully, some kind souls have noticed that as well, and have written up better documentation, found here. I wonder why it's not accepted yet.

Basically, what the compiler modules does is parse a python source file (or string) and give you back the abstract syntax tree for it. You can then use compiler.walk, using your own visitor, to do as you please (Interestingly, while pyflakes uses compiler, it does its own walking).

Auto-complete

The question I had was: "Can we get enough information from the structure of Python code to generate autocompletion lists?". It seems like in many cases, the answer is yes, we can. As a matter of fact, professional IDEs like Wing IDE do it already. I will attribute the lack of other IDEs (that I know of) to do it as "lack of effort", as Yegge himself says (no offence!). (I know that Vim's python omnicompletion tries, but it never worked for me in an acceptable manner).

So how would it work? We can easily gather from an AST most of the properties and methods a class defines. We can the look for clues in the usage of a name to try and figure out its type, and therefore its autocompletion suggestions.

Example: Imagine having the following class:

class Dog(object):
    def bark(self):
        pass
    def play_dead(self):
        pass
    def jump(self):
        pass

then, somewhere else in the code, perhaps in another module:

def interact(animal):
    animal.jump()
    animal.

where the dot . represents the autocompletion request. It is clear that animal has a very big chance of being an instance of Dog. We can the present Dog's properties and methods, complete with calltips. Even if we had a Kangaroo class that defined jump, the completions could include both classes, with some kind of indicator. Better than mindlessly listing every possible keyword, as Vim does now!

In fact, in some cases, we can be 100% sure that a name is an instance of a specific class: If we encounter a constructor within the same scope, or an isinstance check. The other, harder case is to do data-flow analysis and examine the argument in calls to interact, to add more information.

The quote

As I'm writing this, an extension the the familiar Duck typing phrase comes to mind:

If it walks like a duck and quacks like a duck, I'd call it a duck, and this is the only duck that I know of, so there!

Conclusion

Sure, it will be very hard to be absolutely certain. When someone defines __getattr__ (or for even more evil, __getattribute__ -- see an earlier post about the differences), all bets are off. The question is, do we care? I personally don't. I seem to manage without any kind of help, other than keyword completion, so any kind of help will make my life easier, at least. Expect to hear more soon!

Comments

A git-svn tutorial

Sat, 23 Aug 2008 13:07:52 +0200

Introduction

I have been using git-svn successfully for some time now, and I am generally pleased about the productivity improvements I get. I am documenting my workflow here, to serve as an example to anybody else that could be struggling with the admittedly complicated git commands.

Prerequisites and background

I'm using the MSYS version of Git, v1.5.5, on Windows Vista. This packaging is a good Windows citizen, giving you shell integration (if you want to), context menus in directories and two extremely useful GUI tools. Download from here (direct link).

Unfortunately, in v1.5.6 git-svn is broken, and the maintainers are waiting for someone to provide patches. This is one risk you're taking when using open-source software I guess. 1.5.5 works good enough for me, so I don't mind staying there for a while.

Setup

You start your adventures with git-svn by cloning an existing Subversion repository:

git svn clone url://path/to/repo -s

The -s flag assumes that your repository uses the "trunk, branches, tags" convention. If not, you have to specify manually which directories represent branches and tags, if you want Git to know about them.

This will take a long time, as it will fetch every single revision from SVN and commit locally. If for any reason it stops, you can resume with git svn fetch.

I always setup Git to not translate line endings for me, using git config core.autocrlf false. We don't have a strict policy about line endings, so I don't want to complicate matters.

When everything is finished, you should have the default master branch setup, and pointing to your SVN head. Verify that by running gitk --all. You should see a commit tree, with the SVN head tagged as remotes/trunk and with the master branch label next to it.

Day to day usage

Branching

In your day to day usage, you will be working against a local branch. There are two ways to setup one:

The graphical way. Fire up gitk, select the revision you want to branch from (most of the time it would be remotes/trunk), right click, select 'Create branch here...'. You then have to "checkout" that branch, which you can do by right clicking its label.
The command line way: git checkout -b a_branch_name. Note that the branch will happen from the where your current working copy is based on. I'm sure there is a way to specify a revision, but I'm using gitk all the time, since I like to see visually what's going on.

Normal usage

Once you have a branch, you can start editing code, and committing to Git as needed. I use Git GUI for that, as it gives a vastly better UI to review changes.

There are many ways to use Git: I usually commit when I have finished a "unit" of coding, amending to add small changes as I see fit. Git GUI is tremendous help, showing at a glance what's going on.

Interacting with Subversion

There are two things you want to do with Subversion: update your working copy with remote changes and committing your changes to the SVN repository. Here's how to do them:

Updating

To update your working copy, you use git svn rebase. This will make your history seem as you've really branched from the current SVN head, instead of your actual branching point. This means that your changes are still grouped together, and you don't have other people's changes mixed up with yours.

Dealing with conflicts

Occasionally, you may have a merge problem. Things here get tricky. You can setup a mergetool that will show you the correct files, but it's a bit of voodoo. I find that editing the offending file in a text editor gives enough information. The on-screen (command-line, actually) instructions are very helpful, and you can always undo everything by issuing git rebase --abort.

Committing changes

This is very simple: git svn dcommit will send every commit as a separate checkin into Subversion, complete with message. If you realize in horror that a commit message 3 commits back is wrong, or if you don't want to pollute your Subversion tree with 15 commits, you can use git rebase remotes/trunk --interactive which allows you to revisit every local commit you've made, changing messages, squashing commits together and more.

Various other useful bits

git gui blame somefile will give you a nice clickable blame, so you know where to point the finger.
git svn fetch will fetch svn revisions (branches included!) without updating your current branch.
git-svn doesn't handle SVN externals. I just checkout the external using plain SVN, and add the directory in .gitignore.
Speaking of .gitignore, you can generate one by running a git svn command which eludes me right now.

Wrap-up

Git has turned out to be very useful, if only as a glorified backup and undo system. It's also very fast and very space efficient, so you can have lots and lots of branches, and a complete history of your project, in almost the same space as an SVN checkout. Using the remote mechanism, sharing trees between two machines is very fast an easy.

Whenever I have to use plain Subversion, even with the very helpful TortoiseSVN front, I feel very inefficient. Truth be told, for some operations like tagging and branching in Subversion, I can't be bothered to learn the Git equivalent, so I will never completely drop Subversion!

Missed anything? Got suggestions? Let me know in the comments!

Comments

Scripting Vim with Python

Sun, 10 Aug 2008 18:22:28 +0200

I've created and released a small Vim plugin that will try to mimic TextMate's behaviour to insert the closing pair of quotes, brackets, parentheses, braces etc. Download simple pairs! (I've also created one for Emacs)

I used Vim's built-in python scripting support, and seeing that it wasn't documented very well, I present some useful patterns here.

Python inside Vim script

The first thing to know is how to define a block as Python code inside Vim script. You can do it like that:

python << endpython

# your python code here

endpython

endpython can be whatever you want, as long as it is in the beginning of the line by itself. Other people use EOF, but I find it confusing.

Interfacing with Vim

Inside Python code, you can import vim and you have access to a lot of vim objects, the most useful IME being:

vim.command - execute a vim command
vim.eval - evaluate a vim expression and return the result
vim.current.window.cursor - get the cursor position as (row, col) (row is 1-based, col is 0-based)
vim.current.buffer - a list of the lines in the current buffer (0-based, unfortunately)

So to get the current line, you can use:

import vim
(row, col) = vim.current.window.cursor
line = vim.current.buffer[row-1] # 0 vs 1 based
prevChar, nextChar = line[col-1], line[col] # will IndexError if at start or end of line

Of course, you can also update the cursor position and change the line contents:

vim.current.window.cursor = (1, 0) # move to top left
vim.current.buffer[0] = "hello, world" # change first line
vim.current.buffer.append("last line!")
del vim.current.buffer[3] # also works with slices

You have to always check the bounds, or you will get IndexErrors, which are very annoying.

Python inside a Vim function

This is where things become interesting. In the simple pairs, I wanted to use the <C-R>=expression functionality, that will eval the expression and insert the results into the buffer, from insert mode. Turns out that python expr is not a valid Vim expression, so I had to wrap my Python functions inside Vim functions. This is the way to do it:

function! MyCoolFunction(anArg)
python << endpython
import vim
anArg = vim.eval("a:anArg")
# do important stuff
vim.command("return 1") # return from the Vim function!
endpython
endfunction

You use vim.eval to get the values of the vim function arguments, then you use vim.command to return from the vim function. Doing a plain Python return will not work.

You can of course set the values of variables as well, and use them later in vim script:

python << endpython
vim.command("let l:something = 1")
endpython
if l:something == 1
     return 'hi'
else
     return 'bye'
endif

Miscellaneous

You can freely mix and match vim script and python code inside a vim file, as long as you use the python << endpython markers. Each Python block is executed in the same context, so you can define your helper functions and do whatever initialization you want in a big python block in the beginning, and then just call functions and access global state as needed.

One drawback is that the tracebacks you get when you have an error don't give you line numbers (the filename is <string>), but you can still understand what went wrong from the messages themselves.

Comments

How to NOT approach people

Mon, 21 Jul 2008 23:24:07 +0200

So, you are standing in a street, asking passers-by to support your noble cause.

Here's how NOT to do it, from today's personal experience.

Pick a very busy high street road, 20 minutes before the shops close.
Ask me if I know your organization, then explain in teacher's terms what it is, when I don't respond with a 100% accurate answer.
When I tell you I really have to catch the shops, and I'll catch you on the way back, avoid the remark (it's obvious that your shift ends the same time as the shops close)
When I propose to sign-up over the Internet, inform me that "I will make your day", and go home feeling OK!
When the above doesn't work, inform me that if you don't bring in a signup by the end of the day, you may lose your job.
When I inform you (in an annoyed manner) that I don't have my bank details with me, assume a surprised and shocked stance. We both know I'm lying to get you off my back, but it's not my fault.

So, in a nutshell, in order to get people to give you money for a good cause, belittle them, appeal to their basest of emotions (you'll go home feeling OK!), then picture it as a deal: "You give me da' money, you make ma' day, you go home feeling OK, how 'bout that? It's a win-win man!".

/Rant over.

Comments

iPhone, MobileMe and 3rd party applications

Sat, 21 Jun 2008 12:52:03 +0200

MobileMe's ancestor, .Mac, had an SDK for 4 years now, although, judging from the number of applications that are actually using it, you couldn't guess that.

I think this was partly because .Mac wasn't that useful before, and because it was a chicken-and-egg problem: Developers didn't want to tie their application to a paid subscription service, and users didn't see much value in a service that only a handful of apps used.

This equation changes with the iPhone. People will want to sync data from their Desktop to their iPhone and vice-versa, the killer app being, of course, email, calendars and contacts. If a developer wants to create an iPhone application as a sidekick to his desktop application, he has to tackle sync, and MobileMe gives him a no-pain solution.

In fact, given the coming notification API, I think that MobileMe will give developers a "server in the cloud", if they don't want to implement one on their own. Example:

Imagine a "To do list" application, that can be used both on the Desktop and the iPhone. You are home, ready to leave, and you jot down a few Todos for the day on your Mac. The app pushes them up to the cloud, and a badge appears on your iPhone (3 new Todos). Editing or completing tasks on your iPhone will transparently sync back to your desktop.

I can see this happening for simple apps that only want to sync data back and forth. The example breaks down as soon as you want to add extra smarts to your iPhone application: a notification for an approaching due date. To do something like this, your server will have to issue a notification to Apple's servers to be pushed down to your iPhone. The moment you have to create your own server to support your iPhone application, the MobileMe gains break down: you might as well implement sync yourself and rid your users from a subscription.

Of course, as waffle pointed out, Snow Leopard's Calendar Server will support push notifications independently of Mobile Me, so perhaps Apple has a plan for more advanced uses. Time will tell.

Comments

PyObjC and NSOutlineView

Sat, 07 Jun 2008 21:16:55 +0200

I've been stumbled a bit just now, trying to populate an NSOutlineView from PyObjC.

When I was starting, I tried to do everything with Cocoa Bindings. Alas, it was too complicated for my small brain (although I was happily binding my NSTableView with very little fuss), so I went and created my own datasource.

Using Python, it's very simple. All you need class that implements the following:

def outlineView_numberOfChildrenOfItem_(self, outlineView, item):
def outlineView_isItemExpandable_(self, outlineView, item):
def outlineView_child_ofItem_(self, outlineView, index, item):
def outlineView_objectValueForTableColumn_byItem_(self, outlineView, tableColumn, item):

You have to always keep in mind that if item is None then it means it's your root item, and you should act accordingly: I wanted to create an Finder-like source view, so I actually had a table of plain strings (eg. PLACES, SHARED and so on).

Everything was working fine: After some false starts, I could get my labels to appear, but as soon as everything was up, I got dumped in the debugger with no information.

Tip: bt FTW

Typing bt in the XCode debugger will give you the backtrace of the offending call. In my case, it wasn't helpful (something about pythonifying). Trying continue I got: Received BAD_ACCESS (or something like that).

Diving a bit into the PyObjC examples (as I should've done earlier), I've noticed that you can't pass python objects in an NSOutlineView, because it doesn't retain it's values, so everything goes away and the next time round in the message loop there is no reference to my strings.

So the solution is to wrap everything in an NSString:

def NSStringify(l):
    return map(lambda s: NSString.alloc().initWithString_(s), l)

ROOTS = NSStringify(["AUTHORS", 'TAGS'])

Final Implementation

FWIW, my data source looks like this:

def outlineView_numberOfChildrenOfItem_(self, outlineView, item):
    if item is None:
        return len(ROOTS)
    if item == "TAGS":
        return len(self.tags)
    if item == "AUTHORS":
        return len(self.authors)
    return 0

def outlineView_isItemExpandable_(self, outlineView, item):
    if item in ROOTS:
        return True
    return False

def outlineView_child_ofItem_(self, outlineView, index, item):
    if item is None:
        return ROOTS[index]
    if item == 'AUTHORS':
        return self.authors[index]
    if item == 'TAGS':
        return self.tags[index]
    return None

def outlineView_objectValueForTableColumn_byItem_(self, outlineView, tableColumn, item):
    if isinstance(item, NSManagedObject):
        if item.entity().name() in ['Tag', 'Author']:
            return item.Name()
    return str(item)

(as you can tell, self.authors and self.tags are CoreData collections - more on that in another post).

Perhaps this post will help someone who is struggling with PyObjC.

Comments

Explosion: Aftermath

Thu, 05 Jun 2008 01:39:37 +0200

This would make a good movie title, I think.

Anyway, my blog is back up, having survived a 3-day downtime due to a huge explosion in a datacenter.

Luckily, nothing was lost thanks to Webfaction's automated backup which worked reliably.

Anyway, I was thinking about overhauling this blog (it needs a bit of refreshing) so given what it managed to survive, I'll probably tag it as "The Exploding Blog" or something like that. Any better ideas?

Comments

PyObjC Hello World

Sat, 17 May 2008 00:07:15 +0200

I'm planning to do a screencast for this, but knowing myself, I'll just put the code here in case I run off to a different direction tomorrow...

So, I present a lightly annotated hello world app in PyObjC! Only works on Mac OS X 10.5.2.

Download PyObjC tutorial (24 KB)

Contains:

One IBAction
One IBOutlet
A QTMovieView, a label and a button
Cocoa Bindings to Python attributes
Notifications and selectors

When you roll your own, here are some tips:

Don't forget that IBOutlets must be class level
Do init(self), not __init__(self)
Always initialize things you bind to (None will do), or you will get nasty obscure errors
if you have other Pythons installed (eg. Macports), do: alias macpython= /System/Library/Frameworks/Python.framework/Versions/2.5/bin/python and use that. Also remove other Pythons installed in /Library/Frameworks or Xcode might get confused.
If the app isn't launching, do macpython main.py and check for syntax errors (you can't launch it)
If you don't get the expected behaviour, check the console for swallowed errors.

Comments

Whither PyObjC?

Thu, 15 May 2008 18:20:31 +0200

If you like Python and you like Mac OS X, you should really check out PyObjC.

It's a bridge that allows you to create first-class Cocoa applications using Python. It also has support from Interface Builder and XCode in Leopard and it's already bundled with Apple's Developer Tools!

Unfortunately, if you followed the above link, it doesn't seem there's a lot of stuff going on. The examples are outdated and there hasn't been any update since January.

If you look more closely, there have been some recent checkins in the SVN trunk - the last one was on 5 May, which isn't that far away. So the project isn't dead/dying. It just needs some pushing and exposure.

The owner (or at least the person that's done most of the latest work), Ronald Oussoren hasn't written anything in his weblog for months, and indeed the only post there describes how he was using a private repository for the Leopard version, probably for NDA purposes. It's also weird that PyObjC is not hosted in Mac OS forge, Apple's open source repository.

Anyway, this post isn't a whine - it's more like a call to arms. PyObjC needs more visibility and more people using it. More activity in the mailing list, more tweets and more blog posts. I'm working on a small QTKit app to explore the API, and I plan to release it as a tutorial. If I have the time I'll make a screencast too. There's also work been done on bringing other examples up to date. Update: Example available here

If you have any experience using the latest version of PyObjC, please, jump on the mailing list and contribute!

I'm also planning on tracking the SVN repository with git - look for an announcement of the mirror in the coming days. Update: Git repository available here

Comments

Looking for the mouse

Sun, 27 Apr 2008 21:14:28 +0200

I just read this in Google Reader, and just starring it or sharing it seemed a very small thing to do. What I really wanted is to shout out of the window 'YES! THIS MAN GETS IT!'.

I was having dinner with a group of friends about a month ago, and one of them was talking about sitting with his four-year-old daughter watching a DVD. And in the middle of the movie, apropos nothing, she jumps up off the couch and runs around behind the screen. That seems like a cute moment. Maybe she's going back there to see if Dora is really back there or whatever. But that wasn't what she was doing. She started rooting around in the cables. And her dad said, "What you doing?" And she stuck her head out from behind the screen and said, "Looking for the mouse."

Gin, Television and Social Surplus

He calculates that a project like Wikipedia takes roughly 100 million work-hours to be realized. That's the amount of hours a US-sized population spends every weekend just watching ads.

I don't have a television - do I do anything productive with the time spent? Not much, but still, I click and type instead of zap.

Comments

Painless local commits: a VCS comparison

Sat, 05 Apr 2008 13:43:58 +0200

At Resolver Systems, we practice Agile techniques such as Continuous Integration, Pair Programming, Iterations and more.

One fundamental rule of Agile Programming is that There should never be a broken trunk. This means that before checking in we must make sure that every test passes. This makes checking in a very expensive operation: 4 hours for a full test run in a single machine. In the meantime, you go on working on that tree, making things very difficult to keep track of, without some form of version control.

In this post I will investigate various Version Control Systems that can be run locally: Most of them are distributed, but I don't really care about it - I don't want to replace Subversion (yet). I just want to be able to check in every hour and have a local version history, before checking in to the central Subversion repository.

I repeat: I don't want to replace Subversion. (yet). A typical workflow that commonly occurs is:

Checkout a working copy from Subversion
Hack and slash, editing and adding files.
Promote a tree to check-in status
cp tree tree-build
tree-build/run_unit_tests
go on working in tree
if unit tests pass, check in tree-build
svn update tree - This is where we get conflicts, wailing and gnashing of teeth.

After reading up a bit on the general idea of DVCSs, the workflow I will be testing is:

Import trunk (complete with history) from subversion
Make a local branch (tree) to work on
Hack and slash, editing and adding files
Commit locally as needed
Branch tree-build off tree
Run unit tests in tree build
Continue to hack and slash in tree
Check in from tree-build to subversion
Merge tree and tree-build back together, with no subversion conflicts.
Repeat

There could be minor variations of the above, and I don't really need the complete history from subversion. Hopefully the general idea is clear.

The systems I will try out are Mercurial v1.0, SVK v2.0.2, Darcs v1.0.9, Arch v1.3.5, Git v1.5.4, and Bazaar v1.3.

Requirements:

Easy install on Windows XP and Vista, preferrably without Cygwin.
SVN integration - either by supporting empty .svn directories or understarding the SVN protocol so it can communicate directly.

At the end of the day, Subversion is what everyone else will be working on in the office, so everything must end up there.

To be continued...

This article has been on the backburner for a long time, due to the fact that I aimed to test everything and then post my findings. This could take months, as every system has its own intricacies and it takes time to be able to compare between them. I will post a new entry when I have the first results ready!

Comments

Presenting truly international dates in Django

Tue, 25 Mar 2008 14:22:25 +0200

If you use a language that's a bit more complex than English (or most western languages, that is), you'll probably have some problems displaying dates that actually don't look like computer-generated.

Django-localdates deals with this problem, and tries to tackle the general problem of internationalizing dates.

The main issue is, if you want your application to be truly international, you have to find a translator that will translate your templates, messages, emails and everything else that a user might see. However, translating dates isn't that straightforward.

The easiest example is US vs British date format. US citizens write the month first, while the British write the month after the date. So, if you have a template that uses now "d F Y", it will look good for a British person, but not for an US person. A Russian would be appaled, because Март isn't the same as марта!

django-localdates tries to overcome all this by defining translatable date formats. Every language has a "formal" way to display dates, a short way, a numeric way and so on. So if we could just specify "This date should be formatted to be short" or "This day is the full deal", and have each translator specify what full or short means, we could solve this problem.

Here is the usage:

{{ datenow|ldate:"{FULL_DATE}" }} 

{{ datenow|ldate:"{NUM_DATE}" }}

and so on and so forth. Give it a try!. Instructions here.

Comments

On extreme programming

Sun, 23 Mar 2008 14:18:25 +0200

It's been over a month now at Resolver Systems, and since I have to describe what's it like to work there to a lot of friends and relatives, I'll document here what it feels to work in an Extreme Programming, using Agile techniques (I promise, no more buzzwords!).

So, programming using the above methods (see, no buzzword) basically boils down to two or three things:

You work together, on the same machine, with a different person each day
You write two tests before you write the actual feature
You have daily, short, stand-up meetings

So, without further ado, here's my experiences and thoughts for each of the above:

Pair Programming

This is by far the biggest change in the day-to-day experience as a programmer. A usual stereotype presents programmers as sad, lonely geeks that pound a keyboard at huge speeds, illuminated by the monitor glow, oblivious to the outside world in a Nirvana-like state.

You can forget all about sad, lonely geeks in Resolver Systems (judge from the faces! BTW, this needs updating :)! The fact alone that you share a desk with another human being for most of the working hours brings back a much-needed social contact in programming. As an aside, at Resolver Systems each and every one of us has a huge desk with two monitors to call his own, so we don't feel homeless.

We don't pound the keyboard at insane speeds, because it turns out that the actual coding a programmer will do is very little compared to the actual thought and design (even more so in a succinct language like Python). (Cynics will observe that as every desk has a different keyboard, we can't easily touch type anyway.) The biggest part of a day's work will be spent discussing, arguing, drawing diagrams and general designing. This leads to better overall architecture and code quality, but also, perhaps more importantly, leads to happier and sharper developers that have enjoyed a good intellectual argument with another human being.

I could go on and on about the positives of pair programming, but it also has minor drawbacks. The biggest one for me is that you don't go into the "Zone" nearly as much as when you code alone. This is countered a bit by assigning "spiking hours" to developers, when they work alone on problems that the best solution isn't obvious, so they'll hack around alone to come up with something that works. They will later pair with someone to wrap the hacky code in tests and general scrutiny.

Test Driven Development

Test Driven Development, or TDD is all about writing test code that covers a defect or a new feature before writing the actual production code.

In Resolver Systems we have two kinds of tests: Functional Tests and Unit Tests. Functional testing will launch up the actual application, move the mouse, click on buttons, type on the keyboard and check that everything is behaving as expected. Unit testing will check that an actual class or function will produce the expected results when given a set of arguments.

While Functional Tests have clear rules about what you can or can't do, some of our Unit Tests will seem more Functional, but the essence is clear: Functional Tests test a feature from the user's point of view, Unit Tests test the code from a programmer's point of view.

So, back to the day-to-day experience, when we have a new feature to implement or defect to fix, we'll go and write the Functional Test first. This, by definition, has to fail, since the feature isn't there yet or the defect isn't fixed. It's important that we actually run the test and see it fail, to make sure we test for the right thing.

After that, we'll dive into the code, to see where the fix should go. This should be obvious enough that doesn't take a long time. If we aren't sure, then perhaps this is a good candidate for a spike. In the case of a big change or a cross-cutting feature, this might take even a whole day (perhaps more). If we have to change the way a function works, we'll go and change the unit test first, see it fail, then change the function. If we have to write a new function, we'll write a test for it first, see it fail, then implement it.

Of course, this isn't always straightforward, and sometimes we'll cheat, write the function, see that it works reasonably well, then comment it out, write the tests and un-comment it bit by bit. I'm not very experienced yet in the best practices of TDD, and as days pass I'm sure it'll be more clear on how to do it.

To be continued...

This has been a long one, so I'll cover meetings and other things in a next post.

Comments

Switching to Emacs!

Thu, 28 Feb 2008 08:37:40 +0200

In the office I have to use Windows instead of my preferred OS X - no wonder, since we're developing a Windows application using .NET and IronPython.

I was quite used to TextMate by now (a very nice and powerful editor), but it has no Windows version, and it's alleged replacement, e isn't polished enough for my tastes.

After a bit of research, it seems that Emacs could suit my needs very well, since it's infinitely customizable, support python out of the box, and can be extended by either elisp or Python. It also helps that TextMate was based on Emacs, so there isn't going to be much weirdness.

This is a work in progress, so I'll state my goals below, in order of precedence.

Make it behave like a normal Windows application. The goal is to have someone used to TextPad sit on my desk and being able to do most tasks without cursing and swearing. Mostly Done using EmacsW32
Make it run a bunch of Python scripts related to our dev environment, like "Run Test suite", "Run single Test", etc. The output should go to an emacs buffer and be clickable. Also, have it jump from a module to its test and back. Mostly Done using PyMacs and a bunch of python scripts. Some nastiness there - will post further details. No clicky yet - will research that.
Have a TextMate like Python mode, with auto-indenting, snippets etc.
Have a symbol browser that will allow me to jump to a function/method in the current buffer.
Have a "Go to definition" function Mostly Done using ctags. I have to run ctags manually, but I'm sure I can automate this.

That's well enough for an Emacs newb. I'll post my findings and .emacs file as I work on it. A lot of this may already be built-in (I know there's a fairly comprehensive Python mode).

Some resources I've found, in no particular order

UPDATE: It seems that my emacs file is mature enough now. I'll copy it here for future reference. Please comment if you see anything stupid:

(setq load-path (cons (expand-file-name "C:\\elisp") load-path))

(require 'python)

(require 'linum)
(require 'highlight-symbol)
(require 'highline)

(ido-mode 1)

(defun my-python-hooks ()
  (linum-mode 1)
  (highlight-symbol-mode 1)
  )

(add-hook 'python-mode-hook 'my-python-hooks)

(global-set-key (kbd "C-S-h") 'highlight-symbol-at-point)

;; move cursor one line when going past end of page
(setq scroll-step 1)

;; jump to definition
(defun find-tag-noconfirm ()
(interactive)
(find-tag (find-tag-default)))
;; TODO: if no tags are found, try and run ctags -Rn in the source root
(define-key global-map  [f3] 'find-tag-noconfirm)
(define-key global-map  (kbd "S-") 'pop-tag-mark)

;; Camel case
(c-subword-mode 1)
(setq cua-keep-region-after-copy t)

;; no big prompts
(fset 'yes-or-no-p 'y-or-n-p)
;; no startup message
(setq inhibit-startup-message t)

(when (load "flymake" t) 
  (defun flymake-pyflakes-init () 
    (let* ((temp-file (flymake-init-create-temp-buffer-copy 
               'flymake-create-temp-inplace)) 
       (local-file (file-relative-name 
            temp-file 
            (file-name-directory buffer-file-name)))) 
      (list "pyflakes" (list local-file))))

(add-to-list 'flymake-allowed-file-name-masks 
           '("\\.py\\'" flymake-pyflakes-init)))

(add-hook 'find-file-hook 'flymake-find-file-hook)

;;;
;; Smart Tab

(defvar smart-tab-using-hippie-expand nil
  "turn this on if you want to use hippie-expand completion.")

(global-set-key [(tab)] 'smart-tab)
(defun smart-tab (prefix)
  "Needs `transient-mark-mode' to be on. This smart tab is
minibuffer compliant: it acts as usual in the minibuffer.

In all other buffers: if PREFIX is \\[universal-argument], calls
`smart-indent'. Else if point is at the end of a symbol,
expands it. Else calls `smart-indent'."
  (interactive "P")
  (if (minibufferp)
      (minibuffer-complete)
    (if (smart-tab-must-expand prefix)
        (if smart-tab-using-hippie-expand
            (hippie-expand nil)
          (dabbrev-expand nil))
      (smart-indent))))

(defun smart-indent ()
  "Indents region if mark is active, or current line otherwise."
  (interactive)
  (if mark-active
      (indent-region (region-beginning)
                     (region-end))
    (indent-for-tab-command)))

(defun smart-tab-must-expand (&optional prefix)
  "If PREFIX is \\[universal-argument], answers no.
Otherwise, analyses point position and answers."
  (unless (or (consp prefix)
              mark-active)
    (looking-at "\\_>")))

;; Use external python programs, using pymacs
(setenv "PYMACS_PYTHON" "C:\\Python25\\python.exe") 
(load "C:\\elisp\\pymacs.el")
(autoload 'pymacs-apply "pymacs")
(autoload 'pymacs-call "pymacs")
(autoload 'pymacs-eval "pymacs" nil t)
(autoload 'pymacs-exec "pymacs" nil t)
(autoload 'pymacs-load "pymacs" nil t)

;;(eval-after-load "pymacs"
;;'(add-to-list 'pymacs-load-path "C:/elisp/python"))
;; The above doesn't work, so I hack around.
(pymacs-load "PYMACSPATH")
(if (pymacs-load "rsl-util")
    (define-key global-map "\M-r" 'rsl-util-run-resolver)
  (define-key global-map  [f5] 'rsl-util-run-ironpython-file)
  (define-key global-map  [f6] 'rsl-util-run-single-test)
  (define-key global-map  [f4] 'rsl-util-jump-to-test)
  )

(python-setup-brm)

(defun close-window-and-buffer()
  (interactive)
  (kill-this-buffer)
  (delete-window)
  )

(setq skeleton-pair t)
(global-set-key "[" 'skeleton-pair-insert-maybe)
(global-set-key "(" 'skeleton-pair-insert-maybe)
(global-set-key "{" 'skeleton-pair-insert-maybe)
(global-set-key "\"" 'skeleton-pair-insert-maybe)
(global-set-key "'" 'skeleton-pair-insert-maybe)

;; Define some additional "native-Windows" keystrokes (^tab, Alt/F4, ^A, ^F, ^O,
;; ^S, ^W) and redefine (some of) the overridden Emacs functions.
(global-set-key [C-tab] 'other-window)
(global-set-key [M-f4] 'save-buffers-kill-emacs)
(global-set-key "\C-a" 'mark-whole-buffer)
(global-set-key "\C-f" 'isearch-forward)
(global-set-key "\C-z" 'undo)                   
(global-set-key "\C-o" 'find-file)
(global-set-key "\C-s" 'save-buffer)
(global-set-key "\C-n" 'make-frame)
(global-set-key "\C-w" 'close-window-and-buffer)
(global-set-key (kbd "C-S-o") 'open-line)
(global-set-key (kbd "C-S-w") 'kill-region)
(define-key global-map (kbd "RET") 'newline-and-indent) ; For programming language modes
(define-key isearch-mode-map "\C-f" 'isearch-repeat-forward)

(custom-set-variables
  ;; custom-set-variables was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 '(column-number-mode t)
 '(cua-mode t nil (cua-base))
 '(emacsw32-max-frames nil)
 '(emacsw32-mode t)
 '(emacsw32-style-frame-title t)
 '(global-highline-mode t)
 '(highline-priority 1000000)
 '(htmlize-view-print-visible t t)
 '(ido-default-file-method (quote other-frame))
 '(ido-enable-flex-matching t)
 '(ido-ignore-directories (quote ("\\`CVS/" "\\`\\.\\./" "\\`\\./" "\\`SVN/")))
 '(make-backup-files nil)
 '(menuacc-active t nil (menuacc))
 '(recentf-mode t)
 '(show-paren-mode t)
 '(speedbar-directory-button-trim-method (quote trim))
 '(speedbar-track-mouse-flag t)
 '(speedbar-use-imenu-flag nil)
 '(swbuff-y-mode t)
 '(transient-mark-mode t)
 '(w32-meta-style (quote emacs))
 '(w32-print-menu-show-print nil)
 '(w32-print-menu-show-ps-print nil)
 '(w32shell-shell (quote cmd)))
(custom-set-faces
  ;; custom-set-faces was added by Custom.
  ;; If you edit it by hand, you could mess it up, so be careful.
  ;; Your init file should contain only one such instance.
  ;; If there is more than one, they won't work right.
 '(highline-face ((t (:inherit highlight :background "#EEEFEF"))))
 '(linum ((t (:inherit shadow :background "#EEEFEF" :slant italic :weight extra-light)))))

In my elisp file, I have:

linum.el
python.el
sym-comp.el
pymacs.el
highlight-symbol.el
highline.el

and in elisp/python I have my rsl-util.py scripts that are project specific.

Please share!

Comments

Porting to numpy

Sat, 09 Feb 2008 17:52:19 +0200

As you may know (you probably don't :), I've built a greeklish to greek converter for my Diploma thesis, using Python.

It worked well enough (for a diploma thesis), but it was slow. I mean, really, really slow. It was coded in Pure Python (tm) and had no optimizations whatsoever. The main part of the code was an implementation of a Hidden Markov Model:

In a regular Markov model, the state is directly visible to the observer, and therefore the state transition probabilities are the only parameters. In a hidden Markov model, the state is not directly visible, but variables influenced by the state are visible. Each state has a probability distribution over the possible output tokens. Therefore the sequence of tokens generated by an HMM gives some information about the sequence of states.

I was using dicts mainly, to map between a state and a probability, but what I really wanted was a matrix. I went and replaced my dicts with lists of lists, which yielded a performance improvement (a dict lookup may be constant-time, but an index in a list will be faster, because its constant is smaller). I also did some cacheing and other minor stuff, like instead of waiting for KeyErrors or IndexErrors I went and filled up the "matrix", which also helped. Unfortunately, when all that is done, you realize that Python isn't a language designed for number crunching. You have to use something else to do the heavy lifting.

Why numpy?

A usual argument of dynamic language fans when confronted with the "dynamic languages are slow" issue, is, well "you can always rewrite the critical parts in C". Python is supposed to make that easy, using ctypes or SWIG or other libraries. I don't know, I'm not really comfortable doing C, and doing numeric stuff in C seems scary to me, as you'll have to deal with precision and all that weird stuff.

So I decided to use numpy, as the de facto standard when doing numerical applications in Python. It was very straightforward: Instead of using:

a = [[1,2],[3,4]]

you can use:

a = array([[1,2],[3,4]])

and lo and behold, all your code still runs!

Not so fast, mister

Well, actually it runs slower! Huh? How come?

Apparently, just switching to numpy arrays isn't good at all, if you still are looping over them in Python. Nooo, you have to rewrite all your functions to take advantage of the numpy speedup. This can be quite tricky and will lead to hair pulling.

The walkthrough

I'll repeat here the steps I've took to implement the forward algorithm as described in L. Rabiner's HMM tutorial. It's simply 3 equations (19, 20, 21), mostly doing sums and multiplications.

I'll spare the irrelevant details. Assuming A, B, pi are the parameters of the HMM, N is the number of the hidden states and given obs, a list of observation indices, here is the naive code, mostly copying from the book:

#19
T = len(observation_list)
a = zeros((T,N))
for i in range(N):
    a[0][i] = pi[i]*B[obs[0]][i]

#20
for t in range(1,T):
    for j in range(N):
        for i in range(N):
            a[t][j] += a[t-1][i]*A[i][j]
        a[t][j] *= B[obs[t]][j]

#21
prob = 0
for i in range(N):
    prob += a[T-1][i]

The pattern is roughly, the sum (or product) of something over i, from 1 to N, turns into a for i in range(N). Let's see how this can be rewritten to use numpy.

First of all, don't delete the old function! We'll interleave the new code and sprinkle assertions in between to make sure we don't make mistakes. If you have unit tests (you should!), you can make sure later that everything still ticks, but while rewriting it helps to take baby steps.

For function #19, the numpy code is this:

a_ = zeros((T, self.N))
a_[0] = pi*B[obs[0],:]

assert((a==a_).all())

It's apparent that the conversion "algorithm" is:

Use the numpy element accessor [x,y] instead of [x][y]
Replace "i" (the counter) with ":"
If you have just a vector, remove the square brackets entirely.
Remove the for loop
Assert that each element of the numpy array is the same with naive array.

Things to remember:

The "*" operator for numpy arrays is element-wise multiplication, instead of dot product.
The ":" accessor works a bit like a slice-operator. I like to thik of it as wildcard

What the above code is doing boils down to:

"Multiply each element of pi with the corresponding element of the slice obs[0] of B, and assign the resultant array (a vector, really) to a_[0]"

With that in mind, we can easily deduce that for function #21 the code will be:

prob_ = sum(a_[T-1,:])

assert(prob_==prob)

Which means, "sum the elements of the slice T-1 of array a_"

Function #20 is a bit more complex, and I'll go step-by-step:

First, there is a sum:

for i in range(self.N):
    a[t][j] += a[t-1][i]*A[i][j]

Which can be easily replaced with

a_[t][j] = sum(a_[t-1,:]*A[:,j])

using the above algorithm. We now have:

for t in range(1,T):
    for j in range(self.N):
        a_[t][j] = sum(a_[t-1,:]*A[:,j])
        a_[t][j] *= B[obs[t]][j]

There is also a product (a_[t][j] *= B[obs[t]][j]), that can be swapped with a_[t,:] *= B[obs[t],:]

The code now becomes:

for t in range(1,T):
    for j in range(self.N):
        a_[t][j] = sum(a_[t-1,:]*A[:,j])
    a_[t,:] *= B[obs[t],:]

Only one loop to go! However, that sum is trouble. When I tried to apply the usual "replace-i-with-:" routine, I got an AssertionError (that's why you have assertions!). After some minutes with pen and paper, it occured to me that the sum operator reduced the array to a single number, when clearly I expected a vector. So I needed an operator multiplied two arrays, did per row sums and return an array.

OK, duh, that's the dot product. My linear algebra is rusty, I know. So now the code becomes:

for t in range(1,T):
    a_[t,:] = dot(a_[t-1,:],A)
    a_[t,:] *= B[obs[t],:]

Or, in one line:

for t in range(1,T):
    a_[t,:] = dot(a_[t-1,:],A)*B[obs[t],:]

UPDATE: Of course, when implementing the viterbi algorithm, I came across this piece of code:

    for t in range(1,T):
        for j in range(self.N):
            max_daij = 0.0

            for i in range(self.N):
                tmp_daij=d[t-1][i]*A[i][j]
                if tmp_daij>=max_daij:
                    max_daij=tmp_daij
                    psi[t][j] = int(i)
            d[t][j] = max_daij*B[obs[t]][j]

It seems that max and argmax should be used. This translates to:

    for t in range(1,T):
        max_daij=amax(d_[t-1,:]*A.T,1)
        psi_[t,:] = argmax(d_[t-1,:]*A.T,1)
        d_[t,:] = max_daij*B[obs[t],:]

Things to notice: I used from numpy import max as amax so as not to complicate things. Also note the .T syntax. This is the transposed array. It's quite cheap as an operation, as it returns just a transposed view to the data. I should've done this testing months ago. Sigh.

Conclusion

That's not bad. The final result has less cruft, at the expense of more concentration. I actually kept the interleaved version so I can understand what's going on if I revisit the function.

I hope I haven't done anything stupid, since the data I use aren't that complicated, and in my experience, you really need big datasets to test against when developing numerical applications.

Comments

Η Ελλάδα ταξιδεύει

Sun, 03 Feb 2008 13:01:00 +0200

Η Ελλάδα ταξιδεύει, ρυπαρό καράβι, και θυμήθηκα το όραμα του Καζαντζάκη όπως το αποτύπωσε στον “Ζορμπά”: «και μέσα στο βαπόρι οι τετραπέρατοι Ρωμιοί, τα μάτια τα αρπαχτικά, τα ψιλικατζίδικα μυαλά, οι μικροπολιτικοί καβγάδες, ένα ξεκούρδιστο πιάνο, τίμιες φαρμακερές κυράτσες, μοχθηρή, μονότονη επαρχιώτικη μιζέρια. Σου ερχόταν να πιάσεις το βαπόρι από τις δύο άκρες, να το βουτήξεις στη θάλασσα, να το τινάξεις καλά-καλά, να φύγουν όλα τα ζωντανά που το μολεύουν – άνθρωποι, ποντίκια, κορέοι – και να το ανεβάσεις πάλι απάνω από τα κύματα, αδειανό και φρεσκοπλυμένο».

από τον Νίκο Δήμου

Comments

Regex - Look behind you

Tue, 29 Jan 2008 02:41:27 +0200

In response to Michael's post , about a nasty regex...

Yesterday I fooled around with lookbehinds and lookaheads and what not. Very weird stuff :)

If I've understood the problem correctly I think the solution could be:

 >>>import re
 >>>s='test\ntest2\ntest3\r\n\n\ntest4\r\ntest5\n\n\n\n'
 >>>r= '(?<!\r)\n'
 >>>re.sub(r,'\r\n',s)
 'test\r\ntest2\r\ntest3\r\n\r\n\r\ntest4\r\ntest5\r\n\r\n\r\n\r\n'

Comments

Switched to Webfaction

Sat, 26 Jan 2008 10:14:43 +0200

If you can see this, then the move to Webfaction was successful!

My subscription with a2b2 is about to expire and I don't think I really needed the VPS... So I switched to webfaction. The blog is the first piece - I'll move the other projects GreeklishOUT! and PlanetCocoa next this weekend.

Comments

New job, and a move

Thu, 24 Jan 2008 23:44:49 +0200

Well, it's official so I can break the word: I've got a job with Resolver Systems and I'll be moving to London in about 3 weeks!

Resolver Systems is a small startup (I guess it's started now!) that creates a spreadsheet using IronPython. What does it do that excel doesn't? Why, of course, it exports your complete spreadsheet, data and formulae as Python code! And you can use Python to do whatever you want with your data, right inside the program.

If you've created or used an Excel file that resembles spaghetti (you know, formulas, macros and hidden cells all over), I suggest you give it a try! You can see the screencast, it's only a minute. I'm sure you'll be hooked.

Did I mention that it's free for non-commercial use? :)

Anyway, Resolver Systems is based in London, which means I'm moving away! I've already found a small flat in Swiss Cottage, a lovely area. I'm flying out on 13 February.

Can you tell I'm excited?

Comments

My 2008 resolutions

Mon, 31 Dec 2007 17:04:14 +0200

A new year is coming in a few hours, and though the milliseconds passed since 1/1/1970 don't care a lot, it seems that people do. So here are my resolutions for 2008:

1. I will write more to this blog

It seems that I've failed the mission of this blog, which is to force me to finish what I start in order to write about it. There are a lot of loose ends in need of tying, and I intend to tie them and blog about them. If I'm lazy or too busy I will at the least put something up, be it a book review, an opinion or a link to something I find interesting.

2. I will become a better developer

The first step was the hardest, but I think that there are good times ahead. A bit cryptic, I know: stay tuned for official news :)

3. I will tie up loose ends

See #1. Nothing better than crossing things out of a check list.

Well, that's enough for a year don't you think?

A happy and prosperous happy new year to everyone!

Comments

Python's getattr

Mon, 10 Dec 2007 06:26:56 +0200

An interesting question

So, yesterday I was asked an interesting question:

Int: Are you familiar with Python's getattr?
Me: Um, yes?
Interviewer clarifies what it's all about
Int: So I now want you to implement __getattr__ in such a way that when a method is called with the prefix print it'll print it's name before calling it.

Before that, we had a discussion about decorators, where I created a decorator called printme:

def printme(func):
    def decorated(*args, **kwargs):
        print func.func_name #well, it was "print func", but now I know better :)
        return func(*args,**kwargs)
    return decorated

So after a bit of confusion (writing code like that in a piece of paper with two pairs of eyes looking at you can be tricky), I ended up with something that may have looked like this:

class Foo(object):
    def __getattr__(self, name):
        if name.startswith('print_'):
            f = getattr(self, name[6:]) #actually, I did super(Foo, self) but that won't work.
            return printme(f)
        return getattr(self, name)

    def hi(self):
        print "Hello!"

Can you spot the error?

.
.
.

Well, we didn't have access to a console, so we didn't. If you're lazy and didn't pasted the above into a console, here are the results:

>>> f = Foo()
>>> f.hi()
Hello!
>>> 
>>> f.print_hi()
hi
Hello!

So far, so good. But:

>>> f.blah()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 6, in __getattr__
  File "<stdin>", line 6, in __getattr__
  .
  .
  .
  File "<stdin>", line 6, in __getattr__
  File "<stdin>", line 6, in __getattr__
  File "<stdin>", line 6, in __getattr__
RuntimeError: maximum recursion depth exceeded

Seeing that this was a topic I wasn't sure about, I came home and searched about it, which resulted in this blog post. So, after this quite big introduction, a small tutorial on getattr

`getattr`, `getattr` and `getattribute`

So, what is this all about? Python, as you may know is a Strong & Dynamic typed language. We don't care about Strong here, but the Dynamic part is very interesting. Dynamic languages don't care at the compile time the nature of the constructs that are used - they do all the work at run time. So, for example, when you call method "bar" on instance "foo" (foo.bar()), the compiler won't check if there is a void method named bar in the class tree of foo.

So, how does it work? There has to be support for looking up methods at runtime. This is what Python uses getattr, __getattr__ and __getattribute__ for. (Objective-C has similar support, though I haven't got that far yet. I don't know about other languages.)

In essence, consider the following snippet:

>>> class Foo(object):
...     def hi(self):
...             print 'Hello there!'
... 
>>> globals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', 'Foo': <class __main__.Foo at 0x54c30>, '__doc__': None}

What is interesting here is the entry Foo at the globals dictionary. Let's keep that and move on.

>>> f = Foo()
>>> globals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', 'Foo': <class __main__.Foo at 0x54c30>, '__doc__': None, 'f': <__main__.Foo instance at 0x60ad0>}

Again, f is mapped to a Foo instance, in the globals dictionary.

>>> dir(f)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'hi']

What's this? Our instance has an attribute named hi. How can we access this?

>>>f.hi
<bound method Foo.hi of <__main__.Foo instance at 0x60ad0>>
>>> f.hi()
Hello there!

Nice, but boring. This isn't using any features of dynamic typing. Let's see what getattr can do.

`getattr`

>>> getattr(f, 'hi')
<bound method Foo.hi of <__main__.Foo instance at 0x60ad0>>

That looks familiar, but now we're using a string to specify what method we want.

>>> getattr(f, 'hi')()
Hello there!

Nice, right? But this isn't getting us any closer to the answer of the above question, so let's proceed. What we want to do is somehow intercept the built-in atrribute lookup (methods are, after all, attributes) and do some work of our own.

`getattribute`

This is where it gets a bit complicated. You may have noticed a method called __getattribute__ in the results of dir, above. So at first glance you'd want to implement this, but wait! Let's check what the reference has to say:

__getattribute__( self, name)
Called unconditionally to implement attribute accesses for instances of the class. If the class also defines __getattr__(), the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError. This method should return the (computed) attribute value or raise an AttributeError exception. In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, "object.__getattribute__(self, name)".

That is, if you implement this method, you are overriding the built-in attribute access mechanism! So tread carefully, and notice that you can easily trigger infinite recursion. Also, this method is only available with new-style classes.

`getattr`

Back to where we started. Let's see what the reference is saying about __getattr__:

__getattr__( self, name)
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.

Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.

So __getattr__ will be called only when an attribute is not found. This gives us a hint to why we got infinite recursion in the first example. Let's revisit that:

def __getattr__(self, name):
    if name.startswith('print_'):
        f = getattr(self, name[6:]) #actually, I did super(Foo, self) but that won't work.
        return printme(f)
    return getattr(self, name)

This will be called if there is an attribute lookup that fails. In the given specifications, we suppose that someone will try to call print_whatever. However, if something doesn't start with print_, I tried to get the normal behaviour, which is to raise an AttributeError calling it again. I should've either raised AttributeError manually, or used the default implementation, like this:

return object.__getattribute__(self, name)

Of course, this works only for new-style classes, but if you're implementing this for something new, why go old-style?

Final Thoughts

This probably came out more boring than I thought, though it was a good clarification for me. The uses for a technique like this are endless. The first that comes to mind is a Rails-like Active Record class that is empty but responds to messages that correspond to table columns. Another is a proxy object, that routes certain methods over to the real thing, perhaps with cacheing of results and so on.

Comments

Leopard Calendar Store

Thu, 15 Nov 2007 19:27:54 +0200

So, over at the Achaia there was a request for a little to-do application, so I've stepped up to the challenge...

This was an excellent opportunity to create a real application that will see some use. The results so far are really encouraging:

Bindings are really cool, but I think they need to be a bit more extensible. I have to study the documentation a bit closer. Also custom drawing (via subclassing) in Cocoa is a bit hit-or-miss. Maybe I haven't found the documentation for that, either, but I had to get a piece of code from iLifeControls to see how it all works.

I have no idea why NSPopUpCell will not display either an attributedString or an image when in a table view. Weird.

Calendar store seems a bit beta and rough about the edges, but it does what it says...

So stay tuned for the imminent release of TodoLing beta...

Some Arsers mentioned donating, so I thought, what the heck, if anyone wants to send me money:

UPDATE: Get the beta version here!

Comments

Messing around with windows

Mon, 24 Sep 2007 22:22:38 +0200

I wanted to create a custom window, but NSBorderlessWindow is ugly, because you have to implement most of the functionality yourself. Taking a page from Uli, I extended that functionality a little bit.

I present you with my hacky box:

UPDATE:

Anyone who has tried to use the code below, would get this result:

A method went missing. I've added it to the code below.

Notable things:

This is just a textured window. All usual functionality (resizing, exposé, shortcuts) is there.
Notice the location of the close/miniaturize/zoom buttons. They are moved to the right.
The resize handle is also moved to a better location.
The window title (aptly named, "Window") is the usual one, set in IB.
No borders are drawn.
When resized, the image scales with the window.
I can't draw :)

To do this, download the above mentioned class, and add the following methods:

- (struct _NSPoint)_closeButtonOrigin
{
    NSPoint point =  [super _closeButtonOrigin];
    point.x += 35.0;
    point.y -= 5.0;
    return point;
}
- (struct _NSRect)_titlebarTitleRect
{
    NSRect res = [super _titlebarTitleRect];
    res.origin.y-=6.0;
    return res;
}
- (struct _NSRect)_maxXminYResizeRect
{
    NSRect res = [super _maxXminYResizeRect];
    res.origin.x-=13;
    return res;
}
// to make the borders disappear
+ (void)drawBevel:(struct _NSRect)fp8
          inFrame:(struct _NSRect)fp24
 topCornerRounded:(char)fp40
bottomCornerRounded:(char)fp44
{
    return;
}

Can't help with the drawing skills, sorry. You have to be born with them.

Of course, all this is undocumented and will probably break tomorrow... But still, it's fun and a much better way than reimplementing everything in a borderless window!

Comments

Προσοχή στη Viva phone!

Fri, 14 Sep 2007 01:44:39 +0200

Η Viva phone, η γνωστή εταιρεία που μοιράζει κάρτες συνομιλίας με εξωτερικό σε χρέωση αστικής κλήσης, έχει κάποια πολύ ενδιαφέροντα ψιλά γραμματάκια...

Συγκεκριμένα, το κουπονάκι (τουλάχιστον αυτό που έχω στα χέρια μου, το πήρα από το Mall), γράφει

με αστική χρέωση*!!!

Προσέξατε το αστεράκι; Απο κάτω εξηγεί:

*Κόστος αστικής κλήσης προς Viva-Telecom Revolution

Πόσο άραγε είναι αυτό το κόστος; Στο κουπόνι δεν το γράφει, και νομίζω ότι αυτό και μόνο φτάνει για να γίνει καταγγελία. Έψαξα στη σελίδα τους, και στο κομμάτι "Χρεώσεις της Υπηρεσίας" το λέει -προς τιμήν τους αυτή τη φορά- καθαρά:

Δηλαδή, χρέωση 0,0364 € ανά λεπτό (με Φ.Π.Α) με χρέωση ανά δευτερόλεπτο.

Σίγουρα είναι φτηνότερο από απευθείας εξωτερικό, αλλά άλλες υπηρεσίες όπως το Skype και το Voipbuster είναι πολύ φτηνότερες: €0,017 ανά λεπτό (χωρίς ΦΠΑ) και €0,010 ανά λεπτό (με ΦΠΑ) αντίστοιχα.

Από την εμπειρία μου, το Skype δουλεύει αρκετά καλά. Δεν έχω δοκιμάσει ακόμα το Voipbuster, θα το δοκιμάσω μόλις μου τελειώσουν τα λεφτά στο Skype, δηλαδή αύριο μεθάυριο (ας όψεται η ξενιτιά!).

Comments

Cocoa tutorials list

Wed, 12 Sep 2007 10:08:28 +0200

Andy Matuschak had a great idea: let's start organizing cocoa tutorials by tagging them with cocoatutorial in del.icio.us.

I've stumbled upon quite a few tutorials (unfortunately, most where old and obsolete) when I've started learning Cocoa, and I think a central repository is a great idea. Cocoadevcentral and CocoaBlogs are two excellent resources, but it seems as progress has stopped. This is what led me to start Planet Cocoa.

I think that Planet Cocoa and cocoatutorial complement each other well: The first as a way to stay up-to-date with new posts about Cocoa-stuff (I've tried to make it as code-specific as possible), the second as a big archive of interesting stuff. Both need your love, so if you have lots of Cocoa articles in your bookmarks, give them a cocoatutorial tag! If you blog about Cocoa, send me an email at planet@planetcocoa.org so I can add it to Planet Cocoa!

Comments

The joy of Eclipse RCP

Fri, 17 Aug 2007 11:53:35 +0200

I've decided to run again an RCP project I've created one year ago. So, I've downloaded the new Eclipse 3.3 with the RCP/PDE bundles, point it to my old workspace and... tore my hair out in frustration.

Word to the wise

I've forgotten, but now I remember again: Don't try to develop on Eclipse RCP. I like the IDE, it's very powerful, but the RCP platform is so much over-engineered that it's not funny anymore.

I don't object to over-engineered and complication frameworks, but this has to stop. When you cannot understand why your project (that ran under 3.2) refuses to start, and there are 5 or 6 places you have to check (only to find nothing, mind you), there are problems. Can they be overcome? I don't know.

Java is not dynamic enough, XML is not strong typed.

The main issue, as I see it, is that Java is static typed. So, a lot of fancy runtime configuration had to be bolted on using huge XML files and special code that parsed them. This is, in a sense, what the Eclipse platform is all about. You can activate and deactivate plug-ins at will, you can plug-in into pretty much everywhere and provide custom behavior and whatnot.

The catch is that this tends to be very complicated, very fast. They've taken the age old mantra of frameworks:

Simple things should be easy, complicated things should be possible.

And they've turned it on its head:

Simple things are complicated, complicated things are easy.

Why? Well, because getting tangled into a myriad of configuration files and special secret extension points is surely a thing you'd have to bear with in order to get advanced functionality, but you have to do it even for the simplest case.

What to do?

I understand that the Eclipse Foundation is now a big organization with a lot of stuff under its belt, and it has to follow through. But I think they must take a step back, and really simplify the Rich Client Platform. If only it was:

Simpler to grasp
More documented
Refactored for the common case

I think that most of the issues I'm facing would be resolved.

Nomenclature

Just for fun, here is a list of jargon I had to remember, just to try to run my misbhaving application:

product, bundle, application, plugin, osgi, realm, registry, extensions, extension points, dependencies, runtime packages, MANIFEST.MF, plugin.xml, build.properties, plugin_customization.ini, config.ini, my.product, features, fragments, Activator, Buddy, Buddy-Policy

Nice huh?

Comments

Using NSImageView with Core Data and Cocoa Bindings

Wed, 15 Aug 2007 23:23:30 +0200

I've been toying around with Cocoa, Bindings and Core Data, and I have three words to say:

ABSOLUTELY.FUCKING.AMAZING.

Wow. This technology is sorely missing from every other toolkit! It is amazing how much you can accomplish without any code! I'll try to re-implement my failed application in Cocoa, just for the hell of it.

Anyway, I found that in order to save image as data to a Core Data model, and be able to view it in an NSImageView, you have to do a little mucking around:

First, define the attribute in your entity as Binary Data. Obvious.
Drop an NSImageView somewhere on your window. Make sure it is enabled.
In the Bindings options, bind it to data (not value!), and use the NSUnarchiveFromData value transformer.

Done! Value transformers aren't heavily documented, but I got through. The problem was that Apple refers to NSUnarchiveFromDataTransformerName which is a constant that points to NSUnarchiveFromData, while Interface Builder uses the value.

Anyway, small peeve :)

Comments

New MacBook - First impressions

Wed, 15 Aug 2007 15:39:23 +0200

So, I got a MacBook. I've been using Mac OS X for two weeks now, and I can post my impressions, as a previous 100% Windows XP user.

Hardware

General & Display

The MacBook is a cool little machine. I use it mostly with an external screen, keyboard, mouse, speakers, usb hub so I will not comment on the build quality, keyboard, trackpad, screen etc. You can find all these kinds of comments all around the Internet. I only have to say that the screen is very bright, in my setting I have no problem with the glossiness. There are a lot of configuration options for the screen arrangement. I have my Samsung SyncMaster 205BW as my main screen in its native resolution (1680x1050), and to the left I have the MacBook propped using its display as a secondary, in its respective native resolution (1280x800). Very nice.

Speed

I find the responsiveness very fast --- a lot faster than my old laptop that had a Pentium M@2 GHz. I have 2 GB of memory, and I only find I'd like more when I have VMWare Fusion running, which allocates 512MB of RAM to the virtual machine. Hard drive performance is as expected from a laptop drive: bad :)

Accessories compatibility

I've used two external drives (had to reformat one to FAT32 and the other to HFS+ in order to be able to write), an iPod Shuffle, a Sony Ericcsson K750, two MS Keyboards (2000 & 4000), an MS Laser Mouse 6000, a Canon Pixma iP4200 printer and a USRobotics 9108 ADSL Modem/Router/Print server.

My experience:

Drives work well. You can only read NTFS, not write, that's why I reformatted.
Shuffle works well (duh!)
The SE K750i syncs over Bluetooth, and via a hack I can send/receive SMSs and dial contacts in Address Book. The USB cable allows me to mount the filesystem natively, but I get some weird kernel panics so I'll have to look into it.
MS Keyboards and Mouse: I actually find that the Mac version of the IntelliPoint and IntelliType are actually better than the Windows versions! All the extra keys can be remapped to most useful functions.
The Canon iP4200 printer works well plugged in (using the Canon drivers). However, I've had major difficulties using the printer when plugged into the USRobotics print server - I've searched around for a solutions, I'll post my results here.
The wireless router works fine.

Software

General

I've studied a lot about the differences between Mac OS X and Windows, so I was prepared for most pitfalls. Some that got me (and still get me):

The behavior of Home/End/Page up/Page down. I hope I'll get used to it sometime. I know I can change it, but I don't want to (yet).
The modifier keys. Ridiculous. Ctrl is Control, Alt/⌥ is Option, ⌘/ is Command. The fact that I'm using a Windows keyboard doesn't help, though the IntelliType software supports switching around the keys so that they match the physical layout on the Mac keyboard.
The Finder is a bit wonky - I now understand the FTFF crowd --- I hear that come October it'll be better.
I've had a few problems with Greek support --- mostly font issues. It seems that most of the built-in fonts don't have Greek glyphs, and this is very annoying, as Firefox doesn't replace the fonts but instead displays "?". The first run of MS Word test drive installed some more fonts, and this resolved all of my issues about Firefox. Still, the lack of built in fonts makes some applications awkward to use as I only get one or two fonts for everything. I'll have to look into this further.
I got some Kernel Panics (the equivalent of the Windows BSOD), for no apparent reason --- not good.

However, overall I enjoy the experience, and I think it is indeed a better system than Windows for my daily use.

Replacement apps

I've had no trouble. The only apps I haven't used before in Windows are the iLife suite, the iWork suite, Transmission for torrents, Textmate for editing (very nice), XCode and family. I also use QuickSilver, mostly as a launcher for now.

Development

I've found that Python is a first-class citizen on the Mac, as there are a lot of bridges, such as an AppleScript bridge , the PyObjC bridge and a lot of other stuff I haven't looked up . I had to install the latest version of Python but it was really very simple.

I really love Interface Builder. Once I figure out how to use the PyObjC to use Python instead of Objective-C I'll be very happy. I'll wait for Leopard before I dig into Objective-C programming, as I don't really want to use a non-garbage collected language right now...

Overall experience

I like it :) I'll post of my findings and thoughts here, so keep your eye on this blog...

Comments

New MacBook, new hand...

Fri, 03 Aug 2007 06:10:56 +0200

In other, less cryptic words, I bought a MacBook and I spend most of my time familiarizing myself with the new goodness.

Also, I finally had the operation in my elbow, that will hopefully drive the annoying numbness in my left hand away. Unfortunately, 18 or so stitches, a crutch and lots of bandages mean that I do all my typing with one hand. Let me tell you, computers weren't designed to be used one-handed (all you sickos can stop giggling now :).

Anyway, I'm back from Lesvos, I'll have the crutch removed in 6 days, and hopefully I'll have the MacBook setup for all kinds of software development until then.

Oh, and stay tuned for new posts regarding my experience with OS X :) (hint: I love it so far, mostly)

Comments

Announcing django-localdates

Tue, 17 Jul 2007 14:18:55 +0200

localdates is a django app that brings local date presentation to django, by providing custom date filters that can use local-flavored format strings.

Reasoning

I'm Greek, and the greek language is a lot more complicated than english. There are genders and cases and more stuff that change the suffixes of the words, depending on the use. This is very annoying when trying to create a multilingual site, or when using a multilingual framework like Django. As Malcolm Tredinnick , one of Django's lead developers has put it:

(1) The fundamental reason this (date formats) is a problem at all is because we are attempting to construct grammatical sentences out of short fragments. A guiding principle in creating translatable strings is to create complete strings as often as possible because constructing sentences from fragments is very locale specific.

From: the relevant discussion, continued from ticket 4147

One of the most visible aspects of this problem is, as you may have guessed, date presentation. Date is a very common function in most sites, and having it appear correctly is a major issue. Up to now, if you wanted a date to look correct in Greek, the only option was to use an all-numeric format like this: 2/8/1982 (that's my birthday, but I bet you aren't sure if I was born on August 2 or February 8. More on that later on.)

The problems

I've described the one problem above -- mainly, the peculiarities of grammar and syntax present in many languages. I've hinted about the second problem in the last paragraph: The way people around the earth prefer to structure a date phrase. Greeks (and Germans, and British AFAIK) prefer to write dates like DD/MM/YYYY, while folks from the states prefer MM/DD/YYYY. There are lots of different formats, which is my point: I'm not supposed to know them all!.

Django comes with translations for many languages, so if I want to display the admin application in, say, Portuguese, I can, because the nice django folks from Portugal have translated it. But the dates will still look wrong, because you can't translate a date format string.

Actually, I tell a lie. You can do that. In django.utils.translate_real, you can find some interesting code that defines some default formats. I only found that today, I'm not sure when it got in or how should I use it.

But anyway, here's my take:

Most of the time, we just use standard forms of date presentation, like formal date, abbreviated date, numeric date and so on.
Developers don't know what the correct date presentation for each locale is - the translators know that.
Django's dateformat can't handle all the peculiarities of international date formats, because of point (2) - the developers can't be expected to know that.

Ergo, we need a way to let translators specify their date formats and implement their language's way of displaying them. Here is where localdates comes in.

Introducing localdates

Localdates, for the end user (a developer who likes his nice dates) is just a replacement for the date filter, called ldate:

I was born on {{my_birthday|ldate:"{FULL_DATE}"}}!

Or maybe:

Γεννήθηκα στις {{my_birthday|ldate:"j {Fp}, Y"}

Update: You can also create your own translatable date strings, like this:

My birthday is on {{my_birthday|ldate:"{_(MY_BIRTHDAY)}"}

You have to put manually this msgid (MY_BIRTHDAY) in the django.po file of your project. The ldate filter will handle the translation.

You'll notice how one can specify a standard string ({FULL_DATE}), and it works out automagically. You'll also notice how there is a new format string that is enclosed in brackets ({Fp}). Here is where the translators come in.

New format strings

These new format strings are language-specific. The translators decide what their names will be, and how will they work. Developers who create sites in one language can use them freely, while others that create multilingual sites should use the standard strings.

Localdates looks in django.contrib.localflavor to find a module named dateformat, that has a class named LocalDateFormat. Within this class are two-letter functions that both define and implement the special format strings. Example:

from django.contrib.localdates.local_dateformat import DefaultLocalDateFormat  
from local_dates import MONTHS_POS, MONTHS_DIR

class LocalDateFormat(DefaultLocalDateFormat):

    def Fp(self):
        return MONTHS_POS[self.data.month]

    def Fd(self):
        return MONTHS_DIR[self.data.month]

In this case, Fp returns a month in its posessive form and Fd returns a month in its accusative form.

Request for comments

I presume that most of the functionality between languages will be the same. It is my goal to put that in a common place so that after the initial development we won't have duplication. So please please anyone that uses django that has the same issues as I do, please give the localdates a try.

You can download the source from:

http://code.google.com/p/django-localdates/source

There are complete usage instructions here:

http://code.google.com/p/django-localdates/wiki/Usage

Conclusion

This is a work in progress. I'd like some feedback from users that use other languages. Also, please forgive me for using django.contrib.localdates for the name. This app fits nicely in the contrib section of Django and I like it placed there, as the localflavored dateformats can all have a common base. It'd be great to be included in the Django trunk, but I'm not responsible for that.

Comments

Creating a portable development environment

Wed, 04 Jul 2007 21:56:53 +0200

I spend much time on other people's computers - I'd like to be able to hack for 30 minutes on an idea, but unfortunately, it isn't easy. I have to download and install numerous software packages, like python, django, subversion, an SSH client etc. So I decided to put all that in a flash drive and take it around with me.

Here is what it includes:

Portable Python (python + django)
Subversion - standalone binaries
Notepad2
μTorrent
Mac-on-a-stick
Portable apps suite that includes:
- Putty
- Firefox
- OpenOffice
- 7-Zip
- Gaim
- Clamwin
- Sudoku
- VLC player

Not too shabby!

I'll play around with it and I'll update this post. Of course the ultimate test is to use it on another computer, and that'll happen next week.

Comments

Supercharging get_next_by_FOO

Tue, 03 Jul 2007 15:34:49 +0200

There is a neat piece of functionality in Django that will allow you to traverse your object graph by date:

Get next or previous by FOO, where FOO is a Date or DateTime field:

For every DateField and DateTimeField that does not have null=True, the object will have get_next_by_FOO() and get_previous_by_FOO() methods, where FOO is the name of the field. This returns the next and previous object with respect to the date field, raising the appropriate DoesNotExist exception when appropriate...

However, these methods use the default manager, that may be not what one wants.

Don't override

You could override these methods, but admin and many other contrib applications rely on them to return every single object, so you'd break them. Maybe this should be fixed in the trunk by specifying two default managers: one named all and another one named objects. The first could be used by admin-like applications, like admin or databrowse, and the other for applications that display stuff to users.

Anyway, until this is done, we can't override, so instead we'll add new methods.

DIY

Fortunately, we can reuse the functionality of these nice methods, since they accept filter arguments. But, in the principle of DRY, duplicating the filter arguments in many places isn't nice. What I prefer doing is, in each manager that overrides get_query_set, to define a filter_dict property:

class PublicEntriesManager(models.Manager):
    def __init__(self, *args, **kwargs):
        self.filter_dict = dict(is_public=True, pub_date__lte = datetime.now)
        super(PublicEntriesManager, self).__init__(*args, **kwargs)

    def get_query_set(self):
        return super(PublicEntriesManager, self).get_query_set().filter(**self.filter_dict)

...which, I reuse at the get_query_set. I can now use that everywhere that I need to provide the same functionality, without using the manager. In our case, in get_next_entry:

def get_next_entry(self):
    return self.get_next_by_pub_date(**Entry.public.filter_dict) 

def get_previous_entry(self):
    return self.get_previous_by_pub_date(**Entry.public.filter_dict)

You can of course reference other properties as well:

def get_next_in_lang(self):
    return self.get_next_by_pub_date(language=self.language, **Entry.public.filter_dict ) 

def get_previous_in_lang(self):
    return self.get_previous_by_pub_date(language=self.language, **Entry.public.filter_dict )

Templates

{% if entry.get_previous_in_lang %} 
<span>
{% trans 'Previous entry:' %} 
<a href="{{entry.get_previous_in_lang.get_absolute_url}}"
title="{{entry.get_previous_in_lang.title}}">{{entry.get_previous_in_lang.title}}</a> 
</span>
<br />
{% endif %}

Conclusion

What I like about Django is that it forces me to add methods to my models so I can use them in the templates. If I could pass arguments to a method call in the template, I'd be tempted to hard-code the filter arguments in the template, forcing me to hunt around for references if I wanted to change something.

Adding smarts to the model class is the way to go, IMHO.

Comments

Django does Oracle!

Mon, 25 Jun 2007 13:43:21 +0200

I just caught up with the Django timeline, and I saw changeset 5519!

Congratulations, boulder sprinters!

I haven't tested it at all, but it's nice to have.

Comments

Safari 3.0.2 for Windows XP works for me

Sun, 24 Jun 2007 22:21:25 +0200

Apple released version 3.0.2 for Safari.

This fixes some weird bugs I've had with bold text not rendering, and bookmarks being completely broken. So I'm now using Safari 3.0.2 to post this :)

However, not all is well...

Middle-click doesn't close tabs
My Intellimouse back-forward side buttons do not work
Ctrl-Tab, Ctrl-Shift-Tab doesn't switch tabs
I haven't got used to the different font smoothing yet; I understand it is a matter of preference.
No spell checking
No plugins (adblock would be nice)

I'll probably find more as this is a beta, but it's good to know that I can view how pages look in Safari, plus the resizable Textareas are great for blog posts! If only there was spell checking...

Comments

Why aren't there any good RAD frameworks for rich clients?

Sat, 16 Jun 2007 04:04:34 +0200

I mean, seriously. For web-apps we have tens (maybe hundreds) of frameworks that abstract away the painful details for creating data-based applications, then get out of the way as you add custom behavior and so on.

I mean, using Django, you can whip-up some models, generate the appropriate database schema, play around in admin, maybe use databrowse, use nice html forms, add a dash of HTML+CSS and you get maybe 80% of the functionality. Then you add your custom views, report generation, a bit of AJAX --if necessary-- and you have a nice data-based web application ready for deployment.

Why am I ranting about this ? First, for a bit of personal history...

Ye not-so-olde days

I started out working as a software developer in a laboratory in University of Patras. We developed web applications using JSP and later Struts. For database access we used Borland's QueryDataSets that managed a bit of the complexity --- you still had to write your own SQL --just SELECTs, though--, but you defined some metadata in the columns that you could use later for display purposes. We had a lot of custom code that did introspection so we could easily display QDSs as tables. No need to say, it was messy, but it did the job.

The problem with that approach was, of course, duplication. Oh my God was there duplication. To add a field to a table, you had to:

Make the change in the DDL
Search the code for references to that table and...
Change SQL queries
Add the necessary QDS columns
Go to Struts' ActionForm classes and add the new field
Go the JSP template and write up the HTML that displays that field

That is not all. We had of course created utilities that acted as "ghetto" bindings. If the variables in an ActionForm had the same name with the columns in a QueryDataSet, you could populate the one from the other, and vice versa. The problem was, sometimes you had to do more than that, and it was tedious, boring, repetitive... an ideal bug generator.

Fast-forward

Anyway, despite all the above, I've learned a lot there, we were doing cool stuff, and, of course, being in academia we afforded to have terrible bugs that nowhere else would be acceptable. I'm sure that if we had time and went back to start again, we'd do things very differently.

So I tried, in a little project of mine, to do things correctly. I set out to create a "little" desktop application that would manage the patients for a dietician friend. I wanted to leverage as many third-party libraries as possible. I only knew Java well enough back then, and I used Eclipse extensively, so going with Eclipse-RCP was a bit of a no-brainer. A seemingly nice platform, using SWT (I wanted to have a native look), it had wizards, an update manager, background tasks, preference panels, toolbars etc. So far so good.

Whither ORM?

So I looked around for an ORM-like solution. The big thing around was Hibernate. I looked into that, but it seem very complex and an overkill for my needs. I've seen what Rails did with ActiveRecord, and I wanted something like that. To my despair, there was nothing that simple available for Java (at least, nothing I've found at the time). So I went with iBatis: it used plain SQL, which I knew well, and POJOs, which seemed simple enough.

That's where things started going downhill: I got lazy, and I wanted to find a tool that will introspect my database schema, and give me some nice JavaBeans I can use for CRUD operations. I eventually found Abator that generated all the crud for me...

Oh, the horror! For each table, Abator generated one huge xml descriptor file and two Java classes. Talk about duplication. It used the names from the database, too, so to change the POJO name, you had to change it everywhere, but if you changed the schema and regenerated from Abator, you had to do it again. I eventually wrote my own descriptors, so I was back to square one...

Databinding ? Let's see...

To my shocked surprise, there was no framework in Eclipse RCP for binding to forms. Actually, there was, but it was in alpha, you had to get it from CVS and deal with its bugs too. I was brave and did that, and had to update my code once or twice. Worked good enough, but I had to be very verbose about things.

To display the fields in the forms, I used Java annotations, like display_text, foreign_key, boolean_value etc. It was pretty good, but a bit tiresome to use, and of course since I wrote it myself it had a lot of rough edges.

CRUD - roll your own

Soooo, to come round to the introduction of this long rant: I had to roll my own CRUD solution, mostly from scratch. Add an object to the database, delete it, present all objects in a table, simple stuff like that. This took me a lot of time --- if this was a real client project, it would take me at least twice the time I estimated. Plus, it had a lot of bugs and lacked simple features.

The Django approach

Now, what Django has to do with this ? Well, I realised that I could now re-write all this using Django in about 2-3 weeks of 40 manhours. Really, it isn't anything complicated: CRUD with a twist (the forms are a bit more complicated, so I can't use the admin), a basic dashboard the presents information about each patient. I wrote most of the model definition in about 30 minutes. And I get all the metadata I had to implement myself before, plus a basic form creation framework that I can further customize...

Conclusion

To be honest, a lot of the mistakes of the past could be attributed to me: I was inexperienced, and a bit foolish to go reinvent the wheel. But, in defense, I couldn't find something to do what I wanted, so what's a poor coder to do ?

Or maybe the libraries 2 years ago weren't as mature. Or cool ORMs (like Rails' ActiveRecord and Django's Models) weren't widespread enough so people can see the bliss in working with them. Maybe Java developers were all drowned in uber-frameworks lice Eclipse RCP, Hibernate and Spring to see the simplicity. Perhaps I didn't know where to look.

But, one thing is for sure: For now on, it's Python and Django for me. You can install locally, create wrappers and who knows where this can lead. Or, mif you like that stuff, you can factor out the web-specific stuff of Django, and use IronPython or PyObjC to go to the desktop again --- if there is absolute need.

The only desktop client solution I've seen so far that looks promising is what Apple is doing with CoreData, Bindings etc. In fact, when Leopard comes out in October I'll get myself a MacBook and start to play around there. But the fact is, you can't develop data-based apps just for the Mac, unless you sell Mac minis to run your software as well --- it's all about market share. Though if the niche is viable, I'd love to start digging in.

Whew. That was a long rant. Comments welcome...

Comments

Going International, pt. 4: Middleware and context processors

Sun, 03 Jun 2007 16:39:37 +0200

Following up to the part 3 of my series about implementing a multilingual interface for this blog, I now present the things to make the language-magic happen: Middleware and context processors.

Disclaimer

When I wrote this code, I wasn't aware of the threadlocals pattern. In the next iteration of the code, I'll make use of it. For now, it works good enough so it stays.

Views, part 2

Those who read carefully or tried to use the code I posted on my previous post, would notice that I was referencing request.url_lang, which of course would lead to an error, since it's not obvious on when and where it's set. So let's present this:

Middleware

Here's a little helper function that does this:

re_language = re.compile(r'^/(?P<lang>[a-z]{2})/.*')
def put_language(request):
    m = re_language.match(request.path)
    if m:
        lang = m.group('lang')
        request.url_lang = lang
        #print request.url_lang
    else: raise Exception()
    return None

You can see that it uses a regular exception, so it's trivial to set it up to suit different needs.

Now we have another problem, that our URLconfs and views would have to be aware of this URL scheme, and it would make it tedious. So, we go and rewrite request.path like this:

class LanguageMiddleware:
    def process_request(self, request):
        for key in exceptions:
             if request.path.startswith(key): 
                 request.url_lang = request.LANGUAGE_CODE
                 return exceptions[key](request)
        try:
            put_language(request)
            request.oldpath = request.path
            request.path = request.path[3:]
        except Exception:
            if request.POST: return None
            return HttpResponseRedirect("/"+request.LANGUAGE_CODE+request.path)          
        if request.url_lang != request.LANGUAGE_CODE:
            request.different_lang = True
        return None

Warning

Django documentation states clearly that I shouldn't treat request.path as a writable property. Nevertheless I do. So this may break in the future.

This code does many little things:

It sets the request.url_path attribute
It removes the language part of the URL that django will use.
It stores the request.path into request.oldpath. I use this in my 404 template to show the URL a visitor will actually see. Try to put a random address and see that the path displayed is the correct one.
It redirects requests that don't have any language information set, so that there are no links and searches directly. Try to delete the language from the URL, and see what happens.
It sets a flag if the URL language is different from the detected language, so we can present a message to the visitor. Try replacing the en part of the url for this post with el.

Finally a wart. For some views, I don't want all this to happen, so I've setup an exceptions table:

exceptions = {
                 '/i18n/': ret_none,
                 '/static/': ret_none,
                 '/feeds/': ret_none,  
                 '/accounts/': ret_none,          
                 '/en/about/': put_language,                
                 '/el/about/': put_language,                
                 '/google': ret_none,                
                 '/robots.txt': ret_none,                
                 }
   ret_none = lambda x: (None)

This isn't so good, since if I want to add a new flatpage (like /el/about/) I'll have to add another entry. I should move this to the database somehow, but I haven't had the time to figure it out. If you have an idea, please tell :)

Context processors

Nothing big, really:

def language_note(request):
    try:
        if request.url_lang != request.LANGUAGE_CODE:
            if request.LANGUAGE_CODE == 'el':
                note = 'Βλέπετε αυτή τη σελίδα στα Αγγλικά. Είναι επίσης διαθέσιμη στα <a href="/el%s">Ελληνικά</a>'%(request.path,)
            else:
                note = 'You are viewing this page in Greek. It is also available in <a href="/en%s">English</a>'%(request.path,)        

            return {'language_note':note}
    except AttributeError:
        pass
    return {}

I should make this handle more than one available language, but I have no need right now, so I won't ;)

Epilogue

That's all for now. This series is over. I have some ideas about making this a bit simpler, but I have no time to fool around right now. In fact, if I have to make another multilingual site, I'll probably use django-multilingual and put some effort in improving that.

It was good scratching the itch, though :)

Comments

Ημέρα Αμαλίας Καλυβινού

Fri, 01 Jun 2007 06:00:00 +0200

Το βρήκα από το in.gr, και το δημοσιεύω και εγώ...

«Να γίνουν εξαίρεση οι αλμπάνηδες ρε παιδιά, όχι ο κανόνας...»

Η ιστορία μιας κοπέλας που πολέμησε με το ελληνικό κράτος και έχασε. Πάμε να φύγουμε όσο είναι καιρός...

Comments

I cut myself on Django's bleeding edge

Mon, 28 May 2007 22:16:46 +0200

I just updated the django code to the latest revision, and a whole host of errors broke up...

Luckily, I found the culprit fast here. Google is my friend indeed :)

Comments

Problem with django feed

Mon, 28 May 2007 16:20:13 +0200

I fell into the obligatory problem where non-public items leak out to public feeds. Sorry to everyone, I've updated the feed.

The root of the problem was not using the custom public Manager I have created to deal with this problem in the django-tagging application. It accepts only Model classes. I'll probably hack it up to suit my needs...

Comments

Beware: Python markdown limitations

Mon, 28 May 2007 12:22:05 +0200

I use markdown to edit the posts in this blog, using the python markdown library. However, I discovered a nagging problem that will cause to me switch away, to something like ReST.

This is an issue that dates back to 2006, and it's very annoying for blogs. You can't put images inside links! This means that the often-used thumbnail image that links to the full-sized original image, has to be done in plain old HTML, which sucks.

Here's an example:

This should render the google logo linking to google:

[![the google logo][logo]][google]
[logo]: http://www.google.com/images/logo.gif
[google]: http://www.google.com/ "click to visit Google.com"

But instead it renders (Actual output):

[][google]

Source

[<img src="http://www.google.com/images/logo.gif" alt="the google logo"/>][google]

(Taken from here)

Disclaimer: This is about the Python engine of markdown. Other implementations of markdown work fine, as far as I can tell.

So, should I just bite the bullet and switch to ReST ?

Comments

Matplotlib rocks!

Thu, 24 May 2007 21:57:41 +0200

Today I discovered the excellent python module for plotting matlab style, matplotlib. It allowed me to quickly visualize some data I am crunching for my diploma thesis.

Here are some quick pics:

For the curious, this is an experiment using Self-Organized Maps (or SOMs). You start with some random nodes, then as you feed them random data, they unravel to something resembling a grid.

I plan to use it for word clustering. I'll post more about my diploma thesis as I get more close to finishing it.

Comments

Αργαλειός λέξεων

Tue, 15 May 2007 02:22:23 +0200

Διάβασα αυτές τις μέρες το μυθιστόρημα του Χρήστου Χωμενίδη "Το σπίτι και το κελλί", ένα πραγματικά εκπληκτικό διήγημα που αποτυπώνει την ιστορία της 17 Νοέμβρη, ή της "Εταιρείας", σε ένα παράλληλο σύμπαν. Ο τόπος, ο χρόνος και οι πολιτικές συνθήκες είναι οι ίδιες, ενώ τα πρόσωπα και οι λεπτομέρειες είναι διαφορετικά, όχι όμως τόσο διαφορετικά ώστε να μην μπορεί κάποιος να διακρίνει τους παραλληλισμούς.

Αυτό όμως που κάνει την πλοκή τόσο συναρπαστική δεν είναι τόσο η εξιστόρηση μιας (λίγο-πολύ) γνωστής πλέον ιστορίας μιας τρομοκρατικής οργάνωσης, όσο ο θαυμασιος τρόπος με τον οποίο ο Χωμενίδης πλέκει σίγα-σιγά τις σχέσεις μεταξύ των χαρακτήρων.

Ένας διπλωμάτης, ένας οινοποιός, ένας μποεμ τύπος, ένας ψαράς, ένας γεωπόνος, μια νηπιαγωγός είναι οι κλωστές που ξεκινούν χωριστά, πλέκονται, διασταυρώνονται, μπερδεύονται και χωρίζουν ξανά για να ενωθούν με νέο σχήμα κάποιες σελίδες αργότερα. Η σχέσεις διανθίζονται με οικογενειακά συμπλέγματα, έρωτες, απιστίες και αμφιβολίες, και ο αναγνώστης εύκολα ταυτίζεται με γνώριμες παρόμοιες καταστάσεις.

Ξύπνησα σήμερα το απόγευμα και το είδα σχηματικά, την ώρα που ούτε ξύπνιος ήμουν, ούτε όμως και κοιμισμένος:

Ένας αργαλειός, όπου κλωστές ταξιδεύουν πάνω κάτω, αραχνοΰφαντα νήματα τις φέρνουν κοντά και τις απομακρύνουν. Ένα ζευγάρι ενώνεται και ακολουθεί κοινή πορεία, μια επίσκεψη όμως μιας τρίτης κλωστής τους χωρίζει... Κλωστές που ήταν πολύ μακρυά, ξάφνου, με έναν απίθανο συνδυασμό ενώνονται και επηρρεάζουν την πορεία η μία της άλλης.

Στο μυαλό μου σχηματιζόταν καταπληκτικές εικόνες... Τι να σου κάνει ο κινηματογράφος.

Θα ήταν ενδιαφέρον βέβαια να προσπαθήσω να αποτυπώσω τις διαδρομές αυτές σε εικόνα. Ένα ενδιαφέρον project για ένα μεγάλο Σαββατοκύριακο...

Comments

Going International, pt. 3: Implementation of multilingual content

Mon, 14 May 2007 15:31:27 +0200

Introduction

As I wrote in the two previous posts, I wanted the URLs of this site to contain the language the content is in. To do this, I had to write some custom code, including models, managers, middleware and context processors.

Models

The first step on having multilingual content is to define a model that can support it. I thought long and hard and it boils down to these conclusions:

The same content in different languages shares some data, like the slug, the tags and the publication date
The different data are the title, the summary and the body.
Therefore, I have to find a way to "inherit" the shared data to the actual entries, while keeping the different data apart.

Model inheritance in Django is somewhat rough, and so I went with KISS: Keep it simple, stupid, and went with the simplest thing that might just work:

First, define a master entry:

 class MasterEntry(models.Model):
  slug = models.SlugField(
  unique_for_date='pub_date',
  prepopulate_from=('title',),
      help_text='Do not add language info here.'
  )
  pub_date = models.DateTimeField()
  tags = TagField()    
  enable_comments = models.BooleanField(default=True)    

  class Meta:
     ordering = ('-pub_date',)
     get_latest_by = 'pub_date'

  class Admin:
     list_display = ('pub_date', 'slug')
     list_filter = ('enable_comments', 'pub_date')
     search_fields = ['slug','tags']
     date_hierarchy = 'pub_date'

  def save(self):
     super(MasterEntry,self).save()
     for entry in self.entry_set.all():
         entry.save()
  def __str__(self):
     return self.slug

Pretty basic (although I removed some methods to help clarify things) except of the save method: it iterates over all the child entries and saves them.

Then, define the child entry:

class Entry(models.Model):
  title = models.CharField(maxlength=200) 
  summary = models.TextField(help_text="One paragraph. Don't add &lt;p&gt; tag.")
  body = models.TextField()

  master = models.ForeignKey(MasterEntry, 'id')
  language = models.CharField(maxlength=2, choices=settings.LANGUAGES)

  objects = models.Manager()
  public = EntriesManager()

  slug = models.SlugField(
    unique_for_date='pub_date',
    prepopulate_from=('title',),
    blank=True,
    editable=False

  )
  pub_date = models.DateTimeField(blank=True,editable=False)
  tags = TagField(editable=False)    
  enable_comments = models.BooleanField(blank=True,editable=False)    
  is_public = models.BooleanField(default=False)        

  def save(self):
     self.slug=self.master.slug
     self.pub_date=self.master.pub_date
     self.tags = self.master.tags
     self.enable_comments = self.master.enable_comments
     super(Entry, self).save()

  class Meta:
      ordering = ('-pub_date',)
      get_latest_by = 'pub_date'
      unique_together=(('master','language','slug'),)
      verbose_name = _('Entry')
      verbose_name_plural = _('Entries')

  class Admin:
      list_display = ('title','master', 'pub_date', 'is_public',)
      list_filter = ('enable_comments', 'pub_date')
      search_fields = ['title', 'summary', 'body']
      date_hierarchy = 'pub_date'
      fields = (
          (None, {
            'fields': ('master','language','title', 'summary','body','is_public')
          }),
      )

  def __str__(self):
      return self.title+" - ("+str(self.master)+")"

  def get_absolute_url(self):
      if self.is_public:
          return "/%s/blog/%s/%s/" % (self.language,self.pub_date.strftime(
           "%Y/%m/%d").lower(), self.slug)
      else:
          return "/%s/blog/preview/%d" %(self.language,self.id)

The child entry has a reference to a master entry. It also has a language field that defines the language it is in. It then goes on to define the same fields the master entry has, which are then overridden in the custom save method. I had to do this in order to be able to use generic views (which require a pub_date field) as I couldn't get them to use the master entry's pub_date directly. I also toyed with properties, but it didn't worked out for me, so I just went with what worked. Notice that the inherited fields are not editable, hence they are not visible in admin.

You can see now that changes in the MasterEntry propagate down to the Entry , via its custom save method.

I also add a custom manager named public. More on that in...

Managers and Views

Defining the models is half the job - custom managers allow you to access advanced functionality with less code, and also to plug into places like generic views and feeds easily. Here are my managers:

class PublicEntriesManager(models.Manager):
  def get_query_set(self):
      return super(PublicEntriesManager, self).get_query_set().filter(is_public=True)

class EntriesManager(PublicEntriesManager):
  def in_language(self, lang):     
      queryset_lang=super(EntriesManager, self).get_query_set().filter(language=lang)
      return queryset_lang

  def get_entries(self,pub_date,slug):
      qs = Entry.public.filter(pub_date__range=(
        datetime.combine(pub_date, time.min), 
        datetime.combine(pub_date, time.max)
        ),
        slug__exact=slug)
      entries = {}
      for e in qs: entries[e.language]=e
      return entries

  def archives(self):
      return Entry.public.all().dates('pub_date','month','DESC')

  def has_entries_missing(self,the_lang, entry_qs):
      if isinstance(entry_qs,models.Manager):entry_qs=entry_qs.all()
      master_entries = MasterEntry.objects.filter(entry__in=entry_qs)
      return _compare_entries_lang(the_lang,master_entries)

Plus the _compare_entries_lang function:

def _compare_entries_lang2(the_lang, master_entries=MasterEntry.objects): 
    if isinstance(master_entries,models.Manager):master_entries=master_entries.all()
    for me in master_entries:
        if me.entry_set.filter(is_public=True).count() < len(settings.LANGUAGES):
            #this master entry has entries in less than the supported languages
            #lets find out if they are missing in the requested lang
            if not me.entry_set.filter(language=the_lang,is_public=True).count():
                #yep, they do
                for lang,dummy in settings.LANGUAGES:
                    if lang!=the_lang:
                        return lang
    return None

This is quite big, so I'll break it down:

PublicEntriesManager is simply a manager that filters out the non-public entries. This way, in my view code I can use Entry.public.all() and I can be sure I get only the public entries. This saves some headaches and prevents silly mistakes (like, you show only the public objects on site, but display everything in your feeds).

EntriesManager is quite more complex. I have some complicated language-related functionality here, and it manages it all:

If you get to a page (like a month archive or the index page) that detects that there are more entries available, but not in the current language, it displays a notification.
Then, if you request an entry that is not available in your current language, it displays the available entry with a notification.
Lastly, if you request an entry in a language other than your own, it displays it, along with a notification that you might prefer viewing the content in your own language.

I begin with the simple ones: archives is used to make a listing of dates that have public Entries, so I can create the archive listing on the right. in_language filters the queryset to return only the entries in a language.

To explain has_entries_missing, here is some code that accesses the EntriesManager:

def entry_index(request):
  queryset= Entry.public
  entries_missing=Entry.public.has_entries_missing(request.url_lang, queryset)
  archive_blog_dict = {
    'queryset': queryset.in_language(request.url_lang),
    'date_field': 'pub_date',
    'template_name':'entry_archive.html',
    'extra_context':{'entries_missing':entries_missing}
    }   

  return archive_index(request, **archive_blog_dict)

def entry_month(request, year, month):
   date = datetime.date(*time.strptime(year+month, '%Y%m')[:3])
   first_day = date.replace(day=1)
   if first_day.month == 12:
       last_day = first_day.replace(year=first_day.year + 1, month=1)
   else:
       last_day = first_day.replace(month=first_day.month + 1)
   lookup_kwargs = {'pub_date__range' : (first_day, last_day)}   
   queryset= Entry.public
   entries_missing = Entry.public.has_entries_missing(request.url_lang,  
       queryset.filter(**lookup_kwargs) )
   archive_month_blog_dict = {
     'queryset': queryset.in_language(request.url_lang),
     'date_field': 'pub_date',
     'template_name':'entry_archive_month.html',
     'month_format':'%m',
     'allow_empty':True,
     'extra_context':{'entries_missing':entries_missing}
     }
  return archive_month(request, year, month, **archive_month_blog_dict)

The has_entries_missing method checks a QuerySet of Entries to see if they are available in both languages. If they are not, it returns the language code that has the extra entries. I use it in my templates to create a link to the page that displays the missing entries. I now have only two languages, but I think it will work with more than two. I'll let you know when I try it :)

Finally, the get_entries method returns a dictionary with languages as the keys and entries as the values, for a given pub_date and slug. This can be used to check if a single entry is available in one or both languages, and display the relevant information. The view that uses this is:

def entry_detail(request, year, month, day, slug):
   pub_date = datetime.date(*time.strptime(year+month+day, '%Y%m%d')[:3])
   entries = Entry.public.get_entries(pub_date,slug)
   entry_blog_dict = {
       'queryset': Entry.public.all(),
       'date_field': 'pub_date',
       'template_name':'entry_detail.html',
       'template_object_name':'entry',
       'month_format':'%m',
       'extra_context':{'available_languages':len(entries)-1} # so to be either 0 (False) or 1 (True)

       }
   try:
       entry = entries[request.url_lang]
       entry_blog_dict['object_id']=entry.id
   except KeyError:
       entry = entries.values()[0] # we assume only 2 languages active
       entry_blog_dict['extra_context']['language_not_found']=entry.language
       entry_blog_dict['object_id']=entry.id
   return object_detail(request, year=year, month=month, day=day, slug=slug, **entry_blog_dict)

Not the best looking code, but it works ;)

That's all for now

This post got bigger than I expected, and I still haven't covered the middleware and context processors. I'll cover these two in the next post. Until then, please leave a comment and let me know what you think about this!

UPDATE

I guess I should put my money where my mouth is :) I found out that the the _compare_entries_lang function didn't work as expected. I've replaced it with a better, but maybe slightly slower version.

Comments

Going international, pt. 2: URL design

Sun, 13 May 2007 05:43:18 +0200

To help myself developing the international part of this blog, I've made some usage scenarios, which I write down here, partly to not forget them but mostly to help find any mistakes and omissions. So here they are:

The users:

Giannis (Greek, computer savvy user, has his browser correctly set)
Takis (Greek, lives abroad, doesn't have access to the language settings)

I could also define two English users in similar situations, but there's no point as it turns out that they just mirror each other.

I have to point out a seemingly little-used preference of web browsers, the language selection preference. This sets the HTTP header 'Accept-Language' so that a server can know, erm, the language preference. This is basic stuff but not many users bother to change from the default setting.

URLs and design

The pages:

/el/1/ Direct link to a greek page, content isn't available in english.
/en/2/ Direct link to an english page, content isn't available in greek.
/el/3/ Direct link to a greek page, content is available in english.
/en/3/ Direct link to an english page, content is available in greek.

Also, there are two little flags on the top right of the site that allow a user to explicitly select the language of his choice. I think that this is the standard way to provide language selection. I don't quite like drop-downs, as you have to deal with how to provide the language name (in English? in the same language?). Flags are unobtrusive, universal (to those who care), and don't take up much space.

Expected results

The expected results:

Giannis, Takis having selected greek:

/el/1/ - Everything in greek.
/en/1/ - Menu in greek, content in greek, notification (greek) that the content isn't available in english.
/el/2/ - Menu in greek, content in english, notification (greek) that the content isn't available in greek.
/en/2/ - Menu in greek, content in english.
/en/3/ - Menu in greek, content in english, notification (greek) that the content is also available in greek.
/el/3/ - Everything in greek.

Takis, Giannis having selected greek:

/el/1/ - Menu in english, content in greek.
/en/1/ - Menu in english, content in greek, notification (english) that the content isn't available in english.
/el/2/ - Menu in english, content in english, notification (english) that the content isn't available in greek.
/en/2/ - Everything in english.
/en/3/ - Everything in english.
/el/3/ - Menu in english, content in greek, notification (english) that the content is also available in greek.

When Giannis selects greek and Takis selects english, there are no changes.

Algorithm

So a pattern emerges:

The language part of the url defines the language of the content and
The browser preference defines the language of interface (menu and messages)

Also, the browser preference determines the language of the root page ( ie. orestis.gr/, without a following language code). A visit to the root page actually returns a redirect.

Conclusion

I think that this is a sound approach: If we try to be RESTful and Google friendly, a unique resource is named with a unique URL: The URL part define the language of the content. There is a slight anomaly when the content isn't available in all languages, where I consider bad manners to give a 404, so I just display the alternate version.

Is there a better way to handle this ? Please leave your comments!

Comments

How to add a preview page

Mon, 07 May 2007 06:23:13 +0200

When using the django web administration page to add blog posts, it's nice to be able to preview the post to catch layout glitches, typos and markup issues. The best way to do this is to have a preview page.

To add a preview page, the model should have:

a boolean field that determines whether the post is public or not
a smart get_absolute_url function.

Separate your posts

The first is easy:

    is_public = models.BooleanField(default=False)

Make your URLs smart

The second:

 def get_absolute_url(self):
    if self.is_public:
        return "/%s/blog/%s/%s/" % (self.language, 
           self.pub_date.strftime("%Y/%m/%d").lower(), self.slug)
    else:
        return "/%s/blog/preview/%d" %(self.language,self.id)

The django admin uses the get_absolute_url function to put a "Display on site" button on an object's page. So when you have public entries, you get to see what all the world sees, but when an entry is not public, the get_absolute_url function returns a special url.

Add a pre-view

The final piece to the puzzle is to have a view that is protected:

   @login_required 
   @user_passes_test(lambda u: u.is_staff)
   def entry_preview(request, entry_id):
      entry = get_object_or_404(Entry, pk=entry_id)
      return render_to_response('entry_detail.html', {'entry':entry}, 
        context_instance=RequestContext(request))

I use the is_staff property, but this can easily be changed to something else.

Don't forget to add the relevant url to your urls.py :

(r'^preview/(?P<entry_id>\d+)/$', entry_preview),

That's all!

You gotta love Django. I got from idea to implementation, and of course blog post in about 30 minutes! And, most of which I was hunting a bug in my middleware which trashed the account login redirect.

Comments

Going international, pt. 1

Sun, 06 May 2007 22:14:35 +0200

Introduction

From the start, I was interested in making the content posted here available in two languages: Greek and English. This is no small feat, and I thought long and hard on how to make it possible and get it working just the way I like it. So here is a rundown of how internationalization works here:

The prerequisites are not many:

Make the content available in both languages.
Make the interface available in both languages.

I have slacked and made the tags English-only. Maybe I'll fix that later on, but it'll do for now.

Approaches

Now, you have two choices about making a site handle international visitors. You either:

let them decide what language they prefer, by providing the choices somewhere in your interface or,
try to use their request data (like the IP address, or the language preference sent by their browser) to decide their preferred language.

The first approach is the most foolproof, as a user explicitly chooses his language. He can't be wrong, can he? But, there is a snag: what is the first page he sees? If it is in his own language, there's no problem. But if it is in another language, he'll get confused and probably leave. Advanced users may scan the interface for visual clues to change the language (if they know they have this option). With first impressions being so important, you shouldn't scare your visitors away.

This is solved by the second approach, mostly. However, the problem there lies in that a) the location of the user not always matches their language, b) most users never set their language preference in their browser anyway (I have no source for this, it's just a gut feeling) and c) the user has no visible choice in your site. If you guess wrong, he can't change it, so he leaves.

Solution

Well, this leaves us with the only sensible approach: Combine these two. So this is what I've done, of course aided by Django.

We first try to find implicit facts that point to a language. In Django, this is done using the LocaleMiddleware:

LocaleMiddleware tries to determine the user’s language preference by following this algorithm:

First, it looks for a django_language key in the the current user’s session.

Failing that, it looks for a cookie called django_language.

Failing that, it looks at the Accept-Language HTTP header. This header is sent by your browser and tells the server which language(s) you prefer, in order by priority. Django tries each language in the header until it finds one with available translations.

Failing that, it uses the global LANGUAGE_CODE setting.

This is a pretty good approach, as it handles both new and returning visitors.

We also provide a simple interface, the two little flags on the right edge of the header, to give the option of setting the language explicitly. This should cover us but...

Search engines

Unfortunately, we have to deal with search engines, too. Crawlers don't have sessions, nor accept cookies, nor emit Accept-Language HTTP headers. So they'll reach only your default language. If we continue this reasoning, you'll have issues with incoming links too: When I link to a page that is in Greek, I'll expect my link to point in the same version. So we have to somehow put the language information on the URL too. Since we have everything translated, it makes sense to put the language in the lowest level possible:

http://www.example.com/en/
http://www.example.com/el/
http://www.example.com/fr/

etc.

So now, if someone links to a page, he also links to a language. So far so good. What about our first page, without any paths appended ? What would come up if someone bookmarked http://www.example.com ? Of course we will use the discovery mechanism described above and redirect him to the language we guess. This will work with crawlers too, as they handle redirects. The crawler will also find the language setting flags and will happily index all our languages, since now they have distinct URLs.

Final thoughts

There's also the possibility that a visitor will follow a link that is in another language than his own. We could try to redirect him to his preferred language, but this will put us back in the first, half-baked approach. Plus, he'll never be able to view content in another language, unless he clears his session and cookies, and change his preferred language in his browser. Instead, we'll just give him a discreet message in his preferred language notifying him that the content he is viewing is also available in another language, and give him the link to it.

Disclaimer

I haven't implemented any of this at the time of writing. I'll try to, and present the code and results in the part 2 of this article.

Comments

Καλώς ήλθατε

Fri, 27 Apr 2007 12:00:00 +0200

Επιτέλους αποφάσισα να φρεσκάρω το blog μου - νέος σχεδιασμός, νέες λειτουργίες, με 1 λέξη κα-τα-πλη-κτι-κο (και ελπίζω όχι πληκτικό :)

Γιατί όμως να το κάνω αυτό; Ποιος νοιάζεται; Βασικά εκτός από μένα (και είμαι αρκετά εγωκεντρικός για να είναι αυτό αρκετό) νοιάζεσαι εσύ αγαπητέ αναγνώστη, που κάθεσαι και διαβάζεις τις ασυναρτησίες που γράφω (την πάτησες ;). Εντάξει, δεν είναι αρκετό αυτό για να σου κρατήσει το ενδιαφέρον, οπότε όριστε, θα κάνω μια προσωπική αποκάλυψη (τρρρρρρρρ-πσχσ! (τύμπανα)):

Είμαι ΠΑΠΑΤΡΕΧΑΣ!

Ναι ναι, ΠΑΠΑΤΡΕΧΑΣ! Έχω καταπιαστεί με χίλια-δυο πράγματα και τα έχω αφήσει όλα στη μέση. Φωτογραφία, μουσική, ιστιοπλοΐα, τα κοινά, υπολογιστές (αυτό και αν έχει ζουμί), gadgetάκια, όλα με ενθουσιάζουν για λίγο καιρό, διαβάζω, μαθαίνω, ξοδεύω και μετά τα παρατάω και πιάνουν σκόνη, είτε κάπου στο χαοτικό δωμάτιο μου, είτε στον επίσης χαοτικό εγκέφαλο μου. Γι' αυτό λοιπόν αποφάσισα να το ρίξω στο γράψιμο, μπας και ασχοληθώ λίγο παραπάνω με τα ενδιαφέροντα μου. Και για προστιθέμενη αξία, μένει η πληροφορία και on-line μήπως ενδιαφέρεται και κανένας άλλος. (Στην πραγματικότητα, με έπιασε το παπατρεχιλίκι μου (παπατρεχάλα;) και ήθελα να φτιάξω δικό μου μηχανισμό για blog, και πρέπει να γράψω διάφορα για να γεμίσει. Το είπα και ξαλάφρωσα.)

Αυτά λοιπόν για τώρα. Θα ακολουθήσει μια σειρά με άρθρα στυλ "Πώς γυρίστηκε" όπου θα περιγράφονται αναλυτικά όλοι οι μηχανισμοί που χρησιμοποιούνται για να λειτουργήσει αυτό το blog. Μείνετε στο κανάλι σας...

Comments

Welcome

Fri, 27 Apr 2007 12:00:00 +0200

I've finally managed to refresh my blog - new design, new functionality, oh my! I hope you like it :)

But why do this? Who cares? Well, apart from me (and I'm really self-centered enough to make this enough), you care, my dear reader, sitting there reading my mumblings (there you are ;). OK, that's not enough to keep you interested, so I'll make a genuine self announcement:

I'm a jack of all trades!

...and master of none. I've dabbled in lots and lots of stuff, only to abandon them later on. Photography, music, sailing, politics, computers (this is big), gadgets, you name it. I get all worked up about one thing, I study it, learn it, spend money on it and after a while the excitement has wore off and I leave it gathering dust somewhere in my chaotic room or in a corner of my chaotic brain. So this is why I decided to start writing, hoping that I will spend some more time doing things I really like. And, to add some value, all the information will be available on-line to everybody interested. (Well, really, I just wanted to create my own blog software, and I need to write stuff to fill it. There you go. I feel much better.)

That's all for now. I'll follow up with a series of posts that will describe the "making of" of this blog. So stay tuned...

Comments