None of these are with sinar project.

1. Jom Pantau is ushahidi installation, for reporting election issue in Malaysia http://pru13.info/ushahidi/
2. An election websites http://backup.myaddress.com.my/
3. Malaysia crime http://www.malaysiacrime.com/
4. Do something good for linking volunteers to organization http://beta.dosomething.gd/
5. eparliment

Project sinar project that we are currently working on

1. Malysian Representation http://reps.sinarproject.org/
2. The api from reps project. https://github.com/Sinar/Zelus
3. Curruption http://accountable.sinarproject.org/
4. Billwatcher.

Got a feeling we going to have more on the list.

Data Analysis with Open Source Tools

Recently I have been reading the book Data Analysis with Open Source Tools, by Philipp K. Janert. This is a book for anyone that is working with Data. Partly because of the big data buzz, partly is the work I do in Sinar Project. But this is not a book on big data, just data analysis in general.

Unlike other book that deals with data, this don’t cover specific tools, rather use tools as a example, and this books covers a few. But this book focuses on experiences and methods of doing data analysis.

The book have 18 chapter, covers a variety of topic, from statistics, to simulation, clustering etc. Each with enough theory to understand, enough insights coming from the authors experience, enough hands on from demo for us to pick up from there.

While this book don’t covers tools comprehensively, but the method, insights in this book make it worth keeping as a reference for doing data analysis.

So here is an example.

import re

class BleepedString(str):

    def __new__(cls, data=''):
        '''
            this will forced inherited object to return new object
            assigned value in data must be bleeped
        '''
        data = cls.bleep(data)
        new_obj = str.__new__(cls,data)
        return new_obj

    def __iadd__(self,data):
        return self.__add__(data)

    def __add__(self,data):
        data = self.bleep(data)
        return super(BleepedString,self).__add__(data)

    def __str__(self):
        data = super(BleepedString,self).__str__()
        return self.bleep(data)

    def __repr__(self):
        data = super(BleepedString,self).__repr__()
        return self.bleep(data)

    @classmethod
    def bleep(cls):
        if not type(data) == str or type(data) == unicode:
            data = str(data)
        pattern = r'nsfw_word' 
        compiled = re.compile(pattern,re.MULTILINE)
        data = re.sub(compiled,'bleep',data)
        return data

A few thing to take note here.

1. Notice that there is a __new__ method? In python object lifecycle, the first thing to be called is actually __new__ not __init__. __new__ will choose whether whether to replace __init__ or use a new one. In this case, I set the current object cls to str, with value bleeped
2. Now if object is not setup in __new__ yet, how do object access the bleep method? Introducing, classmethod. Classmethod is a method the bind to a class, not an object. This is useful if you want to do call a function before a class is initiated.
3. Classmethod binds to the child class, not the parent class it inherited from, and you need the first parameter to be a class. It can be set with a @classmethod on a method
4. Now because we also want the string to be appended to BleepedString object, to also be bleeped, so we need to override __add__, so when string appended via + operator, will also be bleeped. Interestingly __iadd__ is not inherited, so I need to add __iadd__ method too, so that I can do +=
5. Notice that inside __add__ I use bleep method like a normal method. That is the reason is set a @classmethod decorator in bleep method, I need the method to be available without need to be instantiated, while able to use it like a normal method in a object.
6. Actually just override __add__ and __iadd__ is not enough to override the str behavior. Because the parent class str uses __repr__ to provide value to be used in a + and += operation, which is why I override __repr__ method, by replacing the return value to bleeped string.
7. Then we need to override __str__ so that it return a bleeped string.

This is just a modified example what I used in my work, and have a very different and useful function. But it capture the same idea, overriding the default string behavior and return value.

Cheers!

1. Construction Industry Development Board dataset have been release, you can find it here.
2. CPB is not dead, and the survey is over for long time, in which I too participate. Waiting for the parliament to see how.
3. I have join Julython with a new project.
4. Python Meetup Malaysia coming next 2 week. And still going on
5. Join pycon, and realize that I am still interested in big data.
6. Join Super Happy Dev House, know some very awesome people there.

The group is a diverse one, from experienced developer using python for sometime, to new python user. We have the usual introduction, then we have introduction to Python Malaysia.

Then a talk by Yours Truly, on experience of using bottle micro framework and pyes.

Then have a talk on Mezzanine by renyi khor

And Felix shows us how one can use to use tornadio to provide socket.io for django apps. Which is awesome, actually. Again no slide, but the repo is below

https://github.com/felixleong/tornadio-with-django

and Finally a talk by Khairil Yussoff, which is about Plone Dexterity framework, and also plone development in general. He don’t have slides, but he did hands on demo on plone development and how things work.

Then in between we have some networking session. Which is good, because this meetup is not just about Python, it is about the people that uses or want to try/use it. We have everybody not just the presenter that share their experience, which is awesome.

In the end of the day, this is a good event(I hope). But by July, python.my will finally be updated, things will gone more smoothly for us.

Notes: A few times running an event I still can’t get it right. A few thing is missing is, video, proper photo, groups shots, etc. Maybe I should add a list.

p.s The next meetup will be on May 26, in MindValley, just beside Bangsar LRT station. Detail is http://www.eventbrite.com/event/3414166865

As from the link above, the Computing Professional Bill is dropped for now, asking for the industry to sort it out first. The war on it is not quite ended. One reason(or excuse depends on which side you pick) of the existence of this bill is the dropping of quality of IT professional in Malaysia, and the quality of project I suspect. Registering people will definitely not helping in bringing up quality.

If one of the measurement of quality is how much we knows. Then browsing the Malaysian job sites, I come to the conclusion that we don’t know that much. Almost all job revolve around .net and php, with a few php and c/c++. Yet we knows that the industry is more than that. I mean where is the job on python, ruby and rails or node.js, where the job for CMS’es?

Does it mean that the industry in Malaysia don’t know about it? Gone to local tech meet up, only a few that use stuff that is not PHP and .net. When people develop a buzz in nosql, or node or devops etc. Where is the Malaysian? Are there only that few people will blog about this? Are there only that few people that can go to the meetup, join communities? Are there only that few people in this neighbourhood that cares about this?

Maybe the real problem in IT in Malaysia, it is really just that most of the people involve in IT in Malaysia, don’t care. That even with help of a legislation, it doesn’t help improve it, but only to make things worst.

p.s This is very programmer specific, I apologize. But this is the group of people I have seen the most.

p.p.s Yes I know hack weekend exist, but there is only one hackathon in the whole Malaysia?

First time have to produce an app, and maintaining the server all by myself. Previously can work with the sysadmin, now the guys in sinar is pretty busy with a day job.

So first thing first, the thing with virtualenv is, running it in a cron job can be a pain(it is not that bad). Well, billwatcher is using buildout, so it uses less command, buildout have pretty much an isolated environment. So it look like this, more on why I don’t direct to /dev/null:

0 0 * * * /path/to/Malaysia-Bill-Watcher/bin/python /path/to/Malaysian-Bill-Watcher/billwatcher/loader.py >/dev/null 2>&1

Second thing is, the last deployment, I didn’t add a full path to the sqlite database. This is important, so that the data is written in the right place.

Third, because this is my first time running cron(don’t ask why). I finally found the log file to cron.

less /var/log/cron

Except, my cronjob is run at midnight, and a few cronjob run already. Then I found zcat, because the older cronjob is /var/log/cron.n.bz2. n is the number.

zcat /var/log/cron.0.bz2|less

I read it, make sure it is running, I’m happy, then I realized that I kinda polluting the email on my system which I don’t read often. So comes the MAILTO directive. Thus my final cron job look this way

MAILTO=”me@email.com”
0 0 * * * /path/to/Malaysia-Bill-Watcher/bin/python /path/to/Malaysian-Bill-Watcher/billwatcher/loader.py >/dev/null 2>&1

And the newbie sysadmin live happily ever after… Until something else break, and there is a new project coming soon…

p.s running billwatcher at midnight is a bad idea, might be a better idea doing it in the morning..

• we going to scrape the page twice, one time for english version, one time for malay
• now we need to have a way to save 2 translation of the same bill title
• in which I don’t want to use a new table in db for that, gettext exist for a reason
• at the same time, we need to save 2 different url for pdf for the language too,
• now not just we need support the language, we need to load these into elasticsearch,
• because we have to support 2 language, we have to load into elasticsearch twice, one for each language
• while telling it, “it is really the same thing”
• and did i tell you we need to translate the html template too? which the bottle own template language don’t have support for that

Most is relatively easy, the bill title is tougher, things can easily go out of hand, if using gettext, not really the issue of size, the compiled database is pretty ok.

The most interesting problem, how to index multiple language in elasticsearch while referring to the same bill.

http://billwatcher.sinarproject.org/

What have changed:

• Move to a buildout based infrastructure
• Move to supervisor for starting up the web app.
• Adding a service wrapper for elasticsearch, but elasticsearch is not controlled by supervisor,
• Social media button!!!!
• Moving code to use GPL for license
• Change size of div to accommodate firefox view of pdf

The source code is at: https://github.com/sweemeng/Malaysian-Bill-Watcher

What next, bug fix, localization.

So I have participated in the Global Android Developer Camp a.k.a GADC, the experience have been great. This years event in Kuala Lumpur is held in iTrain. Kuala Lumpur is one of many location that the event is held.

GADC is an event where android developer comes together to create app or tools for Android. Other than giving us prizes, and talks on writing android app, this year we are given a chance to develop for the Smart Watch, only the winning team that develop on it can keep it though.

Team Plastique

A few team have created apps, 2 interesting project is Credit Card reward tracking, and a tower defense game. Team Plastique have created an reward tracking, and they manage to push it to the smartwatch, so they deserve the watch. Then we have a Tower Defence Game, which have a very interesting way to do in app credit. Ultimately the winner is Team Plastique.

Not surprisingly, GADC have more people trying to build their first mobile app, and a few that started out from iOS. Which left the organizer as the few people that have real experience in writing android app.

The event is a fun one, we can mingle around with like minded folks and share experience. The organizer are willing to share experience and knowledge. The only complain is, Sony Ericsson should do a better job setup the smartwatch as it would help us the test the code on a real hardware better.

Me, I didn’t win, but I did get a lot of knowledge and experience. I did finish my app. You can see it on the left, it uses JSch to connect to a linux box, and run a one liner, and diplay result to watch. It is buggy though.

Thanks to Nazrul, Wenjiun, Bernie, Mike for making this event great. The image I have on the post is courtesy of Nazrul and Wenjiun, I spent too much time hacking that day, sorry

Want to see more photo and related page go to

• GADC KL Page on google+
• Wenjiun’s google plus gallery
• Nazroll’s gplus page
• Bernie’s gallery
• iTrain’s gallery on facebook