Gan, Zhi

Context Manager: One important advantage of Python

2014-05-30T16:25:00.001-07:00

In Python, context manager is a relatively new feature which can greatly simplify your program. It's pretty easy to implement the context manager with class or function. Following is a runnable example:

 1 from contextlib import contextmanager
 2 
 3 @contextmanager
 4 def tblock():
 5     yield "hello", "world"
 6 
 7 
 8 def main():
 9     print 'start method'
10     for i in range(5):
11         with tblock() as (a, b):
12             if i % 2 == 0:
13                 continue
14             print "bbb ", a, b, i
15 
16     print 'end of method'
17 
18 main()

In above example, we defined a context manager tblock(), which does nothing but return two strings. When with statement is executed (line 11), the tblock() will be called firstly. Code in tblock() is executed until yield statement is executed. When yield statement is executed, the code in with block will be executed. After with block, the left part of tblock() will be executed as the final step.

So you might want to ask, "what's the difference between a context manager and a normal Python function?". Actually, you can find the answer from above example. Although code in with block is invoked by tblock(), its execution flow is tightly related to the caller of with block. That means:

return statement in with block will return from wrapping function ( main () defined at line 8).
continue statement will jump to next iteration of for loop defined at line 10.

The result code can be much more natural, intuitive and less disruptive. One recent real world usage of context manager in my project is to provide a common exception handler for external API calls. Pseudo code looks like:

 1 from contextlib import contextmanager
 2 
 3 @contextmanager
 4 def api_guard():
 5     try:
 6         yield
 7     except TransientError as e:
 8         logging.warning(e)
 9         raise Retriable('can retry')
10     except:
11         raise
12 
13 
14 def execute(task):
15     with api_guard();
16         call_dropbox_api(task)
17

The context manager provides similar abstraction as Ruby block, which is pervasively used in many ruby applications. Arguably, the syntax of Ruby block is a little bit more intuitive. Following is a simple sample of definition and usage of Ruby block:

 1 def exception_handler(&block)
 2     begin
 3         block.call
 4     rescue ZeroDivisionError
 5         puts "I got ZeroDivisionError and I swallow it"
 6     end
 7 end
 8 
 9 def main()
10     puts "Begin of program"
11     exception_handler{
12         return
13     }
14     puts "End of program"
15 end
16 
17 main()

Both Python context manager and Ruby block can be used to greatly simplify program. Another simple yet powerful example is the 'each' function of Ruby array:

1 arr = [1, 2, 3, 4, 5]
2 arr.each { |a|
3     print a -= 10, " "
4 }

Creating Google SpreadSheet and populating data with Python

2014-01-19T17:15:00.000-08:00

This task can be completed in two steps. First step, we use Google Drive API to create the spreadsheet. Second step, we use Google Spreadsheet API to modify the spreadsheet. Here is the code:

Step 1:
oauthToken = gdata.gauth.TwoLeggedOAuthHmacToken( consumer_key.encode('ascii'), consumer_secret.encode('ascii'), userid.encode('ascii') )

docsclient = gdata.docs.client.DocsClient(source="Arbitrary string")
docsclient.auth_token = oauthToken
docsclient.ssl = True
docclient = auth.getDocsClient(domain, owner

Step 2:
def getSpreadsheetService(domain, userid, prefs=None):
consumer_key = 'XXXXXXXX'
consumer_secret = 'XXXXXXX'

spservice = gdata.spreadsheet.service.SpreadsheetsService(source="Arbitrary string")
spservice.SetOAuthInputParameters(gdata.auth.OAuthSignatureMethod.HMAC_SHA1, consumer_key.encode("ascii"),
consumer_secret.encode("ascii"), two_legged_oauth=True, requestor_id=userid)

return spservice

client = auth.getSpreadsheetService(domain, owner)
worksheet_key = 'od6'
headers = ["timestamp", "row", "email", "type", "context", "message"]
for i, header in enumerate(headers):
client.UpdateCell(row=1, col=i+1, inputValue=header, key=spreadsheet_key, wksht_id=worksheet_key)

for log in logs:
row = {
"timestamp": log.created_at.isoformat(),
"row": log.rowNum, "Email": log.email,
"type": log.type,
"context": log.context,
"message": log.message
}
client.InsertRow(row, spreadsheet_key, worksheet_key)

Migrate data from mysql to mongodb (1)

2012-11-17T23:40:00.000-08:00

It looks as difficult as moving mountains when I start to move some of my data from MySQL to MongoDB. It's even more difficult as I'm dealing with an application used by client everyday. And it's absolutely unacceptable to break the SLA just for some mysterious data migration work that business people never care.

Secondly, I'm not 100% sure about the quality of the new code. Some of our old DB queries took weeks to become correct, fast and robust. I don't think we can build Rome on MongoDB in one day. Obviously, we need a relatively long period of time to make our new data access layer matured enough. And test team should be an important part of this transition.

Even simple things become complex if quality matters. In order to make sure this is a smooth and painless transition, we split a simple task into a not so simple sequence:

Modify our data access layer to write to MongoDB as well as MySQL
Create script to migrate existing data from MySQL to MongoDB. Python is our choice but any modern script language should work too.
Gradually modify 100+ queries so that they query from MongoDB instead of MySQL. As data are stored in both data store, all existing MySQL queries work as before, thus we don't need to change all queries in one day.
After all queries are moved from MySQL to MongoDB, drop MySQL tables and completely switch to MongoDB.

搬到山上了

2012-11-16T20:41:00.001-08:00

为了应对最近家庭人口的快速增长，我们再次搬家了

Data are created unequal in MongoDB

2012-11-16T20:34:00.000-08:00

I’m really happy that we chose MongoDB from the very beginning of my current project. After several months of practicing, we became very satisfied with this elegant open source software. Now the only thing I’m feeling pity is that we should adopt it more bravely.

There are already overwhelming talks about it all through the Internet. Some of its best features, such as document data model, advanced query operations, indexing support, map/reduce, are well discussed. While all of these features are very good and enjoyed by my team, our biggest surprise comes from the flexibility provided by MongoDB to manage data consistency differently under a unified framework.

Enterprise often utilizes tons of data, which is created, collected, distributed, consumed at everywhere, every moment. These endless types of data have many different requirements to the data management system, while we can categorize them under the CAP theorem. CAP theorem from Brewers states that any networked shared-data system can have at most two of three desirable properties:

Consistency (C) equivalent to having a single up-to-date copy of the data;
High availability (A) of that data, and
Tolerance to network partitions (P).

This is a somewhat despairing theorem for data scientists and practioners who always pursue perfectness. Of cause, we want our system have all three attributes. And this theorem told us it’s impossible. It’s really bad, right?

Fortunately, MongoDB provides a way to save us from this imperfect world. While large amount of data are created, collected, arranged, integrated, and consumed, they are not equal. From the terminology of CAP theorem, they can be categorized as:

Drop partition tolerance. One way to do this is to put everything one machine, or in one specialized rack.
Drop availability. On encountering event, affected service simply waits until data is consistent.
Drop consistency. When partitions communication has problem, nodes just return its data, which might be staled.

It’s so good that MongoDB provided fine-grained control in this aspect. For each read and write operation, application can specify preference for each single operation. This is really handy cause we can access the same set of data with different policy at different moment. Thus we can reduce application latency further by reducing slow synchronized operations as more as possible.

Now a major portion of our data follows the popular “eventually consistency” paradigm, thus access to them are very fast. Still, a minor portion of data adopts the strong consistency paradigm. Access to these data is slow but always up-to-date. And best of all, all the data are living under the same roof of MongoDB peacefully.

Beautify XML and JSON in VIM

2012-10-11T19:32:00.002-07:00

map <f8> :% !xmllint --format -
map <f9> :%!python -m json.tool

How to highlight document comments in XCode

2012-08-18T01:49:00.002-07:00

There are multiple solutions to add document comments to Objective C code, such as Doxygen and AppleDoc. They can generate API doc from the source code. To java programmer, it's javadoc for Objective C.

By default, the "document comments" and "document comment keywords" are not highlighted in XCode. That's not a big problem, cause we can easily enable the syntax highlight by following steps:

1. Firstly, change XCode "Font & Color" preference to use different color to display "document comments" and "document comment keywords", as shown in blow image:

2. Secondly, modify "Applications/Xcode.app/Contents/OtherFrameworks/DevToolsCore.framework/Versions/Current/Resources/Built-in languages.pblangspec" and add following snippets to Objective-C configuration:

DocComment = "*";
DocCommentKeywords = (
"@author",
"@param",
...
);

3. After above two steps, we can see the document comments and specified keywords are highlighted in XCode:

Verify the atomic operation of MongoDB with PyMongo

2012-03-25T01:35:00.001-07:00

The purpose of this experiment is to verify the atomic operation in MongoDB. So firstly, I created a collection which has one record like:

{ "_id" : ObjectId("4f6d82...0"), "name" : "jjj", "num" : 0, "version" : 0 }

In above record, "version" attribute is used to keep version of the object. And for each modification, this version number will increase by "1". The existence of the "version" attribute is for preventing ABA problem.

Then I created a small python script for modifying this record with the "Compare And Set" operation:

from pymongo import Connection

conn = Connection()

db = conn.test

coll = db.atomicNum

sum = 0

for i in range(100000):

while(True):

atomic_num = coll.find_one()

version = atomic_num["version"]

num = atomic_num["num"]

coll.update({"name":"jjj", "version":version},{"$inc":{"num":1, "version":1}})

status = db.last_status()

if status["updatedExisting"]:

break

else:

sum = sum+1

print "RETRY: " + str(sum)

print coll.find_one()

I launched two instances of this python script. The script prints many lines of "RETRY:", but the final result is correct:

{u'_id': ObjectId('4f6d828b0618e88800000000'), u'num': 200000.0, u'name': u'jjj', u'version': 200000.0}

儿子上学一个多月了

2012-03-06T02:14:00.003-08:00

儿子在新西兰这边上学一个多月了，目前看来一切都好。虽然刚开始的时候我们很担心语言问题，不过对于我儿子来说好像不成问题。当然，他英语还有待提高，但是他在学校比较外向，相当敢说，有时候听他和隔壁的洋人小孩聊天，听得我直冒汗。

稍微总结一下吧：

学校教的东西比较少，主要是讲故事、阅读什么的。我们就当成英语课程在上了，呵呵
早上8点上课，下午3点下课，怎么接送孩子呢？头疼
气氛比较自由，小孩之间合作和沟通的机会很多。讲故事的时候，小朋友们都坐在地上听，很随意。
小孩都经常光着脚上学，在学校的游乐场中象猴子一样玩耍。见过的每个小学都有很大的草地，以及利用率很高的游乐设施。
儿子学数学都以为是game时间，因为上课的方式都是做游戏，玩色子什么的
老师穿着很随意，美术课穿的衣服都被画的五颜六色的
3年纪以上可以出去camping
课间吃饭都在外面草地上吃饭，毫无组织纪律

总的来说，儿子还是挺喜欢这边的学校的。

How to use different SSH for different git repositories

2012-03-06T01:45:00.001-08:00

In order to separate different SSH key for different git provider, we need to modify the underlying SSH config files, as shown below:

vim ~/.ssh/config
Host bitbucket bitbucket.org
Hostname bitbucket.org
IdentityFile ~/.ssh/id_bb

How to install python package to your home directory with easy_install

2011-12-30T20:11:00.000-08:00

If you are a cautious python user like me, who is not very keen to pollute global package hierarchy, this article is for you! This is pretty easy, but it takes me sometime to figure it out. Firstly, Mac OS Lion already has Python 2.7 installed. And it already support locating packages inside one special directory under user's home. So we only need to create a file named .pydistutils.cfg under user's home directory and put following content into it:

[install]
install_lib = /Users/ganzhi/Library/Python/2.7/lib/python/site-packages
install_scripts = ~/Application/bin

After that, easy_install will install packages to the location specified by "install_lib" property. In my case, it's "/Users/ganzhi/Library/Python/2.7/lib/python/site-packages". We should assign a path, which allows "write" operation and is a part of PATH environment environment, to "install_scripts"property. After that, we can install python packages by following command:

easy_install jinja2
easy_install PIL

搬到新屋子了

2011-12-10T23:02:00.001-08:00

经过两个周末的忙碌之后，我搬到了一个新的房子里面。主要原因是为了准备儿子的入学，不得不去到一个好学校的附近。在之前的房子住了一个多月，走的时候也颇为留恋。房子住得非常舒服，首先要感谢Mike和Emily的诸多照顾。在Remuera房子里，什么都不用操心，水、电、宽带一应俱全。今天搬过来之后，一切都需要自己负责了，麻烦啊...

说太多也没用，还是上图吧

房子后面有一小片花园

今天刚申请了Telecom的宽带，这个不用说太必要了，接下来是水、电和气了...

新公司工作一月体会

2011-12-01T01:10:00.001-08:00

到新公司报道已经一个月有余了，这个过程怎么说呢，痛并快乐着吧。新公司让我喜欢的有以下几点：

基本没有政治斗争，小公司没精力搞这些。老板大权在握，员工之间利益没有冲突，没有必要搞政治。
用的工具非常顺手，都是小而精的工具。小公司怎么选择工具呢，最大的考量是员工的工作效率，其次是成本。基于互联网的服务在这方面非常有优势。基本不会出现由于工具原因造成效率降低的现象。
不用加班。由于公司文化的原因，基本不会有加班。这一个月来的经验，德国人真的很敬业。
对人比较尊重，人情味相对浓厚一些。

目前来说，还没有适应的地方也有很多：

语言问题，由于英语现在是第一语言，所以使用频率更高，要求也更高了。目前还是感觉障碍很多。
需要了解的技术和产品很多，由于小公司人少，所以每个人需要多了解一些。这个无可厚非了，只是需要改变一下学习习惯。

希望能尽快度过这个过渡时期，尽早进入得心应手的时代 :)

寻访Justices of Peace

2011-10-28T04:17:00.000-07:00

Justices of Peace是我来新西兰之后才了解到的。就我粗略的理解来讲，Justices of Peace是当地有声望的一些志愿者，他们充当公证员的角色。当你需要证明你某些方面的承诺，例如支持你的直系亲属拿到新西兰的visa时，会需要Justices of Peace来签字认证。有点类似于中国的公证员。但是Justices of Peace不收取任何费用和礼物。

昨天我上Google找到了周围的几个JP的地址和联系方式，不过心里一直在犹豫。一来觉得自己英文不好担心解释不清楚，二来觉得冒昧打扰别人不太好。有点举棋不定，是要去图书馆找那种责任在身的JP还是找在家的JP呢？

不过随后我想到要尽早让家人过来，所以就下定决心给人家(Mr. Gover)打电话了。在一阵令人紧张的沟通之后，对方终于弄懂了我的意思，然后非常慷慨的承诺帮忙。我随后确认时间，对方说你现在过来就可以。于是，我就激动的拎着书包就冲了过去。首先经历了找路一关，由于我的新SIM卡这一阵不能用（可能是电话被联通锁机了），所以我就只好一边找路一边前行了。

靠着我那奇差的找路天赋，最后终于在天黑之前找到了Mr. Gover的家，还好这里天黑得晚。Mr. Gover的家签名的铭牌上刻了Justices of Peace的字样。主人家对这个称号还是非常自豪的。在一阵自我介绍和我表达了希望和家人尽早团聚的心情之后，JP非常爽快的帮我签名了。Mr. Gover人非常好，非常和蔼。在签字盖章之后，他还表示以后有事情随时找他帮忙。结束之后，Mr. Gover拿出一个本子记录我的一些信息，因为新西兰移民局会找他验证这个。他的本子上满满的都是找他帮忙的人的记录，相当多！我想他肯定花了很多自己的空闲时间来做这个Justices of Peace。做完这个之后，我很快就和主人道别离开了，因为不希望过多的占用他的休息时间。回来的路更黑了，不过这时的心情已经不一样了。

有点感慨，在一个陌生的国家，一个陌生人都能这样愿意帮助你。

BTW，这里有个Justices of Peace的列表：www.yellow.co.nz/justices_of_the_peace

新西兰游记 (2)

2011-10-27T02:45:00.000-07:00

上班两天了，我上班的地方在Surrey Crescent，住的地方在Ladies Mile。开车过去非常快，只需要14分钟，不过坐公车就非常不方便了。充分体会到了新西兰公共交通的不完善，跟国内比完全不是一个数量级的！公交车半小时才有一趟，还相当的昂贵。现在每天上班都需要1个半小时。不过坐公车的人也不多，这几天我还没有见过有人站立的。更不要说北京的城铁比了。

上了两天班，大致上什么也没干，光看文档了。周围的同事都是5～6点准时下班，真是太让人舒心了。在这里，项目经理会把时间安排得相对合理，不会把人压榨得非常厉害。嗯，不管世界怎么变，不加班才应该是常态。

新西兰风光优美，城市整洁。据说很多人家里都有小船。这是在公车上拍到的帆船俱乐部的港口，停靠着一排排的小船。生活真够愉快的。

天空永远都是那么蓝，而且直到晚上8点多才天黑，很适合小孩放学之后玩耍：

在countdown超市看到的的青岛啤酒：

最大的担心，现在英语有点不够用了，希望能早点提升上来吧。

新西兰游记 (1)

2011-10-24T13:33:00.000-07:00

今天是到达新西兰的第二天，对新西兰还没有什么感觉。从北京经香港转机到Auckland，一路上大概花了20个小时。其实从北京有直飞的飞机，不过不是每天都有。因为周一是这边的劳动节，所以在家里过完周六才走的。

有朋友问我怎么过来的新西兰，我这边的大致经过如下：

拿到Job Offer，这一步会比较难。如果没有新西兰的朋友帮忙介绍。
办理工作签证，在北京的新西兰有一个专门的机构办理，在东直门附近。我办理的是Work to Resident签证，20天就拿到了。签证机构会联系给你offer的雇主来确认工作的真实性。现在IT算是新西兰的短缺机能，所以签证比较好办。
买机票，单程的机票大概在5000人民币左右。

由于刚住下没多久，所以对Auckland还没有什么感觉。不过房东Mike和Emily两人都很好，相信应该会相处的比较愉快。

It's very easy to create GWT projects with Spring Roo

2011-09-28T06:48:00.000-07:00

Spring Roo is a very cool open source tool, which can great reduce the time for forging a workable application. After executing below steps, a website can be established:

project --topLevelPackage com.ibm.sde --projectName FeatureManagement --java 6
persistence setup --provider HIBERNATE --database HYPERSONIC_IN_MEMORY
enum type --class ~.domain.FeatureStatus
enum constant --name ENABLE
enum constant --name DISABLE
entity --class ~.domain.Feature --testAutomatically
field string --fieldName displayName --notNull
focus --class com.ibm.sde.domain.Feature
field date --fieldName created --type java.util.Date
field enum --fieldName status --type com.ibm.sde.domain.FeatureStatus
gwt setup
logging setup --level INFO

After that, execute "mvn gwt:run" to execute the application. Spring Roo is still very young, so it has several problems:

It has no support to multiple module project yet
If you move classes to a new folder in Eclipse, it brings a little chaos

But still, this tool looks promising. When it become 2.0, 3.0, or 4.0, it can be a very handy tool for java software developers.

"How can I keep from singing" - 儿子诗歌朗诵的题目

2011-09-10T20:53:00.000-07:00

我挺喜欢的一首诗歌，为了儿子的诗歌朗诵，又找了个中午翻译版，稍微改了改：

我的生命在无尽的歌声中流淌

尽管人世间仍然充满了忧伤

我听见了那来自远方的声音

那清晰的歌声带来了新生的曙光

虽然生活中充斥着争吵和喧嚷

那音乐却始终在我耳边回响

她早已进入了我的灵魂深处

没有什么东西能让我停止歌唱

虽然那暴风雨在我身边隆隆作响

我心中的真理却仍和从前一样

虽然那黑暗正悄悄吞噬着大地

我心中的歌声却把整个世界照亮

再大的风暴也不会让我惊慌

只要我能坚持自己的理想

既然爱是整个世界的主宰

那还有什么能让我停止歌唱？

夏威夷旅行记（4）

2011-07-09T07:04:00.000-07:00

夏威夷旅行记（3）

2011-07-09T06:38:00.000-07:00

夏威夷旅行记（2）

2011-07-09T06:11:00.000-07:00

夏威夷旅行记（1）

2011-07-09T04:27:00.000-07:00

今年我非常幸运的获得了公司的Best Of IBM大奖，获得了去夏威夷的机会。

发图片吧：

夏威夷的机场很小：

机场入口

Honolulu的酒店很好，给我们免费升级到了海景房：

海景

传说中的火烈鸟

从Lotus Notes同步联系人到Gmail的简易方法

2011-05-03T00:34:00.000-07:00

周末买了iPhone 4，又面临从就手机同步“联系人”到新手机上的工作。网上查了一下，发现很多收费的工具，看上去都很不靠谱。随后研究了一下Lotus Notes和Gmail，找到了一个很简单的方法。
1. 首先，进入到Lotus Notes的Contacts功能里面，然后export所有的联系人到一个.csv文件里
2. 随后，在Gmail的Contact管理页面，Import这个.csv文件
3. 设置iPhone4使得其和Google同步calendar和contacts：

在iphone（ipad）的设置中将Exchange账号设置为Gmail账号的话，可以实现iphone自带日历和此Gmail账号的Google Calendar同步。

完成！

一个多线程Java程序的潜在bug

2010-10-14T18:01:00.000-07:00

多线程编程的难点就在于程序的正确性难以判断。Jeremy Manson发现了今天我们要介绍的这个在多线程环境下有bug的程序。首先，下面的Java代码在多线程环境下是正确的吗？


class String {
  private int hashcode;
  public static int hashcode() {
      if (this.hashcode == 0) {
               int h = ... compute hash ...
               this.hashcode = h;
       }
       return this.hashcode;
   }
}

初看上去似乎没有什么问题。尽管有一个data race存在，两个线程可能同时计算并保存hash值，但应该不影响正确性。

问题在于编译器可能进行把以上代码翻译成为如下所示：


class String {
   private int hashcode;
   public static int hashcode() {
       int h = this.hashcode;
       if (this.hashcode == 0) {
           h = ... compute hash ...
           this.hashcode = h;
       }
       return h;
   }
}

上面代码中我们用h来缓存了hashcode，随后，我们判断如果hashcode不等于0，那么就返回h。问题在于，在做第二个判断之前，hashcode的值可能已经被另外的线程改变，因此这个程序有可能返回一个0。这种错误即使是有多年经验的程序员也很难发现。

IBM中国的新家

2010-10-10T05:08:00.000-07:00

Google Earth上的盘古大观模型: