I'm a lumberjaph

The state of HTTP's libraries in Python

2012-02-17T00:00:00-08:00

Hey! I’m alive!

I’ve started to write some Python for work, and since I’m new at the game, I’ve decided to start using it for some personal project too.

Most of what I do is related to web stuff: writing API, API client, web framweork, etc. At SAY: I’m working on our platform. Nothing fancy, but really interesting (at least to me) and challenging work (and we’re recruting, drop me a mail if you want to know more).

Writing HTTP requests with Python

httplib

httplib is part of the standard library. The documentation says: “It is normally not used directly”. And when you look at the API you understand why: it’s very low-level. It uses the HTTPMessage library (not documented, and not easily accessible). It will return an HTTPResponse object, but again, no documentation, and poor interface.

httplib2

httplib2 is a very popular library for writing HTTP request with Python. It’s the one used by Google for it’s google-api-python-client library. There’s absolutly nothing in common between httplib’s API and this one.

I dont like it’s API: the way the library handles the Response object seems wrong to me. You should get one object for the response, not a tuple with the response and the content. The request should also be an object. Also, The status code is considered as a header, and you lose the message that comes with the status.

There is also an important issue with httplib2 that we discovered at work. In some case, if there is an error, httplib2 will retry the request. That means, in the case of a POST request, it will send twice the payload. There is a ticket that ask to fix that, marked as won’t fix. Even when there is a perfectly acceptable patch for this issue. (it’s a “WAT” moment). I’m really curious to know what was the motiviation behind this, because it doesn’nt makes sense at all. Why would you want your client to retry twice your request if it fails ?

urllib

urllib is also part of the standard library. I was suprised, because given the name, I was expecting a lib to manipulate an URL. And indeed, it also does that! This library mix too many different things.

urllib2

urllib2. And because 2 is not enough, also …

urllib3

urllib3. I thought for a moment that, maybe, the number number was related to the version of Python. I’ll spare you the suspense, it’s not the case. Now I would have expected them to be related to each other (sharing some common API, the number being just a way to provides a better API than the previous version). Sadly it’s not the case, they all implement different API.

At least, urllib3 has some interesting features:

Thread-safe connection pooling and re-using with HTTP/1.1 keep-alive
HTTP and HTTPS (SSL) support

request

A few persons pointed me to requests. And indeed, this one is the nicest of all. Still, not exactly what *I*’m looking for. This library looks like LWP::Simple, a library build on top of various HTTP components to help you for the common case. For most of the developers it will be fine and do the work as intented.

What I want

Since I’m primarly a Perl developer (here is were 99% of the readers are leaving the page), I’ve been using LWP and HTTP::Messages for more than 8 years. LWP is an awesome library. It’s 16 years old, and it’s still actively developed by it’s original author Gisle Aas. He deserves a lot of respect for his dedication.

There is a few other library in Perl to do HTTP request, like:

AnyEvent::HTTP : if you need to do asynchronous call
Furl : by Tokuhiro and his yakuza gang

but most of the time, you end up using LWP with HTTP::Messages.

One of the reason this couple is so popular is because it provides the right abstraction:

a user-agent is provided by LWP::UserAgent (that you can easily extends to build some custom useragent)
a Response class to encapsulates HTTP style responses, provided by HTTP::Message
a Request class to encapsulates HTTP style request, provided by HTTP::Message

The response and request objects use HTTP::Headers and HTTP::Cookies. This way, even if your building a web framework and not a HTTP client, you’ll endup using HTTP::Headers and HTTP::Cookies since they provide the right API, they’re well tested, and you only have to learn one API, wether you’re in an HTTP client or a web framework.

http

So now you start seeing where I’m going. And you’re saying “ho no, don’t tell me you’re writing another HTTP library”. Hell yeah, I am (sorry, Masa). But to be honest, I doubt you’ll ever use it. It’s doing the job I want, the way I want. And it’s probably not what you’re expecting.

http is providing an abstraction for the following things:

http.headers
http.request
http.response
http.date
http.url (by my good old friend bl0b)

I could have named it httplib3, but http seems a better choice: it’s a library that deals with the HTTP protocol and provide abstraction on top of it.

You can found the documentation here and install it from PyPI.

examples

A few examples

a client

With this, you can easily build a very simple client combining thoses classes, or a more complex one. Or maybe you want to build a web framework, or a framework to test HTTP stuff, and you need a class to manipulate HTTP headers. Then you can use http.headers. The same if you need to create some HTTP responses: http.response.

I’ve started to write httpclient based on this library that will mimic LWP’s API.

I’ve started to document this library and I hope to put something on PyPI soon.

Weekly Review - 2012-06

2012-02-12T00:00:00-08:00

Lot of documentation for http and httpclient this week, and the first release (with some help from James Rowe) for `http` on PyPI.

I’ve started to play with google app engine, I’ll probably push two applications with their code soon.

Weekly Review - 2012-05

2012-02-05T00:00:00-08:00

Managed to get some work done on the Spore implementation for Python and http.

closed gh-2 on httpclient: added a new client (httpclient.simple, equivalent to LWP::Simple)
closed gh-23 on http: fix a bug with `append` on Url.Path
closed gh-21 on http: add a setter for `content` on Response
closed gh-24 on http: when the path is empty, set it to ‘/’ so the final url is `http://foo.com/` instead of `http://foo.com`
closed gh-22 on http: small bug on the Url class. When the path was unicode, we splitted on each character instead of the default separator
closed gh-17 on spyre: simplified the interface to create a new client
closed gh-14 and gh-13 on spyre: switched to `http` and `httpclient` for spyre!
closed gh-18 on spyre: since we’re using `http`, we don’t need a custom Response class anymore

Also, some code cleanup in bot projects, added a few tests, and created a few tickets for some future tasks.

Weekly Review - 2012-04

2012-01-29T00:00:00-08:00

Nada. Shame on me :(

Weekly Review - 2012-03

2012-01-22T00:00:00-08:00

Mostly worked on http and httpclient this week, and started to work on some Java project (lost two days to learn how to use Maven and eclipse, not very proud here …)

closed gh-19 on http: added the setup.py
closed gh-18 on http: added the license file (MIT)
closed gh-17 on http: use http.Url when an Url is used
closed gh-12 on http: merged the code for fluffyurl inside http’s repository
closed gh-11 on http: created a new dist (httpclient) for the client’s code
closed gh-5 on * httpclient added the license file (MIT)
some documentations for http and httpclient

Weekly Review - 2012-02

2012-01-15T00:00:00-08:00

I didn’t do much this week. Here is a quick list:

published an AnyEvent client for Peer39’s API
created a bunch of tickets for Spyre
closed gh-6 on fluffyhttp: renamed fluffyhttp to http
“closed gh-9 on
fluffyhttp":https://github.com/franckcuny/fluffyhttp/issues/9: Now some headers (like Last-Modified) can store datetime object and are properties. If called as a property they will return the object, if they’re called via the get method from the Header object, they will return a string.
start documenting fluffyurl

I should be able to do more work next week (and Monday is an holiday, I’ll try to put more work on Spyre).

Weekly Review - 2012-01

2012-01-08T00:00:00-08:00

What I’ve done this week:

applied Brian’s patches for Net::HTTP::Spore
remove a useless regex in Net::HTTP::Spore
rewrite the way Headers are handled withing `http’
some documentation for `http’ * (Response objects)
refactoring of the Header class to add some properties when required

StarGit

2011-06-20T00:00:00-07:00

Last year I did a small exploration of GitHub to show the various communities using GitHub and how they work. I wanted to do it again this year, but I was lacking time and motivation to start over. A couple of months ago, I got a message from mojombo asking me if I was planning to do a new poster. This triggered the motivation to work on it again.

This time I got help from Alexis to provide you with an awesome tool: a real explorer of your graph, but more on this later ;)

And of course, the poster. Feel free to print it yourself, the size of the poster is A1.

The data

All the data are available! Last year I got some mails asking me for the dataset. So this time I asked first if I could release the data with the code and the poster, and the anwser is yes! So if you’re intereseted, you can download it.

The data are stored in mongodb, so I provide the dump which you can easily use:

wget http://maps.stargit.net/dump/github.tgz
tar xvzf github.tgz
cd github
mongorestore -d github .

Now you can use mongodb to browse the imported database. There is 5 collections: profiles / repositories / relations / contributions / edges.

Methodology

Last year I did a simple “follower/following” graph. It was already interesting, but it was also really too simple. This time I wanted to go deeper in the exploration.

The various step to process all this data are:

using the GitHub API, fetch informations from the profiles.
when all the profiles are collected, informations about the repositories are fetched. Only forked repositories are kept.
“simple” relations (followers/following) are kept and used later to add weight to relations.
tag user with the main programming language they use. Using the GitHub API, I was able to categorize ~40k profiles (about 1/3 of my whole dataset).
using the GeoNames API, extract the name of the country the user is in. This time, about 55k profiles were tagged.
fetch contributions for each repositories
compute a score between the author of the contribution and the owner of the repo
add a weight to each edges, using the computed score and “+1” if the developer follow the other developer

For all the graphs, I’ve used the following colors for:

Ruby
JavaScript
Python
C (C++, C#)
Perl
PHP
JVM (Java, Clojure, Scala)
Lisp (Emacs Lisp, Common Lisp)
Other

Exploring

Feel free to do your own analysis in the comments :) For each map, you’ll find a PDF of the map, and the graph to explore using gephi (in GEXF or GDF format).

but first, some numbers

I’ve collected:

123 562 profiles
2 730 organizations
40 807 repositories

This took me about a month in order to collect the data and to build the adapted tools.

Accounts creations

The following chart show the number of account created by month. “Everyone” means the total of accounts created. You can also see the numbers for each communities.

On the “Everyone” graph, you can see a huge pick around April 2008, that’s the date GitHub was launched.

For most of the communities, the number of created accounts start to decrease since 2010. I think the reason is that most of the developers from those communities are now on GitHub.

languages

(Keep in mind that these numbers are coming from the profiles I was able to tag, roughly 40k)

Ruby: 10046 (28%)
Python: 5403 (15%)
JavaScript: 5282 (15%) (JavaScript + CoffeeScript)
C: 5093 (14%) (C, C++, C#)
PHP: 3933 (11%)
JVM: 3790 (10%) (Java, Clojure, Scala, Groovy)
Perl: 1215 (3%)
Lisp: 348 (0%) (Emacs Lisp, Common Lisp)

Those numbers doesn’t really match what GitHub gave, but it could be explained by the way I’ve selected my users.

country

United States: 19861 (36%)
United Kingdom: 3533 (6%)
Germany: 3009 (5%)
Canada: 2657 (4%)
Brazil: 2454 (4%)
France: 1833 (3%)
Japan: 1799 (3%)
Russia: 1604 (2%)
Australia: 1441 (2%)
China: 1159 (2%)

The United States are still the main country represented on GitHub, no suprise here.

If you are interested in the “geography” of Open Source, you should read these two articles: Coding Places and Investigating the Geography of Open Source Software through Github.

companies

Looking at the “company” field on user’s profile, here are some stats about which companies has employees using GitHub:

ThoughtWorks: 102
Google: 66
Mozilla: 65
Yahoo!: 65
Red Hat: 64
Globo.com: 55
Twitter: 53
Facebook: 45
Yandex: 43
Intridea: 34
Microsoft: 33
Engine Yard: 32
Pivotal Labs: 29
MIT: 28
Rackspace: 27
IBM: 24
Caelum: 23
Novell: 22
GitHub: 22
VMware: 22

I didn’t knew the first company, ThoughtWorks, and I was expecting to see FaceBook or Twitter as the company with most developpers on GitHub. It’s also interesting to see Yandex here.

Global graph (1628 nodes, 9826 edges)

(download PDF, download GDF)

The main difference with last year, is the android / modders community. They’re developing mostly in C and Java. The poster has been created from this map.

Ruby (1968 nodes, 9662 edges)

(download PDF, download GDF, download GEXF)

This is still the main community on GitHub, even if JavaScript is now the most popular language. This graph is really dense, it’s not easy to read, since there is no real cluster in this one.

Python (1062 nodes, 2631 edges)

(download PDF, download GDF)

Here we have some clusters. I’m not familiar with the Python community, so I can’t really give any insight.

Perl (608 nodes, 2967 edges)

(download PDF, download GDF, download GEXF)

I really like this graph since it show (in my opinion) one of the real strength of this community: everybody works with everybody. People working on a webframework will collaborate with people working on Moose, or an ORM, or other tools. It shows that in this community, people are competent in more than one field.

The Perl community is about the same size as last year. However, we can extract the following informations:

the Japaneses Perl Hackers are still a cluster by themselves
miyagawa is still the glue between the Japanese community and the “rest of the world”
other leaders are: Florian Ragwitz (rafl), Andy Amstrong (AndyA), Dave Rolsky (autarch)
some clusters exists for the following projects:
- Moose
- Dancer

As we can see on the previous charts, the number of created accounts for the Perl developpers is stalling.

United States (2646 nodes, 11344 edges)

(download PDF, download GDF, download GEXF)

This one is really nice. We can clearly see all the communities. There is something interesting:

C and Ruby are on the opposite side (C on the left, Ruby on the right)
Python and Perl are also opposed (Perl at the bottom and Python at the top)

I’ll let you take some conclusion by yourself on this one ;)

France (706 nodes, 1059 edges)

(download PDF, download GDF, download GEXF)

We have a lot of small clusters on this one, and some very big authorities.

Japan (464 nodes, 1091 edges)

(download PDF, download GDF, download GEXF)

There is three dominants clusters on this one:

Ruby
Perl
C

The Ruby and Perl one are well connected. There is a lot of japanese hacker on CPAN using both languages.

StarGit

StarGit is a great tool we built with Alexis to let you explore your community on GitHub. You can read more about the application on Alexis’ blog

It’s hosted on dotcloud (I’m still amazed at how easy it was to deploy the code …), using the Perl Dancer web framework, MongoDB to store the data, and Redis to do some caching.

Credits

I would like to thanks the whole GitHub team for being interested in the previous poster and to ask another one this year :)

A huge thanks to Alexis for his help on building the awesome StarGit. Another big thanks to Antonin for his work on the poster.

French Perl Workshop 2011

2011-05-08T00:00:00-07:00

The call for paper for the French Perl Workshop is open. This event will be held the 24th and 25th of June in Paris. As always, this is a free conference.

Where

The workshop will take place in Cité des Sciences in Paris. This is the same place as for last year OSDC.fr. The venue is easy to access using the subway (line 7), and there is a nice park to take a break or take a lunch.

Talks

Yes, we need more talks :) We’ve already accepted a little bit more than 10 talks, but we need more talks to fill the schedule. An introduction to Perl will also be organized in addition to the talks. We’re also looking for some talks about Perl’s success in entreprise. If you’re using Perl at work, please, come to share with a captive audiance how you use it and how it’s helpful.

What

We will organize two dinners, one the thursday night, and a second the friday night. You should check the wiki to find more informations. (BooK is also looking for some people to join him to a concert :))

Who

You! of course. This event will be nothing without you, your talk, and your presence. The whole organizers team is working hard to make this event a great conference. I also want to thank Laurent Boivin for all his efforts and pushing us to do our tasks.

If you happen to be in Paris next week during Solution Linux, please, stop by to the French Mongueurs booth.

So, what are you waiting to register ?

new job

2011-04-22T00:00:00-07:00

I’ve worked for nearly four years at Linkfluence. From this summer on, I’m switching to a new job at Say Media, San Francisco.

Working at Linkfluence has been an awesome experience for me. I’ve learned a lot of things, worked on really interesting projects, and worked with an awesome team.

Linkfluence is hiring

Linkfluence is always looking for developers to join the team. If you want a Perl job in France, and want to work on interesting topics such as:

social media monitoring
web crawling
data indexation
data visualization

Linkfluence is the company you want to join. If you’re interested, send your CV to Camille Maussang (CTO at Linkfluence): camille.maussang at linkfluence.net.

Technologies at Linkfluence

We’re typically using the following technologies to solve our various problems:

Solr and ElasticSearch to index the contents of various social media (blogs, tweeter, etc)
Riak to store the contents
a lot of Perl (and not the Perl of your (dear) grandmother, but Catalyst, Moose, DBIx::Class, Dancer, perl-5.12, …)
Redis
PostgreSQL
Gephi

Open Source at Linkfluence

Developers here are encouraged to contribute to open source projects, to publish code on GitHub, and talk about the things they do.

The company also sends the developers to various open source conferences:

YAPC::Europe
FOSDEM
Belgium Perl Workshop
French Perl Workshop
OSDC.fr

You’re encouraged to talk at those conferences, and you can prepare your talks during work time. Linkfluence is also working closely with the French Perl Mongueurs, and tries to organize events with them (technical meetings, Dancer hackaton, etc).

Science at Linkfluence

Most of the developers in the company are coming from a scientific background (but this is not a requirement). Some of them do publish scientific papers on topics like social media and graphs analysis.

Stabilité globale et diversité globale dans la dynamique des commentaires de Flickr (Raux, Prieur, 2011)
Describing the Web in less than 140 characters (Raux, Grunwald, Prieur, 2011)
Essai de géographie de la blogosphère politique française (Cardon, Fouetillou, Lerondeau, Prieur 2011)
Two paths of glory – Structural positions and trajectories of websites within their topical territory (Cardon, Fouetillou, Roth 2011)
Clustering based on random graph model embedding vertex features (Zanghi, Volant, Ambroise)
Strategies for Online Inference of Network Mixture (Zanghi, Picard, Miele, Ambroise)
Approches modèles pour la structuration du Web vu comme un graphe (Zanghi)

If you want to work with smart people on interesting topics like graphs, and help them to implement solutions or experiment with some algorithm, you’ll feel at home there.

My future

I’ll move to San Francisco this summer to start my new job at Say. I have a lot to do before moving, but I’m very excited with this new opportunity. Sadly, I’ll probably miss YAPC::EU, but I should do YAPC::NA next year \o/.