Apparatus

Twisted.web vs Tornado, part deux

Matt Heitzenroder — Thu, 17 Sep 2009 16:18:12 +0000

The first time I conducted a performance test between Twisted.web and Tornado, the results were not very meaningful in retrospect. I have three good reasons for redoing the performance tests:

Apache Benchmark errors – the twisted.web test would crash ab once the concurrency reached a certain level, thus not revealing the real performance of twisted.web.
Unrealistic server environment – I did test of my MacBookPro, not a dedicated server.
Controversy – the ensuing conversation of twisted.web vs tornado warrants a more accurate test with better tools and hardware.

This time, I performed the test with the aid of my friend Travis (thank you!) using The Grinder, a legitimate load testing tool and several virtual machines from the Rackspace Cloud. The servers were Ubuntu Jaunty 9.04 with 2GB or RAM and quad-core 2GHz processors.

The Grinder script that I used to conduct the test just did a simple request. The Grinder test and preferences are available for download. I changed the IP address in the Grinder test to switch between twisted.web and tornado. The following command was used to start the tests:

1	$ for i in 100 200 300 400 500 750 1000; do java -Dgrinder.threads=$i -cp grinder-3.2/lib/grinder.jar net.grinder.Grinder grinder.pref; done;

I used the same, simple ”hello, world” response apps for twisted.web and tornado as I did for the previous test. Each app was run as a single threaded application. I also used Python 2.6.2.

The raw data that resulted from The Grinder and html output from the GrinderAnalysis is available for download.

The following results were compiled into a spreadsheet from the GrinderAnalysis tool. The X-axis is an individual test that represents number of concurrent requests. The Y-axis is the mean response time in milliseconds. Lower mean response time is better.

Below are examples of the response times graphs (blue) that I used to compile the above spreadsheet. The entire output from GrinderAnalytics is available for download, as mentioned above. Transactions per second (TPS) is the graph in the green and is maximum number of iterations of the grinder test that successfully ran in a one second interval.

100 Concurrent Requests

500 Concurrent Requests

1000 Concurrent Requests

Twisted.web vs Tornado Performance Test

Matt Heitzenroder — Fri, 11 Sep 2009 15:43:39 +0000

If you look about this blog, there have been a number posts about the twisted framework. Dan, Igor, and I have been playing with txAMQP for sometime. We’ve never run any twisted code in a high-scale production environment, we’re just enthusiasts. So when I read the comments from the Tornado announcement and in the HackerNews thread, I asked #twisted.web on freenode:

“Who runs a large-scale production environment with twisted.web?”

The channel member responded that I speak to abstractbill since he’s written twisted code for justin.tv. He had already commented on hackernews and therefore, I already knew where he stood. Users were asking and I just needed to see for myself:

“What’s the performance difference between Twisted.web and Tornado?”

So I performed the following tests using Apache Benchmark the same test that finiteloop used. However, I varied concurrent connections in addition to the benchmarked performance of requests per second.

The performance test was performed on my MacBookPro Intel Core Duo 2.4GHz, 4GB of RAM. Each python webserver was run in one, single thread. The code for the tests is available for Twisted.web and Tornado. Both webservers benchmarked by increasing the concurrent connections until the servers closed the connection and prevented ab from completing the test.

Tornado outperforms Twisted.web, but in the scale of those numbers the difference is negligible as finiteloop suggests. What is noticeable is that after 50 concurrent connections, twisted.web’s webserver began to close the connection and not complete the test.

The results do not really stop me from wanting to continue using twisted.web, however there is some room for improvement in the twisted.web framework. I applaud Facebook for releasing Tornado as open source and FriendFeed for create a really nice async webserver in python. I look forward to it’s continued development.

Memory Profiling in Twisted Code with Heapy

Matt Heitzenroder — Tue, 01 Sep 2009 05:20:42 +0000

I was prototyping an HTTP server using the Python framework Twisted. After cobbling my code together, I was demoing to Igor and Dan. Igor, decided to flood the HTTP server with requests to see how it would hold over time. Turns out we were leaking memory.

A short Google later and a nice StackOverflow thread, we decided to profile the HTTP server with Heapy. Heapy is part of package called Guppy. Simple install is done like this

1	$> pip install guppy

The code below is an example of a simplified HTTP server. I blatantly put in a memory leak for the purposes of demonstrating on how I found it.

1 2	from twisted.web import resource, server from twisted.internet import reactor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

memory_leak = []

class RootResource(resource.Resource):
def render_GET(self, request):
# Obvious Memory Leak
memory_leak.append(request)
return "OK"

def getChild(self, name, request):
return self

if __name__ == '__main__':
port = 8080
site = server.Site(RootResource())
reactor.listenTCP(port, site)

# Dubugging Memory - outputs file to heap_profile.txt from cwd
from guppy import hpy
h = hpy()
def memory_profile(h, reactor):
f = open('heap_profile.txt', 'a')
f.write('Memory Statistics: \n%s' % h.heap())
f.write('\n\n')
f.write('Traversing Heap: \n%s' % h.heap().get_rp(40))
f.write('\n\n')
f.close()

# Print profile every 5m after execution for 1h
for increment in xrange(60, 3600, 300):
reactor.callLater(increment, memory_profile, h, reactor)
# Kill reactor a minute later to give heap time to write.
reactor.callLater(3660, reactor.stop)

# Now open the page repeatively
from twisted.web.client import getPage
from twisted.internet.task import LoopingCall
gp = LoopingCall(getPage, 'http://localhost:8080')
reactor.callLater(1, gp.start, 0.1)

1	reactor.run()

If you look through the comments you’ll be able to see that I create a function called memory_profile(), which just spits out two profiles from heapy to a file. A few lines past this, you can see I use the reactor’s callLater() to call at 300s increments. The last thing I do is kill the reactor with reactor.callLater(3660, reactor.stop). I give heapy 60 seconds to write the last profile to the file (which is more than plenty of time).

The following output is two profiles, one at 1 minute, another 5 minutes after.

Memory Statistics:
Partition of a set of 104996 objects. Total size = 9162436 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  43217  41  2949300  32   2949300  32 str
     1    590   1   988840  11   3938140  43 dict of twisted.web.server.Request
     2   4369   4   848588   9   4786728  52 dict (no owner)
     3  18370  17   757704   8   5544432  61 tuple
     4   8059   8   470316   5   6014748  66 list
     5   4724   4   321232   4   6335980  69 types.CodeType
     6    590   1   309160   3   6645140  73 dict of twisted.internet.tcp.Server
     7    204   0   307344   3   6952484  76 dict of module
     8   4754   5   285240   3   7237724  79 function
     9    467   0   209740   2   7447464  81 type
<222 more rows. Type e.g. '_.more' to view.>

Traversing Heap:
Reference Pattern by <[dict of] class>.
 0: _ --- [-] 105004 (Queue.Queue | __builtin__.PyCObject | __builtin__.cell ...
 1: a      [-] 590 dict of twisted.web.server.Request: 0x15e18a0..., 0x15e1b98
 2: aa ---- [-] 590 twisted.web.server.Request: 0x1564b98, 0x15d8850...
 3: a3       [-] 1 list: 0x12634b8*590
 4: a4 ------ [S] 1 dict of module: __main__
 5: b      [-] 18370 tuple: 0x4a76f0*1, 0x4a7730*1, 0x4a77f0*1, 0x4a7810*1...
 6: ba ---- [S] 4724 types.CodeType: site.py:82:makepath...
 7: bb      [-] 590 dict of twisted.internet.tcp.Server: 0x15e3710..., 0x162f310
 8: bba ---- [-] 590 twisted.internet.tcp.Server: 0x157ef90, 0x15e3150...
 9: bbaa      [^ 1] 590 dict of twisted.web.server.Request: 0x15e1b98...

Memory Statistics:
Partition of a set of 261479 objects. Total size = 25045732 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0   3543   1  5938068  24   5938068  24 dict of twisted.web.server.Request
     1  87519  33  4638728  19  10576796  42 str
     2  19145   7  2917228  12  13494024  54 dict (no owner)
     3  46449  18  2620132  10  16114156  64 list
     4   3543   1  1856532   7  17970688  72 dict of twisted.internet.tcp.Server
     5   7086   3   992040   4  18962728  76 dict of twisted.internet.address.IPv4Address
     6   7086   3   992040   4  19954768  80 dict of twisted.web.http_headers.Headers
     7   7086   3   992040   4  20946808  84 dict of twisted.web.http_headers._DictHeaders
     8  21326   8   864116   3  21810924  87 tuple
     9   4724   2   321232   1  22132156  88 types.CodeType
<226 more rows. Type e.g. '_.more' to view.>

Traversing Heap:
Reference Pattern by <[dict of] class>.
 0: _ --- [-] 261479 (Queue.Queue | __builtin__.PyCObject | __builtin__.cell ...
 1: a      [-] 3543 dict of twisted.web.server.Request: 0x15e18a0..., 0x15e1b98
 2: aa ---- [-] 3543 twisted.web.server.Request: 0x1564b98, 0x15d8850...
 3: a3       [-] 1 list: 0x12634b8*3543
 4: a4 ------ [S] 1 dict of module: __main__
 5: b      [-] 3543 dict of twisted.internet.tcp.Server: 0x15e3710..., 0x162f310
 6: ba ---- [-] 3543 twisted.internet.tcp.Server: 0x157ef90, 0x15e3150...
 7: baa      [^ 1] 3543 dict of twisted.web.server.Request: 0x15e1b98...
 8: c ---- [-] 25140 list: 0x4a8850*1, 0x4a8878*2, 0x4a88a0*2, 0x4a8ee0*202...
 9: ca      [^ 1] 3543 dict of twisted.web.server.Request: 0x15e1b98...

I am looking for a pattern here. Specifically what I am interested in is:

Partition of a set of 104996 objects. Total size = 9162436 bytes.

Then again in the second profile:

Partition of a set of 261479 objects. Total size = 25045732 bytes.

I can see that my number of objects is rapidly increasing and I can see that we went from ~9MB to ~25MB in very short order.

Now to figure out where it is coming from, I want to “Traverse” the heap. This will help me go in the right direction. It’s pretty clear from as I look at the change from the “Traversing Heap” from the first set to the second set, that the Request object is becoming increasing an issue.

Twisted is extremely well written and does not have memory leaks “out of the box”, so the issue is in my code. Thankfully I do not have complex code and it’s very easy to see what’s going on with the Request object. Appending the Request object to a global list is pretty stupid… thank god it’s only an example :P

The major lesson learned from my experience is to profile memory often and to set up automated tests to do so. Had our development gone much longer without examining our code, we would have likely pulled our hair out. Also critical for us to find the source of our leak, was to let the test run for a length of time and try to find a pattern.

Using prefetch with txAMQP and RabbitMQ 1.6.0

Dan Reverri — Fri, 14 Aug 2009 00:23:37 +0000

I’m hoping this might be useful to someone. I wrote a quick script that uses “channel.basic_qos(prefetch_count=1)” with txAMQP and RabbitMQ 1.6.0. It is available here:
http://app.arat.us/blog/wp-content/uploads/2009/08/prefetch.py

You can run it as follows:

1
2
3
4
5
6
7
8
9
10
11

python prefetch.py localhost 5672 / guest guest path/to/specs/amqp0-8.xml
* Slow consumer received message: Message 1 *
Fast consumer received message: Message 2
Fast consumer received message: Message 3
* Slow consumer received message: Message 4 *
Fast consumer received message: Message 5
Fast consumer received message: Message 6
* Slow consumer received message: Message 7 *
Fast consumer received message: Message 8
Fast consumer received message: Message 9
* Slow consumer received message: Message 10 *

Notice that the fast consumer grabs twice as many messages as the slow consumer.

I have not found a clean way of ending the script, you have to force quit with CTRL+C. Let me know if you have any suggestions.

Py2JSON – Python Classes To JSON-Schema SMD

Igor Guerrero — Tue, 28 Jul 2009 12:00:42 +0000

As we work with Python and JSON-Schema, we found the necessity to create JSON-Schemas from Python classes, but we didn’t find any existing libraries to make this task easier, so we wrote one :)

Py2JSON is a library that will create from a given Python class a JSON-Schema string that you can later save to a file for example.

Development is done in Github: http://github.com/igorgue/py2json

You can report issues or enhancement request here: http://github.com/igorgue/py2json/issues

Getting and Installing Py2JSON

1
2
3
4
5

igor@igorlaptop:~$ git clone git://github.com/igorgue/py2json.git
igor@igorlaptop:~$ cd py2json/
igor@igorlaptop:~/py2json$ python setup.py install
igor@igorlaptop:~/py2json$ python -c 'import py2json'
igor@igorlaptop:~/py2json$

If you don’t get any error then you’ve successfully installed py2json (sudo might be required in some UNIX environments). Here a example, using py2json to generate a JSON-Schema string:

Examples

1
2
3
4
5
6
7
8
9
10

>>> from py2json import Py2JSON
>>> class Foo(object):
... """this class does foo"""
... def bar(self, string='this is a string'):
... print string
...
>>> smd = Py2JSON(Foo)
>>> smd.schema
'{"services": {"bar": {"target": "bar", "parameters": [{"default": "this is a string", "optional": false, "type": "string", "name": "string"}]}}, "envelope": "JSON", "target": null, "transport": "REST", "additionalParameters": true}'
>>>

To exclude a method you can use the exclude parameter:

1
2
3
4
5
6
7
8
9
10
11

>>> class Foo(object):
... """this class does foo"""
... def bar(self, string='this is a string'):
... print string
... def bar2(self, integer=123):
... print integer
...
>>> smd = Py2JSON(Foo, excluded_methods=['bar2'])
>>> smd.schema
'{"services": {"bar": {"target": "bar", "parameters": [{"default": "this is a string", "optional": false, "type": "string", "name": "string"}]}}, "envelope": "JSON", "target": null, "transport": "REST", "additionalParameters": true}'
>>>

You can use it to store the schema in a string too:

1
2
3
4
5
6
7

>>> f = open('foo.js', 'w')
>>> smd = Py2JSON(Foo)
>>> f.write(smd.schema)
>>> f.close()
>>> print open('foo.js').read()
{"services": {"bar": {"target": "bar", "parameters": [{"default": "this is a string", "optional": false, "type": "string", "name": "string"}]}}, "envelope": "JSON", "target": null, "transport": "REST", "additionalParameters": true}
>>>

Py2JSON is a work in progress, suggestions are welcome, patches are awesome :), right now it lacks of a lot of things; since we do not need them right now are not implementing, such as storing attributes.

Installing Git on ChunckHost

Matt Heitzenroder — Thu, 23 Jul 2009 17:56:31 +0000

Igor pointed me to ChunkHost’s free Beta recently, so I signed up.

First and foremost, I have no known affiliation with any related to ChunkHost.

Secondly, my first impression of ChunckHost was great – the signup and automation was completely fluid. It was very easy to signup and receive root to a freshly minted Ubuntu 8.04. All I had to do was put in my account information and then click sign-up and root was emailed to me within minutes. +1 ChunkHost.

I decided to write this blog partly about them. One reason is because of the swift and smooth customer experience. The other reason is because my steps to install git might be different than yours because of the vendors Virtual machine.

The first thing I wanted to do as I christen the server was to install git from source. These are the steps I did to complete that task:

Getting Git

1	root@roder:~# apt-get update

1
2
3
4
5
6
7
8
9

root@roder:~# apt-get upgrade

root@roder:~# apt-get install build-essential

root@roder:~# apt-get install wget

root@roder:~# wget http://kernel.org/pub/software/scm/git/git-1.6.3.3.tar.bz2

root@roder:~# tar -xjf git-1.6.3.3.tar.bz2

1	root@roder:~# cd git-1.6.3.3/

Now I have the latest source it’s time to compile it.

Make

1	root@roder:~/git-1.6.3.3# ./configure

1	root@roder:~/git-1.6.3.3# ./make

At this point I had an issue where I received this error:

1	'Z_DEFAULT_COMPRESSION' undeclared here (not in a function)

I realized after reading a forum thread I figured I needed to do this:

1	root@roder:~/git-1.6.3.3# apt-get build-deps git-core

For good measure, I built openssl after reading through of that thread.

1	root@roder:~/git-1.6.3.3# apt-get install openssl

Then I was able to make

1	root@roder:~/git-1.6.3.3# make && make install

Success. If you have questions or tips, please post them. I hope this helps!

Installing ZeroMQ v1.0.0

Matt Heitzenroder — Wed, 15 Jul 2009 07:18:47 +0000

ZeroMQ released 1.0.0 yesterday (congrats!) and I decided to take it for a spin. I needed to create a buildslave anyway and I decided to capture my steps as I installed ZeroMQ. The following steps were completed on an Ubuntu Hardy and Python 2.6.2

Simple Install

I am not sure if it’s necessary or not for my application, but I wanted to include support for both AMQP and Python.

1
2
3

root@roder:~/src/zmq-1.0.0 # ./configure --with-python --with-amqp
root@roder:~/src/zmq-1.0.0 # make
root@roder:~/src/zmq-1.0.0 # make install

After that I was done… time to fire it up

1	root@roder:~/src/zmq-1.0.0 # zmq_server

But it returned this error

1 2	zmq_server: error while loading shared libraries: libzmq.so.1: cannot open shared object file: No such file or directory

Oh yeah… need to update the libraries. Let me try again:

1 2	root@roder:~/src/zmq-1.0.0 # ldconfig -v root@roder:~/src/zmq-1.0.0 # zmq_server

Voila! Now to play with it.

…to be continued.

How-To for github_buildbot.py

Matt Heitzenroder — Tue, 14 Jul 2009 06:00:17 +0000

GitHub & Buildbot - together at last

github_buildbot.py is a small web server that responds to JSON that is POST’ed to it from GitHub.com and initiates a build on Buildbot instance. You may find the latest source code at GitHub.

Getting Started

In short:

start github_buildbot.py
add your github_buildbot server url to a post-receive service hook in GitHub (ie: http://example.com:4000)
configure your buildmaster’s master.cfg to point to /tmp/your_project_name.git
commit and enjoy

How it works

Creates a webserver that listens to port 4000
Accepts JSON POSTs from GitHub
Clones a bare repository locally in a specified directory or the system temp directory by default
Submits changesets to Buildmaster to build

How it works: verbose

Start it up

To start github_buildbot.py, locate the source of Buildbot on your buildmaster and change directory to the ./contrib. Ensure that github_buildbot is executable and then start it by:

1	./github_buildbot.py

By default, github_buildbot.py starts a webserver on port 4000. This can altered by doing something like:

1	./github_buildbot.py -p 8888

1	./github_buildbot.py --port=8888

The local clone

The Buildbot buildmaster requires that the source directory is locally accessible when you configure the master.cfg in the buildmaster.

from buildbot.steps.source import Git

1
2
3

from buildbot.steps.source import Git
f1 = factory.BuildFactory()
f1.addStep(Git(repourl='/tmp/apparatus.git'))

The default for github_buildbot.py is to create a “mirror” (which in essence, is the same as “bare”) repository in the operating system’s temporary directory. For example, the /tmp directory in POSIX systems. Putting source code in the /tmp directory isn’t necessarily good for everyone, so further down I will discuss how to specify the directory. When github_buildbot.py clones the github repository, it will name the local mirror after the name of the project and append ‘.git’ to it. In our case, the resulting local mirror is ‘apparatus.git’. Therefore, when configuring the Buildbot, specify the directory (by default the OS temp dir) and project name appended with a ‘.git’

Private or Public?

If you are using a private repository, it is critical that you start github_buildbot.py with a user whose ssh identity matches a contributor on the GitHub repository.

Something different

In order to specify another build directory other than the operating system’s temporary directory (ie: /tmp), you may use the following flag:

1	./github_buildbot.py -d /path/to/dir

1	./github_buildbot.py --dir=/path/to/dir

If, for some reason, that the buildmaster is not localhost:9989, you may specify the buildmaster like this:

1	./github_buildbot.py -m HOST:PORT

1	./github_buildbot.py --buildmaster=host:port

It is also possible, that you have an alias for github.com in your .ssh/config directory – especially if you have multiple accounts on one machine. If you need to specify another “location” other than github.com, you can do so like this:

1	./github_buildbot.py -g hostname

1	./github_buildbot.py --github=hostname

Configuring GitHub

GitHub has made this dead simple with Service Hooks. Add the URL to a post-receive service hook, including the port, to the service hooks section in GitHub.

Service Hooks

Post Receive URLs

Inputing the Post-Receive URLs

Onceyou have the github_buildbot.py web server running and the service hook enabled in GitHub, you may want to test using “Test Hook” in GitHub. This will send the last few commits to the specified URL. You can then debug as necessary if there are any issues.

Help Me

If you still need help, probably best to join the #buildbot IRC channel on irc.freenode.net .

Otherwise, you can do this:

1	./github_buildbot.py --help

1	./github_buildbot.py -h

--help of github_buildbot.py

However, if you’re feeling lucky, try turning on logging:

1	./github_buildbot.py -L debug

1	./github_buildbot.py --level=debug

You can find all log entries in your operating system’s temporary directory with the filename “github_buildbot.log”. However, if you’d like to configure that, you can do that too!

1	./github_buildbot.py -l /path/to/new_log_file.log

1	./github_buildbot.py --log=/path/to/new_log_file.log

GitHub & Buildbot Redux

Matt Heitzenroder — Tue, 07 Jul 2009 11:32:58 +0000

The first time I posted about this was after I hastily assembled a github/buildbot integration.

--help of github_buildbot.py

I knew then that I wanted to take a few more passes to clean it up. I did that with the help of Dan over the weekend.

Now it’s much better. If you’re interested in the changes, you can see a dramatic difference by viewing the diff.

However, the easiest way to see the newly refactored code is by looking at ./contrib/github_buildbot.py.

One of the best new features about the new code is it’s ability to control it from command line parameters.

It’s also sporting a nice amount of refactored code, which means we deleting a lot of overly complicated code. Two things made that possible, Dan and Github’s service hook in JSON.

When the local directory is created, it is no longer a working directory with a branch checked out. Now the buildmaster’s local repository is just a bare clone (thanks Igor). I added documentation and cleaned things up with pylint (which I’ll be blogging about my experiences with that shortly) to be more inline with PEP 8.

Finally and most importantly, Dan and I fixed a “changes” bug that would repeat the commits from the JSON n+1 more times.

The importance of unit testing and functional testing

Dan Reverri — Wed, 01 Jul 2009 07:00:21 +0000

I recently read a blog article, Benefits of automated functional testing, which included a section arguing that unit testing was not as important as functional testing. I agree that functional testing is important (and the author provides several good arguments for it) but I am concerned about the negativity against unit testing. The author makes the following argument in favor of functional testing which I think is a great argument in favor of all testing:

The complexity that does arise in this kind of software is all down to interactions between components, messy and changing business requirements and how the whole blob integrates at runtime.

I think the “interactions” issue is one that can be addressed by unit tests. The interaction between components boils down to one component making a request of another component; for example calling a method with an agreed upon set of arguments. The requesting component will have an expectation for the return of the method. If the called method does not meet the expectations of the request then a failure will occur. Unit tests can be used to verify that a method returns in an expected way.

With unit tests in place developers can confidently re-factor their code knowing that any violated expectations will be revealed when running those tests. It would be nice to account for every possible scenarios and test for every fringe case but that is unrealistic. Unit tests can be used to make an agreement with other developers saying “this code will do A when given B at all times”; developers are free to go outside the bounds of these expectations but do so knowing that their use of that unit of code is untested.

I’ve been sitting on a draft of a post regarding the importance of both functional and unit testing. I’ve included it below:

I’ve been thinking about a way to explain the importance of both functional and unit testing. I’ve been trying to come up with an appropriate analogy and have settled on the following:

Let’s imagine a particular software application is a physical structure such as a bridge. The materials used in the construction of the bridge are analogous to the units of a software application. The shape of the bridge is analogous to the functional operations of a software application.

When determining the strength of a bridge, the shape of the bridge plays a significant role, however, a determination of strength is not complete without knowing the material being used to construct the bridge.

Unit tests measure the quality of material used to build the application.

Functional tests measure the strength of the application structure.

The overall quality of the application is dependent on both.