I was looking around for ways to keep Nose from searching for tests in certain directories. You see Nose is a nosey little critter that will scour every nook and cranny of your directory structure looking for unittests to run. Really, it wants your code to work and it loves making dots. But, see I knew better and knew if that little Nose test discoverer ventured down a few rabbit holes it would end up in a world of pain–segfaults or worse. I just wanted to avoid the whole mess and just have Nose exclude a few directories from its massive testhunt.

I asked a few people I know who use Nose regularly about my options. “Write your own plugin,” was the consensus. Nose’s architecture makes it very easy to write plugins for just about every aspect of its behavior…check out its plugin api. The project documentation is also very helpful in this regard.

I spent most of my time trying to figure out how to test my testing plugin. In the end it only took a couple of hours to go from Nose novice to having written a packaged, testable Nose plugin. So, here’s a quick example of how you would use this new nose-exclude plugin:

 $ ls test_dir
dir_with_tests    dir_with_bad_tests
there_be_dragons_here    more_tests

In this example, I want Nose to ignore a couple directories and not even bother searching them. You would run:

 $ nosetests --exclude-dir=test_dir/dir_with_bad_tests \
--exclude-dir=test_dir/there_be_dragons_here test_dir

You could further specify to exclude subdirectories if you wanted that level of control. There is also an –exclude-dir-file= option available that allows you to specify a file containing paths to be excluded.

$ nosetests --exclude-dir-file=exclude_dirs.txt test_dir

nose-exclude is pretty simple and covers a fairly basic use case so hopefully others will find it useful. For now nose-exclude is available on bitbucket, but will be up on PyPI shortly.

Bookmarks:

Almost two years ago I was in a rather unlikely situation in that I was running a software engineering department containing both a C# team and a Python team. The Python group was focused on building scientific computing and NLP-type applications, whereas the C# team was focused on building web applications.

A few of us Python folks in the department had already started playing around with Django–building internal web applications and projects outside of work. It did not take long for us to realize the power of Django and how quickly we were able to produce high-quality applications with little effort. This was my (strong) impression, but in order to propose a corporate platform shift I was going to need some data to support my claims.

It slowly dawned on me that I had a perfect test bed. Here we had two teams using different technology stacks within the same department. The same department. That means they shared the same development processes, project management tools, quality control measures, defect management processes. Everything was the same between these groups except for the technologies. Perfect! So like any good manager I turned my teams into unwitting guinea pigs.

The Hypothesis

We can accomplish more with Python + Django than with C# + ASP.NET given the same amount of time without sacrificing quality

Measuring Productivity

For the sake of this study, I defined productivity as a normalized team velocity: how many story points were completed / developer / week. I record the normalized team velocity for each team’s sprint for later analysis.

For those of you unfamiliar with the concept story points I highly recommend Mike Cohn’s Agile Estimation and Planning.

WAIT! You can’t compare story points between teams!

I hear this a lot. Yes, you can. The problem is that most people do not bother creating a common scale or continually calibrate their estimations (within or between groups). Generally, it’s way more work than most groups need to deal with and it doesn’t deliver much utility to most groups so it isn’t often discussed or practiced.

The methods described below should outline the additional calibration work that was performed to ensure a common estimation scale between the two teams.

Methods

Both teams continued business as usual working on projects in parallel. Each sprint consisted of 3-4 developers. It is worth noting that Team ASP.NET did not make use of MS MVC Framework, but they did use Linq-to-SQL for its ORMy powers.

Special care was taken to maintain linkage between the two team’s effort estimates. During sprint planning, each team would use a common story point calibration reference when making estimates. In order to detect any potential deviations in calibration, during several planning poker sessions I included stories that had already been estimated during previous sprints or by the other team; no significant deviations were found.

At the end of each sprint I would calculate the normalized developer velocity ( # of completed story points / developer / week ). These values were recorded for both teams. It should be noted that only Django-based sprints were used in analysis for Team Python.

I recorded results for approximately 6 months.

Results

Normalized Developer Velocities: C# + ASP.NET and Python + Django

The above histogram shows the distribution of normalized velocities associated with each completed sprint. The table below summarizes the distribution of velocities associated each team.

The distribution of velocities between the two samples are similarly shaped, but have clear differences in their mean. The average velocity of a  C#/ASP.NET developer was found to be 5.8 story points/week. A Python/Django developer has an average velocity of 11.6 story points/week. Independent t-tests reveal these differences as being statistically significant (t(15) = 4.19, p<7.8e-4).

Discussions and Conclusion

Given our development processes we found the average productivity of a single Django developer to be equivalent to the output generated by two C# ASP.NET developers. Given equal-sized teams, Django allowed our developers to be twice as productive as our ASP.NET team.

I suspect these results may actually reflect a lower bound of the productivity differences. It should be noted that about half of the Team Python developers, while fluent in Python, had not used Django before. They quickly learned Django, but it is possible this fluency disparity may have caused an unintended bias in results–handicapping overall Django velocity.

Epilogue

The productivity differences quantified by our findings were then included as part of an overall rationale to shift web-based development platforms. Along with overall velocity differences, the costs associated with maintaining each environment were considered: OS licensing and database licensing for development and production environments, as well as costs associated with development tools. I’m happy to say we are now a Python and Django shop.

Updated:

Several good questions over at Hacker News

Bookmarks:

I presented Fabric as a tool that I use for automated deployment and maintenance of servers. I can publish and update multiple servers with differential instructions based on their defined roles (e.g. Apache web server, MySQL database server, Nginx server, etc.). We briefly touched on some other cool things you can do like database maintenance, database dumps, backups, Django contribs for remotely working with Model objects, and even toggling maintenance modes for your website.

The next part of the presentation dealt with another use I’ve found for Fabric: bootstrapping a server. The ease of automating distributed systems made Fabric an ideal tool for lightweight configuration of a newly installed OS. Go ahead and have it apt-get/yum/emerge whatever packages are going to be required–again as specified by its defined role. For instance, I have a few fabfiles that will prep an Ubuntu or a RHEL servers for serving Django apps with Nginx, Apache+mod_wsgi, and MySQL.

So, the most common questions and comments I’ve received have been along the lines of “Why not just build packages and meta packages for each server configuration?” This is absolutely an approach that one could take and a powerful one at that. I’ll be the first to admit I am not a packaging whiz so the thought of building a slew of .debs and then maybe parallel .rpms if we have to migrate to a new platform based on the client or project is not super appealing, but luckily I work with people smarter than myself who enjoy that sort of thing.

The sweet spot I’ve found for bootstrapping servers with Fabric is where your server configuration may not be final, the project maybe short term, you may still have to switch between distros, and you do not have access to VMs that can be cloned.  Fabric is lightweight and available for your more ephemeral projects that may not necessitate the effort required to build custom packages.

For instance, one place Fabric is handy is prepping Amazon EC2 servers for quick compute jobs. Now I could go ahead and create custom AMIs for every one-off project, but Fabric makes it easy to spin a server up and then bootstrap it with the required packages for a particular project. Nice and lightweight.

I would love to hear about other tools people are using out there. It was nice that after my presentation a bunch of people shared their tools and methods for bootstrapping servers. Og Maciel and other rPathers demonstrated rPath’s rBuilder Online and how it can be used to easily build packages and entire system images or VMs based on sets of packages and a few clicks. Tobias McNulty of the Caktus Consulting Group also demonstrated his work on Mallet, which can be used to quickly build and configure Debian-based servers. Mallet is also available through PyPI.

Overall a great turn out and I learned a lot. Thanks folks!

Bookmarks:

Thanks again to the Caktus Consulting Group, Carrboro Creative Coworking, MetaMetrics, Capstrat, Karen, and everyone who showed up to make the event happen.

I managed to be perfectly eclipsed in this picture…

Bookmarks:

We are organizing a Django sprint here in the Triangle on December 12th and 13th 2009 (Saturday & Sunday). The sprint is being hosted by Caktus Consulting Group over at Carrboro Creative Coworking.

The goal is to get folks familiar with the Django development process and knock out some features and tickets associated with the upcoming release of Django 1.2.

You can find more details on the Sprint Wiki: http://code.djangoproject.com/wiki/Sprint200912TriangleNC

Sign Up Here…

Sponsors

Thanks to Caktus Consulting Group, Carrboro Creative Coworking, and MetaMetrics for helping sponsor this event.

Hope to see you there!

Bookmarks:

We were anxiously looking forward to Alex Gaynor’s multidb efforts, but needed something in the near term to help our site scale. After looking at a few options, including building our own, we settled on Mike Malone’s django-multidb. I first heard about Mike’s django-multidb in his Scaling Django Presentation. It was a perfect solution for our needs: it gave us the ability to manage master-slave databases within Django, it was very simple, and offered just the right amount of flexibility.

The Problem

We started receiving sporadic UnicodeEncodeErrors. The tracebacks were reporting that the system was unable to encode certain Unicode strings into latin1. Latin1? Who wants latin1? We use utf8 as our standard character set for both Django and MySQL (client & server).

So where was this latin1 encoding request sneaking in from? No rogue .encode(’latin1′)s were popping up in codebase searches. The Django MySQL backend certainly looked like it was doing its job, but we had to validate that the cursors being generated were in fact being set with the appropriate charset. We followed the path back and eventually started intercepting a few MySQLdb cursors. Once we started debugging and probing cursors it became clear that they were in fact using latin1 as the default character set.

You keep using that word. I do not think it means what you think it means.

I learned an interesting tidbit along the way that explains why it took us awhile to diagnose the problem. When you don’t specify a character set or encoding, MySQL’s default encoding is called “latin1″. Except by “latin1″ MySQL does not mean “latin1″ of ISO 8859-1 fame, but rather the Windows cp1252 code page. This occurs even though MySQL does know what cp1252 is and is fully capable of honoring that character set separately by name. Really.

So, imagine MySQLdb asking the MySQL server what charset it prefers and the server replies “latin1″. MySQLdb then says, “Awesome, I know latin1,” and they go on chatting and passing information. This works just fine until the MySQL server passes back a bytestring representing a string once stored in its database containing the Unicode entity U+2019 ( ’ ). This RIGHT SINGLE QUOTATION MARK can easily be encoded in cp1252, but it cannot be represented by latin1 (The real ISO 8859-1 one). MySQLdb receives the cp1252-encoded bytestring and attempts to decode as if it were latin1 and lo and behold it throws an exception for attempting the impossible.

Now imagine the encoding mess occurring within a SQL query. MySQLdb opens a connection with the server, agrees to communicate using the latin1 charset, and then prepares to send a query containing a right single smart quote (U+2019). It takes the Unicode query string and attempts to .encode(’latin1′).  BLAM! Encoding error.

The Solution

The problem ended up being an inconspicuous bug in django-multidb that restored Django’s backend cursor settings to system defaults, which resulted in any preferred character sets being ignored. The latin1-cp1252 confusion was then free to crop up.  I know many folks including myself looked right over the code and bug many times without noticing. No worries. Mike Malone has already patched the django-multidb repository over at github. So, at this point you just need to update your project with the latest django-multidb code and you should be good to go.

Regarding the MySQL-MySQLdb latin1 debacle, it seems simple enough to make MySQLdb call MySQL’s bluff. Maybe another time…until then I think I will avoid this issue and stick with explicitly defining utf8 as my character set of choice.

Bookmarks:

Django’s growing popularity

First off, it’s important to note that Django adoption is growing nationwide. The included chart shows the number of posted “Django” jobs found on Indeed.com over the past few years. Notice a trend? Jacob Kaplan-Moss estimates the Django community may have grown somewhere on the order of 2-3x from 2007 to 2009. I definitely believe it and wouldn’t be surprised if it were higher. Between the volume of phone calls from recruiters and the number of people I run into using or talking about Django its popularity is definitely on the rise in the Triangle.

Django in Action

Here’s a short list of shops in the Triangle who use Django in their day-to-day development:

• MetaMetrics ( Lexile )
• Caktus Consulting Group
• Capstrat
• American Research Institute
• Semiconductor Research Corporation
• Viget Labs
• rPath

I know there are other closeted folks out there using Django without full corporate blessing or knowledge. If there are other groups out there who would like to make this list please let me know.

A Triangle Django Users Group?

A new Google Group TriDjUG (twitter: @TriDjUG) was recently created in order to help foster a healthy Django community. At the same time some good discussion erupted from the TriZPUG mailing list. Why bother splitting our local Python community? While the intention was never to split away, some good cases were made for operating under the umbrella of TriZPUG. Strengthen the Python community. One exciting sentiment that came from TriZPUG members was that non-Django Python users were interested in Django and wanted to learn more during regular TriZPUG meetings. That would give us a captive audience at an already catered and organized event. Sounds good.

Surely there is some Django-specific fun to be had… For starters, we’re looking to sponsor local Django-related sprints maybe including one for the upcoming Django 1.2 release. For our community building/growing merit badges a couple of us are developing a Django Bootcamp; let’s continue to grow.

So it sounds like it’s up to us local Djangonauts to step up, participate in TriZPUG, and build greater awareness. People want to hear about our technologies, so let’s share. If you have other ideas we’ld love to hear them. If you haven’t yet introduced yourself swing by the mailing list or irc (#trizpug) and say hello.

Bookmarks:

Your Trusty SVD Tool

SVD is one of those tools that every machine learning practitioner and computational geek will pull out at some time or another. It’s a powerful matrix factorization technique that allows you to get at the matrix’s eigenvectors and eigenvalues. One of reason it tends to be used so often is the fact that it can be used on those pesky M x N matrices, which us data junkies tend to generate.

For most small problems I can just use scipy and numpy’s svd and never give it a second thought. LAPACK’s suite of SVD routines power the svd functions of scipy, numpy, and MATLAB among others. It is developed for dense matrices and processes them in their entirety. What happens when you start dealing with problems in high-dimensional space? Those dense representations and full processing are expensive. So, when your problem space is better suited for sparse matrices you tend to run into not enough memory, non-convergence…no SVD.

At the time I was considering a problem that would be well-suited for incremental training, meaning I did not want to have to rerun the entire dataset through SVD after adding a small set of new data; GHA lets you avoid that sort of inconvenience and approximates the same outcome (as far as my problem was concerned).

GHAPACK

GHAPACK is written by Genevieve Gorrell and based on her work using GHAs to perform Latent Semantic Analysis (LSA).

“Offline” Calculations

My first order of business upon joining the project was to get the offline training working. This allows you to compute a pseudo-SVD based on a massive matrix without having to load the whole thing into memory.  No more out-of-memory segfaults. Now, you’re just limited by the resiliency of your hardware. This is now working.

Memory Management

I addressed a few memory leaks, but will likely do some restructuring to optimize memory management.

Resource Library

I would like the core of the GHA magic to be extracted into a library that others could embed in their own projects. So, I intend to move core functionality into a library and restructure the existing apps into commandline tools that utilize those libs.

Performance

GHA, off-the-bat, is not known for its speed compared to some other eigen decomposition approaches. Besides that, there is room for some major gains in performance. Let’s see what we can squeeze out of GHAPACK and perhaps lean on things like BLAS.

Testing Framework

A testing framework that objectively keeps track of performance gains, while ensuring computational integrity through unit testing always makes refactoring work that much less stressful.

Lots of work to do and a hearty thanks to Professor Gorrell for letting me join her efforts.

Other SVD Resources

There are other SVD libraries out there that will carry you farther if SVD is what you really want and not necessarily the means to an end.

ScaLAPACK has parallel SVD code, which creams LAPACK’s performance when you have access to multiple cores and/or MPI. ARPACK and SVDPACK both offer Lanczos-based SVD solutions for sparse matrices with ARPACK being well-suited for parallel processing.

Bookmarks:

Ken Schwaber replied to Jason’s article:

When Scrum is mis-used, the results are iterative, incrementa​l death marches. This is simply waterfall done more often without any of the risk mitigation techniques or people and engineerin​g practices needed to make Scrum work.

The article also sparked Bob Martin to write an essay “Dirty Rotten ScrumDrels” in response to some of the Scrum scapegoating that has been going on recently. Check out the comments for some Uncle Bob-Shore dialog.

Things like self-reflection and process and work reviews are all practices that people naturally adopt to improve themselves. I think successful developers tend to do this anyways in order to stay ahead of the curve. When you have mediocre teams flailing about who are looking for a silver bullet and turn to things like Scrum who don’t already do this sort of thing, it is easy for them to brush that off as a triviality; if they haven’t already seen the value of it, it just gets lost in the noise.

I share Mr. Shore’s frustration in seeing Agile and Scrum being blamed for the shortcomings of teams and improper implementations, when you’ve seen the real thing work over and over.

Bookmarks:

Protosprout ( http://protosprout.com ) is a NC-based company founded by Justis Peters that is looking to bring local entrepreneurs together and foster an energized, start-up community. I think it was a successful first meeting. Justis pulled folks together and lead some interesting and informal discussions including how the community could work together to provide support and value to one another. Ideas included things like skill and service bartering, community forums, general networking, official Protosprout mentorship and startup programs. A local skill bartering system would be great for a lot of startups ( ala programmer meets designer ) as would a tighter knit community for bouncing ideas off of or even testing your prototypes.

Part of the evening included time for pitching your startups and Q&A. Even Wayne Sutton got up and pitched an idea. I’ll take a moment to mention one really cool company that pitched: Durham TechShop ( http://www.durham.techshop.ws )

The TechShop will be a a DIY workshop with a slew of tools and machines for any number of tasks ( electronics, woodworking, plastics, machining, welding, 3d printing, even blacksmithing! ). They’ll offer classes so that you can get certified to use the various tools and then you’re off to create whatever you want. I am really excited to see this in action, so all you folks with checkbooks contact Scott Saxon over at the TechShop and get this puppy funded!

I met a lot of interesting people that night and look forward to attending the next Protosprout meeting. You should check it out over at http://protosprout.com . I think Justis is doing a great job getting this organized.

You can also follow Protosprout on Twitter  ( http://twitter.com/protosprout )

Bookmarks: