Stefan Arentz

O Canada, Eh!

stefan — Sun, 11 May 2008 15:48:19 +0000

Row, Row, Row Your Boat

stefan — Tue, 15 Apr 2008 17:47:18 +0000

Our stuff is already on it’s way to Toronto! By boat. Hope my books don’t get seasick.

Learning Solaris

stefan — Mon, 14 Apr 2008 20:34:38 +0000

I’ve decided to spend my time on more useful things than learning historic programming languages like Lisp. (Although that was a lot of fun and it definitely changed my thinking and coding style in a positive way!)

My goal for the coming months is to master Open Solaris. What I want to learn is a good mix of system administration and application development, debugging and performance tuning tricks.

I actually started using Solaris 2.x many years ago. That was the first release that came after SunOS 4.1.4. The last BSD-like OS that came out of Sun before they made the switch to SysV.

We ran Solaris on a Axil 320 Sparc clone, on which we actually hosted a complete ISP setup. From email to web to usenet. Those where the days. Soon however when FreeBSD and Linux became more popular halfway the 90s, we stopped using Solaris and I haven’t really looked back at it.

But now in 2008, Solaris is back in a big way. Solaris 10 is a free download and runs on Intel or AMD hardware and Open Solaris is even completely open source. Very nice change of plans and a lot of incentive to take a look at it.

To get started I found two promising books on Safari, my digital bookshelf.

The first, Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris , is more oriented towards software developers and system administrators who are interested in analyzing, debugging and optimizing Solaris and the software running on it.

The second, Solaris 10 System Administration Exam Prep, is a pretty in-depth book about anything related to system administration. From disks to network interfaces, from user accounts to Zones.

I don’t have a real and dedicated Solaris box running yet. Instead I now simply run the latest Solaris Express Developer Edition on my MacBook Pro under VMWare Fusion. Works great and is much easier to restore when you accidentally destroy it.

More later. Under the Solaris category.

SOAP vs REST at AWS

stefan — Thu, 10 Apr 2008 09:42:50 +0000

I found an interesting quote about SOAP API usage for Amazon’s Simple Storage Service:

We are continuing to support our existing SOAP APIs. That said, given that SOAP requests currently make up < 1% of our request volume in the US, we made the pragmatic decision to not extend the SOAP APIs as part of our EU launch. This allowed us to make our EU location available sooner than we could have otherwise.

It does not specifically say whether this is just for S3 but I think it is a good indication of REST popularity.

All my AWS code is also using the REST API’s. Much simpler and you don’t have to use bloated SOAP frameworks to get things done.

Using Spring 2.5 Auto Detection of Components

stefan — Wed, 09 Apr 2008 12:48:19 +0000

Spring 2.5 has some great improvements that make configuration much simpler. One that I very much like is the auto detection of components in the classpath. The Spring 2.5 documentation on this subject is worth reading. It also gives a nice and simple example.

For the 2.0 branch of the Polar Rose Web Service Foundation I have added an AnnotationWebServiceController that uses this new Spring 2.5 functionality.

This makes creating a web service very very simple. All you have to do to get going with a public web service is annotate your Actions with @WebServiceAction and tell Spring to scan for component.

This is an example action:

package com.foo.action.
@WebServiceAction(name = "Hello", version = "2008-04-01")
public class HelloAction implements WebServiceActionHandler
{
    public Object execute(WebServiceActionContext context, Parameters parameters)
        throws ActionHandlerException
    {
        return “Hello, ” + parameters.getName();
    }

    public static class Parameters
    {
        private String name;

        public String getName() {
            return name;
        }

        @WebServiceParameter
        public void setName(String name) {
            this.name = name;
        }
    }
}

The following Spring configuration is enough to setup the web service controller:

I think this is a really nice way to auto-detect and configure ‘top level’ components. It removes a lot of XML configuration. Which I don’t mind, but less is always better.

If you want to play with the framework, you can check it out from Google Code.

Started working on Web Service Foundation 2.0

stefan — Mon, 07 Apr 2008 04:26:43 +0000

I’ve created a 2.0 branch in the polarrose-wsf project where I want to try out some new ideas.

Some things on the list:

Merge the wsf and wsf-examples into one maven project as modules (already done)
Merge the projects on Google code
Add the wsf-client module to the project
Depend on Spring 2.5 for auto detection of action handlers. Just like Spring 2.5 can do with @Controller annotated Spring MVC controllers.
Remove as many dependencies as possible (commons-*)
Remove the need for prototype scope beans by seperating handler and parameters (work in progress)
Write proper JavaDoc for at least all top-level code

I’ve already made a rough implementation of the seperation of Action and Parameters task. The ActionHandler interface now looks like this:

public interface WebServiceActionHandler
{
   Object execute(WebServiceActionContext context, Parameters parameters)
       throws ActionHandlerException;
}

The Parameters parameter is new. So that means you need to define a bean that contains the fields to which the request parameters are mapped. For example the AddNumbersAction action from the example project now becomes:

public class AddNumbersAction implements WebServiceActionHandler
{
   public Object execute(WebServiceActionContext context, Parameters parameters)
       throws ActionHandlerException
   {
       int total = 0;
       for (Integer number : parameters.getNumbers()) {
           total += number;
       }
       return total;
   }

   public static class Parameters
   {
       private List numbers;

       public List getNumbers() {
           return numbers;
       }

       @WebServiceParameter
       public void setNumbers(List numbers) {
           this.numbers = numbers;
       }
   }
}

I like to use inner classes to keep the action and parameters together but if you don’t like that then you can also do this of course:

public static class AddNumbersParameters
{
   private List numbers;

   public List getNumbers() {
       return numbers;
   }

   @WebServiceParameter
   public void setNumbers(List numbers) {
       this.numbers = numbers;
   }
}

public class AddNumbersAction implements WebServiceActionHandler
{
   public Object execute(WebServiceActionContext context, AddNumbersParameters parameters)
       throws ActionHandlerException
   {
       int total = 0;
       for (Integer number : parameters.getNumbers()) {
           total += number;
       }
       return total;
   }
}

Personally I think this is a little cleaner design than what I did for the 1.x version.

The speed also went up!

1.2-SNAPSHOT ~ 1200 requests/second
2.0-SNAPSHOT ~ 2100 requests/second

This is because the prototype beans are gone of course; they don’t have to be initialized every time. I can probably get the speed up more by caching some things that are looked up every time with introspection.

Spring 2.5 has a lot of interesting new features. I really hope to make WSF 2.0 much more Spring friendly and make it possible to get a web service up and running with as less code and configuration as possible.

Huge flaw in Ubuntu Dapper’s Python Crypto Module

stefan — Thu, 03 Apr 2008 14:49:28 +0000

This is rather serious. Consider this:

% dd if=/dev/zero of=data bs=1 count=2679
% sha256sum data
a25f4ccc56ddf88a4fb3e11baec5838c5181a496f376cdd569f8fc782f8fdcdf  data

A file of 2679 zeros and its SHA-256 hash. Nothing special.

Well, not exactly .. now look at the equivalent Python code:

$ python
Python 2.4.3 (#2, Oct  6 2006, 07:49:22)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Crypto.Hash import SHA256
>>> data = open("data", "rb").read()
>>> print SHA256.new(data).hexdigest()
14a59554dba8be8ac9aa03eea67a026b3773eb674d22719123be41ca774319a3

Oops!

So it seems the Python Crypto module generates wrong SHA-256 hashes for files that have a size of (N*8)-1. But only for files of reasonable size.

We discovered this the hard way because we cache images based on their SHA256 hash. All cache entries with wrong hashes had this (N*8)-1 file size.

I’m not sure if this is just limited to the Ubuntu Dapper Drake / X64 install that we use. For the record, this is the package I’m talking about:

$ aptitude show python2.4-crypto
Package: python2.4-crypto
State: installed
Automatically installed: yes
Version: 2.0.1+dfsg1-1ubuntu1
Priority: optional
Section: python
Maintainer: Andreas Rottmann 
Uncompressed Size: 557k
Depends: libc6 (>= 2.3.4-1), libgmp3c2, python2.4
Description: cryptographic algorithms and protocols for Python
 A collection of cryptographic algorithms and protocols, implemented for use from Python.
  Among the contents of the package: 

    * Hash functions: MD2, MD4.
    * Block encryption algorithms: AES, ARC2, Blowfish, CAST, DES, Triple-DES.
    * Stream encryption algorithms: ARC4, simple XOR.
    * Public-key algorithms: RSA, DSA, ElGamal, qNEW.
    * Protocols: All-or-nothing transforms, chaffing/winnowing.
    * Miscellaneous: RFC1751 module for converting 128-key keys into a
      set of English words, primality testing.

This sure taught us a lesson. Never trust (crypto) code until you have unit tested it yourself.

I’m trying to figure out where to report this.

Twisted Amazon SimpleDB

stefan — Sat, 08 Mar 2008 15:52:09 +0000

Now that SimpleDB is out of it’s secret stealth beta I thought it was a good idea to make my Amazon Python modules available on Google Code.

I’ve written these for Polar Rose but as usual they are happy to release things like this under an open source license.

What the package contains is two things:

Command line utilities to control SQS and SDB. I’ve found it extremely useful to be able to do things like creating domains and running queries from the command line during development or API exploration.
Python modules to use SimpleDB, SQS and S3 in a Twisted environment. Twisted is a really nice Python framework to do asynchronous IO.

Good starting points on the wiki are:

The SimpleDB code is pretty stable and I encourage others to play with it. The SQS and S3 code needs some love. I have to check if it is still up to date with the current web service versions. I’ll do that soon and I will also try to make a really official release.

Enjoy!

Primitives

stefan — Thu, 14 Feb 2008 20:51:43 +0000

There is an excellent article on the Mathematic blog by Stephen Wolfram, who is still, after 21 years, the main architect of the Mathematica core.

If one does a good job at finding the primitives, then one can build a very broad system that gives one not just the functionality one was first thinking about, but lots more too.

This is very much what we are doing at Polar Rose now. Except our primitives are internal web services that operate on names, faces and images. Combining these web services for experiments and new ideas is now possible in just a few lines of Python.

Let Emacs spell-check your code

stefan — Sun, 10 Feb 2008 11:02:27 +0000

I just discovered flyspell-prog-mode. It works on comments and string constants in your code.

You can enable this mode automatically by doing something like:

(add-hook 'c-mode-common-hook '(lambda () (flyspell-prog-mode)))

Note for OS X users. It uses ispell, which does not come with OS X. Fortunately that is easily solved by using the MacPorts.