My goal for the coming months is to master Open Solaris. What I want to learn is a good mix of system administration and application development, debugging and performance tuning tricks.

I actually started using Solaris 2.x many years ago. That was the first release that came after SunOS 4.1.4. The last BSD-like OS that came out of Sun before they made the switch to SysV.

We ran Solaris on a Axil 320 Sparc clone, on which we actually hosted a complete ISP setup. From email to web to usenet. Those where the days. Soon however when FreeBSD and Linux became more popular halfway the 90s, we stopped using Solaris and I haven’t really looked back at it.

But now in 2008, Solaris is back in a big way. Solaris 10 is a free download and runs on Intel or AMD hardware and Open Solaris is even completely open source. Very nice change of plans and a lot of incentive to take a look at it.

To get started I found two promising books on Safari, my digital bookshelf.

The first, Solaris Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris , is more oriented towards software developers and system administrators who are interested in analyzing, debugging and optimizing Solaris and the software running on it.

The second, Solaris 10 System Administration Exam Prep, is a pretty in-depth book about anything related to system administration. From disks to network interfaces, from user accounts to Zones.

I don’t have a real and dedicated Solaris box running yet. Instead I now simply run the latest Solaris Express Developer Edition on my MacBook Pro under VMWare Fusion. Works great and is much easier to restore when you accidentally destroy it.

More later. Under the Solaris category.

We are continuing to support our existing SOAP APIs. That said, given that SOAP requests currently make up < 1% of our request volume in the US, we made the pragmatic decision to not extend the SOAP APIs as part of our EU launch. This allowed us to make our EU location available sooner than we could have otherwise.

It does not specifically say whether this is just for S3 but I think it is a good indication of REST popularity.

All my AWS code is also using the REST API’s. Much simpler and you don’t have to use bloated SOAP frameworks to get things done.

For the 2.0 branch of the Polar Rose Web Service Foundation I have added an AnnotationWebServiceController that uses this new Spring 2.5 functionality.

This makes creating a web service very very simple. All you have to do to get going with a public web service is annotate your Actions with @WebServiceAction and tell Spring to scan for component.

This is an example action:

package com.foo.action.
@WebServiceAction(name = "Hello", version = "2008-04-01")
public class HelloAction implements WebServiceActionHandler
{
    public Object execute(WebServiceActionContext context, Parameters parameters)
        throws ActionHandlerException
    {
        return “Hello, ” + parameters.getName();
    }

    public static class Parameters
    {
        private String name;

        public String getName() {
            return name;
        }

        @WebServiceParameter
        public void setName(String name) {
            this.name = name;
        }
    }
}

The following Spring configuration is enough to setup the web service controller:

<beans ...>
  <context:component-scan base-package="com.foo.action"/>
  <bean id="annotationWebServiceController" class="com.polarrose.wsf.controller.AnnotationWebServiceController">
    <property name="version" value="2008-04-01"/>
  </bean>
</beans>

I think this is a really nice way to auto-detect and configure ‘top level’ components. It removes a lot of XML configuration. Which I don’t mind, but less is always better.

If you want to play with the framework, you can check it out from Google Code.

Some things on the list:

• Merge the wsf and wsf-examples into one maven project as modules (already done)
• Merge the projects on Google code
• Add the wsf-client module to the project
• Depend on Spring 2.5 for auto detection of action handlers. Just like Spring 2.5 can do with @Controller annotated Spring MVC controllers.
• Remove as many dependencies as possible (commons-*)
• Remove the need for prototype scope beans by seperating handler and parameters (work in progress)
• Write proper JavaDoc for at least all top-level code

I’ve already made a rough implementation of the seperation of Action and Parameters task. The ActionHandler interface now looks like this:

public interface WebServiceActionHandler<Account extends WebServiceAccount, Parameters>
{
   Object execute(WebServiceActionContext<Account> context, Parameters parameters)
       throws ActionHandlerException;
}

The Parameters parameter is new. So that means you need to define a bean that contains the fields to which the request parameters are mapped. For example the AddNumbersAction action from the example project now becomes:

public class AddNumbersAction implements WebServiceActionHandler<DummyWebServiceAccount, AddNumbersAction.Parameters>
{
   public Object execute(WebServiceActionContext<DummyWebServiceAccount> context, Parameters parameters)
       throws ActionHandlerException
   {
       int total = 0;
       for (Integer number : parameters.getNumbers()) {
           total += number;
       }
       return total;
   }

   public static class Parameters
   {
       private List<Integer> numbers;

       public List<Integer> getNumbers() {
           return numbers;
       }

       @WebServiceParameter
       public void setNumbers(List<Integer> numbers) {
           this.numbers = numbers;
       }
   }
}

I like to use inner classes to keep the action and parameters together but if you don’t like that then you can also do this of course:

public static class AddNumbersParameters
{
   private List<Integer> numbers;

   public List<Integer> getNumbers() {
       return numbers;
   }

   @WebServiceParameter
   public void setNumbers(List<Integer> numbers) {
       this.numbers = numbers;
   }
}

public class AddNumbersAction implements WebServiceActionHandler<DummyWebServiceAccount, AddNumbersParameters>
{
   public Object execute(WebServiceActionContext<DummyWebServiceAccount> context, AddNumbersParameters parameters)
       throws ActionHandlerException
   {
       int total = 0;
       for (Integer number : parameters.getNumbers()) {
           total += number;
       }
       return total;
   }
}

Personally I think this is a little cleaner design than what I did for the 1.x version.

The speed also went up!

• 1.2-SNAPSHOT ~ 1200 requests/second
• 2.0-SNAPSHOT ~ 2100 requests/second

This is because the prototype beans are gone of course; they don’t have to be initialized every time. I can probably get the speed up more by caching some things that are looked up every time with introspection.

Spring 2.5 has a lot of interesting new features. I really hope to make WSF 2.0 much more Spring friendly and make it possible to get a web service up and running with as less code and configuration as possible.

% dd if=/dev/zero of=data bs=1 count=2679
% sha256sum data
a25f4ccc56ddf88a4fb3e11baec5838c5181a496f376cdd569f8fc782f8fdcdf  data

A file of 2679 zeros and its SHA-256 hash. Nothing special.

Well, not exactly .. now look at the equivalent Python code:

$ python
Python 2.4.3 (#2, Oct  6 2006, 07:49:22)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Crypto.Hash import SHA256
>>> data = open("data", "rb").read()
>>> print SHA256.new(data).hexdigest()
14a59554dba8be8ac9aa03eea67a026b3773eb674d22719123be41ca774319a3

Oops!

So it seems the Python Crypto module generates wrong SHA-256 hashes for files that have a size of (N*8)-1. But only for files of reasonable size.

We discovered this the hard way because we cache images based on their SHA256 hash. All cache entries with wrong hashes had this (N*8)-1 file size.

I’m not sure if this is just limited to the Ubuntu Dapper Drake / X64 install that we use. For the record, this is the package I’m talking about:

$ aptitude show python2.4-crypto
Package: python2.4-crypto
State: installed
Automatically installed: yes
Version: 2.0.1+dfsg1-1ubuntu1
Priority: optional
Section: python
Maintainer: Andreas Rottmann 
Uncompressed Size: 557k
Depends: libc6 (>= 2.3.4-1), libgmp3c2, python2.4
Description: cryptographic algorithms and protocols for Python
 A collection of cryptographic algorithms and protocols, implemented for use from Python.
  Among the contents of the package: 

    * Hash functions: MD2, MD4.
    * Block encryption algorithms: AES, ARC2, Blowfish, CAST, DES, Triple-DES.
    * Stream encryption algorithms: ARC4, simple XOR.
    * Public-key algorithms: RSA, DSA, ElGamal, qNEW.
    * Protocols: All-or-nothing transforms, chaffing/winnowing.
    * Miscellaneous: RFC1751 module for converting 128-key keys into a
      set of English words, primality testing.

This sure taught us a lesson. Never trust (crypto) code until you have unit tested it yourself.

I’m trying to figure out where to report this.

I’ve written these for Polar Rose but as usual they are happy to release things like this under an open source license.

What the package contains is two things:

• Command line utilities to control SQS and SDB. I’ve found it extremely useful to be able to do things like creating domains and running queries from the command line during development or API exploration.
• Python modules to use SimpleDB, SQS and S3 in a Twisted environment. Twisted is a really nice Python framework to do asynchronous IO.

Good starting points on the wiki are:

• GettingStarted
• XmlRpcAndSimpleDatabaseExample
• SimpleDatabaseService
• SimpleDatabaseServiceCommandLineTools

The SimpleDB code is pretty stable and I encourage others to play with it. The SQS and S3 code needs some love. I have to check if it is still up to date with the current web service versions. I’ll do that soon and I will also try to make a really official release.

Enjoy!

If one does a good job at finding the primitives, then one can build a very broad system that gives one not just the functionality one was first thinking about, but lots more too.

This is very much what we are doing at Polar Rose now. Except our primitives are internal web services that operate on names, faces and images. Combining these web services for experiments and new ideas is now possible in just a few lines of Python.

You can enable this mode automatically by doing something like:

(add-hook 'c-mode-common-hook '(lambda () (flyspell-prog-mode)))

Note for OS X users. It uses ispell, which does not come with OS X. Fortunately that is easily solved by using the MacPorts.