Service Oriented Architecture is primarily about business and not technology

This also reflected in the recent SOA Manifesto which states as its very first described value :

Business value over technical strategy

Allow me to straight away start picking some holes into this :

1. Anything that a business does – whether it is soa, software architecture, building architecture or simple plant and machinery design, to the extent (which is exactly 100%), technology serves the business goals, all technology activities (and non-technology as well) are at the end of the day about achieving business objectives and therefore about business. So why single out architecture? And even more so why single out SOA?
2. Architecture is also about business. But its not the same as saying its primarily about business and not so much about technology. For a moment lets step away from Software/Hardware Architecture and look at Building Construction Architecture. The legendary creation of Ayn Rand – Howard Roark, for all his eccentricities and seemingly portrayed egocentric and egotistic behaviour did meet the test of business objectives to the extent of making the residents of his creations extremely satisfied. And at no point would you gather the impression that he in any manner put construction technology to any secondary position to his business context and objectives. At the end of the day thats what architecture is. It is not about making one of business or technology more important than or subservient to other. Its about effectively mapping the two to provide a strong technology solution appropriate to the business needs.

I suspect one of the important causes here is that people have forgotten that the A in SOA stands for architecture, and therefore shorn of architecture, business and technology can be seen to be competing in a non win-win form.

So if SOA stands for Service Oriented Architecture, then I must submit that architecture is the art of getting the two working together. And I am of the opinion that an exercise suggesting one is more important than other is an exercise in a field unrelated to architecture. I suspect many practicing software architects will agree with this. I suspect Ayn Rand wouldn’t disagree as well.

Related posts:

1. Service oriented REST architecture is an oxymoron

Apologies for sending this note to you completely unannounced and out of the blue. However I find myself in a peculiar situation of having a very weird name being dumped upon me. While I am indifferent to the name per se, I am greatly pained as I realise that it is a completely inappropriate name. What is even more confounding is the very bunch of people who have happily assigned me the name and continue to popularise it belong to that class of people some of whom actually are extremely particular about accurate nomenclature and have no hesitation in creating a 100 letter class or function name by concatenating 20 words just to make sure the name is unambiguous and conveys the intent clearly.

Ahh.. but I digress and impose upon you without introducing myself adequately first. I am a data storage style. I am not new, but lately far too many a software engineer have started taking a liking for me. Ever since I have been around, I have with great amounts of jealousy watched my cousin the RDBMS being courted by the finest of engineers (in all honesty there were some fine engineers interested in me too, but far too few compared to my cousin). But lately multiple concurrent developments have made a fair amount of attention come my way too.

You see unlike RDBMS, I don’t require that data be clearly split into tables, columns and rows. I can work with data the way it is most naturally represented. As a tree of individual data fields, lists, arrays, dictionaries etc. Also I do not require that you always clearly define each and every possible schema element before being able to store data corresponding to the schema. I can happily accept a schema dynamically or even work without a schema. Some of my early forms were based on key value pairs stored as B-Trees (eg. Berkeley DB). Over the years people have figured out ways to represent the data as a set of decomposed document elements, store data spread across a cluster, replicate it for better availability and fault tolerance, and even perform post storage processing tasks using map-reduce sequences. But really what separates me from my cousin and other storage systems is that I don’t make demands on the data – I take it in its naturally found form and then store it, replicate it, slice it, dice it and glean information out of it. And therein lies my true identity – I will work with data the way the data is best represented with all its arbitrary inconsistencies and inabilities to always clearly specify a constraining schema. And the engineers who’ve spent time with me seem to have enjoyed it quite a bit.

But the horror of it – they gave me a completely inappropriate moniker – ‘NoSQL’. First and foremost I exist to promote a storage style and thats what identifies me. I work with data in its natural and arbitrary forms. Therefore to make it seem like I represent a lack of something else is utterly missing the point. The SQL in NoSQL stands for Structured Query Language, which depends upon Fixed Structure Relational Data. Since I change the very nature of the data being stored, that SQL is not required or relevant is automatic and inconsequential.

Its like calling a under-the-ocean-mountain_range as NoIgloo. Its dead obvious igloos will not be found there. But calling that mountain range NoIgloo is a big disservice to visitors. You use that as a marketing term, attract people, then tell them that NoIgloo actually has nothing to do with Igloos – its got to do with mountains and oceans, and that they need to first unwind all the confusion they created in their minds due to NoIgloo and then go through a phase of reunderstanding mountains and oceans. And while they came prepared for a possibly warmer place given the name NoIgloo – it actually is a wet place so they need to again change their garments and equipment for the journey. A wholely avoidable situation.

Update: Brad Anderson pointed out this interesting post NoSQL: A Modest Proposal which traces the genesis of my name which leaves me very very disappointed. Almost seems to suggest that people are flocking together and naming me not based on something inherently powerful about me – but as a mechanism to demonise my cousin RDBMS. This is most unfortunate, since we actually end up being useful in very different situations and more often than not are likely to complement each other rather than compete with each other. I do hope a better moniker does prevail over time

What I would like is to see a better / more appropriate name for me. Hmm .. call me free form storage, natural persistence or flexi schema storage or perhaps something else even more appropriate (this blog owner prefers “natural persistence”). Each of these conveys far more about me far more accurately than NoSQL does. Basically please please call me something better than NoSQL. So can I request you to carry forward my plea by further forwarding and retweeting this to your friends and ask them how they can so callously call me by such a silly name when they take the utmost precautions in properly naming their classes and methods. Plead with them to stop doing this and please work with others to give me a better name. I think it will cause less confusion over the coming months and years, and the field of software shall recover its glorious tradition of maintaining precision in communication by using accurate naming.

Sincerely,
The one who doesn’t want to be called NoSQL

PS : As a background to this there was an interesting conversation earlier today between this blog owner dnene and Kent Beck on twitter, where Kent so kindly and graciously helped carry forward the thought process of helping identify my essential characteristics, and it is in no small part, thanks to this conversation that I was able to articulate myself and my grief. I reproduce that conversation below. (Update: though in all likelihood Kent’s intent was to help clarify the thought rather than contest the names. In hindsight, it makes sense to ask for permission to reproduce conversations .. even when such are on the public twitter stream – something that wasn’t done in this case. )

Twitter ID	Tweet / Message
dnene	NoSQL is such an inappropriate name. NoTables at least makes a little more sense.
KentBeck	@dnene but what would nosql be called if you wanted to say something positive about it?
dnene	@KentBeck Thats a great question .. still thinking .. best thought so far – FlexiStore (though not good enough yet )
KentBeck	@dnene what can you do with a nosql store that you can’t do with an sql database? why would you be excited to use one?
dnene	@KentBeck I see where u r going with this (a) unconstrained & composite storage (b) store resources not records (c) shard/scale horizontally
	.@KentBeck I think there is a merit in attempting to define nosql in terms of what it is rather than what it isn’t
KentBeck	@dnene there are many more people confused about what datastore to use than who hate sql. the positive approach appeals to the former.
dnene	@KentBeck Agreed .. and I’m aware of many more who wonder why we need a different datastore than the RDBMSs. NoSQL as a name doesn’t help.
KentBeck	@dnene well, why *do* we need a different data store?
dnene	@KentBeck Primary Need : We need support for flexible/arbitrary schemas with complex depths – RDBMSs don’t dance well in this space.
	@KentBeck Secondary Need : Support for deferred processing required for analytics (eg. Map/Reduce). RDBMS don’t do too bad a job here
	@KentBeck Tertiary Need (not one that I’ve felt strongly yet) : Distributed and horizontally scalable storage on commoditized h/w.
KentBeck	@dnene it seems like you’re looking for realistically structured data, not data twisted to fit a formula convenient for mathematicians.
dnene	@KentBeck Yes.. thats it! I’m looking for realistically or naturally structured data storage / persistence. Rocks compared to the term nosql
	@KentBeck Wonder if the term arbitrarily structured makes sense as well. This has been one heck of a conversation/Q&A so far +1:)
KentBeck	@dnene glad you found it helpful. you get bonus points if the opposite of the name you pick is unattractive, a la “structured programming”

Related posts:

1. NoSQL – A fluid architecture in transition
2. Five important trends on the enterprise architect’s radar
3. Stop making SOA complex

What is NoSQL : Some of the characteristics shared by most if not all the NoSQL engines are as follows :

1. Schemaless or Hierarchical Schema Storage NoSQL assumes at its very basis a schemaless or a hierarchichal schema storage system. In most cases this consists of a simple key value pair storage. While some storage engines excel at storing small values (LightCloud/Tokyo Cabinet), some are strong at storing large documents (CouchDB).
2. Distributed storage : This is one of the driving forces of NoSQL growth, though not a distinguishing characteristic of NoSQL. One of the areas these storage systems separate themselves from RDBMS’s is their ability to allow better horizontal scalability. This varies from the simple master-master replication for MongoDB, to multi node sharding using consistent hashing with LightCloud (a la memcached) to a multiple master eventual consistency model of Riak. The basic premise in using some of the NoSQL engines is that storage will scale horizontally.
3. Support for deferred processing :Many of these engines allow for some degree of deferred processing. Whether this be simple lua scripting in case of LightCloud or map-reduce scripts in case of CouchDB, the general assumption is that some amount of latency in computation times is acceptable, and some of the computations (especially related to analytics based views) will be performed post storage.
4. Eventual Consistency : This may seem like a necessary feature of all NoSQL storage systems but it isn’t. While clearly some such as CouchDB (in terms of its map-reduce views) and Riak are better placed for supporting and implementing eventual consistency, it is quite feasible to use others such as LightCloud or MongoDB to implement immediate consistency using a single master-master pair. Suffice to note that eventual consistency is not a necessary side effect of using a NoSQL storage system, though it wouldn’t be incompatible for the two to work together.

But the points I would really like to emphasize are :

• NoSQL is not a direct competitor to RDBMS/SQL : It is actually a solution to many use cases where using RDBMS was perhaps a poor fit. Thus the decision for an architect is not which of the two competing options (RDBMS or NoSQL) Update: one should selectshould be the preferred standard storage strategy, – it simply is which one is the more appropriate storage system for the application under consideration.
• NoSQL is still at a fluid stage of its development : All the NoSQL storage systems (but for LightCloud/Tokyo Tyrant) are still being quite actively developed. These have not reached v 1.0 (Update: MongoDB is at v 1.1) and it is likely that some time will pass before any of these get beyond the beta and release candidate stage and get a 1.0 in-production stamp. While there is a lot of interest, there still is a substantial amount of experimentation in terms of the right feature sets leading to differently focused developments in different storage systems. To an architect this represents an interesting challenge. I think the way to approach this right now is to not use these in mission critical (eg. life or health impacting) systems, and to focus on reasonable expectation management in terms of ensuring the right kind of SLAs around their availability (simply because many of these haven’t yet been put to intense use in production the way say an Oracle or MySQL have been). This is not an attempt to spread “FUD” about NoSQL – far from it it is an exercise in setting appropriate expectations. i would also recommend that it would be appropriate to evaluate the available NoSQL choices only when reasonable SLAs can be worked out for their usage. It is certainly preferred to using NoSQLs rather than using RDBMS’s in an inappropriate manner (large objects serialised into BLOBs or into name-value pair tables). However, I would suggest that you do not deeply bind yourself into a particular NoSQL engine. The future development of most of the storage systems is still unknown to a certain extent, as is the future landscape including any shakeouts. Should one recommend usage of a NoSQL engine – it is important to have a clear plan for switching over to an alternative engine should a need arise in the future. While this is easier said than done, deciding the appropriate level of abstraction to use (ie. code to the API directly vs. use a layer of abstraction for engine independence) is best left to designer / architect to dwell upon.

Related posts:

1. Stop calling me NoSQL
2. Five important trends on the enterprise architect’s radar

Keep in mind that these steps are based on my notes which might be a little incomplete especially around the part where acidbase is installed.

• Initial Configuration : Basic configuration when getting started up.
• Setting up basic security : Basic Security configuration
• Setting up rootkit detection : Setting up rootkit detection
• Setting up Bastille : Setting up Bastille
• Setting up the lamp stack : Set up the LAMP stack including mysql and apache
• Setting up Snort and acidbase : Configure Intrusion detection using Snort and Acidbase.
• Setting up File Integrity using AIDE : Setting up File integrity checks using AIDE

Initial Configuration

These steps cover the initial setup of a server

Setup the hostname

Lets say the hostname we want to setup is vps.

Now update the /etc/hosts file to reflect the hostname and the fully qualified domain name
Replace 12.34.56.78 with the IP address of your host

Updating the ubuntu repositories

You will need to update your ubuntu repositories to include jaunty-updates and universe repositories. This is so that you may install additional packages as required from these repositories as well. In my case, the earlier version of the file /etc/apt/sources.list was as follows.

However please note, that repository selection and its update strategy may be linked to your company or application strategy. Please make sure these steps are consistent with your policy. If not, kindly adapt consistent with your team / organisations policy. Also instead of us.archive.ubuntu.com, you may find other country specific server names. In that case you may want to continue to use the other server name as already listed in your file.

Upon adding jaunty-updates and the universe repositories, the resultant file is as follows.

Now update the sources. This will scan all the repositories

Finally upgrade ie. replace any existing packages which have a newer upgrade

Download the language pack

To add the necessary for the preferred language of your choice add the appropriate language pack. In my case I add support for english (en)

Set the timezone

Set the timezone of the server. You may choose to set it based on server location, or typical user location or to UTC.

That will start a small app, from which you can select the timezone. I selected None of the Above which offered me a choice of timezones based on UTC offsets and subsequently selected UTC.

Setting up Mail sending

I do not need this VPS to act as a mail server. However I do need to have capabilities to send email from this machine. Many unix tools routinely assume the existence of sendmail or equivalent MTA. However that is an overkill in this context. So we shall not be installing sendmail or postfix or exim or any other equivalent. Instead we shall configure this server to be only able to send out mail using an SMTP account on another mail server. For this we shall install a tool called mailx. Note: If you have mailx already installed through another ubuntu package called mailutils, you may either continue with the same (in which case you will need to configure the remainder of the mail stack correspondingly eg. sendmail) or remove mailutils and add heirloom-mailx

We shall also configure a global configuration for sending out mail. In my case its all right to always send mail using only one account irrespective of the process or user who is sending it.

Note that in the above configuration, we shall be placing the mail account password in clear text. Make sure it is a mail account you do not use for any other purposes and that its password is not the same as used for any other purposes. Now enter the following as contents of the /etc/nail.rc file. Obviously change the relevant fields to appropriate values. Note that this file is configured for sending mail via gmail. You may need to configure it differently based on your own SMTP configurations.

You can try testing whether this got set up successfully. Enter the following (replace youremailid@youremaildomain.com by the email id where you would like the mail to be sent to)

Basic Security

In this section we shall make some basic configuration changes with a view to enhance the system security.

Mounting the shared memory as read only

Open and edit the file /etc/fstab to add an entry to mount shared memory in read only mode. The reason we do it is because many exploits use shared memory to attack other running services.

If you have a good reason to make shared memory writeable skip this step.

Now add the following line at the end of the file

Tightening the passwords

One of the easiest exploits is to attempt a brute force login using dictionary based attacks. In order to ensure strong ie. non-guessable passwords we shall update the password checking policy so that it allows only strong passwords. A simple way to ensure that is to ensure a reasonable minimum length and to ensure multiple character classes.

First lets install a new pam authentication module pam_cracklib. To install the same run the following

Answer ‘Y’ to the prompt it asks for regarding continuing.

Note: if you did not add the universe repository to your sources.list file, you will not be able to install libpam-cracklib. In that case you will need to skip this step.

This should’ve resulted in the file etc/pam.d/common-password having an entry for pam_cracklib.so and pam_unix.so. Update the pam_cracklib.so entry to add one more requirement ie. minclass=4.

In my case, the resultant two lines in /etc/pam.d/common-password are as follows. Note that I added the minclass=4 clause manually.

There. You now have a strong password scheme which will conduct a whole range of password checks in addition to ensuring that the password has a minimum length of 8 and each new password has at least one each of the four character classes. The four character classes are lower_case, upper_case, digit and special_characters (the last one being any non alpha-numeric character)

Creating the first user

Note: if you have already created at least one more non root user this step is not required. We are primarily creating the new user so that we shall eventually allow sudo and remote ssh login privileges to the user and disable remote ssh privileges for the root user.

Setup the first new user. One of the reasons you should create a new user is so that it will afford you the ability to allow him to perform root actions through sudo, and thus subsequently allow you to disable root access over ssh. By default when one creates a new user, another group gets created with the same name as well. In this case we shall create a new group “dev” and then create a new user associated with that group “someuser“. Use the groupname and the username as you would like to setup when executing the commands below. In the commands below we create a new home directory for the user, associate the /bin/bash shell with his account instead of the default /bin/sh, (I just prefer bash to the plain sh) and finally set the password for him.

We shall also create the .ssh directory for the user which we shall be using later

Now we shall create the keypair for the user to log in to the host remotely. Note that if you are going to do this for multiple users, then you might want to have each user run the next step locally and then copy over his public key onto the server before continuing to the ssh tightening operations described later.

The user should do the following on his local workstation from which he is most frequently likely to connect to the server (not the server that we are hardening).

Note: the part after -C in ssh-keygen is just a comment to identify the keys – enter something to identify the user and his machine.
Also make sure not to keep the passphrase blank though ssh-keygen will allow a blank passphrase. The reason is that if the user’s local machine is compromised the attacker can then get an easy access to the server being hardened.

change someuser and some.host.com below based on the user id and vps name correspondingly

Now the user should himself ssh to the remote server and on the remote server move his public key into the authorized_keys file. So execute the following command after being connected to the VPS

At this stage the user can disconnect from the VPS and attempt to reconnect using ssh. If all works well, he should get connected to the vps in a manner where it does not prompt him for a password but instead he does get prompted for the passphrase to his private key (assuming he did set one).

This stage of updating the authorized_key file can also be performed by an administrative user / root once we later reconfigure ssh to only allow public key based logins.

Enabling the user to perform sudo operations

We shall enable any group who belongs to the group ‘admin‘ to be able to conduct root operations through using sudo.

First create a group ‘admin‘. subsequently associate the user with that group as well. Note : For best security ensure you allow associate only a very small number of users with the ‘admin’ group since that will effectively allow them control over the whole machine (assuming you setup the privileges as I subsequently describe below).

Now we shall enable any user who belongs to the admin group to perform root actions by using sudo. To edit the sudo policy file do the following

At the end of the file which is now opened up – add the following line

Note this grants all superuser privileges to the users who belongs to admin group when conducting operations using sudo. You can use the sudo policy configurations to set up far more fine grained set of privileges, but thats beyond the scope of this document.

To test whether the configuration worked successfully, you can login as someuser and execute the following command.

Tightening up ssh

To create the group and associate the users with them perform the following command (use the appropriate username instead of someuser for each user who you would like to allow SSH access).

Now that we have at least one user id which can conduct root operations using sudo, its time to disable root login from ssh. Go open the file /etc/ssh/sshd_config. Make the following changes

This one disables root login for ssh.
PermitRootLogin yes to PermitRootLogin no

Adding the following line allows only users of a particular group to login. Note: if your user count is small, you could use AllowUsers instead. We shall be separately creating the group and associating the users with the group.
AllowGroups sshlogin

This change reduces the login grace time (time before a user needs to login after making the ssh login request). This is to allow better protection to sshd against DOS or brute force attacks.
LoginGraceTime 20 to LoginGraceTime 20

This changes the port number that sshd listens on from the default 22. Changing the default port takes away the ability of an attacker to easily guess the port on which to attempt to penetrate the system via ssh.
Port 22 to Port 9999 (or any other suitable number)

This change disables X11Forwarding over the SSH connection. It will result in you not being able to run X11 GUI applications remotely. If you need that flexibility, do not make this change
X11Forwarding yes to X11Forwarding no

The next change disables password authentication (thus allowing only users with their public key being stored in the corresponding authorized_keys folder to connect using ssh). Thus passwords, which need to be transferred to the server in clear text no longer are a valid authentication mechanism. The only available choice is public key based authentication.
PasswordAuthentication yes to PasswordAuthentication No

The next change disables password authentication (thus allowing only users with their public key being stored in the corresponding authorized_keys folder to connect using ssh). Thus passwords, which need to be transferred to the server in clear text no longer are a valid authentication mechanism. The only available choice is public key based authentication.
PasswordAuthentication yes to PasswordAuthentication No
UsePAM yes to UsePAM no

The following change of uncommenting the banner allows a welcome message (not really a welcome one ) to be displayed to SSH logins. It does not improve any aspect of the physical security but is more from a legal notice perspective.
#Banner /etc/issue.net to Banner /etc/issue.net

Now go update the contents of the file /etc/issue.net to something similar to following

Restart the ssh daemon and you will no longer be able to connect as root over ssh directly.

Disabling su

As per the policy being followed, only users belonging to the ‘admin‘ group will have any special privileges. For all other users, we shall attempt to lock down the system to the extent feasible. As an early step we shall disable the ‘su‘ (switch user) command for all but those users belonging to the admin group. Note : since this disables sudo for a brief duration (between the commands), first switch over as a root before executing these commands

Install rootkit detection

Install chkrootkit

Now create this file /etc/cron.daily/chkrootkit.sh and enter the following contents (replace sever.domain.com with the servername and emailid where you would like the report to be sent to).
Since the mail is rather verbose, and may lead to useful alerts getting ignored, I’ve removed the lines with the common messages. You may change or not use the grep commands as per your preference.

Allow execute permissions on the file, and test once by running .. you should get a mail with the report.

Installing Bastille

Bastille is a comprehensive package for security configuration.

Bastille Bug? and workaround

You may need to perform this workaround if your /etc/debian_version contains 4.0 and the current ubuntu bastille package has not resolved the issue. Ideally one should just need to install bastille from the ubuntu repositories

On Ubuntu 9.0.4 (jaunty) as of the day this document was written, I received the following error
~& ERROR: ‘DB5.0′ is not a supported operating system.

I’ve had a difficulty to install Bastille on Ubuntu 9.0.4. This is due to the fact that Bastille does not support the debian version number in /etc/debian_version. As per Bastille bug reports this got fixed in 3.0.8, but did not work for me in Bastille 3.0.9. Hence the following is a work around to install bastille.

This workaround involves downloading lynx to access internet, downloading the debian package directly, and then installing the debian package. When you reach the web page it will show you a list of mirrors. Download the .deb package from one of the mirrors

Install psad which we shall need later and perl-gtk which we shall need to configure bastille with the following command

You will notice an error message :

To resolve the same execute the following :

Now run bastille in an interactive mode

The resultant dialog with bastille is too long and I am just showing the final configuration file which is produced as a result of bastille configuration. I’ve used the port 22 as a proxy for the SSH port which I assume you’ve changed to some other value – use the other value instead of 22

Installing the Lamp Stack

Let us use the not so well known tasksel command to download all the packages necessary for a lamp stack. Note that tasksel is simply a convenience tool around apt-get and allows one to install a whole bunch of packages based on a class of necessity – in this case a LAMP (Linux,Apache,MySQL,PHP) stack.

During the installation you will be required to create the password for the mysql root user. In interest of tight security do make sure to create a really strong password.

This actually sets up the basic LAMP stack and that should get you operational.

Securing Apache

Change the user and group apache runs as

You may be aware that installation of apache resulted in a new userid and group being created (both called www-data). I just find it a little bit more comforting to change the well known usernames to something that are less predictable. In the steps below I change www-data to apache but you may want to use something a little less predictable than apache.

We shall need to change the same in the apache configuration file. Edit the file /etc/apache2/envvars and change the two lines as follows

Install and configure mod_security

Earlier when we configured, bastille and through it the iptables firewall, we allowed incoming public traffic on the SSH, HTTP and HTTPS ports. We’ve already covered the tightening of SSH security. However we still do not have a good way to control HTTP and HTTPS traffic. The solution to that is installing the apache module mod_security. It allows us abilities to inspect and if necessary reject or redirect HTTP/S traffic. The full description of mod_security is beyond the scope of this document, but a good example of how it can be used further to implement application level security as well can be had from the document Securing web services with mod_security. Another useful overview page is Secure your Apache2 with mod-security.

We shall start off with installing mod_security.

You should notice a new file in /etc/apache2/conf.d called security. These are the global mod_security settings.

Change the following variables in the file to make server identification difficult. Note that while these settings do not enhance the security directly, they do make it harder for an intruder to easily identify the specific server and configuration, thus making it harder for him to attempt configuration specific exploits.

Also uncomment the following four lines. Note that this may require you to configure other web applications which are not configured appropriately.

While not related to mod_security, to minimise server information leakage, you may consider changing these two variables to the shown values in /etc/php5/apache2/php.ini

As mentioned above, there is a whole range of additional controls you can enforce using mod_security. Depending upon your specific requirements feel free to leverage mod_security much more.

Disable apache modules you do not need

In general it is better to turn off unrequired modules unless you really need them. The default configuration installs a number of modules which may not be required. The modules that have been enabled are in the /etc/apache2/mods-enabled directory which are all symbolic links to the modules that have been installed in the /etc/apache2/mods-available directory. Note that turning off some modules may affect some web applications you install, in which case you may choose to subsequently turn them on only if necessary. New modules can be enabled with sudo a2enmod module_name command and enabled modules can be disabled by sudo a2dismod module_name command.

Depending upon my requirements, I disabled the following modules

Chrooting apache

This is an advanced topic that I shall skip. Note that if you do setup chrooting, there’s a lot of additional considerations which will need to be observed, considerations especially around shared libraries and configuration files which will make the overall process of configuration extremely difficult.

Setting up snort and acidbase

Author’s Note: Careful – Very likely, there were a few things that I might have missed or incorrectly noted especially in the context of acidbase and following these instructions may not get acidbase running properly. You may have to do some steps outside these notes to get it all working properly. Unfortunately I am unable to rerun the process on a fresh server so am unable to immediately note the possible lacunae in the notes

Now that you have a mysql server going, create a new database for snort. In the following section I use snortdb as the database, snort as the user and snortpwd as the password for the database. However I do encourage you to replace the same with some non easily guessable names and passwords.

Here’s the commands to create a new snort database and user. In the following example ‘$‘ is the unix prompt while ‘>‘ is the mysql prompt.

Now we shall install snort.

The installation process will require you to enter the network that you wish to protect. Since in this case we are only protecting a single machine, enter the IP_Address/32 eg. 12.34.56.78/32

The process will prompt you whether you wish to configure the database. Select “Yes” .. the script will proceed but terminate abnormally with the error “subprocess post-installation script returned error exit status 6″

Now execute the following commands to setup the tables in the database. At the end you shall open the snort configuration file to enter the database configuration parameters

Uncomment the line that starts with output database: log, mysql and add the configuration information at the end as shown

Now remove the file as shown below to indicate that the database configuration has been done and start snort. The subsequent command confirms that snort daemon is indeed up and running

Now, lets install acid which is a web based application to be able to view snort alerts

Select No when it prompts you for configuring the database. By default it uses the database and userid ‘snort‘ and I prefer to keep these different from a predictable default, in which case acidbase database initialisation will fail.

Edit the file /etc/dbconfig-common/acidbase.conf.

Set the following values

The default configuration allows acid data to be accessed only from the machine where it is involved. Since this shall be a VPS which we shall be accessing remotely, it is time to relax the constraint. If you have a good idea of the network IPs that you will be accessing the acid web application from, update the same in the allow from directive in /etc/acidbase/apache.conf, else relax it fully to allow from all.

Also protect the ‘/acidbase‘ url by http basic authentication. The resultant section in the file /etc/acidbase/apache.conf looks like :

You will need to create a user id / password pair for the app or point AuthUserFile above to another file where you already have the ones set up for your server.

To do so you shall need to run the following command :

If you need additional user ids to be installed use the command as shown above again with different user ids – but make sure not to use the -c switch since thats only used the first time to create the password file.

Now start the web application by restarting apache

Goto the web page http:vps_fully_qualified_domain_name/acidbase/base_db_setup.php, click on ‘Create Base AG//’ and you are on your way.

File integrity checking using AIDE

Install AIDE

Update the aide configuration file etc/default/aide to update your email id where you would prefer the integrity check reports sent (Look for the variable MAILTO)
Now initialise the aide configuration

Now run the process (which will be running once daily)

This process takes a long time (about 5 mins for me) and will at the end email you a report if any files changed compared to the ones in the default database.

Related posts:

1. Opera Unite : A model for server disintermediation on the internet

Service orientation is neither essential for, nor is it the intention of REST.
Not only is REST not service oriented, service orientation is irrelevant for REST

There. But why was it so important to state that ? Allow me to quote from the blog post I referred to.

I say not surprisingly because RESTafarians have no clear position on “service”, they just say REST is the right way to build a Service Oriented Architecture. Yet, REST has no concept of “service” anywhere, just resources and their shiny uniform interface, links and bookmarks. Indeed there are no services in REST. Just read the thesis.

and it further goes on to state

But I digress, let’s go back to “services”. Even Bill, in this REST-* proposal is talking about creating a RESTful interface to non RESTful services. That certainly begs the question, how can a service be non RESTful since REST is all about SOA and replaces in its entirety WS-*.

The essential issue here is the flawed assumption that REST attempts to be service oriented or it is all about SOA. Its not. And why so ? Since it is resource oriented. And whats the difference ? Read on, because that’s what this post attempts to address.

Service

Wikipedia describes a service as follows :

the term service refers to a set of related software functionality, together with the policies that should control their usage.

OASIS (organization) defines service as “a mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and is exercised consistent with constraints and policies as specified by the service description.”

Now lets attempt to understand a service in a little more dumbed down fashion. Lets hark back to the good old construct of flow charts and process charts. In these charts one basically divided an overall set of functionality into discrete set of functionalities and chained them together through some sequences and decision points. As an example if we were to consider a simplified retail outlet system, it would consist of steps that would support (a) ordering items, (b) receiving and reviewing items, (c) selling items. In a SOA world, these could be mapped into a Ordering Service, Receipt Service and Sales Service (you could of course come up with better names and further decomposition). But each service is essentially one of the decomposed tasks of a larger workflow. If the interface to such service could be standardised and documented it would help it to be reusable across multiple contexts. And to the extent such services are reusable across multiple workflows, the advantage of Service Orientation become obvious. And finally if such a service interfaces are exposed over the web – it is a web service. At the end of the day, each service is a reusable, composable task (or tasklet) performer.

Resources

But REST does not attempt to be service oriented. Thats because it does not view the process as a sequence of tasks to be performed. It views it as a sequence of resources under modification. To put it differently, it views the process as a set of actors who exchange resources (or documents for better visualisation) and carry out activities based upon receipt of such resources. Though not as equally apt as a process chart, the analogy here would be a data flow diagram. And what might such resources be ? Well in the above scenario, there’s a Purchase Order, a Goods Receipt and an Invoice. Those are the essential abstractions that REST focuses on. These are Resources. Just like Services where there’s no one valid set of abstraction of services, one could work out a different set of resources rather than those I listed. But the bottom line is that the essential abstractions are resources *not* services.

How are they different ?

You could build a system either way – as services or resources. In terms of being able to successfully build, deploy and maintain a piece of software, both REST and SOA are likely to be equally successful at building the software. But the essential vocabulary through which they decompose their various parts (and therefore describe their interface elements) will be different. And how is that different ?

Let us imagine the ordering service we talked about above. One way to build a SOA ordering service is to establish a interaction procedure which combines an overall protocol and a series of steps (Service API). To reduce potential errors, there is a document upfront which describes in adequate detail how such an interaction should be conducted, what are the data elements to be exchanged at each stage, and what are the necessary sequencing requirements between various steps for such interactions to be concluded successfully (WSDL). The focus here is the tasks being done and the protocol for the task instructions. In case of REST the essential construct will be exchange of one Purchase Order. The purchase order would have sufficient in band instructions about the fact that it is a purchase order and the attributes it has (in-band metadata), and formal documentation if any would be restricted to the structure of the purchase order and its data than than to the sequencing, flow or any protocol level activities. (Thats why sometimes REST looks deceptively simple to be treated as just another CRUD).

More often than not when called upon to describe a service, the description will describe what the service does, and the service interface will mirror the steps required to perform the activities. Resources on the other hand will simply describe themselves and anyone who looks at a resource description will be none the wiser about what processing exactly happens behind the scene.

Thats why I believe even if both REST and SOA can be used to build software effectively, the essential focus on resources as the central abstraction makes REST much easier to use for the clients. But thats just my opinion – you may form your own.

You cheated! REST meets the service orientation definitions you listed above

Yeah, kind of (in theory). There is one way where REST over HTTP is service oriented. Imagine a document service which could store, update, fetch or delete documents. Now replace document with resource in the earlier statement. Thats your typical HTTP service that REST works off of to implement a resource management service – but thats just a single service which is standardised for REST over HTTP. And all REST implementations will be service oriented to that extent. However the sheer simplicity and ubiquity of this service makes the associated service orientation of REST rather uninteresting and thus largely ignorable.

So next time one wants to debate the merits for REST and/or SOA – feel free to add to the tons of stuff thats already written. But don’t measure REST based on service orientation. Service orientation is largely irrelevant for REST. And that per se does not make one better than other – it just makes them different.

Note: There were many other points in the blog post I referred to that I would want to offer different opinions on. But in this case, I believed it was important to keep this post focused on an essential thought that I really wanted to emphasize.

Related posts:

1. Service Oriented Architecture is primarily about business and not technology. Bollocks!
2. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
3. ReST : SOA, WOA or ROA ?

But the goal of this post is to neither encourage nor discourage the use of polyglotism. It is to suggest that at the end of the day even if polyglotism really stimulates the technogeek sensors in us, it needs to be evaluated not in technology terms, but in business and economic terms. A point that is also partially supported by The plague of polyglotism. Unless we use tools like GWT and hibernate to keep us monoglots, (which is likely to be a very small proportion), polyglotism is the default state that we shall operate in. So its not worthwhile to fight that trend per se. The important aspect is to ensure that we understand the costs of supporting each new language in the mix. And the reason to do so is to evaulate whether the benefits of adding the language outweigh the costs in the medium to long term. So here are some of the costs (in no specific order)

• Syntax and libraries : This is probably one of the lowest costs even though it is the most apparent. For every language we add to the mix, generally for a long running project, at least three developers (if you have reasonable risk management objectives) need to learn the new language syntax and the libraries they work with. Along with this is the cost of books, training courses and materials etc. (yes for many enterprises, many developers but those purely self motivated will need to be formally provided all of these).
• Idioms and Philosophies : Knowing the language is not adequate. You need to understand its philosophy, its design intent and its typical idioms. Being able to use a language to write code is not the same as being able to use it idiomatically. And while that may not constrain you in over small code bases in the short term, over a period of time for reasons such as performance, consistency and ability to understand other libraries that you may reuse, you will be forced to deal with these aspects of the language as well. The book Dreaming in Code talks about how Java programmers who started writing python could not write idiomatic python code, and thus a fair degree of code had to be redesigned or rewritten. And that is by no means the only such instance. I suspect similar trends are likely to be found if java programmers move to writing ruby or scala code. And its a trend you might have likely observed when people comfortable with writing relatively simpler languages such as Visual Basic or PL/SQL or even more complex languages like C++ moved to writing Java. Knowing the language is not the same as knowing it idiomatically – and that takes time. Which means – do not take on high risk activities when you absorb a new language, it will take a few iterations for your team to absorb the new language – far far more time than it will take to read a book or attend a training course.
• Skills to manage multiple idioms in one head : Unless you are going to have different programmers for different languages, the programmers working on multiple languages at a time will not only have to deal with the multiple idioms, but will need to concurrently apply them. In my experience, this is not a trait we are born with, and it takes some time and experience with it. It will take some time for them to learn that the best practices are not identical across various languages. It will take even more time to figure out why they are different. So there is a good likelihood, you will not be in a situation to have all your developers work on all the languages simultaneously – especially some of the Junior ones.
• Skill portfolio management will become more difficult : For typical enterprise or ISV teams, the managers need to ensure adequate skill availability across all teams. As the number of languages in use increase, this management process will become harder. Which means there is an automatic economic disincentive which rises is in magnitude, each time you add yet another language to the mix.

So is polyglotism good or bad ? I submit such a value attribute cannot be assigned to polyglotism per se. It needs to be really evaluated in your context, preferably beyond a project window into the medium or a long term. What is indeed obvious is that each new language will bring in some very strong capabilities and aspects where it is superior to the other languages in the mix. Equally obvious is that each new language will bring in associated costs. Just like there is no perfect language, there is no perfect number of languages that make sense in every context. Just like you do your homework in terms of language selection, make sure you do the same for your language portfolio not just in technology but in management and economic terms as well.

No related posts.

In my understanding which hasn’t changed for many years – a duct tape programmer is a programmer who always takes the shortest route from point A to point B without giving regards to longetivity or maintainability of the solution. When he applies a duct tape – he is ticking off a job as “done”. He has not accounted for how long will the solution last. Is it even the right solution or is it just something that on the face of things seems right. He is only pragmatic enough to make sure he can get home early or be in his bosses good books. Thats where his pragmatism ends. Thats why the term has a negative meaning. Thats why it is not meant to be a compliment.

Now what happens when you associate it in the way Joel just did in his blog post ? Suddenly it no longer seems to be a negative term. It gives rise to confusion. If someone now calls me a duct tape programmer – I’ll be tempted to take it as a compliment. On the other hand – when someone writes a shitty crappy piece of code – I now have to figure out a new term – since I can no longer term it duct tape programming. I mean if I do and he responds a link to the post – what chance would I stand compared to someone as esteemed as Joel.

I understand as bloggers we want to make a point. I also understand that there are ways to make the point in a manner that really really stands out. But at times there are risks and negative consequences when we go down that path. These include confusing the juniors, the post getting misunderstood, misconstrued or even misquoted. These also include possible positive spin on essentially negative characteristics. Lets stick to calling pragmatic programmers as pragmatic programmers and leave duct tape programming to the realm of the essentially short sighted, fragile and often brittle programming that it really stands for.

Related posts:

1. Jerk programmers can’t be managed tactically
2. So were Jeff / Joel / Uncle Bob discussing happiness and fitness ?
3. Java : the perpetually undead language

So I have been reading a lot about Scala lately and even opinions about how it will be a long term replacement for Java. I’ve also read some interesting writeups about Scala adoption such as On Scala’s Future and A Tipping Point for Scala. While I used to code a lot, my responsibilities today require me to interact with and address a lot of issues including those faced by our customers, our development teams and also engage with my peers and superiors on many other difficulties bedeviling our organisation. This gives me little time to try out Scala. I know I should be doing that, but sincerely I do not have the time. So I rely on the feedback of my team, the trade journals and other influential architects within and outside my organisation.

I have heard about many developers switching from Java to Python / Ruby. However I have heard of relatively only a smaller number of large Java shops which have done the shift – most of the switch stories I’ve heard reflect a smaller sized teams. I can feel the excitement Scala has generated amongst the development teams – the brevity, the functional programming model introduction, the exciting stuff being done concurrently et al. I have no doubt that, given so much excitement it must really be a good language.

To introduce my organisation – it is one of those shops which service many projects concurrently. Given the tremendous business and growth, I must confess we do not always have the luxury of being able to hire the most top notch talent. We do have a lot of projects we use Java for, and thats a language our customers are comfortable with. I’ve had some of the senior people check out Scala to gain some feedback into the language. But at this stage I must say I am inclined to evaluate the shift but not convinced enough to do so. I am sure that I could if convinced drive the change to Scala incrementally. However my fear stems from the fact that if things don’t turn out well, despite all the great advice I’ve received – its going to be my rear end on the line. So here’s some of my concerns regarding evaluating the shift to Scala and there are many of them, so some of you might be able to help me through this thought process.

• Functional Programming : I’m sure in many ways it rocks. But my guys tell me they are not sure how to use it in the typical bread and butter applications which read from database, do some processing and write back to the database. Does Functional Programming help me in this context ? Will my team scale into being able to write functions with no side effects assuming thats a desirable goal ? What if they tie themselves up in knots and my release to the customer is risked ? I can’t afford that. Is functional programming even desirable in such contexts ? So I am not sure if in these contexts I should just ditch functional programming and work with just normal imperative programming capabilities of Scala. I am so confused, and afraid.
• Different Syntax : While Scala runs on the JRE, its syntax is very different from Java. From what I could gather, it is much easier for a Java programmer to read (make sense of) simple Python code than to read Scala code. Is it true ? So even if I do get compatibility in terms of the runtime environment, would I be picking up a language that is syntactically so different a language that it would involve a substantial relearning curve ? I remember when we had to learn Java and Javascript. For better or for worse these were indeed relatively minor modifications of the C/C++ syntax, compared to what I sense as the syntactic shift between Java and Scala. Am I wrong ? If so, could you help point me to resources which help me understand that Scala code is not much different than Java ?
• Sample code : Guys, I need your help. I need to see some good sample code. Some code which reflects how a typical application is architected, designed and programmed in Scala. And I don’t need it for a complex multi threaded actor based processing – I just need to see simple J2EE server based departmental applications maybe a simple recruitment tracking or library maintenance application. If I find a good one, I’ll just take it and give it to my team and say – there, thats how we’re largely going to build it, and even if we make a few changes along the way we at least have a reasonable template that we can build from.
• Dumbed down environment : I remember my great adventures with C and vi and make. But my team today is very different. They want great IDEs. They must have syntax highlighting, autocompletion and nice refactoring capabilities. If I ask them to move, some of them might be excited about the change and be willing to overcome these short term hurdles. But there are some of them who will not be keen to do so and may be disinclined to support such a shift. And at the end of the day my ability to conduct this shift is a function of my ability to carry a large proportion of them along with me. Even when I considered a shift from svn to git, the IDE support was a big issue even though quite obviously git capabilities were really exciting. I couldn’t push along that change, and in this case we are talking of changing the language.
• Is this a good time to shift to Scala ? I remember the early adopters of Java from 1996 thru 2001. While they gained a lot of experience, JRE and J2EE really matured only post JRE 1.3. Scala seems to be coming out with so many enhancements so fast, I am not sure if it has stabilised. I am told there is a 2.8 coming out in a few months. So if I train my team and Scala continues to change rapidly will I have to keep on retraining my team regularly ? And what about the customers I take to production. Will the frequent upgrades mean I end up supporting multiple customers on multiple versions of Scala ? Maybe Scala is stable but it would be helpful for someone important enough to make a clear statement that there are no new major shifts anticipated anytime soon and that these version shifts are likely to be no faster than the JRE version upgrades (which were fast enough).
• Support from peers and superiors : I remember the day I decided to shift to Java. What made the move easy for me was the sheer fact that Java was a big paradigm leap away from the then dominant C++. Not only was it cross platform with binary compatibility thrown in for good measure, Sun ensured that it made all the right noises to appeal to the enterprise architects and all the business managers. I see the senior developers in my team clamouring for the shift to Scala, but my peer managers and my superiors don’t display even the fraction of the enthusiasm they displayed during the Java shift. The implication for me is that the risk cover I get when I order the shift is far lesser than what I had when I made the move to Java. Which means if things don’t quite work out well, I’m really going to be screwed.
• Business friendliness : I understand all the nice talk about the technical excellence of Scala. But I really need to translate all these great language features into a projected ROI that I can use to convince others about. So I would like to see actual case studies of applications that were moved to Scala and what impact it had on the time and cost so that I can use it to compute my ROI. And what scares me is that learning curve may risk the initial applications long enough to push my breakeven point of shifting to Scala well beyond a 12 month and perhaps even a 24 month period. I fear things might not be as difficult but in absence of known studies, I am likely to lean towards projecting a worst case scenario rather than an optimistic one.

So folks, I am asking for your help. And while a lot of you may think that people like us who balk at the thought of limited IDE support are wimps, please remember that 80% of us don’t fit into the top 20%. And if you would like Scala to be popular, you need us as much as we need you. And if you are not too sure, please remember Lisp and Smalltalk are great languages as well.

Related posts:

1. A brush with Functional Programming and Scala
2. Java : the perpetually undead language

Apart from some tweets I saw on the topic and some twitter conversations, the blog posts which perhaps set off the thought were

• CRUD is bad for REST by Arnon Rotem-Gal-Oz
• A summarisation of the same on InfoQ : Is CRUD bad for REST?
• Can We REST For A Minute? 6 Lessons From The CRUD vs. Hypermedia Debates by Dion Hinchcliffe

The underlying fear and rationale for these posts makes a lot of sense – the fear of creating real dumb passive and shallow applications. I submit, that the problem however is not CRUD – it is resource identification and scoping, and CRUD is not only good for but is the right way to build intelligent, active and deep applications.

CRUD supports Uniform Interface : The primary reason why CRUD gets used is because it supports a uniform interface. At the end of the day, a consistent Create/Read/Update/Delete or POST/GET/PUT/DELETE interface makes things easy. It makes things easy for the development team because of the consistency it introduces in their applications. It makes things easy for the clients who have a simple and consistent interface to deal with. At the interface level CRUD breeds consistency, and at the risk of broad generalisation, consistency is good.

So why do we end up creating shallow applications at times with REST ? CRUD in general works with simple forms built on simple tables. Quite often this style of programming gets elevated into simple forms over simple domain objects. Standardised CRUD helps a lot at the lower end of application development and most database driven application developers are likely to have at some stage in their early development life attempted to build a small CRUD library or framework to help themselves substantially. The reason why we are likely to be ending up creating shallow applications is not because we apply CRUD, but because we continue to apply CRUD on tables or simple domain objects. And therein lie the distinctions

• REST is not about CRUD on tables – its about CRUD on resources
• CRUD is the interface – not the implementation

I attempt to bring up the difference in the example that I detail below.

Simple Account Transfer Example

Lets say we want to build the software to transfer amount X from account A into account B. Lets further specify that a transfer is not effected immediately and requires one more explicit approval. Lets also specify that while a transfer is waiting to be approved, it could be amended. Thats the simple scenario that we shall deal with.

In order to implement this, we shall define a datastructure / table / object for Account which shall contain a field called balance. Further there shall also be Transfer table / object which shall contain the fields sourceAccount, destinationAccount, amount and status. The possible status values shall be Initiated and Completed.

In a simple service oriented application we shall perhaps have a transfer service. Ignoring error handling, SOA wrapping etc., the service interface will probably boil down to the following equivalent Java interface.

public interface TransferService
{
    public Long transfer (Long sourceAccountId,
                                Long destinationAccountId,
                                BigDecimal amount);
    public Transfer get(Long transferId);
    public void amend(Long transferId,
                             Long sourceAccountId,
                             Long destinationAccountId,
                             BigDecimal amount);
    public approve(Long transferId)
}

Lets think of these might get modeled in a REST environment. The important thing to remember is – don’t think about services or functions or methods – think about what are the resources you choose to expose using a simple CRUD interface.

# The following creates a new transfer. The returned data shall include
# the URI of the new transfer, and the URI to approve it
POST /transfer
# The following retrieves the status of a current transfer. If it has not
# been approved the returned data shall include the URI to approve it.
GET /transfer/${id}
# The following modifies the transfer. The returned data shall also
# include the URI to approve it
PUT /transfer/${id}
# The following approves and further processes the transfer. It shall
# return the URI for the transfer
POST /transfer/${id}/approve

While most of this seems all right – what sticks out like a sore thumb to me is the approve URI. Its just so SOAish / RPCish. Plus at least the way this particular interface has been implemented, there is no way to access the approval specific information, without actually accessing the transfer. Hence I suggest that we define a new resource TransferApproval to account for the same.

# The following creates a new approval. If successfully executed
# the transfer is complete and no future amendments or approvals
# are allowed. the returned data shall include the TransferApproval
# URI and the transfer URI
POST /transfer/${id}/approval
# The following gets an existing approval
GET /transfer/${id}/approval/${approvalId}

Please note that the “${approvalId}” at the approval URI simply wasn’t required – since there exists a 1-1 relationship with the transfer. I just included it for easier understanding. If I had to implement the functionality as is I would choose to skip it however if I knew I would very soon need to build in multi-stage approval (as in most banking systems), I would keep it so that each approval against a transfer can also be listed.

But the really interesting method above is the POST. This is a seemingly simple new (in RDBMS parlance) insert into TransferApproval table. But if you are building a REST service, you might be tempted to encourage your clients to not only create the new TransferApproval resource, but also go back and update the Transfer table to update a status to Approved. That would be a smell. Once the POST on the approval is processed, all side effects on other tables should be handled while servicing the POST request. In other words the POST request is not just an insert – its an insert with an associated trigger to conduct all the necessary downstream processing. And its essential one looks at request servicing in this manner so that CRUD can be used effectively. Servers should be designed this way, and clients should anticipate it and we should be on our way to build non-shallow applications.

So finally – CRUD is good. It makes things easy for the clients. Stick to CRUD. Just remember that it is CRUD on resources and not on tables, and the resources shall handle all the downstream changes necessary so that you don’t have to. And finally CRUD is the interface, not the implementation.

Note: Some might notice that this post is just a much more detailed elucidation of one of my earlier posts – REST is the DBMS of the internet.

Related posts:

1. REST is the DBMS of the Internet

• CNet.com : Judge orders Microsoft to stop selling Word
• Crunchgear : Texas Judge rules Microsoft can’t sell Word anymore
• Techdirt.com : Judge Bars Sale Of Microsoft Word For Patent Infringement (Though It Won’t Stick)

I also came across many other tweets bemoaning the verdict and expressing the opinion that it is a sad turn of events – it being related to storage of documents as XML. I did take a quick look at the said patent : Patent 5787449.

And therein I thought there’s seems to be a big misunderstanding (at least to my lay reading). The patent itself has nothing to do with storing data or documents or XML. Its got to do with a particular implementation of data storage which requires maintaining a metadatametacode map. I shall use an example from the patent itself.

Lets say the XML is as follows :

<Chapter><Title>The Secret Life of Data</Title><Para>Data is hostile. </Para>The End</Chapter>

The patent suggests a storage which would maintain a metacode map as follows :

The metacode map essentially stores a tag along with its position. You can also see the same clearly on page 15 of the patent document on google patents.

My reading suggests that the patent and alleged infringement if any has got nothing to do with storage of XML documents per se. In fact, that the tags being used are XML/SGML like is probably completely coincidental – these could very well be ${chapter} instead of <chapter>. And XML documents are stored with the tags embedded along with the content – this patent actually refers to maintaining a map of these tags and the positions they should be inserted into.

Could I be wrong in my interpretations ? Perhaps, since I haven’t seen anyone else point this out and would prefer to be corrected. But the fact remains that at this point in time as I write this post, I believe that XML and storage of XML documents are completely orthogonal to the patent and the case around it – that centers around a metacode map, and metacode maps are not a characteristic of typical XML storage at all. So there’s probably one big misunderstanding about what this case is about and if the injunction upsets you because you are against patents in principle, thats fair. But if you are disappointed about the possibility that this somehow substantially impacts XML storage, the way I interpret it – there’s no such implication.

I shall keep my fingers crossed and hope no one points out a seemingly obvious flaw in my interpretation.

No related posts.

No related posts.

What perhaps hasn’t been clearly stated but is important is that that the standards for employability here are set by the employing organisation based on its context and preferences which may be different from standards required by many other organisations and I would submit that the statement ought to be looked at in that context as well.

To quote from the article :

The official wanted to know why HCL, a $2.5 billion (revenue) company with more than 3,000 people across 21 offices in 15 states, wasn’t hiring more people in his state. Vineet’s short answer: because most American college grads are “unemployable.” (In fairness to HCL, the company recently announced plans to open a delivery center in another state, North Carolina, and invest $3.2 million and hire more than 500 employees there over the next five years under a Job Development Investment Grant.)

Many American grads looking to enter the tech field are preoccupied with getting rich, Vineet said. They’re far less inclined than students from developing countries like India, China, Brazil, South Africa, and Ireland to spend their time learning the “boring” details of tech process, methodology, and tools–ITIL, Six Sigma, and the like.

As a result, Vineet said, most Americans are just too expensive to train–despite the Indian IT industry’s reputation for having the most exhaustive boot camps in the world. To some extent, he said, students from other highly developed countries fall into the same rut.

So why are some graduates are unemployable ?

Some of these are folks who are working on creating tons of new languages (Java, C#, Scala, Closure, Erlang, Python etc.), Operating Systems (Linux, Mac and Windows), frameworks for web applications, clustering, fault tolerance and scalability, schemaless and distributed databases to ensure availability and fault tolerance at a wide scale, competing messaging architectures that each service a particular problem differently etc. etc. Would you believe it some have set up clusters of  hundreds of thousands of machines and service search requests very rapidly. And some others are working on creating innovative and disruptive models in social networking, application integration, peer to peer networking etc. Many are working out next generation mobile technologies including building (not building on) android, iphone OS and the palm (whatever OS it has). And so many in their spare time are spending a whole bunch of time creating open source software, blogging and micro blogging about all the work they have done and sharing it with the wide world so that they can learn off it. And if these are consumer and technology stories, there are a whole bunch of people building critical business infrastructure architectures and frameworks and solutions as well. Even after you complete reading this post and come back to it a month from now, I shall still be busy trying to figure out what exactly India has been able to deliver that can challenge this. Of course let’s not miss out on the fact that most of these activities are conducted by fairly small sized teams. (I am aware some of the examples I quote could have non-american heritage, but that per se is not likely to detract from the issue)

Let’s call a spade a spade. When you have large or very large software construction and maintenance contracts, there are multiple ways to deal with it. In my experience, many Indian companiess have honed to a fine art the process of recruiting, deploying and juggling large armies of programmers to service such expectations (not all Indian companies are based on the same model). Many companies have indeed managed to acquire, retain and expand customer engagements through a combination of technology and business innovation. And they have done a good job of it in the context they’ve defined for themselves. In my limited understanding and experience it is not technology innovation, creativity or extraordinary technical prowess that is at the top of the list of skills that get deployed (though these are indeed found in sufficient levels across the board and some of the prowess can be impressive) – it’s the clockwork project management and business methodologies, techniques and innovation (and even these do result in projects with delays) with their reliance on “another brick in the wall” that gets the customer serviced.

So, if true, the “employable” was perhaps in that context.  If so based on the earlier two paragraphs, I submit it reflects positively and negatively on the employing organisation. Perhaps even more so than the graduates themselves. I do wonder if “incompatible” would have been a better choice of word. And I wonder what it reflected more poorly upon, as well who should be feeling more sad about it.

Disclaimer : These views are mine, mine alone and should not be puported to be shared by any other people or companies I’ve been associated with in the past, present or the future.

Related posts:

1. Service Oriented Architecture is primarily about business and not technology. Bollocks!
2. Is a large corporate making money off open source or open standards an oxymoron ? In a Sun / Java Context

Here’s what my preliminary look into Opera Unite suggested :

• It has a web server bundled into the browser.
• It allows you to write (what would be in current parlance be called) serverside apps using javascript which are hosted by the web server.
• Sample applications include photosharing apps to share photos on your desktop or fridge which allows internet users to post notes to you.
• It allows these apps to be accessed across a router on a dynamic IP using a subdomain on a centralised operaunite service ..operaunite.com (not sure yet whether across a firewall as well but seems so)
• It allows these applications to be shared / published using the config.xml file (similar to google widgets ?). There’s also a central opera directory for the same, but I don’t think sharing is restricted to that service alone (apple are you listening ?)
• These applications can be further installed by other users of Opera Unite (which is the web server service running inside the Opera browser).

So why is this such a big deal ? Without going into any further elaboration let us just imagine user’s used it for following (for desktop-to-desktop or peer-to-peer) communication :

• Send mails to each other – disintermediates email services
• Send short burst messages to each other – potentially disintermediates twitter
• Share files with each other – disintermediates ftp servers and shares characteristics with gnutella, kazaa etc.
• Share resume, product profiles etc. – disintermediates traditional static web hosts
• Build networks of other interested users – disintermediates linkedin, facebook

The constraint it introduces is that there is no global list of user ids to search from – you need to know the URL upfront (like the facebook vanity URL).

The possibilities are many. As are the interesting uses this model could be put to. And the characteristics of improved privacy, data ownership and control (including fine grained access control or selectivity). And there does remain a potential for malicious actors and virus writers to ride on popular apps to exploit vulnerabilities to tend to their nefarious needs.

As I put it differently in another tweet – this opens up the possibility for a google wave without the google in it.

References :

• Taking the web into our hands one computer at a time : An introductory writeup
• An introduction to Opera Unite : A get started guide
• Opera Unite developer’s primer : A primer for writing server side applications

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. Why I deleted my Facebook data. Commentary on Internet data privacy rules.

Introduction

Programing language selection is often a topic that elicits a lot of excitement, debate and often a bit of acrimony as well. There is no universally superior programming language that one can recommend, so I tend to generally disregard most language opinions which say ‘X language is the best’, without specifying the context under which it is superior. Finally most language debates often deal with the technical issues and not the ROI issues. Hopefully I shall be able to address this topic without being guilty of any of these problems.

The range of languages that fall under Dynamic Programming Languages category is rather extensive. My experience is primarily limited to Python and to a lesser extent PHP, Ruby, Javascript, and Groovy. For the rest of this article, I shall be primarily referring to Python or Ruby when I use the word dynamic languages, though many of the references may continue to be applicable and relevant for a number of other dynamic programming languages.

As I describe the technical characteristics, I shall also continue to attempt to address the business aspects as well, so you might find this article at a little techno-business level. Assuming I am able to excite their interest, the tech guys would not find sufficient technical details and would be hungry to hunt for more, and while the business guys would get a little teased with the possibilities, they will not quite get the ROI served in the traditionally formatted excel spreadsheets. Being aware of that, I continue down this path with a feeling that this perhaps will be the most appropriate level for me to abstract this article to.

Object Oriented : Many dynamic languages support full object orientation. There are many who don’t necessarily buy the benefits of Object Orientation, but it is my strong belief, that once a piece of software grows beyond a certain threshold of complexity and / or size, Object Orientation starts delivering very strong dividends. There are a few areas such as highly complex, algorithmic processing which might be better suited for functional programming. However a majority of the medium-to-large sized web applications are better served by OO. The empirical evidence at least bears out the fact that most of the most popular languages today (except C) are Object Oriented. However this still is a very very large class of languages which in them include C++, Java, PHP, Python, Ruby etc. The one area where some dynamic languages separate themselves from the others is in the notion of “everything is an object”, ie. primitives such as numbers, functions are all objects by themselves.

Business implications: OO code well designed and implemented allows for a substantial reduction in maintenance costs. When working with a team which is up the curve on OO, it is likely to lead to lower costs and time on inital coding as well. On the other hand, both training costs and skill requirements are higher for fully OO languages. If you are already using partialy OO / hybrid languages such as PHP, C++ or Java, and are convinced about OO, using fully OO languages such as Python or Ruby will help you leverage the OO capabilities even further.

Duck Typing : In very loose terms, duck typed languages do not require you to declare an explicit interface. You send an object a message (ie. invoke a function or access an attribute) and if it can respond to it, it will, and if it can’t it will result in an error. Duck typing is a specific typing system which is a subset of a broader system called Dynamic Typing, which often makes for an interesting debate with its counterpart – Static typing : Static and Dynamic Type checking in practice. For people well grounded in static typing alone, this can sometimes seem to be sacrilegious. I am convinced that duck typing makes writing code much much faster for two reasons – a) You now require to write fewer lines of code and b) You often don’t have to keep on regularly waiting for the compiler to do its work. There is also a substantial capability enhancement that dynamic typing makes to the language type system, which allow the frameworks to build dynamic types on the fly. This in turn offers the framework users many more capabilities than frameworks written in other languages. That is why it is nearly impossible to write frameworks like Rails or Django in Java (You can modify the class loaders and use byte code generation to generate the new types, but the compiler can’t see them so you cant use them). That is also why there is a lot of anticipation of using JRuby, Jython and Grails on the JVM since the languages underlying them (Ruby, Python and Groovy respectively) bring the dynamic typing capabilities to the JVM platform.

Business Implications :Writing code is much much faster. Maintenance depending upon the situation can sometimes be more or less difficult in case of dynamic typed languages. Refactoring is usually a lot more difficult in case of dynamically typed languages since the underlying type system is not able to infer sufficiently about the code to help the refactoring tools, as is possible in case of statically typed languages. It is my opinion that a skilled and trained development team using dynamic languages can generally substantially outperform another equally capable team using static languages. Insufficiently or poorly skilled development teams however can lead to very very different kind of pitfalls in these class of languages. In both cases the code becomes difficult to change or maintain due to a) cryptic code in case of dynamically typed languages and b) extremely large code bases in case of statically typed languages. Both are undesirable situations to be in but if I had to choose between one of the two, I would go for being in the cryptic mess since it is at least manageable by bringing in external skilled help.

Business Implications : Read on, they will get covered in the final roundup. They are large and they are positive.

When did these languages say Ruby and Python originate ? Most people are likely to be a little surprised if the answer is in the last millenium. Yet Guido von Rossum started working on Python in 1986 and Ruby was released in 1992. Python has been rather well known within the scientific community and perhaps a bit within the systems / OS utility programming communities for quite some time. However both languages grabbed a large mindshare only post 2005. A big reason for their popularity (especially in case of Ruby’s case) came from the popularity the frameworks which used them. Ruby on Rails for ruby and Django (to the best of my knowledge) for python. These frameworks combined the language capabilities with the learnings of good design practices for internet applications (eg MVC, declarative validations, simple ORM etc) into a simple usable package, which developers could take and build web applications quickly. There are examples of people having built simple web apps within a day and medium complexity apps in 1-3 weeks using these frameworks. The languages are the ingredients, the frameworks are the cooks – a great combination for serving great meals. Now you will find many such frameworks in these languages, including some which have better capabilities for building more sophisticated / complex applications eg. Merb and Pylons.

I am not too sure of how many people are exactly aware of the role of metaprogramming in the frameworks’ successes. I am willing to believe that but for metaprogramming, these frameworks simply would not have achieved anywhere close to the success they achieved. It is metaprogramming which takes the datastructures as defined by a developer and converts it into runtime code implicitly, saving the developer lots of time and effort. So even if most developers don’t actively write metaprograms, their lives are so much easier. Metaprogramming capabilities are also the reason why it is virtually impossible to write similar frameworks in Java. However if you are on the .NET or JVM environments, things are definitely looking encouraging with the possibilities to use IronPython or IronRuby on .NET or JRuby or Jython or Groovy+Grails on the JVM.

Business implications : If you are focused on scientific or desktop or highly algorithmic applications, where python especially is used extensively, you are likely to get benefits from these languages on their own merit alone. For web applications you will see the maximum benefits by using the web MVC frameworks along with the languages. I submit that on the whole you are likely to see very substantial reduction in development, enhancement and maintenance times – sweet music for any end user, investor or project manager.

Increased Business Agility

There is one more reason why I believe these languages are especially helpful. They help by increasing development agility to an extent where it now allows for the business to be more agile. You can get a first prototype version up in weeks, take it around to potential users, and gather feedback on the same. Incorporate elements of this feedback into the next release of working code quickly. The business benefits of such a scenario are tremendous. You might wonder that this is a process issue, so what does it have to do with a language selection. I would submit, that languages which allow changes to be made faster, help support this process in a far superior way. Another equally important facet is the superior risk management. Since you are able to build features with lower investments, you are able to get a series of customer feedbacks into your decision making process much faster. This helps being able to come up with a product that really meets the customer expectations much earlier. This happens by allowing the better features to come in earlier and also by allowing the lesser important or lesser relevant features to be decided to be deferred earlier. That’s precisely the reason why the dynamic languages have found a strong acceptance in the startup world. I believe the increasing agility which is often required in the startup world, is and will continue to be increasingly required of established enterprises. Precisely the reason why I believe these languages will continue to do better in the enterprise space as well. Finally, these languages make it relatively easier to tell your business sponsor – We will work with you on imprecise requirements rather than spending months on nailing down requirements which anyways are likely to change later. This has both a pro and a con especially for outsourcing situations. It is likely to allow for tremendous customer delight in terms of a vendor that works with him in such a flexible manner, yet it does introduce challenges in terms of how the commercials and management of the project are handled.

The reason I would like to especially point out increased business agility is because programmers don’t often visualise or evangelise it much, but when I wear a manager’s hat, it is perhaps the most compelling benefit of these languages.

Concluding

As I said earlier, there is no single universal language which is the best for all scenarios. There are some scenarios where using dynamic languages will not be helpful

A Treemap view of sales of programming language books by O’Reilly Media in 4Q2008. The size of a box represents the total sales of a book. The color represents the increase or decrease in sales compared to same quarter in 2007. Green = increase, bright green = big increase, red = decrease, bright red = large decrease. See full article at O’Reilly Radar for lots of interesting details.

When not to use these languages

You are building a simple / small application and don’t have the available skill sets. One exception to this is where you decide to use it in a simple application to allow yourself a non risky mechanism of building these skillsets.
Extremely High performance requirements. However please make sure that you really need the high performance capabilities of say a C, C++ or Java. In my experience 80% of developers like to believe that they are building highly performant applications where the maximum speed is a must have. Yet the top 10% of them are facing far far more critical performance requirements than the remainder. Unless you are convinced you are in the top 10%, you should certainly consider dynamic languages as an option. Moreover in case of most high performance requirements, these can sometimes be boiled down to a few inner loops / algorithms. Consider implementing the same in C, / Java or other .NET languages (depending upon the choice of your dynamic language interpreter implementation)
You have an architecture standard in place which does not allow using these languages. If you are convinced your applications are better served by using dynamic languages both from your individual application and an overall enterprise perspective, consider taking the feedback to your standards setting body to see if you can pilot a different approach. Also evaluate if the .NET or JVM versions can help you comply with the architecture guidelines.
You are unable to commit to the retraining requirements. While these languages are easy and powerful to use, leveraging that power can require some amount of retraining. If that does not fit your business plans, since the retraining effort could impact immediate and urgent requirements, that could be a reason to not use these languages. However in such situations do consider investing in building this skill sets before you get to another similar decision point.
• You need a very high levels of multithreadinging as opposed to multi processing support. While this is not a typical situation for web applications, you should be aware that most dynamic languages have some limitations in terms of multi threading support. This actually is not necessarily an issue with the language as with the implementation eg. the C implementation of python has the notorious Global Interpreter Lock which constrains you from being able to use more than a handful of threads per processes efficiently. However the same restriction is not present in Jython (the jvm implementation of python). This is likely to be an issue for a miniscule percentage of the web applications market for the primary reason that multi process / shared nothing architecture styles often work quite well for many web applications and they don’t really need multi threading.

First of all lets talk of the investment part. If you get into it in a paced approach, the investment may not be that great. Start with a team size of anywhere between 2-6 people (depending upon your organisation and project size). Think of 15 days of intensive training followed by a 2-6 months coming up the curve effort (more likely 2 than 6). Make sure your first project is not a critical one under tremendous business pressure. This can be subsequently followed by more people getting retrained as necessary. In the longer term it might actually help reduce your incremental investment, since it might be much easier to ramp up new programmers in Ruby or Python than say Java or C#.

Secondly lets look at the incrementally higher costs. You are likely to need people who are a little bit more capable in terms of understanding and debugging the same logic expressed in fewer lines of code (that sometimes can be a challenge) and then be able to modify and enhance the same. This may increase your testing and fixing costs in the earlier days. Finally while the fewer lines of code can make refactoring easier, you could find that your total refactoring costs are a little higher.

Now the returns part. I am convinced that the increased business agility is the strongest return in business terms. Immediately after that is the substantial reduction in development, enhancement and maintenance times. If neither of these benefits are appealing, when contrasted with some other issues that you might perceive, maybe considering dynamic languages in your context is not such a great idea.

One more factor that I would of course encourage you to evaluate from a business perspective are the implications for you if your competition (assuming it is not already using them) started using these languages. The implications would vary from case to case, but it could also help you decide how important this issue is for you.

About the author – Dhananjay Nene

Dhananjay is a Software Engineer with around 17 years of experience in the field. He is passionate about software engineering, programming, design and architecture. He did his post graduation from Indian Institute of Management, Ahmedabad, and has been involved in Senior Management positions and has managed team sizes in excess of 120 persons. His tech blog, and twitter stream are a must read for anybody interested in programming languages or development methodologies. Those interested in the person behind the tech can check out his general blog, and personal twitter stream. For more details, check out Dhananjay’s PuneTech wiki profile.

Related posts:

1. Talk Slides : Programming Language Selection
2. A brush with Functional Programming and Scala
3. Java : the perpetually undead language

1. Why REST ?This is a rather long post which provides a narrative of the history of web and service architectures eventually coming together into web services. It refers to many of the strengths that made web architectures so omnipresent, and to the uneasy coming together of the two architectures as web services. It details many REST characteristics, describes how REST provided a style by which the strengths of the web architectures could be retained even as the processing aspects of service architectures could be supported, and finally enumerates some of the benefits of REST.

   On the coming together of Web and Service oriented architectures.

   Clearly as WWW started getting used far more, people were only too keen to use it for much more than storing or retrieving documents. This led to the development of CGI and subsequently other dynamic web application technologies (eg. LAMP, J2EE etc.) which would allow us to use the web to ‘do something’. Since these were clearly offshoots of the SOA world, being mapped onto the WWW infrastructure, the characteristics of such dynamic applications often had a lot in common with SOA, and they started dropping many characteristics of the traditional static WWW. Thus was born the child of the world wide web and distributed service oriented architectures – web services. This led to newer SOA technologies such as WS-* and SOAP.

   Like the typical scenarios after the discovery of any highly profitable opportunity, the early rush was to leverage the opportunity and it was only a little later when the dust died down, that people started wondering if they had sacrificed something in the heat and dust of the moment. That stock taking resulted in the realisation, that some of the very basic characteristics of the extraordinarily successful internet technologies (FTP / SMTP / WWW) had been diluted, and even if such dilution still allowed immediate progress to have occurred, some of them would need to be corrected to be able to continue the explosive growth that had been seen so far. One such exercise in my opinion is the laying down of the REST architecture style.

   On the aspect that even though REST is not as feature rich as SOA, its strength is the simpler abstractions it employs

   I have generally found that simpler abstractions even though harder to deal with initially, often win in the long run. Notice the fact that the bare bones rendering functionality of HTML/WWW completely trounced the rich UI and application integration capabilities then available (eg. Windows/Java and DCOM/CORBA/RMI). This is not to suggest that the extra capabilities are not required. That is why Rich User Interfaces on WWW continue to be a dominant part of the internet technology wishlist. However the simpler, cleaner and minimalistic abstractions often are far more important than feature richness. A point I would want to make in favour of REST even as I admit that conventional SOA technologies are far more feature rich than REST.

2. REST is the DBMS of the InternetThis post reflects the thought that since REST effective allows one to GET, PUT, POST and DELETE resources, it is similar to being a database which exposes its tables to many applications to SELECT, INSERT, UPDATE and DELETE from. In this analogy, each media type is effectively a new table and the REST interface is primarily of the nature of allowing basic operations on a set of tables (resources).
   

   To summarise the exchange differently
   
   “If WS-* is the RPC of the Internet, REST is the DBMS of the internet“

   To expand on it a bit more :

   Traditional SOA based integration visualises different software artifacts being able to interact with each other through procedures or methods. REST effectively allows each software artifact to behave as a set of tables, and these artifacts talk to each other using SELECT, INSERT, UPDATE and DELETE. (or if you wish GET, PUT, POST, DELETE). And where exactly is is the business logic ? Is it in the stored procedures ? Not Quite. Its in the triggers.

3. Design Characteristics of REST / Resource Oriented Server Frameworks and ClientsThis post dwells into the many aspects of design of a REST or Resource Oriented Serverside Framework and attempts to enumerate a large number of their characteristics. One of the aspects it brings up is the role of the Controller. Since the a resource oriented interface primarily consists of basic primitive operations on resources, it suggests that the controller could either be merged with the Resource or support only basic operations and have a one to one relationship with a resource.
   

   This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

   Again to extend the analogy of a DBMS, it argues that instead of a lot of logic being in the controllers which are the entry points of the interface (stored procedures), the interface now changes to support basic operations on the resources (tables) and that the logic could perhaps be modeled in a separate class of handler functions (triggers).

   Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.

4. ReST : SOA, WOA or ROA ?This post dwells on how consistent REST is with Service, Web and Resource oriented architectures. It argues that REST could perhaps be argued to be SOA only in a most specific form, and that for all practical purposes REST should not be expressed as SOA.
   

   And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

   It brings out some potential inconsistencies in how WOA is currently not only portrayed to be a set of architectural elements in addition to REST but also as a extension of / future of SOA.

   My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

   …

   So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying.

   Finally it refers to ROA as the architecture (which actually is defined with REST as the basis) with which it is most consistent with.

   Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

   Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. REST is the DBMS of the Internet
3. ReST : SOA, WOA or ROA ?

Nice alphabet soup indeed. But what style of architecture does ReST (Representational State Transfer) correspond to.

Before we get into any definitional issues which are hugely referred to in case of such debates – I shall be referring to the currently available definitions and descriptions as available on Wikipedia viz. Service Oriented Architecture, Web Oriented Architecture and Resource Oriented Architecture.

Service Oriented Architecture

As per the definition on Wikipedia (emphasis is mine):

In computing, service-oriented architecture (SOA) provides methods for systems development and integration where systems package functionality as interoperable services. A SOA infrastructure allows different applications to exchange data with one another.

Service-orientation aims at a loose coupling of services with operating systems, programming languages and other technologies that underlie applications. SOA separates functions into distinct units, or services, which developers make accessible over a network in order that users can combine and reuse them in the production of applications. These services communicate with each other by passing data from one service to another, or by coordinating an activity between two or more services.

On the topic of Service Orientation, it further goes on to state

Service-orientation is a design paradigm that specifies the creation of automation logic in the form of services. It is applied as a strategic goal in developing a service-oriented architecture (SOA). Like other design paradigms, service-orientation provides a means of achieving a separation of concerns.

While REST does attempt to solve many similar goals as SOA, I believe there lies an important distinction. The essential focus of SOA is to separate functions or automation services. An example here is to separate an authentication service from authorisation, monitoring, logging etc. A SOA architecture that consists of a number of SOA services assembles such “functionalities” into one feature consistent whole. But is that how REST works ? Only in a very vague sense even if arguably so. REST standardises the functions ie. GET, PUT, POST and DELETE in case of HTTP connectors. It is arguable that REST could be used with a different set of functions, but even in that case the function set is likely to remain consistent. This is further supported by Roy Fielding’s post “It is Okay to use POST” in which he argues

Some people think that REST suggests not to use POST for updates. Search my dissertation and you won’t find any mention of CRUD or POST. The only mention of PUT is in regard to HTTP’s lack of write-back caching. The main reason for my lack of specificity is because the methods defined by HTTP are part of the Web’s architecture definition, not the REST architectural style. Specific method definitions (aside from the retrieval:resource duality of GET) simply don’t matter to the REST architectural style, so it is difficult to have a style discussion about them. The only thing REST requires of methods is that they be uniformly defined for all resources (i.e., so that intermediaries don’t have to know the resource type in order to understand the meaning of the request). As long as the method is being used according to its own definition, REST doesn’t have much to say about it.

So across the board the functions remain the same and the data types (or media types) change. Sounds familiar ? Thats like supporting SELECT, INSERT, UPDATE, DELETE on a broad range of tables each having a different schema. An argument I make in an earlier post REST is the DBMS of the Internet. Moreover Roy further goes on to elaborate that in “REST intermediaries don’t have to know the resource type in order to understand the meaning of the request” and that “method is being used according to its own definition”. Each of this further introduces constraints into traditional service orientation. And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

Web Oriented Architecture

This is not yet a particularly widely used term yet, but I did come across a reference to it on a recent article in InfoQ “REST is a Style – WOA is the architecture“. Looking it up on Wikipedia leads us to the same sources as referred to by InfoQ – articles written by Dion Hinchcliffe. Wikipedia states it as follows

Web Oriented Architecture (WOA) is a style of software architecture that extends service-oriented architecture (SOA) to web based applications, and is sometimes considered to be a light-weight version of SOA. WOA is also aimed at maximizing the browser and server interactions by use of technologies such as REST and POX.

But I just argued in the earlier section that REST is only a very very specific use case of SOA, whereas the statement above says WOA extends SOA to web based applications. If we were dealing with objects and not architectural styles here, an argument that X is a specific (constrained) sub type of Y and X extends Y would instantaneously be flagged off as a violation of Liskov’s Substitution Principle (LSP). So something isn’t quite right here and the whole situation does not add up to a consistent whole for me. I will leave it to the reader to decide whether there exists a flaw in my reasoning here or elsewhere. My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

To be fair I couldn’t quite find the same worlds (WOA extends SOA) in Hinchcliffe’s writings. Referring to one of them “What is WOA ? Its the future of Service Oriented Architecture (SOA)“, he refers to another of his post “Beating a Dead Horse: What’s a SOA Again? All About Service-Orientation…” which in turn states

It’s here that John Reynolds’ well-known SOA Elevator pitch comes tantalizingly close to capturing the essence:

SOA is an architectural style that encourages the creation of loosely coupled business services. Loosely coupled services that are interoperable and technology-agnostic enable business flexibility. An SOA solution consists of a composite set of business services that realize an end-to-end business process. Each service provides an interface-based service description to support flexible and dynamically re-configurable processes.

This business view is right on, and doesn’t mean business in a traditional, white-collar way. In this context, “business” means the actual functionality of the system, apart from technical details.

There we go again on services. But Business Services cannot be GET, PUT, POST, DELETE. I would emphasise again that REST does not expose business services – it exposes some very basic CRUD services. So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying. But it is likely that Hinchcliffe meant that WOA is built on REST and is similar to SOA in terms of the goals when he says WOA is the future of SOA. But honestly I could not quite figure out how exactly he would want to describe the relationship between WOA and SOA.

Resource Oriented Architecture

The wikipedia page states :

Resource Oriented Architecture (or, ROA) is a specific set of guidelines of an implementation of the REST architecture.

REST, or Representational State Transfer (see Roy Thomas Fielding’s Doctoral Thesis “Architectural Styles and the Design of Network-based Software Architectures”), describes a series of architectural constraints that exemplify how the web’s design emerged. Various concrete implementations of these ideas have been created throughout time, but it has been difficult to discuss the REST architecture without blurring the lines between actual software, or the architectural principals behind them.

Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

Related posts:

1. Service oriented REST architecture is an oxymoron
2. Musings on REST
3. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients

Struts, Django, Ruby on Rails. We’ve worked with these and many other similar frameworks. Some time back I started thinking of what would a completely new ground up REST / Resource oriented framework would look like (ground up to ensure it had no legacy design to deal with). Would such frameworks be similar to the ones dominantly used today ? What about the ecosystem that surrounds and interacts with them (client libraries) ? And finally what about the implications on the fine grained object model (assuming there is one) and its relationship with the resource model ? This post deals with some of the thoughts.

There are some specifics the post does not address and is agnostic about :

• Language : I shall be avoiding language issues as much as possible. Wherever I do bring in code constructs these may be assumed to be in Java (or pseudo-Java)
• Convention or Configuration : I think both are valid choices in their appropriate contexts, and I don’t specifically emphasise one over the other in this post

The frameworks mentioned above are not the only ones out there. There are many, and some actually are very REST specific eg. Apache CXF JAX-RS or Restlet. It would certainly be interesting to contrast my thoughts with these, but for reasons of insufficiently detailed knowledge about them, I shall choose to skip it (better to not make any statements than making incorrect ones).

I shall be assuming a HTTP connector with GET, PUT, POST and DELETE as the constant set of operations. These four operations shall be collectively referred to as Resource Operations.

We shall first start with the server side characteristics, and the term ROF shall refer to a Resource Oriented (server side) Framework

A ROF will have a resource oriented interface : Certainly not a profound statement, but it was important to lay that down upfront. So what is a Resource Oriented Interface. Given a particular resource, a Resource Oriented Software will support or consume end points which allow you GET, PUT, POST or DELETE the resource. There is one reason why this particular constraint is relaxed just a little bit. Modern browsers do not support all the four methods easily eg DELETE and make it just slightly hard to use the PUT method. Hence these methods can also be invoked by using a URI segment containing the method name eg. delete.

A ROF will have an abstraction to represent a resource as an end point : Again, that seems to be pretty obvious. But there is a reason why I make it explicitly. In many situations we see controllers acting as end points. To the extent a controller acts as an abstraction for a resource end point which essentially only has the resource operations as public methods, it would fit this requirement. However if I was using an Order as a resource and if I introduced an approve method on the OrderController that would not be consistent with this requirement. That would need to be modelled as an OrderApproval resource which may on successful completion, effect a state change on the Order resource to the status ‘approved’.

This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

Given consistent expectations of the Resource Operations these will actually be auto-magically implemented : Thats a bit of a turnaround from what I was just describing in the earlier paragraph. What I mean to suggest is that the class level methods I just referred to will be implemented within the framework. What the framework will allow are plugins to provide extended functionality at specific points. Thus a “public static Order Order.put(Order order)” method will be implicitly implemented by the framework. But before a put can be processed it needs to be validated. Thus the framework will allow the developer to plug in / override his own implementation for an Order.validate(Order order). There are multiple ways such plug-ins could be implemented. Depending upon the nature of abstraction, it could be an overridden method as I just described, or it could be a standalone method that is registered into the overall workflow (either through convention or configuration). The latter might be especially useful in case one wants to implement the functionality as stand alone methods or in case of functional programming languages. The plugin points provided could be framework specific. eg, One may want to validate a resource for consistency even at it is being read from the database. For the rest of the post I shall refer to these as plugins. In addition, framework will most certainly provide methods for for downstream handling of impact of PUT, POST or DELETE. This is covered in the next point. In case the framework chooses to not deal with persistence, it may choose to allow capabilities for integration with other persistence frameworks.

A ROF will provide capabilities to a developer to override or register methods to handle downstream impact of PUT, POST and DELETE : Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.

Note that the actual invocations to select, insert, update, delete the resource are *NOT* to be programmed by the developer. These are automatically handled by the framework. The developer essentially fills in the necessary logic to the plugin methods (eg. Order.validate) or handlers (eg. Order.onCreate)

A ROF will provide a mechanism to describe or map a resource abstraction to to the actual programming constructs : There are a number of ways this could be achieved. XML, YAML, DSL, Annotation – take your pick. Also the actual class could be defined (as in case of a POJO) and the resource characteristics mapped onto it, or the class may manifest itself at runtime based on metaprogramming around the metadata. Sample possibilities here are Hibernate like Resource-to-Object-to-Relation mapping (using either Annotations or XML) or a a completely metaprogrammed ActiveResource. One important aspect that the framework will need to cover is the situations where a Resource is a composite of many or partial underlying business objects. eg. an Order resource instance could theoretically span one Order instance and many OrderItem instances. Thus a one to one relationship between a resource and underlying business objects (or datastructures) is not assumed. What is assumed is that the framework will allow such relationships to be described or introspected.

A ROF will allow resources to be mapped onto URI or URI segments : This is too obvious an requirement to be explained and is mentioned here only for completeness.

A ROF will allow foreign keys across resources which manifest themselves as URIs to be mapped onto the underlying business object references : Resources refer to each other through URIs. The underlying business objects refer to each other through object references. Given the resource descriptions and URI mappings, the framework should allow for a transparent referencing/dereferencing between such URIs and the object references.

A ROF will allow factory methods for locating or allow injection of other resources / business objects : Within the handler functions, developers will need references to the associated resources or business objects. I say resources or business objects, since the developer may choose to interact with these at a coarse grained (resource) or fine grained (business object) level. The framework should allow the necessary support for such activities.

A ROF may provide additional support for typical aspects of lifecycle (eg. validation) : While I mentioned validate as a possible plugin function. However given the omnipresence of validations, the framework may provide additional support for such activities. Thus the framework may choose to automatically implement such capabilities using the resource descriptions.

A ROF may provide capabilities for domain specific extension of resource capabilities : Certain domains have standardised mechanisms of working with resources. As an example most banking systems based on the four eyes principle require approval activities. While this particular aspect is much tougher than it seems, a ROF may choose to allow extension of such capabilities using template like functions or mix ins. As an example in this situation, once an Order resource is defined, an OrderApproval resource will be automatically made available as will the GET and PUT methods on it (POST and DELETE in this particular case may not be relevant), as will the necessary and appropriate handler functions on OrderApproval.

A ROF will provide capabilities for automatically generating the resource representation from the resource and vice-versa : Resources manifest themselves in multiple possible formats eg. XML, JSON etc. An ROF will allow such transformations between the representation and the resource/business object instance automatically.

A ROF will provide capabilities for assembling more complex representations using templates : In many situations, especially when the representations are being composed for manual (browser based) consumption, additional resources may need to be pulled into a view. A ROF will allow for such assembly of resources to be composed into a final view using templates.

A ROF will allow for introduction of appropriate additional URIs in views using templates : Thanks to HATEOAS (I’ve really avoided it thus far ), the framework will need to allow some mechanism of describing what are the additional context specific URIs to be included in the final representation. The template logic should allow the developer to specify such URIs.

A ROF should allow for the resource / media-type descriptions to be shipped in band with the resource representation : Since REST allows media types to be auto discovered and auto described, the framework should allow for the metadata for such media types to be also presented to the client. While I think it is essential that such in band information should be conveyed on demand, the framework may also optionally support upfront interrogation for media types and their details, which will require such information to be shipped out of band as well. I am not aware of any specific standards around such interrogation APIs so the framework could implement a custom API for the same. The actual metadata could be represented using any of the typical appropriate standards such as RDFa, XML Schema snippets etc.

A ROF should optionally allow support for auto generation of bindings for clients : I really really cringe as I write this. I cringe because to me the great attraction of REST is the simplicity and the ease of introducing incremental integration. The client binding generation (especially if it is statically generated) flies in the face of many accepted lightweight design scenarios. However I think there are likely to be some situation where availability of such client side bindings would be helpful. When possible (eg. with dynamically typed, metaprogramming capable languages like Python or Ruby), such bindings should be dynamic. In such cases the client can automatically introspect the server side media types and make available the necessary client side objects on the fly. In cases where statically typed languages such as Java or Scala are used, the client side may choose to expose everything as generic datastructures (e.g trees of name value pairs) or may allow for generation and compilation of client side bindings. I have no specific thoughts around the API support needed on the client side, except that quite obviously this would include support for the resource construction, resource operations etc. and that they would allow the client to interact with the server using the underlying language constructs rather having to work at a raw HTTP level.

In addition to the characteristics described above, I suspect frameworks will have many other optional characteristics such as support for monitoring, auditing / logging, transaction management, object pooling etc. etc. However these are unlikely to be particularly interesting when focusing on the framework aspects especially from a resource oriented perspective, which is indeed the focus of this post.

Update : InfoQ covered this blog post here : Design Characteristics Of Resource Oriented Server Frameworks

Related posts:

1. Why REST ?
2. Musings on REST
3. REST is the DBMS of the Internet

The Twitter DM’s exchanged were as follows :

@sbidwai : is REST like object oriented implementation for services, where as SOA is procedural ? thinking loud..

@dnene : a slightly better analogy would be SOA is like invoking stored procedures, whereas REST is like invoking SQL on the table

@sbidwai : agreed in parts.. but most impl will hv much more than CRUD.. eg, twitter rest apis..*

@dnene : CRUD is the interface. To extend the analogy, logic is implemented as triggers not SPs. (My Opinion)

* CRUD in our shared vocabulary stands for Create, Read Update, Delete.

As I subconsciously wrote the last DM, it suddenly dawned on me that this was the one concise way to express how REST architectures would impact software designs.

To summarise the exchange differently

“If WS-* is the RPC of the Internet, REST is the DBMS of the internet”

To expand on it a bit more :

Traditional SOA based integration visualises different software artifacts being able to interact with each other through procedures or methods. REST effectively allows each software artifact to behave as a set of tables, and these artifacts talk to each other using SELECT, INSERT, UPDATE and DELETE. (or if you wish GET, PUT, POST, DELETE). And where exactly is is the business logic ? Is it in the stored procedures ? Not Quite. Its in the triggers.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. Musings on REST
3. Why REST ?

It is becoming apparent that even as it becomes popular, REST (Representational State Transfer) is not yet as well understood. This might seem a surprising statement, but a lot of us use REST thanks to many frameworks supporting REST like interfaces, have a sense of what REST like interfaces are like (even if such an understanding is not sufficiently accurate), and exercise our common sense in using such interfaces. Having said that, let me clarify that while the internet is full of documentation about the semantics of REST, its actually quite light on the rationale for REST (including Roy Fielding’s dissertation which is the reference document for REST). Thus I have had to intersperse REST semantics and historical narrative with some personal opinions. So treat this as a part opinion and feel free to question my thought where you think it does not make sense.

Audience

If you are a REST expert, you are likely to have figured out much of this any ways by now. If you would like to understand specific technical semantics about REST, again this may not be the best article to read. However if you are curious about REST and would like to read a perspective on why and how it makes sense read on.

Flow

I shall be meandering through a historical narrative in the first half before starting to make the points I wish to make in the second. Lot of the points I make in the first half are likely to be those you already are aware of. However these are being made to allow an immediate recall when you read the second half. It is quite important to have read the first half to understand the perspective I put together in the second.

Historical Narrative

FTP

I am sure you have used FTP (File Transfer Protocol) a few times even though nowhere as frequently as HTTP (HyperText Transfer Protocol). Let me quickly present some characteristics of FTP

• Given a FTP client, you can connect to any FTP server so long as you have a valid userid/password pair for the server (or anonymously if the server so supports).
• The home directory on connecting to a FTP server is typically your starting point. At this point typically you can execute the ‘ls’ or ‘List Directory’ command to list all the files and directories within the home directory.
• If a file interests you, you can get it by issuing a get command
• If a subdirectory interests you, you can further navigate into that directory by issuing a ‘cd’or ‘Change Directory’ (or often by double clicking on the directory in case of a graphical client).
• If you would like to add a file to the current directory you can issue a ‘put’ or ‘Upload’ command.
• While you have the flexibility to navigate from one directory to another, you soon realise that every file and directory is uniquely addressable by its fully qualified path (either absolute or relative) and you can refer to each file and directory by its path. You are also aware that a valid path will uniquely resolve to only one directory or file.
• At each stage as you navigate into a separate directory, the server allows you to retrieve the list of subdirectories and files within your current directory. It always shows you the current state of that directory. Thus even if you were to list the same directory twice, and someone else uploaded a new file or created a new subdirectory successfully between the two requests, you will see the reference to the new file / directory when you request for the listing the second time.
• At every stage you issue a command, the FTP client+server work together to service the request (or issue the appropriate error message as necessary), and then pretty much forget about what you did. In other word the server keeps no track of and shows no awareness of what you have done earlier in your session (though it does remember who you are primarily from a security perspective.
• To work with a FTP server using a command line client, you primarily need to understand the usage of four commands (verbs) post a successful connection. These are ‘ls’ to list the contents of a directory, ‘cd’ to navigate to a different directory, ‘get’ to download a file, and ‘put’ to upload a file.

So FTP allows us to upload and download files. But does it allow us to ‘do things’ ? Sure, so long as you combine it with a few more pieces in the puzzle. Let us say, we are back in the late 80s (prior to the invention of HTTP) and I want to send and a list of purchase orders collected by a local office to a central office for further processing. This requires the following elements to be added to the mix.

• A shared understanding of where the files will be uploaded, how they will be uniquely named, their specific file extensions (optionally) and the specific format of the file eg. Comma Separated or Lotus 1-2-3 or WordStar or WordPerfect (the popular application software of the day) including the positioning of the various fields in the file, and
• Preferably a daemon process on the central office computer (the FTP server) which regularly scans the directory, parses each file as it comes it, does the relevant processing on it, and generates the appropriate result files and places them in the appropriate directories using the shared understanding of the directory structure and the file naming convention to communicate back the results of the processing.

Ho Hum. This was all stuff you knew. But the reason I brought it up is that FTP and how it was leveraged then has a lot to do with the principles that govern REST as we shall later see.

RPC

Back then in the 80’s FTP wasn’t the only mechanism to transfer data between machines. One more of the many other options was RPC (Remote Procedure Call). It not only allowed you to transfer data across machines, it actually had built into itself, a contract to remotely execute software. Unlike FTP which merely transferred data (in well understood units called files), RPC allowed you to invoke remote procedures by supporting an ability to pass messages which included the message name and the values for all the parameters necessary to be supplied to the message. Unlike FTP which was meant to do data transfer across a network, RPC was geared to do things remotely.

A contrast between FTP and RPC

If the objective of network computing is to use the computers and networks to ‘do things’ one would assume that many more people would use RPC than FTP for the same. While RPC did get used as a technical substrate, at a business processing level FTP got used far more (eg. to send and remotely process the list of purchase orders). There are some important reasons that we need to understand here.

FTP required understanding of very few basic verbs (ls, cd, get, put). Thus the training required to understand FTP semantics was far less than that for RPC. This was partially due to the fact that RPC had a programmatic interface. To the best of my knowledge there are no widely used command line clients for human interaction with RPC services. In addition, each procedure required a set of semantic data (parameter) associated with it. This was no different than FTP which also required similar data to be shipped over the network. Turns out there were a few distinctions. First, the nature of design of RPC services often required combining application data with control data, and there was also often a sequential expectation due to the RPC business transaction being broken up into multiple RPC calls perhaps for the sake of efficiency. Moreover each time, new procedures were added or parameters added, these required programmatic changes. FTP on the other hand was simpler. In most cases the entire data (including some redundant data perhaps) was sent in one block (or file in FTP parlance). By dealing with a file as the least common denominator, the FTP stack decoupled itself from any application specific semantics. Moreover, depending upon the agreed upon format, the files could be edited at either end by by human actors using specific software such as plain text editors or word processors or spreadsheets. Moreover if the formats changed, wherever such files were being manually edited, no programmatic changes were required. Irrespective of the changes you made to the file formats, file processing software, the FTP stack itself did not change – it remained stable.

Less is more

A theme I shall come back to again is many a times less is more. FTP had far fewer training requirements (few basic verbs). FTP did not deal with parameter value formatting (though other pieces of software subsequently might have to). FTP was just so much easier to start working with. FTP did not actually preclude any of the capabilities of RPC from being introduced, it merely allowed this to be added subsequently as additional optional layers (or subsequent elements in the processing pipeline). Finally FTP allowed users to deal with the data in the units and the formats and the tools they understood the best – their day to day application software components and simply focused on only transferring files, while imposing only one requirement – each end should work with a file as a unit, and both ends should understand the file formats. By focusing on file as a unit, each business user could focus on the data he/she wanted to deal with in the format that was most appropriate (an analogy in REST would be a resource .. but I’m getting ahead of myself). And at the end of the day, by doing less, FTP ended up being much more popular and thus doing more.

Email

Other protocols widely used on the net were SMTP / POP which were used for email. Email eventually was considered the killer app for the internet. Similar to FTP email focused on the users getting to learn only a few basic verbs and exchanging the basic unit of data transfer (messages) using these verbs. Again, even though email itself didn’t get things done, it contributed far more heavily than RPC to getting things done, by having other manual or software actors at either end of the messages who did the necessary processing required.

WWW (World Wide Web)

While email was the killer app for the internet, the one that really brought it to masses was the world wide web which was based on the HTTP protocol. While HTTP could be used to ship documents of a variety of types (often classified by their mime-types), the defacto type of document used the HyperText Markup Language (HTML). Unlike FTP and email, this required the authors to understand a new language, but used a simple markup syntax to keep the learning curve to the minimum. It however introduced a very powerful element – the embedded hyperlink. While the earlier technologies supported a uniform identifier for each document / message, the hyperlink allowed references to other documents / messages to be embedded thus converting the document pool into a document web. We now had the ability to navigate from one document to another and such navigation retained the contextual relevance by embedding the hyperlinks. There were other enhancements as well such as introduction of more verbs (eg. POST and DELETE, the latter not really being supported by any of the browsers). Allow me to state the salient points of WWW despite the obvious duplications with some of the points I listed under FTP (for the sake of emphasis). Note that the scenario I describe below is primarily describing static web serving (except to the extent of file uploads) and does not address the presence of a dynamic web application.

• Given a web browser you can connect to any web server optionally using the appropriate authentication credentials.
• Typically the home page of the web server is your starting point. At this point you are shown the document which usually include embedded hyperlinks to other associated documents on the web server.
• You can get/view/download/save a document by clicking on a hyperlink pointing to the document.
• Some web servers may be configured to allow you to browse a directory. Clicking a hyperlink pointing to the directory allows you to see a directory listing which shows all the subdirectories and documents within the current directory. Each such subdirectory or document is also shown as a hyperlink to allow you to navigate to it.
• Some documents have an form including an embedded file field and a button which allow you to upload a new document onto the web server.
• Each document (and directory if directory browsing is turned on) has at least one identifier which uniquely identifies the document – the URL. It is feasible to directly navigate to the document if you are aware of the URL.
• Navigating to a different document often provides you with a different list of embedded hyperlinks which are contextually relevant to the document being viewed.
• At each stage the web server is not aware of any other information about you apart from your authentication credentials, and is not generally aware of your browsing history (except what may be stored for audit purposes on the web server logs).
• As a user the primary skills you need to grasp is the ability to enter a starting URL, and then being able to navigate from document to document by clicking on the hyperlinks. If you are uploading documents, you may in addition need to know how to specify a local file path and press the Submit button to upload the file.
• While HTML documents are the defacto default, the same capabilities can be used to serve any types of documents. The server usually identifies the document types by the registered mime types, and the browser may either render the document itself or call upon the necessary add-on plugin application to render the document based on the appropriate type or may in some cases simply save the document locally in case no such application is available for further processing.
• Usually but not necessarily the document name have the characteristics of a noun

Again the reason I listed these characteristics is that these have a tremendous commonality with those of REST (except that what I refer to as a document above may get referred to as a resource in REST parlance).

SOA : DCE, Tuxedo, CORBA, RMI

Even as the web was evolving other technologies which allowed for more sophisticated remote service invocations were being developed. Along with RPC, these were essentially different technical manifestations of Service Oriented Architecture (SOA) principles. While these are substantial developments in their own right, the relevant points to be made in the context of this article are :

• Each SOA service supported the ability to define a set of service semantics which included the service name, the parameters to the service, an ability to expose the metadata of such semantics, an ability to leverage such metadata and invoking such services either statically or dynamically from a remote client.
• Many services were usually expected to “do something” though quite often some services would simply return the requested data. Usually but not necessarily the services were identified by using ‘verbs’.
• Some of the SOA services allowed maintenance of a client state on the server, and allowed the server to do processing conditional on the client state.
• These technologies almost invariably required some kind of programmatic effort at both the client and the server end. Manual specification of the service parameters and manual invocation of the service was simply not a typical use case. Neither was a default rendering of the results easily available to be manually viewed by an end user.
• Unlike retrieving or storing a document, these services often were expected to have a far more complex functionality.

CGI, dynamic web applications and Web Services

Clearly as WWW started getting used far more, people were only too keen to use it for much more than storing or retrieving documents. This led to the development of CGI and subsequently other dynamic web application technologies (eg. LAMP, J2EE etc.) which would allow us to use the web to ‘do something’. Since these were clearly offshoots of the SOA world, being mapped onto the WWW infrastructure, the characteristics of such dynamic applications often had a lot in common with SOA, and they started dropping many characteristics of the traditional static WWW. Thus was born the child of the world wide web and distributed service oriented architectures – web services. This led to newer SOA technologies such as WS-* and SOAP.

Like the typical scenarios after the discovery of any highly profitable opportunity, the early rush was to leverage the opportunity and it was only a little later when the dust died down, that people started wondering if they had sacrificed something in the heat and dust of the moment. That stock taking resulted in the realisation, that some of the very basic characteristics of the extraordinarily successful internet technologies (FTP / SMTP / WWW) had been diluted, and even if such dilution still allowed immediate progress to have occurred, some of them would need to be corrected to be able to continue the explosive growth that had been seen so far. One such exercise in my opinion is the laying down of the REST architecture style.

REST Semantics

While REST brings back many of the characteristics that made internet so successful back to application design, it should be noted that many of these are not precluded by Web Services or SOA. However what are mandatory characteristics in REST are in some cases missing from but in most cases quite feasible to implement in traditional (non REST) web services by using additional best practices. Also note that each characteristic is not necessarily universally superior. So do evaluate it in your context to see if it makes sense. However before we get to the benefits of REST, a quick synopsis of REST technical characteristics might be in order.

While a full description of the REST technical aspects is completely beyond the scope of this post, I summarise these below. You might notice the strong parallels between the characteristics of FTP and WWW and those of REST even as REST adds a few more capabilities. The reason I portray them in the form below in a manner quite similar to the way I portrayed the characteristics of FTP and WWW is to emphasise that REST actually continues to leverage the same characteristics that made these technologies so popular and globally scalable, even as it just adds those few minimally necessary capabilities to achieve the same scalability for not just transferring documents or rendering pages but to ‘do something’. In other words it brings together the characteristics which made the internet technologies so popular and applies them to the inter application integration, component and service orientation, and application mashup scenarios to allow them to achieve similarly large adoption and to perform the tasks necessary in the given context (or ‘do someting’ as I have continuously referred to).

REST characteristics

• Resource and media types as the basic units : REST treats a resource as the basic unit of data transfer. Such resources could refer to anything in the particular context eg. a flight reservation, an invoice, a video etc.
• Unique resource identifiers :REST requires that each resource have at least one identifier which uniquely identifies that resource. This makes it easy to be able to bookmark resources or make them searchable.
• Each resource has at least one representation :Each resource can be expressed using a variety of representations. This could include HTML, XML, CSV, JSON etc.
• Each resource has often one default manually readable representation : In most cases but not all, each resource has a representation that can be manually consumed using a web browser. Such manual representations are often either XHTML representation with associated CSS, but could theoretically be some custom representation rendered by a delivered on demand custom javascript or Java applet. Note that this is not a requrement of REST but is a practice often followed.
• Each resource has a type. REST supports self describing media types : Each resource has a type (referred to as media type since REST refers to the resource web itself as hypermedia). The type influences the data semantics of the resource, and the type itself can be self documenting using a variety of technologies (eg. one possible way is to specify XML schema descriptors).
• Each resource representation optionally includes contextually relevant hyperlinks to other resources :This not only allows the clients to auto discover associated resources, but also allows the server to clearly communicate the contextually relevant links based on an application state.
• REST resources are indexable and search engine friendly : A consistent resource naming and representation allows for easy indexation and search engine integration.
• REST requires minimal starting point intelligence : Typically one only needs the initial URL for being able to integrate with a REST implementation. All newer resources are often dynamically discovered. Since these media types also document their own metadata, client agents can automatically discover more information about them. Thus media type metadata rather than being compiled into the REST client can be dealt with dynamically or by using code on demand agents for dealing with the appropriate media type (similar to browser plugins)
• REST encourages a uniform interface. :Typically this manifests itself by the minimal verbs being used to describe REST operations.When used with HTTP these are GET, PUT, POST and DELETE. This reduces the intelligence requirements on the client. Additionally clients may be capable of parsing metadata for the resources based on standard formats such as ATOM or XML schemas. The context specific intelligence required on the part of the client is no longer in the verbs it has to understand (method names) but is now in the resource types that it may need to manipulate. Thus if a client can deal with resource identification, resource representation, self descriptive messages and hypermedia, it can start dealing with REST.
• REST supports value addition by intermediate processors : REST supports the scenarios where intermediate processor units can provide additional value addition. These could include processors which provide caching support or those that provide resource enrichment capabilities.
• REST encourages usage of scalable practices :By precluding usage of conversational state and sequential assumptions, REST implementations tend to be easier to scale even as they compromise on efficiency at times (due to redundant data transfer or additional processing requirements)

REST benefits

Having described many of the REST characteristics the following could be interpreted as the benefits of adopting a REST style architecture.

• Default RenderingIn case of most REST implementations, you can quickly provide a default HTML rendering capability. Thus even as you provide a REST interface to allow inter application integration, customers of such an interface do not have to wait for building the programmatic capabilities for leveraging it, they can get started immediately by being able to manually view all the resources and their states manually and by navigating around the interface by using a plain web browser. This substantially reduces the entry barriers for your customers, and allows them to get more conversant with your media types even as they are still figuring out how to programatically leverage the capabilities.
• Self describing / auto discovery of media types and capabilitiesThe traditional web service semantics rely upon clear upfront documentation of media types, their schema and the API semantics. Thus the metadata about the service is often communicated ‘out of band’ from the actual service itself. This is required so that the clients can understand all the valid end points and service semantics up front before they can leverage the services. Not so with REST. Given an initial starting point, REST greatly encourages a contextual provision of the relevant additional interfaces (hyperlinks) as a part of the the document / resource data itself. Thus clients do not upfront need to be aware of all the end points (resource URIs) to be able to leverage the services. Moreover REST supports self describing media types as well. Thus the schema information for the resources can be shipped ‘in band’ with the resource representation itself. This allows for clients to discover new media types or changes to their schema and even allows the default rendering of the same without having to upgrade the programmatic components to leverage the newly discovered or modified media types / schemas. Finally the code on demand capabilities (these are optional) of REST can allow code to be downloaded to automatically parse or render such newly discovered or modified media types.
• Encourages scalability even at the cost of efficiencyAspects such as non maintenance of conversational state, greatly increase the scalability of REST applications even if they do incur a minor cost in efficiency (which can be due to repeated redundant communication of data elements, or additional processing requirements due to preclusion of conversation state). This makes it relatively easier to set up multiple servers as the demand for the REST capabilities increases. Having said that, let me quickly add a caveat that designing clustered applications even if with REST interfaces is not always trivial, and while REST makes it “easier to scale” that should not be confused with “easy to scale”.
• Resource / Data semantics are much easier to understand than Service semanticsTo put it differently an invoice structure is much easier to understand from a data perspective than an invoice processor API. This makes it easy for the clients. This often also makes it easier for the server side implementations. Service semantics often bring in issues of sequence, client state and other control information, most of which can be avoided using REST. Generally speaking expectations are simpler to lay down and meet when specifying resources rather than services.
• Clear naming and accessibility of each resource in your universeWeb services don’t mandate clear unique identifier for all your resources. Thus sometimes it is not possible to reach a particular resource except through a convoluted series of steps. In some cases some resources are inaccessible for ever. As an example, many online shopping experiences end with an invoice being shown, but I have often found it impossible to later on pull up that invoice that was earlier shown to me at the end of a transaction.
• Extensible resource types which are optionally dealt with by clientsNot only are resource types self describing, REST makes it easier to convey additional extensions to such resource types by using additional URIs within the resource representation as well. Thus even as a representation lays down a variety of field values (say for an invoice), there might be other associated resources which might either be optional or variable media types based on the context (eg. purchase order / quality report etc.) which can be easily referenced by simply including their URIs. Such additional information does not require the basic media type to be enhanced or by introducing attachments to the media types. These can be implemented as additional navigable out of band media types. Thus clients don’t have to deal with them, but they can do so easily when they choose to do so. Thus the client has a choice to not deal with the additional media types when they do not make sense in the client’s context.
• Search engine friendlinessWhile resource directories help for smaller scale integration (eg. Yahoo when it started off, attempted to categorise the web), such directories or registries are often found to be tough to scale beyond a particular threshold (thats why Yahoo or Google now provide entry points by allowing us to search through all the web resources). Consistent resource naming and representation make REST resources search engine friendly and allow additional entry points into a REST service based on search criteria. This makes location of newer resources far easier than what might be feasible through a resource registry especially on a large scale.
• Easer layeringWhile it is possible to add intermediate proxy services for enriching the capabilities of a REST implementation, it just makes it seem a lot easier to implement these as and when required when the underlying architecture is REST based. Thus while mashups can be readily implemented using both REST and traditional SOA implementations, I would submit that these are much easier to implement on REST based architectures.

I have used the word scalability above in the context of the ability to service the runtime demands of a larger number of clients. However REST helps makes your software artifacts become scalable in one more way. By providing a basic and minimal uniform interface requirements, REST allows your applications / services / components a low entry barrier path into being a participant in a broad web of similar others who all agree on the basic REST semantics. This substantially increases the potential number of clients to your services since they can leverage these services easily and with low entry barriers. While traditional SOA technologies attempt to provide the universal access to all possible consumers, REST with its emphasis on minimalism, simplicity and low entry barriers actually makes it practical. Similarly REST makes it easy for you to start consuming other services and mashing them up with others to service your clients (pun intended) quickly. Finally REST takes the the very characteristics that made document and message sharing so easy to use and popular (characteristics which are not necessarily found in all conventional SOA implementations) and combines them with the necessary elements to achieve transaction processing, application integration and mashups (use the web to ‘do something’) on a truly global scale even as it makes these capabilities easily available and cost effective to leverage.

Some concluding comments

While not directly relevant as REST rationale or REST benefits, I thought it might be useful to add a few more associated comments within the context of REST usage and adoption.

Simplicity and bottom up adoption

I must confess, my biases show up quite strongly in this paragraph (so feel free to treat this as a partially prejudiced statement). Simplicity is not per se a characteristic of REST. However it does stem from the nature of genesis of the competing options. While most internet technologies using an incremental, evolutionary approach, most SOA technologies have been designed by a committee. This is why the consulting and development budgets required to implement FTP / email / Web especially on a per utility basis are far different than those likely for implementing DCE, Tuxedo, CORBA and SOAP. Part of the reason is also due to the fact that most internet technology adoption is bottom up, while that of SOA often is top down. While top down may seem attractive, it may seem sobering to realise that most top down processes break down beyond a particular scale. Thats why free markets on the whole have trounced centrally planned economies (though some recent happenings do point to limitations of the same as well). Thats why internet scale simple inter application API integration and mashups took off even as intranet scale application integration was mired in budgeting, territorial, enterprise modeling and governance issues. Thats why the LAMP stack (eg. PHP) which hasn’t been particularly strong in the non web arena, is deeply entrenched in the web based application space. Sometimes it just is more productive to quickly implement a simpler technology and incrementally enhance it rather than attempting to cover all possibilities, options, and border conditions by putting a committee in place. At its very core, REST requires only incremental understanding of newer technologies, is easier to incrementally adopt and is less likely to get mired in organisational issues. Precisely the characteristics that FTP, EMail and WWW had.

Simpler abstractions win

I have generally found that simpler abstractions even though harder to deal with initially, often win in the long run. Notice the fact that the bare bones rendering functionality of HTML/WWW completely trounced the rich UI and application integration capabilities then available (eg. Windows/Java and DCOM/CORBA/RMI). This is not to suggest that the extra capabilities are not required. That is why Rich User Interfaces on WWW continue to be a dominant part of the internet technology wishlist. However the simpler, cleaner and minimalistic abstractions often are far more important than feature richness. A point I would want to make in favour of REST even as I admit that conventional SOA technologies are far more feature rich than REST.

REST is not SOA

I must confess, for a long time I believed REST was merely a specific usecase of SOA. However recent thoughts lead me to believe otherwise. There is indeed a reason for such potential confusion. REST based architectures and SOA may often attempt to service similar goals. To the extent of servicing such goals, REST may look like a substitutable component for other SOA technologies such as SOAP. However even as they attempt to meet similar goals, REST attempts to view at your architecture artifacts differently. REST encourages you to view and model your architecture as a set of resources rather than services. There are important implications of this not just in terms of the many benefits I describe above available under REST but also in terms of the design and architecture characteristics of the implementation. Treating REST as just another way to implement SOA sometimes encourages one to miss out on the subtleties. These however are beyond the scope of this post, and I intend to cover the same apart from the implications of REST on software design in my next post.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. REST is the DBMS of the Internet
3. ReST : SOA, WOA or ROA ?

The way I perceived it, Sun was going through similar difficulties. The hardware business was delivering dwindling margins post the dot com bust, and the software business was under threat from upstarts such as Linux which offered a substantially similar software stack at near zero licensing costs. One of the crown jewels asset Sun had in its stable was Java. And while it was (and continues to be) a wonderful asset, it just was incredibly difficult to make money off it. Now java hasn’t been open source exactly for most of its life, but it was a sufficiently open stack nevertheless to cause the same difficulties that open source software would in terms of monetisation.

There’s an excellent blog post written a year and half ago by Michel Bauwens, Can the experience economy be capitalist? which does refer to some of the underlying issues. I suggest you do read it, but would like to quote a few points from it below.

First of all, in the field of the immaterial, we are no longer dealing with scarce goods, but with marginal reproduction costs and non-rival goods. With such goods, sharing does not diminish the enjoyment of the good, since all parties retain their ability to use them. The emergence of peer production shows a new form of creating value, that is in fundamental aspects “outside the market”. Typically, in commons-based production we have a common pool, accessible to everyone (Linux, Wikipedia), around which an ecology of business can form to create and sell scarcities (usually services and experiences). In sharing-oriented production (YouTube, Google documents), we have proprietary platforms that enable and empower the sharing, but at the same time, sell the aggregated attention (a scarcity), to the advertising market. Finally, in the third crowdsourcing mode, companies try to integrate participation in their own value chain and framework.

So the good news is that indeed business is possible. But I would like the readers to entertain the following proposition, nl. That:

1) The creation of non-monetary value is exponential

2) The monetization of such value is linear

In other words, we have a growing discrepancy between the direct creation of use value through social relationships and collective intelligence (open platforms create near infinite value through the operations of the laws of Metcalfe and Reed), but only a fraction of that value can actually be captured by business and money. Innovation is becoming social and diffuse, an emergent property of the networks rather than an internal R & D affair within corporations; capital is becoming an a posteriori intervention in the realization of innovation, rather than a condition for its occurrence; more and more positive externalizations are created from the social field.

Quite eloquently argued here that open platforms create near infinite value which is difficult to be captured by business and money. I still remember companies such as Borland, HP, Sun, IBM making a fair amount of money through selling C/C++ compilers and IDEs (I remember it used to be almost $5000 per seat). However this was in the days when the community capability of creating similar competing software stacks was only in its infancy. No longer so today. Now communities, open standards, open processes and open source are fairly well established sources of delivering asymmetrically significant capabilities at a fraction of the cost, a fraction which is made even smaller when the same is incurred by a small motivated group of individuals or small highly agile companies. There in lies the difficulty. Companies are trained to and built to deliver linear and symmetric capabilities in the context of costs they incur. However they have a relatively poor handle on monetising in scenarios where small teams can deliver exponentially asymmetric capabilities.

The blog post I referred to above, identifies the right area to look at in this context – Identify what is scarce and Monetise the scarcity. So when any software has a sufficiently large utility and can be managed through open processes to be able to satisfy a globally substantial demand at a fraction of the cost (eg. apache httpd), that particular capability (eg. static and dynamic file serving over http) is no longer scarce enough for a commercially focused company to make money out off. Thats why the money shifts where the scarcity is – how to leverage such software and put it to good use. This is where the existing commercial successes around open source are based on eg. RedHat, JBoss, IBM, Oracle etc. They monetise the support, training, services and consulting around such software. In many ways Sun could be argued to be a victim of its own success. It not only went much further than any other large corporate in terms of creating open specifications, it also contributed substantially to sufficient knowledge dissemination and tooling to allow many other individual, SME and large corporates to be able to compete with Sun in these very areas that Sun could’ve monetised. Sun thus created its own competition for monetising the scarcity that got created around Java even as many others cried out hoarsely that Sun simply wasn’t open enough. Will there be an incentive for Oracle to reverse that somewhat ? I suspect that could be a logical option if it focuses on ensuring a good ROI for Sun’s Java investments and assets and more so to be able to make itself more capable in the space around Java than the competition (ie. IBM).

So is a large corporate making money off open source or open standards an oxymoron ? Viewed narrowly yes. Because while it is possible for a small team to make some money with adequate ROI off open source or open standards, it is unlikely to be feasible for large corporates. They really need to figure out what is the new scarcity that gets created off such initiatives, come to a judgment that the opportunity arising from such scarcity is large enough and worthy of their time, attention and investment, ensure that there is no current available capability of anyone else disruptively delivering exponentially asymmetric value in that space, and finally occupy that space before other large corporates do.

It does boil down to the fact that while individuals and small companies may find it easier to monetise open source and open standards, the same is going to be generally tough for large corporates. Or as Michel Bauwens said in the blog post I referred to (in my opinion perhaps a little exaggerated but not entirely off the mark)

For all of this, we will need new policies, major reforms and restructurations in our economy and society.

But one thing is sure: we will have markets, but the core logic of the emerging experience economy, operating as it does in the world of non-rival exchange, is unlikely to have capitalism as its core logic.

Related posts:

1. Java : the perpetually undead language
2. Stop making SOA complex
3. Most american graduates are unemployable because …

A few months ago I was working with using Python for complex data processing scenarios. This was stuff I was very well versed in Java, but was working hard at to make sure that when I wrote code in Python, it would at least pass as half decent pythonic code. My initial struggles with idiomatic Python, soon became a strong interest in functional programming. I initially considered it to probably have a niche applicability in traditional business transaction processing space and of primary importance in algorithmic code, and thats the state where I almost left it. Curiously even after reaching that conclusion I did persist in attempting to figure out how it could be used in a manner where it would be helpful to have functional programming constructs in typical transaction processing scenarios.

Is functional programming useful in traditional business transaction processing ?

One of the aspects of the struggle was that I didn’t want to have the functional programming code merely replace the existing code. I wanted to figure out if it would be possible to replace it in a manner where it would make a substantial positive difference, and that hunt proved a little elusive for a while. Another aspect was the fact that the traditional OO code depended so intensively on state, that to actually try to figure out how the same would work under Functional Programming was really requiring a substantial leap in terms of thought, a leap I had to really struggle with. Finally my language of choice was Python, a dynamically typed language, and some of the literature surrounding functional programming really assumed static typing, especially that related to patterns and constructs in Haskell (eg. Monads). Such patterns were difficult to apply since the underlying expected static typing support was simply at odds with a language like Python.

It was after persisting for a few weeks, that I started to see how one could apply these techniques effectively. I also started figuring out how to Pythonise constructs such as Monads. Once I was past the initial rather steep hurdles, the going became a lot easier, and the progress much much faster. In at least a subset of some typical financial transaction processing scenarios I did get some extremely encouraging (to myself I call them jaw dropping) results. The resultant code suddenly seemed smaller, far far more intuitive to understand and extremely easy to change.

Working with Object Oriented and Functional Programming simultaneously

I am not sure whether it was due to my abilities and experience or due to my inabilities to visualise, I decided that pure functional programming was simply not a choice in this context and thus one would need to combine traditional OO and FP constructs. Having reached that decision, it became a little easier to identify the areas which could be better leveraged by FP. However I believe I still have some ways to go before being able to be confident of writing OO and FP together the way each of them should be.

Scala

My focus on functional programming and fondness to Java also led me to investigate Scala. I made the mistake of actually reading the reference documentation on its web site first. My initial reactions were that it simply was a weird syntax language which was unlikely to be of help in reducing development time even when writing FP code. Thankfully, I did not give up at that stage and continued to read a lot more. I soon realised that type inferencing and other aspects (eg. no getter setters) really resulted in some substantial savings compared to traditional Java code (it probably cuts down the code size to half). And the syntax really wasn’t so cryptic after all, it just required a day to get used to. Moreover one had the entire arsenal of the the FP constructs along with traditional OO constructs to deploy. I did write some processing code and was very happy at the results I got. In this current state, I am excited about learning more about Scala as a language that would be an alternative to Java. It gives me some of the capabilities of more productive languages such as Python, while continuing to leverage the performance and stability of the JVM. Note that Scala’s compatibility with the JVM should be thought of differently from that of dynamic language implementations such as Jython on the JVM. Given Scala’s static typing capabilities, its runtime structures are far more similar to Java and thus are able to retain somewhat similar performance and memory footprint characteristics as traditional Java programs. I suspect the same is unlikely to be found for other dynamic languages on the JVM.

Concluding Thoughts

Prior to this entire exercise, in my mind there were two primary individual preferences to programming languages. For most scenarios, my preferred choice was Python especially due to the sheer speed with which one could write and maintain code, code that was really compact and readable. And if Python runtime performance was not going to be good enough, use Java. Coming out of this exercise, it does seem that there is a high potential that I might want to switch my preference to Scala instead of Java. I have already validated the positive benefits of functional programming and Scala coding in processing scenarios (those that do a lot of computation and processing in the background). I do need to validate the same for typical web based applications, but on the face of things, I believe I have already identified areas where FP can help in typical web application scenarios as well. And now I have two multi paradigm (OO and FP) languages to be able to leverage FP in alongside OO – Python and Scala.

This is something thats definitely going to be my interest area for the next few months and I intend to write many more specific posts on the topic. Should that be of any interest keep listening on this blog’s RSS feed and my twitter stream.

Related posts:

1. Why should I switch to Scala ?
2. Improve your web based software development and maintenance ROI with dynamic programming languages
3. Talk Slides : Programming Language Selection

Background

Per section 3.4.3 of Roy Fielding’s dissertation

The client-stateless-server style derives from client-server with the additional constraint that no session state is allowed on the server component. Each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is kept entirely on the client.

The same thought is further commented upon in section 5.1.3.

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 , such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.
…
Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server’s control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

The potential confusion areas

Given the clear prohibition of storing client state on server, there are some typical idioms which do get challenged as REST unfriendly. eg.

• Logging in to obtain a session token, which is subsequently a validation of an authentication status
• Binding additional attributes to such a session token on the server side with variables such as :
  • Computed and cached access privileges of the user
  • Conversational state eg. fields entered on page 1 of a two page form

The question there is are all he above scenarios inconsistent with REST ? Many articles seem to suggest that that would indeed be the case. as an example one of them is Ajax and REST, Part 1 Section: Violating the “stateless server” constraint

My thoughts

Clearly in the above example computed data such as access privileges, or for that matter user’s time zone information are available to the server even if these are not in the session so long as the server has the user id. So long as a user id is being supplied (or a proxy such as the session token id), such data being stored in the session is simply a server optimisation and thus does not violate REST guidelines since this is application state and not conversational state.

To the extent the data is storing conversational state such as the fact that the user has authenticated himself and the fields that the user may have entered in page 1 of a two page form, such is incompatible with REST guidelines, and if such practices are adopted the resultant design may not be termed fully REST compliant.

So the existence of a user token does not make the design REST “uncompliant”. Storing conversational state in a user session usually associated with such a token does.

One way to ask if a data is conversational or application, may be to ask oneself, that can the request be satisfied properly if it was accidentally routed to a different server in a situation where the cluster was configured for session affinity. In a REST compliant design, the other server will have a capability to still be able to service the request. However this requires the presumption that the cluster is configured for session affinity and session state is not available to other servers. Should one configure the cluster with shared and distributed sessions, the cluster will be able to service the requests even when the APIs were not REST compliant.

The one thing that does leave me a little uneasy is that to be fully REST compliant one will need to always supply a userid / password with each API call instead of a server token. That does have implications on security, implications that I am not entirely comfortable with.

Update :Surya in a comment in earlier post refers to the fact that authentication need not be done through the URI but through the headers or through protocols such as OAuth. So what that leaves me with is the assessment that existence of a token or a session object is not a violation of REST but storage of conversational state in the same is.

Related posts:

1. Fomenting unREST : Is RESTfulness a semantics game ? Why does REST require statelessness ?
2. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
3. Why REST ?

Update : Further breaking news on the topic can be found here.

No related posts.

Background article (not exactly same but on a similar topic) : Improve your web based software development and maintenance ROI with dynamic programming languages

Talk Announcement : Seminar: Strengths and weaknesses of various programming languages – 28th March

Related posts:

1. Improve your web based software development and maintenance ROI with dynamic programming languages
2. Java : the perpetually undead language
3. A brush with Functional Programming and Scala