No related posts.

What perhaps hasn’t been clearly stated but is important is that that the standards for employability here are set by the employing organisation based on its context and preferences which may be different from standards required by many other organisations and I would submit that the statement ought to be looked at in that context as well.

To quote from the article :

The official wanted to know why HCL, a $2.5 billion (revenue) company with more than 3,000 people across 21 offices in 15 states, wasn’t hiring more people in his state. Vineet’s short answer: because most American college grads are “unemployable.” (In fairness to HCL, the company recently announced plans to open a delivery center in another state, North Carolina, and invest $3.2 million and hire more than 500 employees there over the next five years under a Job Development Investment Grant.)

Many American grads looking to enter the tech field are preoccupied with getting rich, Vineet said. They’re far less inclined than students from developing countries like India, China, Brazil, South Africa, and Ireland to spend their time learning the “boring” details of tech process, methodology, and tools–ITIL, Six Sigma, and the like.

As a result, Vineet said, most Americans are just too expensive to train–despite the Indian IT industry’s reputation for having the most exhaustive boot camps in the world. To some extent, he said, students from other highly developed countries fall into the same rut.

So why are some graduates are unemployable ?

Some of these are folks who are working on creating tons of new languages (Java, C#, Scala, Closure, Erlang, Python etc.), Operating Systems (Linux, Mac and Windows), frameworks for web applications, clustering, fault tolerance and scalability, schemaless and distributed databases to ensure availability and fault tolerance at a wide scale, competing messaging architectures that each service a particular problem differently etc. etc. Would you believe it some have set up clusters of  hundreds of thousands of machines and service search requests very rapidly. And some others are working on creating innovative and disruptive models in social networking, application integration, peer to peer networking etc. Many are working out next generation mobile technologies including building (not building on) android, iphone OS and the palm (whatever OS it has). And so many in their spare time are spending a whole bunch of time creating open source software, blogging and micro blogging about all the work they have done and sharing it with the wide world so that they can learn off it. And if these are consumer and technology stories, there are a whole bunch of people building critical business infrastructure architectures and frameworks and solutions as well. Even after you complete reading this post and come back to it a month from now, I shall still be busy trying to figure out what exactly India has been able to deliver that can challenge this. Of course let’s not miss out on the fact that most of these activities are conducted by fairly small sized teams. (I am aware some of the examples I quote could have non-american heritage, but that per se is not likely to detract from the issue)

Let’s call a spade a spade. When you have large or very large software construction and maintenance contracts, there are multiple ways to deal with it. In my experience, many Indian companiess have honed to a fine art the process of recruiting, deploying and juggling large armies of programmers to service such expectations (not all Indian companies are based on the same model). Many companies have indeed managed to acquire, retain and expand customer engagements through a combination of technology and business innovation. And they have done a good job of it in the context they’ve defined for themselves. In my limited understanding and experience it is not technology innovation, creativity or extraordinary technical prowess that is at the top of the list of skills that get deployed (though these are indeed found in sufficient levels across the board and some of the prowess can be impressive) – it’s the clockwork project management and business methodologies, techniques and innovation (and even these do result in projects with delays) with their reliance on “another brick in the wall” that gets the customer serviced.

So, if true, the “employable” was perhaps in that context.  If so based on the earlier two paragraphs, I submit it reflects positively and negatively on the employing organisation. Perhaps even more so than the graduates themselves. I do wonder if “incompatible” would have been a better choice of word. And I wonder what it reflected more poorly upon, as well who should be feeling more sad about it.

Disclaimer : These views are mine, mine alone and should not be puported to be shared by any other people or companies I’ve been associated with in the past, present or the future.

No related posts.

Here’s what my preliminary look into Opera Unite suggested :

• It has a web server bundled into the browser.
• It allows you to write (what would be in current parlance be called) serverside apps using javascript which are hosted by the web server.
• Sample applications include photosharing apps to share photos on your desktop or fridge which allows internet users to post notes to you.
• It allows these apps to be accessed across a router on a dynamic IP using a subdomain on a centralised operaunite service ..operaunite.com (not sure yet whether across a firewall as well but seems so)
• It allows these applications to be shared / published using the config.xml file (similar to google widgets ?). There’s also a central opera directory for the same, but I don’t think sharing is restricted to that service alone (apple are you listening ?)
• These applications can be further installed by other users of Opera Unite (which is the web server service running inside the Opera browser).

So why is this such a big deal ? Without going into any further elaboration let us just imagine user’s used it for following (for desktop-to-desktop or peer-to-peer) communication :

• Send mails to each other – disintermediates email services
• Send short burst messages to each other – potentially disintermediates twitter
• Share files with each other – disintermediates ftp servers and shares characteristics with gnutella, kazaa etc.
• Share resume, product profiles etc. – disintermediates traditional static web hosts
• Build networks of other interested users – disintermediates linkedin, facebook

The constraint it introduces is that there is no global list of user ids to search from – you need to know the URL upfront (like the facebook vanity URL).

The possibilities are many. As are the interesting uses this model could be put to. And the characteristics of improved privacy, data ownership and control (including fine grained access control or selectivity). And there does remain a potential for malicious actors and virus writers to ride on popular apps to exploit vulnerabilities to tend to their nefarious needs.

As I put it differently in another tweet – this opens up the possibility for a google wave without the google in it.

References :

• Taking the web into our hands one computer at a time : An introductory writeup
• An introduction to Opera Unite : A get started guide
• Opera Unite developer’s primer : A primer for writing server side applications

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients

Introduction

Programing language selection is often a topic that elicits a lot of excitement, debate and often a bit of acrimony as well. There is no universally superior programming language that one can recommend, so I tend to generally disregard most language opinions which say ‘X language is the best’, without specifying the context under which it is superior. Finally most language debates often deal with the technical issues and not the ROI issues. Hopefully I shall be able to address this topic without being guilty of any of these problems.

The range of languages that fall under Dynamic Programming Languages category is rather extensive. My experience is primarily limited to Python and to a lesser extent PHP, Ruby, Javascript, and Groovy. For the rest of this article, I shall be primarily referring to Python or Ruby when I use the word dynamic languages, though many of the references may continue to be applicable and relevant for a number of other dynamic programming languages.

As I describe the technical characteristics, I shall also continue to attempt to address the business aspects as well, so you might find this article at a little techno-business level. Assuming I am able to excite their interest, the tech guys would not find sufficient technical details and would be hungry to hunt for more, and while the business guys would get a little teased with the possibilities, they will not quite get the ROI served in the traditionally formatted excel spreadsheets. Being aware of that, I continue down this path with a feeling that this perhaps will be the most appropriate level for me to abstract this article to.

Object Oriented : Many dynamic languages support full object orientation. There are many who don’t necessarily buy the benefits of Object Orientation, but it is my strong belief, that once a piece of software grows beyond a certain threshold of complexity and / or size, Object Orientation starts delivering very strong dividends. There are a few areas such as highly complex, algorithmic processing which might be better suited for functional programming. However a majority of the medium-to-large sized web applications are better served by OO. The empirical evidence at least bears out the fact that most of the most popular languages today (except C) are Object Oriented. However this still is a very very large class of languages which in them include C++, Java, PHP, Python, Ruby etc. The one area where some dynamic languages separate themselves from the others is in the notion of “everything is an object”, ie. primitives such as numbers, functions are all objects by themselves.

Business implications: OO code well designed and implemented allows for a substantial reduction in maintenance costs. When working with a team which is up the curve on OO, it is likely to lead to lower costs and time on inital coding as well. On the other hand, both training costs and skill requirements are higher for fully OO languages. If you are already using partialy OO / hybrid languages such as PHP, C++ or Java, and are convinced about OO, using fully OO languages such as Python or Ruby will help you leverage the OO capabilities even further.

Duck Typing : In very loose terms, duck typed languages do not require you to declare an explicit interface. You send an object a message (ie. invoke a function or access an attribute) and if it can respond to it, it will, and if it can’t it will result in an error. Duck typing is a specific typing system which is a subset of a broader system called Dynamic Typing, which often makes for an interesting debate with its counterpart – Static typing : Static and Dynamic Type checking in practice. For people well grounded in static typing alone, this can sometimes seem to be sacrilegious. I am convinced that duck typing makes writing code much much faster for two reasons – a) You now require to write fewer lines of code and b) You often don’t have to keep on regularly waiting for the compiler to do its work. There is also a substantial capability enhancement that dynamic typing makes to the language type system, which allow the frameworks to build dynamic types on the fly. This in turn offers the framework users many more capabilities than frameworks written in other languages. That is why it is nearly impossible to write frameworks like Rails or Django in Java (You can modify the class loaders and use byte code generation to generate the new types, but the compiler can’t see them so you cant use them). That is also why there is a lot of anticipation of using JRuby, Jython and Grails on the JVM since the languages underlying them (Ruby, Python and Groovy respectively) bring the dynamic typing capabilities to the JVM platform.

Business Implications :Writing code is much much faster. Maintenance depending upon the situation can sometimes be more or less difficult in case of dynamic typed languages. Refactoring is usually a lot more difficult in case of dynamically typed languages since the underlying type system is not able to infer sufficiently about the code to help the refactoring tools, as is possible in case of statically typed languages. It is my opinion that a skilled and trained development team using dynamic languages can generally substantially outperform another equally capable team using static languages. Insufficiently or poorly skilled development teams however can lead to very very different kind of pitfalls in these class of languages. In both cases the code becomes difficult to change or maintain due to a) cryptic code in case of dynamically typed languages and b) extremely large code bases in case of statically typed languages. Both are undesirable situations to be in but if I had to choose between one of the two, I would go for being in the cryptic mess since it is at least manageable by bringing in external skilled help.

Business Implications : Read on, they will get covered in the final roundup. They are large and they are positive.

When did these languages say Ruby and Python originate ? Most people are likely to be a little surprised if the answer is in the last millenium. Yet Guido von Rossum started working on Python in 1986 and Ruby was released in 1992. Python has been rather well known within the scientific community and perhaps a bit within the systems / OS utility programming communities for quite some time. However both languages grabbed a large mindshare only post 2005. A big reason for their popularity (especially in case of Ruby’s case) came from the popularity the frameworks which used them. Ruby on Rails for ruby and Django (to the best of my knowledge) for python. These frameworks combined the language capabilities with the learnings of good design practices for internet applications (eg MVC, declarative validations, simple ORM etc) into a simple usable package, which developers could take and build web applications quickly. There are examples of people having built simple web apps within a day and medium complexity apps in 1-3 weeks using these frameworks. The languages are the ingredients, the frameworks are the cooks – a great combination for serving great meals. Now you will find many such frameworks in these languages, including some which have better capabilities for building more sophisticated / complex applications eg. Merb and Pylons.

I am not too sure of how many people are exactly aware of the role of metaprogramming in the frameworks’ successes. I am willing to believe that but for metaprogramming, these frameworks simply would not have achieved anywhere close to the success they achieved. It is metaprogramming which takes the datastructures as defined by a developer and converts it into runtime code implicitly, saving the developer lots of time and effort. So even if most developers don’t actively write metaprograms, their lives are so much easier. Metaprogramming capabilities are also the reason why it is virtually impossible to write similar frameworks in Java. However if you are on the .NET or JVM environments, things are definitely looking encouraging with the possibilities to use IronPython or IronRuby on .NET or JRuby or Jython or Groovy+Grails on the JVM.

Business implications : If you are focused on scientific or desktop or highly algorithmic applications, where python especially is used extensively, you are likely to get benefits from these languages on their own merit alone. For web applications you will see the maximum benefits by using the web MVC frameworks along with the languages. I submit that on the whole you are likely to see very substantial reduction in development, enhancement and maintenance times – sweet music for any end user, investor or project manager.

Increased Business Agility

There is one more reason why I believe these languages are especially helpful. They help by increasing development agility to an extent where it now allows for the business to be more agile. You can get a first prototype version up in weeks, take it around to potential users, and gather feedback on the same. Incorporate elements of this feedback into the next release of working code quickly. The business benefits of such a scenario are tremendous. You might wonder that this is a process issue, so what does it have to do with a language selection. I would submit, that languages which allow changes to be made faster, help support this process in a far superior way. Another equally important facet is the superior risk management. Since you are able to build features with lower investments, you are able to get a series of customer feedbacks into your decision making process much faster. This helps being able to come up with a product that really meets the customer expectations much earlier. This happens by allowing the better features to come in earlier and also by allowing the lesser important or lesser relevant features to be decided to be deferred earlier. That’s precisely the reason why the dynamic languages have found a strong acceptance in the startup world. I believe the increasing agility which is often required in the startup world, is and will continue to be increasingly required of established enterprises. Precisely the reason why I believe these languages will continue to do better in the enterprise space as well. Finally, these languages make it relatively easier to tell your business sponsor – We will work with you on imprecise requirements rather than spending months on nailing down requirements which anyways are likely to change later. This has both a pro and a con especially for outsourcing situations. It is likely to allow for tremendous customer delight in terms of a vendor that works with him in such a flexible manner, yet it does introduce challenges in terms of how the commercials and management of the project are handled.

The reason I would like to especially point out increased business agility is because programmers don’t often visualise or evangelise it much, but when I wear a manager’s hat, it is perhaps the most compelling benefit of these languages.

Concluding

As I said earlier, there is no single universal language which is the best for all scenarios. There are some scenarios where using dynamic languages will not be helpful

A Treemap view of sales of programming language books by O’Reilly Media in 4Q2008. The size of a box represents the total sales of a book. The color represents the increase or decrease in sales compared to same quarter in 2007. Green = increase, bright green = big increase, red = decrease, bright red = large decrease. See full article at O’Reilly Radar for lots of interesting details.

When not to use these languages

You are building a simple / small application and don’t have the available skill sets. One exception to this is where you decide to use it in a simple application to allow yourself a non risky mechanism of building these skillsets.
Extremely High performance requirements. However please make sure that you really need the high performance capabilities of say a C, C++ or Java. In my experience 80% of developers like to believe that they are building highly performant applications where the maximum speed is a must have. Yet the top 10% of them are facing far far more critical performance requirements than the remainder. Unless you are convinced you are in the top 10%, you should certainly consider dynamic languages as an option. Moreover in case of most high performance requirements, these can sometimes be boiled down to a few inner loops / algorithms. Consider implementing the same in C, / Java or other .NET languages (depending upon the choice of your dynamic language interpreter implementation)
You have an architecture standard in place which does not allow using these languages. If you are convinced your applications are better served by using dynamic languages both from your individual application and an overall enterprise perspective, consider taking the feedback to your standards setting body to see if you can pilot a different approach. Also evaluate if the .NET or JVM versions can help you comply with the architecture guidelines.
You are unable to commit to the retraining requirements. While these languages are easy and powerful to use, leveraging that power can require some amount of retraining. If that does not fit your business plans, since the retraining effort could impact immediate and urgent requirements, that could be a reason to not use these languages. However in such situations do consider investing in building this skill sets before you get to another similar decision point.
• You need a very high levels of multithreadinging as opposed to multi processing support. While this is not a typical situation for web applications, you should be aware that most dynamic languages have some limitations in terms of multi threading support. This actually is not necessarily an issue with the language as with the implementation eg. the C implementation of python has the notorious Global Interpreter Lock which constrains you from being able to use more than a handful of threads per processes efficiently. However the same restriction is not present in Jython (the jvm implementation of python). This is likely to be an issue for a miniscule percentage of the web applications market for the primary reason that multi process / shared nothing architecture styles often work quite well for many web applications and they don’t really need multi threading.

First of all lets talk of the investment part. If you get into it in a paced approach, the investment may not be that great. Start with a team size of anywhere between 2-6 people (depending upon your organisation and project size). Think of 15 days of intensive training followed by a 2-6 months coming up the curve effort (more likely 2 than 6). Make sure your first project is not a critical one under tremendous business pressure. This can be subsequently followed by more people getting retrained as necessary. In the longer term it might actually help reduce your incremental investment, since it might be much easier to ramp up new programmers in Ruby or Python than say Java or C#.

Secondly lets look at the incrementally higher costs. You are likely to need people who are a little bit more capable in terms of understanding and debugging the same logic expressed in fewer lines of code (that sometimes can be a challenge) and then be able to modify and enhance the same. This may increase your testing and fixing costs in the earlier days. Finally while the fewer lines of code can make refactoring easier, you could find that your total refactoring costs are a little higher.

Now the returns part. I am convinced that the increased business agility is the strongest return in business terms. Immediately after that is the substantial reduction in development, enhancement and maintenance times. If neither of these benefits are appealing, when contrasted with some other issues that you might perceive, maybe considering dynamic languages in your context is not such a great idea.

One more factor that I would of course encourage you to evaluate from a business perspective are the implications for you if your competition (assuming it is not already using them) started using these languages. The implications would vary from case to case, but it could also help you decide how important this issue is for you.

About the author – Dhananjay Nene

Dhananjay is a Software Engineer with around 17 years of experience in the field. He is passionate about software engineering, programming, design and architecture. He did his post graduation from Indian Institute of Management, Ahmedabad, and has been involved in Senior Management positions and has managed team sizes in excess of 120 persons. His tech blog, and twitter stream are a must read for anybody interested in programming languages or development methodologies. Those interested in the person behind the tech can check out his general blog, and personal twitter stream. For more details, check out Dhananjay’s PuneTech wiki profile.

Related posts:

1. Talk Slides : Programming Language Selection
2. A brush with Functional Programming and Scala
3. Commentary on Python from a Java programming perspective

1. Why REST ?This is a rather long post which provides a narrative of the history of web and service architectures eventually coming together into web services. It refers to many of the strengths that made web architectures so omnipresent, and to the uneasy coming together of the two architectures as web services. It details many REST characteristics, describes how REST provided a style by which the strengths of the web architectures could be retained even as the processing aspects of service architectures could be supported, and finally enumerates some of the benefits of REST.

   On the coming together of Web and Service oriented architectures.

   Clearly as WWW started getting used far more, people were only too keen to use it for much more than storing or retrieving documents. This led to the development of CGI and subsequently other dynamic web application technologies (eg. LAMP, J2EE etc.) which would allow us to use the web to ‘do something’. Since these were clearly offshoots of the SOA world, being mapped onto the WWW infrastructure, the characteristics of such dynamic applications often had a lot in common with SOA, and they started dropping many characteristics of the traditional static WWW. Thus was born the child of the world wide web and distributed service oriented architectures – web services. This led to newer SOA technologies such as WS-* and SOAP.

   Like the typical scenarios after the discovery of any highly profitable opportunity, the early rush was to leverage the opportunity and it was only a little later when the dust died down, that people started wondering if they had sacrificed something in the heat and dust of the moment. That stock taking resulted in the realisation, that some of the very basic characteristics of the extraordinarily successful internet technologies (FTP / SMTP / WWW) had been diluted, and even if such dilution still allowed immediate progress to have occurred, some of them would need to be corrected to be able to continue the explosive growth that had been seen so far. One such exercise in my opinion is the laying down of the REST architecture style.

   On the aspect that even though REST is not as feature rich as SOA, its strength is the simpler abstractions it employs

   I have generally found that simpler abstractions even though harder to deal with initially, often win in the long run. Notice the fact that the bare bones rendering functionality of HTML/WWW completely trounced the rich UI and application integration capabilities then available (eg. Windows/Java and DCOM/CORBA/RMI). This is not to suggest that the extra capabilities are not required. That is why Rich User Interfaces on WWW continue to be a dominant part of the internet technology wishlist. However the simpler, cleaner and minimalistic abstractions often are far more important than feature richness. A point I would want to make in favour of REST even as I admit that conventional SOA technologies are far more feature rich than REST.

2. REST is the DBMS of the InternetThis post reflects the thought that since REST effective allows one to GET, PUT, POST and DELETE resources, it is similar to being a database which exposes its tables to many applications to SELECT, INSERT, UPDATE and DELETE from. In this analogy, each media type is effectively a new table and the REST interface is primarily of the nature of allowing basic operations on a set of tables (resources).
   

   To summarise the exchange differently
   
   “If WS-* is the RPC of the Internet, REST is the DBMS of the internet“

   To expand on it a bit more :

   Traditional SOA based integration visualises different software artifacts being able to interact with each other through procedures or methods. REST effectively allows each software artifact to behave as a set of tables, and these artifacts talk to each other using SELECT, INSERT, UPDATE and DELETE. (or if you wish GET, PUT, POST, DELETE). And where exactly is is the business logic ? Is it in the stored procedures ? Not Quite. Its in the triggers.

3. Design Characteristics of REST / Resource Oriented Server Frameworks and ClientsThis post dwells into the many aspects of design of a REST or Resource Oriented Serverside Framework and attempts to enumerate a large number of their characteristics. One of the aspects it brings up is the role of the Controller. Since the a resource oriented interface primarily consists of basic primitive operations on resources, it suggests that the controller could either be merged with the Resource or support only basic operations and have a one to one relationship with a resource.
   

   This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

   Again to extend the analogy of a DBMS, it argues that instead of a lot of logic being in the controllers which are the entry points of the interface (stored procedures), the interface now changes to support basic operations on the resources (tables) and that the logic could perhaps be modeled in a separate class of handler functions (triggers).

   Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.

4. ReST : SOA, WOA or ROA ?This post dwells on how consistent REST is with Service, Web and Resource oriented architectures. It argues that REST could perhaps be argued to be SOA only in a most specific form, and that for all practical purposes REST should not be expressed as SOA.
   

   And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

   It brings out some potential inconsistencies in how WOA is currently not only portrayed to be a set of architectural elements in addition to REST but also as a extension of / future of SOA.

   My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

   …

   So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying.

   Finally it refers to ROA as the architecture (which actually is defined with REST as the basis) with which it is most consistent with.

   Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

   Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. ReST : SOA, WOA or ROA ?
3. REST is the DBMS of the Internet

Nice alphabet soup indeed. But what style of architecture does ReST (Representational State Transfer) correspond to.

Before we get into any definitional issues which are hugely referred to in case of such debates – I shall be referring to the currently available definitions and descriptions as available on Wikipedia viz. Service Oriented Architecture, Web Oriented Architecture and Resource Oriented Architecture.

Service Oriented Architecture

As per the definition on Wikipedia (emphasis is mine):

In computing, service-oriented architecture (SOA) provides methods for systems development and integration where systems package functionality as interoperable services. A SOA infrastructure allows different applications to exchange data with one another.

Service-orientation aims at a loose coupling of services with operating systems, programming languages and other technologies that underlie applications. SOA separates functions into distinct units, or services, which developers make accessible over a network in order that users can combine and reuse them in the production of applications. These services communicate with each other by passing data from one service to another, or by coordinating an activity between two or more services.

On the topic of Service Orientation, it further goes on to state

Service-orientation is a design paradigm that specifies the creation of automation logic in the form of services. It is applied as a strategic goal in developing a service-oriented architecture (SOA). Like other design paradigms, service-orientation provides a means of achieving a separation of concerns.

While REST does attempt to solve many similar goals as SOA, I believe there lies an important distinction. The essential focus of SOA is to separate functions or automation services. An example here is to separate an authentication service from authorisation, monitoring, logging etc. A SOA architecture that consists of a number of SOA services assembles such “functionalities” into one feature consistent whole. But is that how REST works ? Only in a very vague sense even if arguably so. REST standardises the functions ie. GET, PUT, POST and DELETE in case of HTTP connectors. It is arguable that REST could be used with a different set of functions, but even in that case the function set is likely to remain consistent. This is further supported by Roy Fielding’s post “It is Okay to use POST” in which he argues

Some people think that REST suggests not to use POST for updates. Search my dissertation and you won’t find any mention of CRUD or POST. The only mention of PUT is in regard to HTTP’s lack of write-back caching. The main reason for my lack of specificity is because the methods defined by HTTP are part of the Web’s architecture definition, not the REST architectural style. Specific method definitions (aside from the retrieval:resource duality of GET) simply don’t matter to the REST architectural style, so it is difficult to have a style discussion about them. The only thing REST requires of methods is that they be uniformly defined for all resources (i.e., so that intermediaries don’t have to know the resource type in order to understand the meaning of the request). As long as the method is being used according to its own definition, REST doesn’t have much to say about it.

So across the board the functions remain the same and the data types (or media types) change. Sounds familiar ? Thats like supporting SELECT, INSERT, UPDATE, DELETE on a broad range of tables each having a different schema. An argument I make in an earlier post REST is the DBMS of the Internet. Moreover Roy further goes on to elaborate that in “REST intermediaries don’t have to know the resource type in order to understand the meaning of the request” and that “method is being used according to its own definition”. Each of this further introduces constraints into traditional service orientation. And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

Web Oriented Architecture

This is not yet a particularly widely used term yet, but I did come across a reference to it on a recent article in InfoQ “REST is a Style – WOA is the architecture“. Looking it up on Wikipedia leads us to the same sources as referred to by InfoQ – articles written by Dion Hinchcliffe. Wikipedia states it as follows

Web Oriented Architecture (WOA) is a style of software architecture that extends service-oriented architecture (SOA) to web based applications, and is sometimes considered to be a light-weight version of SOA. WOA is also aimed at maximizing the browser and server interactions by use of technologies such as REST and POX.

But I just argued in the earlier section that REST is only a very very specific use case of SOA, whereas the statement above says WOA extends SOA to web based applications. If we were dealing with objects and not architectural styles here, an argument that X is a specific (constrained) sub type of Y and X extends Y would instantaneously be flagged off as a violation of Liskov’s Substitution Principle (LSP). So something isn’t quite right here and the whole situation does not add up to a consistent whole for me. I will leave it to the reader to decide whether there exists a flaw in my reasoning here or elsewhere. My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

To be fair I couldn’t quite find the same worlds (WOA extends SOA) in Hinchcliffe’s writings. Referring to one of them “What is WOA ? Its the future of Service Oriented Architecture (SOA)“, he refers to another of his post “Beating a Dead Horse: What’s a SOA Again? All About Service-Orientation…” which in turn states

It’s here that John Reynolds’ well-known SOA Elevator pitch comes tantalizingly close to capturing the essence:

SOA is an architectural style that encourages the creation of loosely coupled business services. Loosely coupled services that are interoperable and technology-agnostic enable business flexibility. An SOA solution consists of a composite set of business services that realize an end-to-end business process. Each service provides an interface-based service description to support flexible and dynamically re-configurable processes.

This business view is right on, and doesn’t mean business in a traditional, white-collar way. In this context, “business” means the actual functionality of the system, apart from technical details.

There we go again on services. But Business Services cannot be GET, PUT, POST, DELETE. I would emphasise again that REST does not expose business services – it exposes some very basic CRUD services. So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying. But it is likely that Hinchcliffe meant that WOA is built on REST and is similar to SOA in terms of the goals when he says WOA is the future of SOA. But honestly I could not quite figure out how exactly he would want to describe the relationship between WOA and SOA.

Resource Oriented Architecture

The wikipedia page states :

Resource Oriented Architecture (or, ROA) is a specific set of guidelines of an implementation of the REST architecture.

REST, or Representational State Transfer (see Roy Thomas Fielding’s Doctoral Thesis “Architectural Styles and the Design of Network-based Software Architectures”), describes a series of architectural constraints that exemplify how the web’s design emerged. Various concrete implementations of these ideas have been created throughout time, but it has been difficult to discuss the REST architecture without blurring the lines between actual software, or the architectural principals behind them.

Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

Related posts:

1. Musings on REST
2. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
3. Why REST ?

Struts, Django, Ruby on Rails. We’ve worked with these and many other similar frameworks. Some time back I started thinking of what would a completely new ground up REST / Resource oriented framework would look like (ground up to ensure it had no legacy design to deal with). Would such frameworks be similar to the ones dominantly used today ? What about the ecosystem that surrounds and interacts with them (client libraries) ? And finally what about the implications on the fine grained object model (assuming there is one) and its relationship with the resource model ? This post deals with some of the thoughts.

There are some specifics the post does not address and is agnostic about :

• Language : I shall be avoiding language issues as much as possible. Wherever I do bring in code constructs these may be assumed to be in Java (or pseudo-Java)
• Convention or Configuration : I think both are valid choices in their appropriate contexts, and I don’t specifically emphasise one over the other in this post

The frameworks mentioned above are not the only ones out there. There are many, and some actually are very REST specific eg. Apache CXF JAX-RS or Restlet. It would certainly be interesting to contrast my thoughts with these, but for reasons of insufficiently detailed knowledge about them, I shall choose to skip it (better to not make any statements than making incorrect ones).

I shall be assuming a HTTP connector with GET, PUT, POST and DELETE as the constant set of operations. These four operations shall be collectively referred to as Resource Operations.

We shall first start with the server side characteristics, and the term ROF shall refer to a Resource Oriented (server side) Framework

A ROF will have a resource oriented interface : Certainly not a profound statement, but it was important to lay that down upfront. So what is a Resource Oriented Interface. Given a particular resource, a Resource Oriented Software will support or consume end points which allow you GET, PUT, POST or DELETE the resource. There is one reason why this particular constraint is relaxed just a little bit. Modern browsers do not support all the four methods easily eg DELETE and make it just slightly hard to use the PUT method. Hence these methods can also be invoked by using a URI segment containing the method name eg. delete.

A ROF will have an abstraction to represent a resource as an end point : Again, that seems to be pretty obvious. But there is a reason why I make it explicitly. In many situations we see controllers acting as end points. To the extent a controller acts as an abstraction for a resource end point which essentially only has the resource operations as public methods, it would fit this requirement. However if I was using an Order as a resource and if I introduced an approve method on the OrderController that would not be consistent with this requirement. That would need to be modelled as an OrderApproval resource which may on successful completion, effect a state change on the Order resource to the status ‘approved’.

This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

Given consistent expectations of the Resource Operations these will actually be auto-magically implemented : Thats a bit of a turnaround from what I was just describing in the earlier paragraph. What I mean to suggest is that the class level methods I just referred to will be implemented within the framework. What the framework will allow are plugins to provide extended functionality at specific points. Thus a “public static Order Order.put(Order order)” method will be implicitly implemented by the framework. But before a put can be processed it needs to be validated. Thus the framework will allow the developer to plug in / override his own implementation for an Order.validate(Order order). There are multiple ways such plug-ins could be implemented. Depending upon the nature of abstraction, it could be an overridden method as I just described, or it could be a standalone method that is registered into the overall workflow (either through convention or configuration). The latter might be especially useful in case one wants to implement the functionality as stand alone methods or in case of functional programming languages. The plugin points provided could be framework specific. eg, One may want to validate a resource for consistency even at it is being read from the database. For the rest of the post I shall refer to these as plugins. In addition, framework will most certainly provide methods for for downstream handling of impact of PUT, POST or DELETE. This is covered in the next point. In case the framework chooses to not deal with persistence, it may choose to allow capabilities for integration with other persistence frameworks.

A ROF will provide capabilities to a developer to override or register methods to handle downstream impact of PUT, POST and DELETE : Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.

Note that the actual invocations to select, insert, update, delete the resource are *NOT* to be programmed by the developer. These are automatically handled by the framework. The developer essentially fills in the necessary logic to the plugin methods (eg. Order.validate) or handlers (eg. Order.onCreate)

A ROF will provide a mechanism to describe or map a resource abstraction to to the actual programming constructs : There are a number of ways this could be achieved. XML, YAML, DSL, Annotation – take your pick. Also the actual class could be defined (as in case of a POJO) and the resource characteristics mapped onto it, or the class may manifest itself at runtime based on metaprogramming around the metadata. Sample possibilities here are Hibernate like Resource-to-Object-to-Relation mapping (using either Annotations or XML) or a a completely metaprogrammed ActiveResource. One important aspect that the framework will need to cover is the situations where a Resource is a composite of many or partial underlying business objects. eg. an Order resource instance could theoretically span one Order instance and many OrderItem instances. Thus a one to one relationship between a resource and underlying business objects (or datastructures) is not assumed. What is assumed is that the framework will allow such relationships to be described or introspected.

A ROF will allow resources to be mapped onto URI or URI segments : This is too obvious an requirement to be explained and is mentioned here only for completeness.

A ROF will allow foreign keys across resources which manifest themselves as URIs to be mapped onto the underlying business object references : Resources refer to each other through URIs. The underlying business objects refer to each other through object references. Given the resource descriptions and URI mappings, the framework should allow for a transparent referencing/dereferencing between such URIs and the object references.

A ROF will allow factory methods for locating or allow injection of other resources / business objects : Within the handler functions, developers will need references to the associated resources or business objects. I say resources or business objects, since the developer may choose to interact with these at a coarse grained (resource) or fine grained (business object) level. The framework should allow the necessary support for such activities.

A ROF may provide additional support for typical aspects of lifecycle (eg. validation) : While I mentioned validate as a possible plugin function. However given the omnipresence of validations, the framework may provide additional support for such activities. Thus the framework may choose to automatically implement such capabilities using the resource descriptions.

A ROF may provide capabilities for domain specific extension of resource capabilities : Certain domains have standardised mechanisms of working with resources. As an example most banking systems based on the four eyes principle require approval activities. While this particular aspect is much tougher than it seems, a ROF may choose to allow extension of such capabilities using template like functions or mix ins. As an example in this situation, once an Order resource is defined, an OrderApproval resource will be automatically made available as will the GET and PUT methods on it (POST and DELETE in this particular case may not be relevant), as will the necessary and appropriate handler functions on OrderApproval.

A ROF will provide capabilities for automatically generating the resource representation from the resource and vice-versa : Resources manifest themselves in multiple possible formats eg. XML, JSON etc. An ROF will allow such transformations between the representation and the resource/business object instance automatically.

A ROF will provide capabilities for assembling more complex representations using templates : In many situations, especially when the representations are being composed for manual (browser based) consumption, additional resources may need to be pulled into a view. A ROF will allow for such assembly of resources to be composed into a final view using templates.

A ROF will allow for introduction of appropriate additional URIs in views using templates : Thanks to HATEOAS (I’ve really avoided it thus far ), the framework will need to allow some mechanism of describing what are the additional context specific URIs to be included in the final representation. The template logic should allow the developer to specify such URIs.

A ROF should allow for the resource / media-type descriptions to be shipped in band with the resource representation : Since REST allows media types to be auto discovered and auto described, the framework should allow for the metadata for such media types to be also presented to the client. While I think it is essential that such in band information should be conveyed on demand, the framework may also optionally support upfront interrogation for media types and their details, which will require such information to be shipped out of band as well. I am not aware of any specific standards around such interrogation APIs so the framework could implement a custom API for the same. The actual metadata could be represented using any of the typical appropriate standards such as RDFa, XML Schema snippets etc.

A ROF should optionally allow support for auto generation of bindings for clients : I really really cringe as I write this. I cringe because to me the great attraction of REST is the simplicity and the ease of introducing incremental integration. The client binding generation (especially if it is statically generated) flies in the face of many accepted lightweight design scenarios. However I think there are likely to be some situation where availability of such client side bindings would be helpful. When possible (eg. with dynamically typed, metaprogramming capable languages like Python or Ruby), such bindings should be dynamic. In such cases the client can automatically introspect the server side media types and make available the necessary client side objects on the fly. In cases where statically typed languages such as Java or Scala are used, the client side may choose to expose everything as generic datastructures (e.g trees of name value pairs) or may allow for generation and compilation of client side bindings. I have no specific thoughts around the API support needed on the client side, except that quite obviously this would include support for the resource construction, resource operations etc. and that they would allow the client to interact with the server using the underlying language constructs rather having to work at a raw HTTP level.

In addition to the characteristics described above, I suspect frameworks will have many other optional characteristics such as support for monitoring, auditing / logging, transaction management, object pooling etc. etc. However these are unlikely to be particularly interesting when focusing on the framework aspects especially from a resource oriented perspective, which is indeed the focus of this post.

Related posts:

1. Musings on REST
2. Why REST ?
3. REST is the DBMS of the Internet

The Twitter DM’s exchanged were as follows :

@sbidwai : is REST like object oriented implementation for services, where as SOA is procedural ? thinking loud..

@dnene : a slightly better analogy would be SOA is like invoking stored procedures, whereas REST is like invoking SQL on the table

@sbidwai : agreed in parts.. but most impl will hv much more than CRUD.. eg, twitter rest apis..*

@dnene : CRUD is the interface. To extend the analogy, logic is implemented as triggers not SPs. (My Opinion)

* CRUD in our shared vocabulary stands for Create, Read Update, Delete.

As I subconsciously wrote the last DM, it suddenly dawned on me that this was the one concise way to express how REST architectures would impact software designs.

To summarise the exchange differently

“If WS-* is the RPC of the Internet, REST is the DBMS of the internet“

To expand on it a bit more :

Traditional SOA based integration visualises different software artifacts being able to interact with each other through procedures or methods. REST effectively allows each software artifact to behave as a set of tables, and these artifacts talk to each other using SELECT, INSERT, UPDATE and DELETE. (or if you wish GET, PUT, POST, DELETE). And where exactly is is the business logic ? Is it in the stored procedures ? Not Quite. Its in the triggers.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. Musings on REST
3. Why REST ?

It is becoming apparent that even as it becomes popular, REST (Representational State Transfer) is not yet as well understood. This might seem a surprising statement, but a lot of us use REST thanks to many frameworks supporting REST like interfaces, have a sense of what REST like interfaces are like (even if such an understanding is not sufficiently accurate), and exercise our common sense in using such interfaces. Having said that, let me clarify that while the internet is full of documentation about the semantics of REST, its actually quite light on the rationale for REST (including Roy Fielding’s dissertation which is the reference document for REST). Thus I have had to intersperse REST semantics and historical narrative with some personal opinions. So treat this as a part opinion and feel free to question my thought where you think it does not make sense.

Audience

If you are a REST expert, you are likely to have figured out much of this any ways by now. If you would like to understand specific technical semantics about REST, again this may not be the best article to read. However if you are curious about REST and would like to read a perspective on why and how it makes sense read on.

Flow

I shall be meandering through a historical narrative in the first half before starting to make the points I wish to make in the second. Lot of the points I make in the first half are likely to be those you already are aware of. However these are being made to allow an immediate recall when you read the second half. It is quite important to have read the first half to understand the perspective I put together in the second.

Historical Narrative

FTP

I am sure you have used FTP (File Transfer Protocol) a few times even though nowhere as frequently as HTTP (HyperText Transfer Protocol). Let me quickly present some characteristics of FTP

• Given a FTP client, you can connect to any FTP server so long as you have a valid userid/password pair for the server (or anonymously if the server so supports).
• The home directory on connecting to a FTP server is typically your starting point. At this point typically you can execute the ‘ls’ or ‘List Directory’ command to list all the files and directories within the home directory.
• If a file interests you, you can get it by issuing a get command
• If a subdirectory interests you, you can further navigate into that directory by issuing a ‘cd’or ‘Change Directory’ (or often by double clicking on the directory in case of a graphical client).
• If you would like to add a file to the current directory you can issue a ‘put’ or ‘Upload’ command.
• While you have the flexibility to navigate from one directory to another, you soon realise that every file and directory is uniquely addressable by its fully qualified path (either absolute or relative) and you can refer to each file and directory by its path. You are also aware that a valid path will uniquely resolve to only one directory or file.
• At each stage as you navigate into a separate directory, the server allows you to retrieve the list of subdirectories and files within your current directory. It always shows you the current state of that directory. Thus even if you were to list the same directory twice, and someone else uploaded a new file or created a new subdirectory successfully between the two requests, you will see the reference to the new file / directory when you request for the listing the second time.
• At every stage you issue a command, the FTP client+server work together to service the request (or issue the appropriate error message as necessary), and then pretty much forget about what you did. In other word the server keeps no track of and shows no awareness of what you have done earlier in your session (though it does remember who you are primarily from a security perspective.
• To work with a FTP server using a command line client, you primarily need to understand the usage of four commands (verbs) post a successful connection. These are ‘ls’ to list the contents of a directory, ‘cd’ to navigate to a different directory, ‘get’ to download a file, and ‘put’ to upload a file.

So FTP allows us to upload and download files. But does it allow us to ‘do things’ ? Sure, so long as you combine it with a few more pieces in the puzzle. Let us say, we are back in the late 80s (prior to the invention of HTTP) and I want to send and a list of purchase orders collected by a local office to a central office for further processing. This requires the following elements to be added to the mix.

• A shared understanding of where the files will be uploaded, how they will be uniquely named, their specific file extensions (optionally) and the specific format of the file eg. Comma Separated or Lotus 1-2-3 or WordStar or WordPerfect (the popular application software of the day) including the positioning of the various fields in the file, and
• Preferably a daemon process on the central office computer (the FTP server) which regularly scans the directory, parses each file as it comes it, does the relevant processing on it, and generates the appropriate result files and places them in the appropriate directories using the shared understanding of the directory structure and the file naming convention to communicate back the results of the processing.

Ho Hum. This was all stuff you knew. But the reason I brought it up is that FTP and how it was leveraged then has a lot to do with the principles that govern REST as we shall later see.

RPC

Back then in the 80’s FTP wasn’t the only mechanism to transfer data between machines. One more of the many other options was RPC (Remote Procedure Call). It not only allowed you to transfer data across machines, it actually had built into itself, a contract to remotely execute software. Unlike FTP which merely transferred data (in well understood units called files), RPC allowed you to invoke remote procedures by supporting an ability to pass messages which included the message name and the values for all the parameters necessary to be supplied to the message. Unlike FTP which was meant to do data transfer across a network, RPC was geared to do things remotely.

A contrast between FTP and RPC

If the objective of network computing is to use the computers and networks to ‘do things’ one would assume that many more people would use RPC than FTP for the same. While RPC did get used as a technical substrate, at a business processing level FTP got used far more (eg. to send and remotely process the list of purchase orders). There are some important reasons that we need to understand here.

FTP required understanding of very few basic verbs (ls, cd, get, put). Thus the training required to understand FTP semantics was far less than that for RPC. This was partially due to the fact that RPC had a programmatic interface. To the best of my knowledge there are no widely used command line clients for human interaction with RPC services. In addition, each procedure required a set of semantic data (parameter) associated with it. This was no different than FTP which also required similar data to be shipped over the network. Turns out there were a few distinctions. First, the nature of design of RPC services often required combining application data with control data, and there was also often a sequential expectation due to the RPC business transaction being broken up into multiple RPC calls perhaps for the sake of efficiency. Moreover each time, new procedures were added or parameters added, these required programmatic changes. FTP on the other hand was simpler. In most cases the entire data (including some redundant data perhaps) was sent in one block (or file in FTP parlance). By dealing with a file as the least common denominator, the FTP stack decoupled itself from any application specific semantics. Moreover, depending upon the agreed upon format, the files could be edited at either end by by human actors using specific software such as plain text editors or word processors or spreadsheets. Moreover if the formats changed, wherever such files were being manually edited, no programmatic changes were required. Irrespective of the changes you made to the file formats, file processing software, the FTP stack itself did not change – it remained stable.

Less is more

A theme I shall come back to again is many a times less is more. FTP had far fewer training requirements (few basic verbs). FTP did not deal with parameter value formatting (though other pieces of software subsequently might have to). FTP was just so much easier to start working with. FTP did not actually preclude any of the capabilities of RPC from being introduced, it merely allowed this to be added subsequently as additional optional layers (or subsequent elements in the processing pipeline). Finally FTP allowed users to deal with the data in the units and the formats and the tools they understood the best – their day to day application software components and simply focused on only transferring files, while imposing only one requirement – each end should work with a file as a unit, and both ends should understand the file formats. By focusing on file as a unit, each business user could focus on the data he/she wanted to deal with in the format that was most appropriate (an analogy in REST would be a resource .. but I’m getting ahead of myself). And at the end of the day, by doing less, FTP ended up being much more popular and thus doing more.

Email

Other protocols widely used on the net were SMTP / POP which were used for email. Email eventually was considered the killer app for the internet. Similar to FTP email focused on the users getting to learn only a few basic verbs and exchanging the basic unit of data transfer (messages) using these verbs. Again, even though email itself didn’t get things done, it contributed far more heavily than RPC to getting things done, by having other manual or software actors at either end of the messages who did the necessary processing required.

WWW (World Wide Web)

While email was the killer app for the internet, the one that really brought it to masses was the world wide web which was based on the HTTP protocol. While HTTP could be used to ship documents of a variety of types (often classified by their mime-types), the defacto type of document used the HyperText Markup Language (HTML). Unlike FTP and email, this required the authors to understand a new language, but used a simple markup syntax to keep the learning curve to the minimum. It however introduced a very powerful element – the embedded hyperlink. While the earlier technologies supported a uniform identifier for each document / message, the hyperlink allowed references to other documents / messages to be embedded thus converting the document pool into a document web. We now had the ability to navigate from one document to another and such navigation retained the contextual relevance by embedding the hyperlinks. There were other enhancements as well such as introduction of more verbs (eg. POST and DELETE, the latter not really being supported by any of the browsers). Allow me to state the salient points of WWW despite the obvious duplications with some of the points I listed under FTP (for the sake of emphasis). Note that the scenario I describe below is primarily describing static web serving (except to the extent of file uploads) and does not address the presence of a dynamic web application.

• Given a web browser you can connect to any web server optionally using the appropriate authentication credentials.
• Typically the home page of the web server is your starting point. At this point you are shown the document which usually include embedded hyperlinks to other associated documents on the web server.
• You can get/view/download/save a document by clicking on a hyperlink pointing to the document.
• Some web servers may be configured to allow you to browse a directory. Clicking a hyperlink pointing to the directory allows you to see a directory listing which shows all the subdirectories and documents within the current directory. Each such subdirectory or document is also shown as a hyperlink to allow you to navigate to it.
• Some documents have an form including an embedded file field and a button which allow you to upload a new document onto the web server.
• Each document (and directory if directory browsing is turned on) has at least one identifier which uniquely identifies the document – the URL. It is feasible to directly navigate to the document if you are aware of the URL.
• Navigating to a different document often provides you with a different list of embedded hyperlinks which are contextually relevant to the document being viewed.
• At each stage the web server is not aware of any other information about you apart from your authentication credentials, and is not generally aware of your browsing history (except what may be stored for audit purposes on the web server logs).
• As a user the primary skills you need to grasp is the ability to enter a starting URL, and then being able to navigate from document to document by clicking on the hyperlinks. If you are uploading documents, you may in addition need to know how to specify a local file path and press the Submit button to upload the file.
• While HTML documents are the defacto default, the same capabilities can be used to serve any types of documents. The server usually identifies the document types by the registered mime types, and the browser may either render the document itself or call upon the necessary add-on plugin application to render the document based on the appropriate type or may in some cases simply save the document locally in case no such application is available for further processing.
• Usually but not necessarily the document name have the characteristics of a noun

Again the reason I listed these characteristics is that these have a tremendous commonality with those of REST (except that what I refer to as a document above may get referred to as a resource in REST parlance).

SOA : DCE, Tuxedo, CORBA, RMI

Even as the web was evolving other technologies which allowed for more sophisticated remote service invocations were being developed. Along with RPC, these were essentially different technical manifestations of Service Oriented Architecture (SOA) principles. While these are substantial developments in their own right, the relevant points to be made in the context of this article are :

• Each SOA service supported the ability to define a set of service semantics which included the service name, the parameters to the service, an ability to expose the metadata of such semantics, an ability to leverage such metadata and invoking such services either statically or dynamically from a remote client.
• Many services were usually expected to “do something” though quite often some services would simply return the requested data. Usually but not necessarily the services were identified by using ‘verbs’.
• Some of the SOA services allowed maintenance of a client state on the server, and allowed the server to do processing conditional on the client state.
• These technologies almost invariably required some kind of programmatic effort at both the client and the server end. Manual specification of the service parameters and manual invocation of the service was simply not a typical use case. Neither was a default rendering of the results easily available to be manually viewed by an end user.
• Unlike retrieving or storing a document, these services often were expected to have a far more complex functionality.

CGI, dynamic web applications and Web Services

Clearly as WWW started getting used far more, people were only too keen to use it for much more than storing or retrieving documents. This led to the development of CGI and subsequently other dynamic web application technologies (eg. LAMP, J2EE etc.) which would allow us to use the web to ‘do something’. Since these were clearly offshoots of the SOA world, being mapped onto the WWW infrastructure, the characteristics of such dynamic applications often had a lot in common with SOA, and they started dropping many characteristics of the traditional static WWW. Thus was born the child of the world wide web and distributed service oriented architectures – web services. This led to newer SOA technologies such as WS-* and SOAP.

Like the typical scenarios after the discovery of any highly profitable opportunity, the early rush was to leverage the opportunity and it was only a little later when the dust died down, that people started wondering if they had sacrificed something in the heat and dust of the moment. That stock taking resulted in the realisation, that some of the very basic characteristics of the extraordinarily successful internet technologies (FTP / SMTP / WWW) had been diluted, and even if such dilution still allowed immediate progress to have occurred, some of them would need to be corrected to be able to continue the explosive growth that had been seen so far. One such exercise in my opinion is the laying down of the REST architecture style.

REST Semantics

While REST brings back many of the characteristics that made internet so successful back to application design, it should be noted that many of these are not precluded by Web Services or SOA. However what are mandatory characteristics in REST are in some cases missing from but in most cases quite feasible to implement in traditional (non REST) web services by using additional best practices. Also note that each characteristic is not necessarily universally superior. So do evaluate it in your context to see if it makes sense. However before we get to the benefits of REST, a quick synopsis of REST technical characteristics might be in order.

While a full description of the REST technical aspects is completely beyond the scope of this post, I summarise these below. You might notice the strong parallels between the characteristics of FTP and WWW and those of REST even as REST adds a few more capabilities. The reason I portray them in the form below in a manner quite similar to the way I portrayed the characteristics of FTP and WWW is to emphasise that REST actually continues to leverage the same characteristics that made these technologies so popular and globally scalable, even as it just adds those few minimally necessary capabilities to achieve the same scalability for not just transferring documents or rendering pages but to ‘do something’. In other words it brings together the characteristics which made the internet technologies so popular and applies them to the inter application integration, component and service orientation, and application mashup scenarios to allow them to achieve similarly large adoption and to perform the tasks necessary in the given context (or ‘do someting’ as I have continuously referred to).

REST characteristics

• Resource and media types as the basic units : REST treats a resource as the basic unit of data transfer. Such resources could refer to anything in the particular context eg. a flight reservation, an invoice, a video etc.
• Unique resource identifiers :REST requires that each resource have at least one identifier which uniquely identifies that resource. This makes it easy to be able to bookmark resources or make them searchable.
• Each resource has at least one representation :Each resource can be expressed using a variety of representations. This could include HTML, XML, CSV, JSON etc.
• Each resource has often one default manually readable representation : In most cases but not all, each resource has a representation that can be manually consumed using a web browser. Such manual representations are often either XHTML representation with associated CSS, but could theoretically be some custom representation rendered by a delivered on demand custom javascript or Java applet. Note that this is not a requrement of REST but is a practice often followed.
• Each resource has a type. REST supports self describing media types : Each resource has a type (referred to as media type since REST refers to the resource web itself as hypermedia). The type influences the data semantics of the resource, and the type itself can be self documenting using a variety of technologies (eg. one possible way is to specify XML schema descriptors).
• Each resource representation optionally includes contextually relevant hyperlinks to other resources :This not only allows the clients to auto discover associated resources, but also allows the server to clearly communicate the contextually relevant links based on an application state.
• REST resources are indexable and search engine friendly : A consistent resource naming and representation allows for easy indexation and search engine integration.
• REST requires minimal starting point intelligence : Typically one only needs the initial URL for being able to integrate with a REST implementation. All newer resources are often dynamically discovered. Since these media types also document their own metadata, client agents can automatically discover more information about them. Thus media type metadata rather than being compiled into the REST client can be dealt with dynamically or by using code on demand agents for dealing with the appropriate media type (similar to browser plugins)
• REST encourages a uniform interface. :Typically this manifests itself by the minimal verbs being used to describe REST operations.When used with HTTP these are GET, PUT, POST and DELETE. This reduces the intelligence requirements on the client. Additionally clients may be capable of parsing metadata for the resources based on standard formats such as ATOM or XML schemas. The context specific intelligence required on the part of the client is no longer in the verbs it has to understand (method names) but is now in the resource types that it may need to manipulate. Thus if a client can deal with resource identification, resource representation, self descriptive messages and hypermedia, it can start dealing with REST.
• REST supports value addition by intermediate processors : REST supports the scenarios where intermediate processor units can provide additional value addition. These could include processors which provide caching support or those that provide resource enrichment capabilities.
• REST encourages usage of scalable practices :By precluding usage of conversational state and sequential assumptions, REST implementations tend to be easier to scale even as they compromise on efficiency at times (due to redundant data transfer or additional processing requirements)

REST benefits

Having described many of the REST characteristics the following could be interpreted as the benefits of adopting a REST style architecture.

• Default RenderingIn case of most REST implementations, you can quickly provide a default HTML rendering capability. Thus even as you provide a REST interface to allow inter application integration, customers of such an interface do not have to wait for building the programmatic capabilities for leveraging it, they can get started immediately by being able to manually view all the resources and their states manually and by navigating around the interface by using a plain web browser. This substantially reduces the entry barriers for your customers, and allows them to get more conversant with your media types even as they are still figuring out how to programatically leverage the capabilities.
• Self describing / auto discovery of media types and capabilitiesThe traditional web service semantics rely upon clear upfront documentation of media types, their schema and the API semantics. Thus the metadata about the service is often communicated ‘out of band’ from the actual service itself. This is required so that the clients can understand all the valid end points and service semantics up front before they can leverage the services. Not so with REST. Given an initial starting point, REST greatly encourages a contextual provision of the relevant additional interfaces (hyperlinks) as a part of the the document / resource data itself. Thus clients do not upfront need to be aware of all the end points (resource URIs) to be able to leverage the services. Moreover REST supports self describing media types as well. Thus the schema information for the resources can be shipped ‘in band’ with the resource representation itself. This allows for clients to discover new media types or changes to their schema and even allows the default rendering of the same without having to upgrade the programmatic components to leverage the newly discovered or modified media types / schemas. Finally the code on demand capabilities (these are optional) of REST can allow code to be downloaded to automatically parse or render such newly discovered or modified media types.
• Encourages scalability even at the cost of efficiencyAspects such as non maintenance of conversational state, greatly increase the scalability of REST applications even if they do incur a minor cost in efficiency (which can be due to repeated redundant communication of data elements, or additional processing requirements due to preclusion of conversation state). This makes it relatively easier to set up multiple servers as the demand for the REST capabilities increases. Having said that, let me quickly add a caveat that designing clustered applications even if with REST interfaces is not always trivial, and while REST makes it “easier to scale” that should not be confused with “easy to scale”.
• Resource / Data semantics are much easier to understand than Service semanticsTo put it differently an invoice structure is much easier to understand from a data perspective than an invoice processor API. This makes it easy for the clients. This often also makes it easier for the server side implementations. Service semantics often bring in issues of sequence, client state and other control information, most of which can be avoided using REST. Generally speaking expectations are simpler to lay down and meet when specifying resources rather than services.
• Clear naming and accessibility of each resource in your universeWeb services don’t mandate clear unique identifier for all your resources. Thus sometimes it is not possible to reach a particular resource except through a convoluted series of steps. In some cases some resources are inaccessible for ever. As an example, many online shopping experiences end with an invoice being shown, but I have often found it impossible to later on pull up that invoice that was earlier shown to me at the end of a transaction.
• Extensible resource types which are optionally dealt with by clientsNot only are resource types self describing, REST makes it easier to convey additional extensions to such resource types by using additional URIs within the resource representation as well. Thus even as a representation lays down a variety of field values (say for an invoice), there might be other associated resources which might either be optional or variable media types based on the context (eg. purchase order / quality report etc.) which can be easily referenced by simply including their URIs. Such additional information does not require the basic media type to be enhanced or by introducing attachments to the media types. These can be implemented as additional navigable out of band media types. Thus clients don’t have to deal with them, but they can do so easily when they choose to do so. Thus the client has a choice to not deal with the additional media types when they do not make sense in the client’s context.
• Search engine friendlinessWhile resource directories help for smaller scale integration (eg. Yahoo when it started off, attempted to categorise the web), such directories or registries are often found to be tough to scale beyond a particular threshold (thats why Yahoo or Google now provide entry points by allowing us to search through all the web resources). Consistent resource naming and representation make REST resources search engine friendly and allow additional entry points into a REST service based on search criteria. This makes location of newer resources far easier than what might be feasible through a resource registry especially on a large scale.
• Easer layeringWhile it is possible to add intermediate proxy services for enriching the capabilities of a REST implementation, it just makes it seem a lot easier to implement these as and when required when the underlying architecture is REST based. Thus while mashups can be readily implemented using both REST and traditional SOA implementations, I would submit that these are much easier to implement on REST based architectures.

I have used the word scalability above in the context of the ability to service the runtime demands of a larger number of clients. However REST helps makes your software artifacts become scalable in one more way. By providing a basic and minimal uniform interface requirements, REST allows your applications / services / components a low entry barrier path into being a participant in a broad web of similar others who all agree on the basic REST semantics. This substantially increases the potential number of clients to your services since they can leverage these services easily and with low entry barriers. While traditional SOA technologies attempt to provide the universal access to all possible consumers, REST with its emphasis on minimalism, simplicity and low entry barriers actually makes it practical. Similarly REST makes it easy for you to start consuming other services and mashing them up with others to service your clients (pun intended) quickly. Finally REST takes the the very characteristics that made document and message sharing so easy to use and popular (characteristics which are not necessarily found in all conventional SOA implementations) and combines them with the necessary elements to achieve transaction processing, application integration and mashups (use the web to ‘do something’) on a truly global scale even as it makes these capabilities easily available and cost effective to leverage.

Some concluding comments

While not directly relevant as REST rationale or REST benefits, I thought it might be useful to add a few more associated comments within the context of REST usage and adoption.

Simplicity and bottom up adoption

I must confess, my biases show up quite strongly in this paragraph (so feel free to treat this as a partially prejudiced statement). Simplicity is not per se a characteristic of REST. However it does stem from the nature of genesis of the competing options. While most internet technologies using an incremental, evolutionary approach, most SOA technologies have been designed by a committee. This is why the consulting and development budgets required to implement FTP / email / Web especially on a per utility basis are far different than those likely for implementing DCE, Tuxedo, CORBA and SOAP. Part of the reason is also due to the fact that most internet technology adoption is bottom up, while that of SOA often is top down. While top down may seem attractive, it may seem sobering to realise that most top down processes break down beyond a particular scale. Thats why free markets on the whole have trounced centrally planned economies (though some recent happenings do point to limitations of the same as well). Thats why internet scale simple inter application API integration and mashups took off even as intranet scale application integration was mired in budgeting, territorial, enterprise modeling and governance issues. Thats why the LAMP stack (eg. PHP) which hasn’t been particularly strong in the non web arena, is deeply entrenched in the web based application space. Sometimes it just is more productive to quickly implement a simpler technology and incrementally enhance it rather than attempting to cover all possibilities, options, and border conditions by putting a committee in place. At its very core, REST requires only incremental understanding of newer technologies, is easier to incrementally adopt and is less likely to get mired in organisational issues. Precisely the characteristics that FTP, EMail and WWW had.

Simpler abstractions win

I have generally found that simpler abstractions even though harder to deal with initially, often win in the long run. Notice the fact that the bare bones rendering functionality of HTML/WWW completely trounced the rich UI and application integration capabilities then available (eg. Windows/Java and DCOM/CORBA/RMI). This is not to suggest that the extra capabilities are not required. That is why Rich User Interfaces on WWW continue to be a dominant part of the internet technology wishlist. However the simpler, cleaner and minimalistic abstractions often are far more important than feature richness. A point I would want to make in favour of REST even as I admit that conventional SOA technologies are far more feature rich than REST.

REST is not SOA

I must confess, for a long time I believed REST was merely a specific usecase of SOA. However recent thoughts lead me to believe otherwise. There is indeed a reason for such potential confusion. REST based architectures and SOA may often attempt to service similar goals. To the extent of servicing such goals, REST may look like a substitutable component for other SOA technologies such as SOAP. However even as they attempt to meet similar goals, REST attempts to view at your architecture artifacts differently. REST encourages you to view and model your architecture as a set of resources rather than services. There are important implications of this not just in terms of the many benefits I describe above available under REST but also in terms of the design and architecture characteristics of the implementation. Treating REST as just another way to implement SOA sometimes encourages one to miss out on the subtleties. These however are beyond the scope of this post, and I intend to cover the same apart from the implications of REST on software design in my next post.

Related posts:

1. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
2. REST is the DBMS of the Internet
3. Fomenting unREST : Is RESTfulness a semantics game ? Why does REST require statelessness ?

The way I perceived it, Sun was going through similar difficulties. The hardware business was delivering dwindling margins post the dot com bust, and the software business was under threat from upstarts such as Linux which offered a substantially similar software stack at near zero licensing costs. One of the crown jewels asset Sun had in its stable was Java. And while it was (and continues to be) a wonderful asset, it just was incredibly difficult to make money off it. Now java hasn’t been open source exactly for most of its life, but it was a sufficiently open stack nevertheless to cause the same difficulties that open source software would in terms of monetisation.

There’s an excellent blog post written a year and half ago by Michel Bauwens, Can the experience economy be capitalist? which does refer to some of the underlying issues. I suggest you do read it, but would like to quote a few points from it below.

First of all, in the field of the immaterial, we are no longer dealing with scarce goods, but with marginal reproduction costs and non-rival goods. With such goods, sharing does not diminish the enjoyment of the good, since all parties retain their ability to use them. The emergence of peer production shows a new form of creating value, that is in fundamental aspects “outside the market”. Typically, in commons-based production we have a common pool, accessible to everyone (Linux, Wikipedia), around which an ecology of business can form to create and sell scarcities (usually services and experiences). In sharing-oriented production (YouTube, Google documents), we have proprietary platforms that enable and empower the sharing, but at the same time, sell the aggregated attention (a scarcity), to the advertising market. Finally, in the third crowdsourcing mode, companies try to integrate participation in their own value chain and framework.

So the good news is that indeed business is possible. But I would like the readers to entertain the following proposition, nl. That:

1) The creation of non-monetary value is exponential

2) The monetization of such value is linear

In other words, we have a growing discrepancy between the direct creation of use value through social relationships and collective intelligence (open platforms create near infinite value through the operations of the laws of Metcalfe and Reed), but only a fraction of that value can actually be captured by business and money. Innovation is becoming social and diffuse, an emergent property of the networks rather than an internal R & D affair within corporations; capital is becoming an a posteriori intervention in the realization of innovation, rather than a condition for its occurrence; more and more positive externalizations are created from the social field.

Quite eloquently argued here that open platforms create near infinite value which is difficult to be captured by business and money. I still remember companies such as Borland, HP, Sun, IBM making a fair amount of money through selling C/C++ compilers and IDEs (I remember it used to be almost $5000 per seat). However this was in the days when the community capability of creating similar competing software stacks was only in its infancy. No longer so today. Now communities, open standards, open processes and open source are fairly well established sources of delivering asymmetrically significant capabilities at a fraction of the cost, a fraction which is made even smaller when the same is incurred by a small motivated group of individuals or small highly agile companies. There in lies the difficulty. Companies are trained to and built to deliver linear and symmetric capabilities in the context of costs they incur. However they have a relatively poor handle on monetising in scenarios where small teams can deliver exponentially asymmetric capabilities.

The blog post I referred to above, identifies the right area to look at in this context – Identify what is scarce and Monetise the scarcity. So when any software has a sufficiently large utility and can be managed through open processes to be able to satisfy a globally substantial demand at a fraction of the cost (eg. apache httpd), that particular capability (eg. static and dynamic file serving over http) is no longer scarce enough for a commercially focused company to make money out off. Thats why the money shifts where the scarcity is – how to leverage such software and put it to good use. This is where the existing commercial successes around open source are based on eg. RedHat, JBoss, IBM, Oracle etc. They monetise the support, training, services and consulting around such software. In many ways Sun could be argued to be a victim of its own success. It not only went much further than any other large corporate in terms of creating open specifications, it also contributed substantially to sufficient knowledge dissemination and tooling to allow many other individual, SME and large corporates to be able to compete with Sun in these very areas that Sun could’ve monetised. Sun thus created its own competition for monetising the scarcity that got created around Java even as many others cried out hoarsely that Sun simply wasn’t open enough. Will there be an incentive for Oracle to reverse that somewhat ? I suspect that could be a logical option if it focuses on ensuring a good ROI for Sun’s Java investments and assets and more so to be able to make itself more capable in the space around Java than the competition (ie. IBM).

So is a large corporate making money off open source or open standards an oxymoron ? Viewed narrowly yes. Because while it is possible for a small team to make some money with adequate ROI off open source or open standards, it is unlikely to be feasible for large corporates. They really need to figure out what is the new scarcity that gets created off such initiatives, come to a judgment that the opportunity arising from such scarcity is large enough and worthy of their time, attention and investment, ensure that there is no current available capability of anyone else disruptively delivering exponentially asymmetric value in that space, and finally occupy that space before other large corporates do.

It does boil down to the fact that while individuals and small companies may find it easier to monetise open source and open standards, the same is going to be generally tough for large corporates. Or as Michel Bauwens said in the blog post I referred to (in my opinion perhaps a little exaggerated but not entirely off the mark)

For all of this, we will need new policies, major reforms and restructurations in our economy and society.

But one thing is sure: we will have markets, but the core logic of the emerging experience economy, operating as it does in the world of non-rival exchange, is unlikely to have capitalism as its core logic.

Related posts:

1. Java : the perpetually undead language
2. Stop making SOA complex
3. Commentary on Python from a Java programming perspective

A few months ago I was working with using Python for complex data processing scenarios. This was stuff I was very well versed in Java, but was working hard at to make sure that when I wrote code in Python, it would at least pass as half decent pythonic code. My initial struggles with idiomatic Python, soon became a strong interest in functional programming. I initially considered it to probably have a niche applicability in traditional business transaction processing space and of primary importance in algorithmic code, and thats the state where I almost left it. Curiously even after reaching that conclusion I did persist in attempting to figure out how it could be used in a manner where it would be helpful to have functional programming constructs in typical transaction processing scenarios.

Is functional programming useful in traditional business transaction processing ?

One of the aspects of the struggle was that I didn’t want to have the functional programming code merely replace the existing code. I wanted to figure out if it would be possible to replace it in a manner where it would make a substantial positive difference, and that hunt proved a little elusive for a while. Another aspect was the fact that the traditional OO code depended so intensively on state, that to actually try to figure out how the same would work under Functional Programming was really requiring a substantial leap in terms of thought, a leap I had to really struggle with. Finally my language of choice was Python, a dynamically typed language, and some of the literature surrounding functional programming really assumed static typing, especially that related to patterns and constructs in Haskell (eg. Monads). Such patterns were difficult to apply since the underlying expected static typing support was simply at odds with a language like Python.

It was after persisting for a few weeks, that I started to see how one could apply these techniques effectively. I also started figuring out how to Pythonise constructs such as Monads. Once I was past the initial rather steep hurdles, the going became a lot easier, and the progress much much faster. In at least a subset of some typical financial transaction processing scenarios I did get some extremely encouraging (to myself I call them jaw dropping) results. The resultant code suddenly seemed smaller, far far more intuitive to understand and extremely easy to change.

Working with Object Oriented and Functional Programming simultaneously

I am not sure whether it was due to my abilities and experience or due to my inabilities to visualise, I decided that pure functional programming was simply not a choice in this context and thus one would need to combine traditional OO and FP constructs. Having reached that decision, it became a little easier to identify the areas which could be better leveraged by FP. However I believe I still have some ways to go before being able to be confident of writing OO and FP together the way each of them should be.

Scala

My focus on functional programming and fondness to Java also led me to investigate Scala. I made the mistake of actually reading the reference documentation on its web site first. My initial reactions were that it simply was a weird syntax language which was unlikely to be of help in reducing development time even when writing FP code. Thankfully, I did not give up at that stage and continued to read a lot more. I soon realised that type inferencing and other aspects (eg. no getter setters) really resulted in some substantial savings compared to traditional Java code (it probably cuts down the code size to half). And the syntax really wasn’t so cryptic after all, it just required a day to get used to. Moreover one had the entire arsenal of the the FP constructs along with traditional OO constructs to deploy. I did write some processing code and was very happy at the results I got. In this current state, I am excited about learning more about Scala as a language that would be an alternative to Java. It gives me some of the capabilities of more productive languages such as Python, while continuing to leverage the performance and stability of the JVM. Note that Scala’s compatibility with the JVM should be thought of differently from that of dynamic language implementations such as Jython on the JVM. Given Scala’s static typing capabilities, its runtime structures are far more similar to Java and thus are able to retain somewhat similar performance and memory footprint characteristics as traditional Java programs. I suspect the same is unlikely to be found for other dynamic languages on the JVM.

Concluding Thoughts

Prior to this entire exercise, in my mind there were two primary individual preferences to programming languages. For most scenarios, my preferred choice was Python especially due to the sheer speed with which one could write and maintain code, code that was really compact and readable. And if Python runtime performance was not going to be good enough, use Java. Coming out of this exercise, it does seem that there is a high potential that I might want to switch my preference to Scala instead of Java. I have already validated the positive benefits of functional programming and Scala coding in processing scenarios (those that do a lot of computation and processing in the background). I do need to validate the same for typical web based applications, but on the face of things, I believe I have already identified areas where FP can help in typical web application scenarios as well. And now I have two multi paradigm (OO and FP) languages to be able to leverage FP in alongside OO – Python and Scala.

This is something thats definitely going to be my interest area for the next few months and I intend to write many more specific posts on the topic. Should that be of any interest keep listening on this blog’s RSS feed and my twitter stream.

Related posts:

1. Commentary on Python from a Java programming perspective
2. Improve your web based software development and maintenance ROI with dynamic programming languages
3. Talk Slides : Programming Language Selection

Background

Per section 3.4.3 of Roy Fielding’s dissertation

The client-stateless-server style derives from client-server with the additional constraint that no session state is allowed on the server component. Each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is kept entirely on the client.

The same thought is further commented upon in section 5.1.3.

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 , such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.
…
Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server’s control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

The potential confusion areas

Given the clear prohibition of storing client state on server, there are some typical idioms which do get challenged as REST unfriendly. eg.

• Logging in to obtain a session token, which is subsequently a validation of an authentication status
• Binding additional attributes to such a session token on the server side with variables such as :
  • Computed and cached access privileges of the user
  • Conversational state eg. fields entered on page 1 of a two page form

The question there is are all he above scenarios inconsistent with REST ? Many articles seem to suggest that that would indeed be the case. as an example one of them is Ajax and REST, Part 1 Section: Violating the “stateless server” constraint

My thoughts

Clearly in the above example computed data such as access privileges, or for that matter user’s time zone information are available to the server even if these are not in the session so long as the server has the user id. So long as a user id is being supplied (or a proxy such as the session token id), such data being stored in the session is simply a server optimisation and thus does not violate REST guidelines since this is application state and not conversational state.

To the extent the data is storing conversational state such as the fact that the user has authenticated himself and the fields that the user may have entered in page 1 of a two page form, such is incompatible with REST guidelines, and if such practices are adopted the resultant design may not be termed fully REST compliant.

So the existence of a user token does not make the design REST “uncompliant”. Storing conversational state in a user session usually associated with such a token does.

One way to ask if a data is conversational or application, may be to ask oneself, that can the request be satisfied properly if it was accidentally routed to a different server in a situation where the cluster was configured for session affinity. In a REST compliant design, the other server will have a capability to still be able to service the request. However this requires the presumption that the cluster is configured for session affinity and session state is not available to other servers. Should one configure the cluster with shared and distributed sessions, the cluster will be able to service the requests even when the APIs were not REST compliant.

The one thing that does leave me a little uneasy is that to be fully REST compliant one will need to always supply a userid / password with each API call instead of a server token. That does have implications on security, implications that I am not entirely comfortable with.

Update :Surya in a comment in earlier post refers to the fact that authentication need not be done through the URI but through the headers or through protocols such as OAuth. So what that leaves me with is the assessment that existence of a token or a session object is not a violation of REST but storage of conversational state in the same is.

Related posts:

1. Fomenting unREST : Is RESTfulness a semantics game ? Why does REST require statelessness ?
2. Design Characteristics of REST / Resource Oriented Server Frameworks and Clients
3. Why REST ?

Update : Further breaking news on the topic can be found here.

No related posts.

Background article (not exactly same but on a similar topic) : Improve your web based software development and maintenance ROI with dynamic programming languages

Talk Announcement : Seminar: Strengths and weaknesses of various programming languages – 28th March

Related posts:

1. Improve your web based software development and maintenance ROI with dynamic programming languages
2. Java : the perpetually undead language
3. Commentary on Python from a Java programming perspective

A month later I ran into this article Tighter Ruby Methods with Functional-style Pattern Matching, Using the Case Gem which again piqued my interest in the same matter. Thats when I looked at the topic once again and came up with the following similar example (similar to the one in the blog post referred to), to demonstrate constructor overloading.

Note that method overloading is not consistent with duck typing approaches for many. However many a time we do run into a situation where conditional logic is required based upon types. As one attempts to bring in a more FP oriented style, I suspect the need for type matching is likely to only increase. As an example you may have a reference to a stream variable which you need to interpret differently based on the format, say xml or json. In such case one approach is to have two functions parse_as_xml(stream) and parse_as_json(stream) What the construct below allows you to do is to have a common function around the two parse(stream) which can switch between the two internally.

class X(object):
    def __init__(self, *val):
        # Matcher functions. These return true or false for the type matching
        # Possible enhancement could be to have  generic function
        check_dict = lambda x : isinstance(x[0],dict)
        check_sequence = lambda x : hasattr(x[0],'__iter__')
        check_args = lambda x : len(x) == 3 

        # Type specific initialisers
        def initialise_from_dict(hash):
            self.first_name = hash['first_name']
            self.last_name = hash['last_name']
            self.age = hash['age']

        def initialise_from_args(*args):
            self.first_name = args[0]
            self.last_name = args[1]
            self.age = args[2]

        def initialise_from_sequence(seq):
            self.first_name = seq[0]
            self.last_name = seq[1]
            self.age = seq[2]

        # Matching data. This is the switching data based on which the appropriate
        # function is called.
        dispatch_data =((check_dict,initialise_from_dict),
                        (check_sequence,initialise_from_sequence),
                        (check_args,initialise_from_args))

        # The switch
        for check,func in dispatch_data :
            if check(val) :
                func(*val)
                break

    def __str__(self):
        return "%s %s(%s)" % (self.first_name, self.last_name, self.age)

print X('hello','world',33)
print X(('greetings','earthlings',44))
print X({'first_name' : 'john','last_name' : 'doe', 'age' : 45})

Essentially it boils down to create a list of function pairs, the first to be executed to check if the second should be performed. Some of the characteristics of this approach are :

• Even thought the example is for constructor overloading, the same could be applied to normal function overloading at both class and module level
• No external packages being used
• The different functions which serve the same intent are actually packaged as nested functions within the function thats being attempted to be overloaded. This allows for a cleaner structure. However that is not a requirement. The functions can be moved into the top level name space and the only change that will be required in the example above is the necessity to explicitly pass the “self” argument as the first argument.

One of the possible enhancements to this could be to have a generic pattern matching function and make the first argument in each of the items in the matching sequence (dispatch_data in above case) be some pattern data which results in the pattern matcher function returning a True or a False.

This is not the only way to overload but I found it amongst the easier and simpler structured approaches.

Related posts:

1. Python from a Java perspective – Part 2 – How duck typing influences class design and design principles
2. Commentary on Python from a Java programming perspective
3. Twitter / HTTP / REST API Invocation Infrastruture using data pipelines

Process Maturity (especially CMM and ISO oriented maturity) focus on ensuring consistent, measured and self improving quality of output. This is achieved through multiple checks and balances, and detailed documentation of processes, and regular audits to identify points of weakness and their redressal. At least to the extent I have been exposed to them, people are looked at as cogs in the assembly line (replaceable and faceless) and focus is on ensuring that the assembly line works at a consistent pace even as people move around from job to job (or from assembly line to assembly line). Thus the focus is on setting expectations at modest levels and ensuring that these expectations can be met even as various events occur which introduce risk into the process, since the process methodology and documentation is geared to help address many of these risks.

Agility on the other hand probably stemmed from some of the concerns around the non-people-centricity and heavy-documentation-focus of these process maturity models. It instead ended up focusing on Individuals and their Interactions and on accepting the necessity of Change and thus adopting it wholeheartedly as an aspect that is inevitable and thus best accepted as a feature rather than a bug. Thus rather than slowing down the delivery, it attempts to substantially speed it up by removing many process steps which may seem to be impediments to progress. While substantially reducing most control and measurement points such as documentation, reviews, audits, they replaced them with steps which allowed quality to be maintained by introducing higher skills and discipline into the people (test driven development, pair programming, daily scrum).

Thus if a development team tells me they follow a process maturity model, assuming they are being completely honest in that statement, it tells me that as a customer I can expect them to set a pace and generally largely stick to it with incremental improvements while shielding me from having to deal with many of the internal risks they battle with.

If on the other hand a development team tells me they follow the agility model and again assuming they are being completely honest, it tells me they have a lot of respect in their own capabilities and to their ability to adaptively deal with internal risk through better earlier risk detection and redressal (Test Driven Development / Continuous Integration etc.) rather than through process oriented controls such as Documents / Audits / Reviews etc. I would also generally expect a much much higher productivity and quality from the team but with a higher statistical variance.

Ceteris Paribus, as a customer I would feel that the Agile Team is likely to introduce more schedule risks in situations of higher person turnover or lesser developer disciplines, while the team that focuses on Process Maturity is more likely to introduce schedule risks in situations of changes in requirements or in situations where a very quick time to market is called for.

What is obvious to me is that these methodologies have been defined with very different attitudes. If we play a “one word that comes to your mind” game with the terms, its “power” with “agility” and “control” with “process maturity”. Both are important and desirable and play a role, but the slider is positioned differently in each of the cases.

It has been rather obvious that agile processes have been occupying a greater mindshare over the past 5 years (the same thing happened with process maturity methodologies in the decade before that). I had been concerned that, as with many other trends in Software, the word Agile might also start getting abused where people start using it quite differently, to leverage on its momentum and the mindshare. It has been happening for a while, but the article Strategies for Scaling Agile Software Development The Agile Process Maturity Model just blew me off. I have only the greatest respect for Scott Ambler and especially for all his writings on Persistence. But in this article all the agile methodologies have been clubbed together as “Level 1 development” and many of the rather heavy weighted processes some of which probably led to the formation of the agile alliance in the first place are listed in the “Level 2 (disciplined) delivery”. I must clarify that agile software development covers all aspects of lifecycle including testing and measurement and agile software methodologies deliver as well (contrary to what the article “seems” to suggest) and rather than building discipline into processes, reviews and audits it builds discipline in the people that form a part of that process.

Since agility and process maturity are based on different attitudes I can’t see how one says I am going to put both the things together unless one intentionally and quite deliberately repositions the slider at a level with offers lesser power than agility and also dilutes process control compared to process maturity. But thats not the impression I got from the article. It seems to suggest that one can take these “level 1″ agile processes add “level 2″ disciplines to it and then add some uncertain “level 3″ elements to them to make things better than where they stand today, something that I am a little skeptical about.

So when a development team tells me they are following the “agile process maturity model (APMM)” I am likely to believe that they are either (a) too confused, (b) want to appear to leverage the agile momentum while using the strict control regime or (c) successfully figured out a way to move their slider somewhere in between with the risk that they lost some of the best elements of the two methodologies while attempting to lose the worst.

Having said that, it will be interesting to read Scott’s future articles on topic and to follow this trend. I have been proven wrong in the past, and it could happen again for me to have to eat my words. But moving that slider between power and control to attempt to achieve the best of both seems a little risky, and you can be sure I am unlikely to be an early adopter for the fear of getting severely burnt.

Update :I forgot to add that amongst the many items listed in Level 2, those such as (throwaway) sketching, whole team, test-driven development (TDD), continuous integration, daily standup meeting are accepted aspects of typical agile methodologies whereas those in the level 2 methodologies which agilists often avoid as actively managed and maintained charts are not even listed eg. in case of RUP – use case diagrams, class diagrams, sequence diagrams, state diagrams. So what is the contribution of other methodologies such as RUP in this case is largely unclear. I do not know the other Level 2 methodologies so can’t comment on the same.

Related posts:

1. Software / IT Terms in early stages of abuse or ripe for Abuse

I have been working on scouring and analysing twitter data for which I have been having to work with continuously accessing twitter on a sustained basis for a number of days. I started refactoring my code recently, and this blog post details the results of that exercise. More specifically in the context of invoking HTTP APIs it deals with the following aspects (many of them which are specifically introduced due to twitter.

• Ability to make HTTP Calls and deal with HTTP error conditions
• Ability to deal with Connection and other failures and allow for auto retries
• Ability to make HTTP Calls in bulk in batches
• Ability to spawn HTTP Calls across multiple threads
• Ability to respect and deal with API Rate Limitations

I have specifically focused on creating a pipelined design which might be an interesting design aspect for many in this situation, and have primarily relied on python based generators for the same though similar functionality could be built using Iterators in other languages as well.

Accessing sites using HTTP Basic Authentication

Twitter is accessed using Python urllib2. Since many API’s often require using basic authentication, we shall set up the HTTP Opener to be able to work with such sites.
To be able to initialise the opener we define a method get_opener() which accepts the username, password, realm and host to be able to setup the opener for the site.

def get_opener(username,password,realm,host):
	"""
	The default urllib2 opener is adequate for non authenticated http api requests.
	However twitter API requires basic authentication in many situations. Hence
	this method initialises the opener to use the appropriate user id / password
	parameters
	"""
	headers = {}
	basic_auth = base64.encodestring('%s:%s' % (username, password))[:-1]
	headers['Authorization'] = 'Basic %s' % basic_auth
	handler = urllib2.HTTPBasicAuthHandler()
	handler.add_password(realm, host, username, password)
	opener = urllib2.build_opener(handler)
	opener.addheaders = headers.items()
	return opener

Making a Basic HTTP call to fetch JSON data

Twitter supports data both in XML and JSON formats. I tend to prefer json since it is much more compact on the network and is more efficient to parse and construct as well. As a result I have function which extracts the data from a URL as presented below. It takes the opener and the URL and returns a constructed data object. Note the class Data which has no predefined attributes but instead dynamically adds attributes to itself based on JSON attributes. Moreover it checks whether the return parameter is a single object or a sequence and acts correspondinglyThe logic actually should work with nested dictionaries (dictionaries within dictionaries) as well, which is currently a pending exercise

class Data(object):
	"""
	This is a generic data object. It is a completely dynamic object meant to
	create itself from a dictionary or a set of nested dictionaries. All keys
	in the dictionary are converted to object attributes allowing for an easier
	and more intuitive access to the object attributes.
	"""
	def __repr__(self):
		return "Generic Data Object:%s" % self.__dict__.__repr__()
	def __str__(self):
		return "Generic Data Object:%s" % self.__dict__.__str__()

	@classmethod
	def load_from_json(self,json):
		data = Data()
		for key,val in json.items():
			data.__dict__[key] = val
		return data  #to allow chaining of calls

def http_to_json(opener,url):
	"""
	This method invokes the remote URL which is expected to revert with a JSON datastream. It returns
	an object with the json keys being attributes of the object along with the corresponding values.
	"""
	try:
		response = opener.open(url)
		stream = response.read()
		# A JSON stream begins with a { or a [
		if (stream[0] == '{') or (stream[0] == '[') :
			jsonobj = simplejson.loads(stream)
			if jsonobj :
				# Some twitter API return an 'error' in case of error
				if stream[0] == '[' :
					data = [Data.load_from_json(obj) for obj in jsonobj]
				else:
					data = Data.load_from_json(jsonobj)
	except urllib2.URLError, e:
		print 'Received Error while fetching twitter record for url : %s = %s' % (url, e)
		raise e
	return data

Simple Client

I think we have enough infrastructure methods at this stage to initiate a simple HTTP invocation to give us an idea of how these could be used. In the code below twitter_user_id and twitter_password will need to be changed to your user id and password. This shows you how a simple twitter call could be made.

if __name__ == "__main__":
	opener = get_opener('twitter_user_id','twitter_password','Twitter API','twitter.com')
	data = http_to_json(opener,"http://twitter.com/account/rate_limit_status.json")
	print data

Automatic Retries

We do know that network communications are not perfect and occasionally we get errors which are not due to the data but due to the network or technical infrastucture. Errors that are unlikely to be repeated if the same function is retried again. Hence we work out a common retry function which will retry certain function invocation based on various retry policy parameters. Note that retry strategies cannot be used when the functions are not idempotent, especially if it is not possible to ascertain if a invocation has been received and processed by the server. However most retrieval calls are idempotent (except for the side effect of twiter api rate limits still getting impacted), so we should be able to use the retry logic in most cases. The retry method is as follows.

def retry(max_tries, check_error_func, break_codes, continue_codes, retry_policy, func,args):
	"""
	This is a wrapper which runs a function in a retry loop . In this case it is assumed to be
	a function thats dealing with HTTP. The arguments are as follows :

	max_tries : maximum number of tries before giving up
	check_error_func : function to check if a repeatable occurred. (Repeatable errors should
		not be retried. Plugin for custom logic.
	break_codes : sequence of http error codes on which no retry should be attempted
	continue_codes : sequence of http_error codes on which retry should be attempted
	retry_policy : default policy in case of http code not matching either of the two sequences
	func : function to be performed / executed
	args : arguments to the function
	"""
	try_num = 1
	success = False
	data = None
	e = None
	while try_num <= max_tries :
		data = None
		e = None
		try :
			data = func(*args)
		except Exception, e :
			pass
		if check_error_func(data,e) : break
		if not check_retry(data,e,break_codes, continue_codes, retry_policy) : break
		try_num += 1
	if e : raise e
	else : return data

There are still some missing gaps we need to fill in in this case. The check error function is domain specific (in this case twitter specific) which checks for an attribute called error in the data object as follows.

def is_twitter_error(data,e):
	"""
	Method to check if the contained data is actually an error based on twitter return
	value semantics
	"""
	if hasattr(data,'error') :
		return True
	return False

We also know that http error code 404 means a not found, and retrying is unlikely to help that. However code 520 is often received when twitter is over capacity and 101 is received in case of network communication failures, good candidates for retrying the http calls. Finally for other error codes I preferred to have a default policy as retry though your choice could be different. Thus the final invocation along with the retry logic looks like follows

rate_status = retry(3, is_twitter_error,(404),(520,101),True,get_rate_status,(opener,))

Bulk Processing Twitter IDs

It is often useful to do processing in bulk, especially when one does have bulk data to process. In such a situation the starting point is the list of data elements to be processed. We shall be attempting to retrieve twitter data in bulk for a large number of user ids. So the starting point is a generator which can return a list of user ids that we need to fetch. For sake of simplicity, I have written a generator which will generate twitter ids as a sequence of numbers, though you could replace it with a more suitable generator as appropriate

def gen(min,max):
	"""
	A generator to generate twitter ids
	"""
	counter = min
	while counter < max :
		yield counter
		counter += 1

Batching

In case of bulk processing, it is often useful to break down the data in a set of batches. To be able to do that we need a generator which can generate a set of batches from an underlying generator of ids (defined above). Usually the batch size could've been fixed, but in case of twitter, API rate limits apply and hence sometimes the batch sizes need to be computed dynamically depending upon the remaining available API calls. Thus we first define a function which decides on the batch size based on the preferred batch size and remaining hits as per twitter. It also uses a keep_unused parameter to ensure that a certain number of API calls are left unused (ostensibly for usage by other programs)

def twitter_batch_sizer(opener,max_size, keep_unused):
	"""
	Method to generate a batch size based on available rate limit, maximum batch size, and number
	of API calls to keep as unused
	"""
	while True :
		rate_status = retry(3, is_twitter_error,(404),(520,101),True,get_rate_status,(opener,))
		if rate_status :
			size = rate_status.remaining_hits - keep_unused
			if size > max_size : size = max_size
			elif size < 0 : size = 0
			yield size
		else : yield 0

Now we also need to create a generator which will generate batches using the underlying generator. However I did run into a peculiar problem. When the underlying generator as completed (StopIteration), the batch generator also needs to complete processing. This required setting up of a booleanfunction attribute, but since function attributes are not mutable, I had to define a BooleanWrapper class which internally contains a boolean attribute and has a false() method which sets the attribute value to false. The new class along with the Batch generator are shown below.The sizer and generator arguments are functions / generators that we have already defined earlier.

class BooleanWrapper(object):
	"""
	This is just a dummy object to represent a boolean. It is wrapped in an object
	since function attributes are not mutable from within a function body. However calling a
	method on an object to change its internal attribute works.
	"""
	def __init__(self):
		self.status = True
	def false(self):
		self.status = False

def batcher(sizer,gen):
	"""
	This object takes a sizer function which computes a size of a batch dynamically, and creates
	a batch out of the generator function supplied by gen. Thus it effectively breaks down the
	data supplied by gen into individual potentially unequally sized batches as specified by the
	sizer function

	To put it differently, it takes a generator and outputs a generator of generators, each
	second level generator being a batch
	"""
	loop = BooleanWrapper()
	def lot(sizer,gen):
		counter = 0
		size = sizer.next()
		if size == 0 :
			loop.false()
			raise StopIteration
		while counter < size :
			try :
				x = gen.next()
				yield x
				counter += 1
			except StopIteration, e:
				loop.false()
				raise StopIteration
	while loop and loop.status:
		try :
			yield lot(sizer,gen)
		except StopIteration, e :
			raise StopIteration

Threading

Now that we have batching support, we might like to be able to multithread the calls. In my situation, I was whitelisted by Twitter so had an API rate limit of 20,000 calls per hour. However I prefer to get the API calls done as soon as possible (say in the first 20 minutes of an hour), and use the remainder of the time to do additional processing. Being able to process 1000 API calls per minute requires support of either multiprocessing or multi threading. Since I was interested in only having one executable (from a manageability perspective) I decided to use the multi threading approach. Note that python has the Global Interpreter Lock (GIL) issue, but thats unlikely to play a role since threads easily yield the GIL to each other when they reach blocking IO points and most HTTP calls are indeed blocking IO.

Having decided on threading, I decided to spawn a set of threads per batch, process the batch, shutdown the threads and continue processing. There are more efficient ways of maintaining a continuously running thread pool, but given the limitations of the rate limit APIs, even the implementation I chose is far more than adequate as far as efficiency goes.

Anyway for any threading operations we either need a Thread subclass with a run method or a function which will be executed in a thread. Given the simplicity of this situation I chose to use a function to retrieve a particular user's information as follows. This function shall be invoked simultaneously by different threads for different user ids.

def threadfunc(threadid, queue, process_func, func_args, global_dict):
	"""
	This method runs takes data item from a queue, invokes another function as specified by
	process_func and its arguments func_args, updates the result in the dictionary global_dict
	with the key being the data item. Since this function is to be intended to be run as a
	standalone thread, the first argument threadid, is a identifier for the thread purely for
	debugging purposes
	"""
	print "Thread %s started " % threadid
	while not queue.empty() :
		try :
			val = queue.get_nowait()
			args = [arg for arg in func_args]
			args.append(val)
			result = process_func(*args)
			print "Thread %s processed %s with result %s" % (threadid,val,result)
			global_dict[val] = result
		except Exception, e:
			print "Unexpected error in thread %s for value %s : %s " % (threadid, val, e)
		queue.task_done()
	print "Thread %s exited " % threadid

Note that I added the print statements just so that you can follow the program when it runs. These are not a part of the final code. It is quite important here to be able to handle all exceptions, and not leave any exceptions unhandled. Unhandled exceptions result in the main program getting terminated, something that is probably not such a great thing to have for a continuously running program. However I haven't really included any error handling code here apart from a simple print statement.

If you noticed this function still doesn't contain our actual twitter userid fetch logic. Thats in another function which is passed as a parameter to the above function

def resultfunc(opener,val):
	"""
	Method to generate user data along with the retry and twitter specific error testing
	"""
	user_data = retry(3,is_twitter_error,(404),(520,101),True,get_user_data,(opener,val))
	return user_data

Having written a function that can be executed in a thread which will retrieve user data, we now need a function which will take a batch, spawn a bunch of threads, feed the data to the threads through a queue, and allow the threaded function to process each element in the queue concurrently. This function is as follows.

def batch_threader(size,gen, process_func, process_func_args):
	"""
	This is a generator which expects a two level generator (generator which returns a generator).
	The top level returns a set of batches, whereas the nested generator supplies a set of
	individual items in the batch. It iterates through all the batches, and for each batch spawns
	a thread pool of the requested size and executes the function process_func along with the
	supplied arguments on each of the items concurrently in the thread pool. It finally returns a
	dictionary with each item being the key and the result of the processing being the value.
	"""
	batch_number = 0
	for batch in gen :
		batch_number += 1
		print "Running Batch : %s with pool size %s" % (batch_number, size)
		# Construct a list to figure out size of queue
		items = []
		for item in batch :
			items.append(item)

		# Initialise queue
		queue = Queue.Queue(len(items))

		# Push Items into the queue
		for item in items :
			queue.put(item)

		# Start threads
		result_dict = {}
		for i in range(size) :
			t = threading.Thread(None,threadfunc,None,
					(i,queue, process_func, process_func_args, result_dict))
			t.setDaemon(True)
			t.start()

		# Wait for all threads to complete processing
		queue.join()
		yield result_dict

Putting it all together
Finally here's the main body which actually assembles the data pipeline and triggers the full processing. Since the typical API rate limit is 100, I have set the keep_unused to 90 (so 10 API calls are likely to be used). I also set the preferred batch size as 6 and thread pool size as 3. This should result in two batches being fetched, the first of size 6 and the next of size 4 (assuming your API rate limit is at 100) through a threadpool of 3.

if __name__ == "__main__":
	opener = get_opener('twitter_user_id','twitter_password','Twitter API','twitter.com')
	for results in batch_threader(3, batcher(twitter_batch_sizer(opener,6,90),gen(0,10)), resultfunc,(opener,)) :
		print "====== Begin Batch ======="
		for key, val in results.items() :
			print "%s was processed with result %s" % (key,val)
		print "====== End Batch ======="

In summary

I would've actually preferred to blog at length about the various design characteristics, but since this post has become really long, I shall probably come back to it in the coming few days. However I would encourage you to note the brevity (about 175 lines of non commented source code) in which we have been able to achieve a lot, and a clear separation of the infrastructural and twitter specific code. Moreover its an example of really putting together a lot of diverse functionalities and capabliities by using building blocks which can be plugged together to work with each other (thanks in no small part to the function objects and generator support of python). The full listing is in the file twitter-http-rest-api.py.

Related posts:

1. A twitter feed in gmail : Write a google gadget
2. Poll : Usage of web services (SOAP / REST / HTTP)
3. Why I deleted my Facebook data. Commentary on Internet data privacy rules.

Scott states

The best place to start is empathy. Why is someone acting like a jerk? There are basic psychological reasons for this: Either they are insecure, they are unhappy, or they are angry about something.

Ok, there is a fourth reason, that they are psychopathic hell spawn put on the earth to torture all living things in a 10 foot radius, especially you, but lets assume that’s not the case for a moment.

That fourth one sounds eerily like me, but I shall pretend thats not the case. Scott further goes on to make a set of suggestions (read his post for details, I am just listing them below)

• Charm to Connect
• Demonstrate your ability to help
• Agree on the roles you both play
• Get help from allies

While definitely in the right direction, these are very tactical steps. And I really am not sure that the jerk programmers can be “unjerked” with these. There are two types of jerk programmers. Those who simply enter the office with a combined sense of arrogance and disdain that the situation is likely to be beyond any reasonable likelihood of any decently workable relationship. Thankfully this is likely to be a very rare set (I suspect they exist, but haven’t ever worked with one). At least half of the remainder are likely to have some issue with the “big picture” work atmosphere. Something that reeeallly bothers them. Something they are finding difficult to reconcile to. Something thats gnawing at them.

I think there are two options in this case. If you can largely agree with the issues, ensure that you not only empathise and help (as Scott mentions) but also make sure you’re clearly seen to be believing and acting towards solving the issues. I think that belief is a very important element which sets up a shared goal and is complemented by the actions and results. The programmer needs to believe you are on the same side (in his opinions the right side) of the issues as him many times even if not completely so. This could be related to number of aspects – technology choices, code or design quality, deadline tradeoffs, deadlines, territorial control issues etc. etc. A programmer is very unlikely to actually promote himself to jerkdom because of the coffee vending machine being located on the wrong side of the aisle or even due to more serious matters such as role ambiguity or disagreement. It usually runs much deeper.

In case you cannot agree with the issues, I think a clear talk needs to be had about the differences, the criteria that are going to be laid down, and the programmer should be encouraged to either adjust to these or attempts should be made to find a role / position where such differences can be minimised. In extreme cases it may be helpful to encourage the programmer to relocate to other teams for a win-win resolution.

To summarise, if a programmer is acting like a jerk, there’s likely to be a much deeper issue. Work with him, be seen to be friendly to and believing in his case and be seen to be on his side many times on the issue and demonstrate at least some tangible progress on the issue at regular intervals. Alternatively in extreme cases find avenues to work independently. There is a risk of lesser tactical resolutions simply prolonging the difficulties for all concerned.

No related posts.

Unlike my earlier post, this does not critique Facebook’s actions but instead attempts to analyse the statements in the context of a newer dimension of data thats mobile on the cloud. This brings in both a set of challenges and opportunites and I shall try carefully to not fall into the FUD zone as I discuss these challenges. Finally it also discusses these statments and its implications on user’s and their awareness education challenges. In this exercise I treat Facebook as one of the early providers who is facing and attempting to address these challenges of “mobile data on the cloud” in terms of its privacy and security implications. But in the mashup world, it is likely to be a challenge that both the users and a number of other service providers have to eventually start dealing with.

Allow me to add my annotations on these statements.

Proposed Facebook Principles

A very nice and clear enunciation.

• Point 1 : Note the specific mention of “service”. Thats what in my opinion is leading to many of these challenges.
• Point 2 : Perfect in principle. However one technical issue needs to be resolved. How does one define “received information“. If my status update is seen by another user and it has appeared on his wall, has he received information ? Probably not since it is quickly clarified that this is particularly outside the Facebook Service. So we are probably talking about an external application to which facebook feeds the data, another user who might have downloaded some data on his desktop, potential third party services such as search engines (which usually aren’t allowed to scour this data) or other analytics services. I think its a fair statement. The only suggestion I would have is that there probably needs to be some amount of user (re)education to understand its implications.
• Point 3 : Clearly refers to the fact that information will be shared with other users and services including through tools (desktop apps / mobile apps ?). Sets up the ground for API usage.
• Point 4 : A great point. Establishes a level of symmetry amongst the parties which was one of my sore points in the earlier modified TOS
• Point 6 : Clearly documents API usage to access data
• Point 10 : Notes that the data is not just mobile across organisations (ie Facebook and its applications) but also across geographic boundaries.

OK, lets quickly review the next statement.

Proposed Statement of Rights and Responsibilities :

Point 2 is an important one to note, especially 2.2 and 2.3. Point 2.2 is wonderful to the extent that it clearly defines that facebook eventually will delete data that a user deletes or in case the user deletes the account. It also specifies that content shared with others may remain until they delete it. Now this content is in all likelihood going to be third party applications and services. Here’s an interesting tripartite arrangement – theres now going to be (for practical purposes) a mini-TOS between the user and Facebook which this statement covers, between Facebook and the third party provider, which I think is likely to be covered within this document as well (well, at least parts of it) and probably one between the user and the third party service. Its the last one which could be worrisome, since the user may now need to worry about each TOS with each third party provider. Most users are unlikely to review each of those terms with the level of detail, yet find their data moving to the third party provider. However, how and when that could get deleted is also addressed later in this statement. Point 2.3 also clearly documents the fact that subject to the user privacy and application settings Facebook assumes a whole bunch of capabilities around such intellectual property. Unlike in the modified TOS, this gives the control to the user to decide how the intellectual property will be treated. Since the control is with the user, I think it is a fair clause to ensure that Facebook itself is not at the receiving end of any intellectual property rights dispute.

In point 3 Facebook attempts to secure the fact that its users do not harvest information or otherwise maliciously gather access to store information that they shouldn’t get access to store. Why the focus on store ? The clauses around using the API are likely to have strict conditions around what data can be stored by the third party application and for what duration (which is true today as well). So its the social or screen scraping or other non-API ways of collecting data that Facebook is trying to secure against.

Point 5 sets up the environment for users to respect property rights of each other. Point 5.7 especially ensures that services clearly document their privacy policies when collecting data and inform the user that the data is being shared with the service and not with Facebook. Point 5.8 ensures that users don’t post other user’s sensitive information and then let it stream over the cloud potentially into other applications and services. Quite reasonable in my opinion.

Point 9 is probably one of the most interesting sections. The important point here is that Facebook has clearly communicated with the user, the undertakings of the applications, websites and services. Coupled with 5.7 and 5.8 above, it essentially creates an environment where the user is clearly aware of the boundaries between Facebook and other services, and that once the data crosses that boundary, that data for all practical purposes is governed by the agreement between the user and the other service. 9.16 was one which did arouse my curiosity a bit. We now start seeing the outlines of why the TOS needed to be modified in the first place.

Finally point 12 also goes to a great distance in assuaging the concerns raised about unilateral and quiet change of TOS earlier. It allows for upfront communication with the users, a reasonable notice period and curiously a vote based user veto (not sure why this really was required, but then I would be surprised if 30% of the user base actually voted).

So what about data sans frontiers ?

In the earlier modified TOS, Facebook had attempted to secure a very large number of rights in order that it would stand protected (if and) when large parts of user’s data start getting shared with other applications, websites and services. Thanks to the user reactions, it has now crafted a very different statement. In this statement while Facebook now assumes far fewer rights, and those it assumes to me seem like part of a fair and symmetric relationship. The way I read and interpret this is that under this statement, Facebook clearly defines a set of boundaries or frontiers (ie. the Facebook Service), and undertakes and provides a mechanism to properly share and control the sharing of such data within such frontiers. However it also makes it abundantly clear, that the user is now adequately aware that it has far lesser control the moment the data crosses such frontiers, and at such a stage this data will now get controlled by the third party services, that the undertakings by the third party service are clearly specified and the user is fully in the loop on this issue. Again seems fair from a corporate perspective and in terms of securing itself from misbehaviour or other inappropriate activities or accidents on the part of the third party services.

The catch is now that the onus is on the user. He has to clearly understand the implications of (perhaps even more detailed) privacy controls, and agree to the fact that once the data crosses the facebook frontiers, facebook has taken adequate precautions in terms of ensuring that the appropriate undertakings having been provided by the users and the services, and that any issues that crop up further could perhaps be bilateral issues between the users and the third party service.

I don’t know that all users are actually in a shape to understand all these implications, and suggest that some mechanisms be put in place to provide users some tutorials or presentations which explain these in as easy and lucid manner as feasible. Users will need to exercise the Caveat Emptor clause before subscribing to / using other services which integrate with Facebook. Users will need to be aware that based on their privacy settings and application requirements, their data is now moving from host to host, company to company, and country to country. On the whole while this does place a far bigger burden of exercising caution on the user, it also puts the user in control and is a far more symmetric statement than the earlier modified TOS.

Finally in an increasingly mashup and service integration driven world, I fully anticipate many more companies and users having to deal with these very issues in other non Facebook contexts as well. This statement is likely to also play a role in the ongoing discussion about the capabilities and the risks of having data in the cloud. If these statements can be agreed upon in a format that is acceptable to all, there’s a good likelihood it will provide the precedent for many more service arrangements being drafted in the future in a relatively similar and consistent fashion.

A programmer is unlikely to be a good lawyer : I am a programmer and have very lay understanding of legal matters. This opinion is based on my understanding but such an understanding could be wrong or inaccurate when looked at by a more skilled person. It is most certainly not a legal opinion and could contain discrepancies, inaccuracies or improper assessments on my part. If you note or find any, please add your comments so that others can benefit from your views as well.

Related posts:

1. Why I deleted my Facebook data. Commentary on Internet data privacy rules.
2. Platform Designers – Think Reach, Cloud, Usability

Terms of Use Update

A couple of weeks ago, we posted an update to our Terms of Use that we hoped would clarify some parts of it for our users. Over the past couple of days, we have received a lot of questions and comments about these updated terms and what they mean for people and their information. Because of the feedback we received, we have decided to return to our previous Terms of Use while we resolve the issues that people have raised. For more information, visit the Facebook Blog.

Mark Zuckerberg also blogged about the same issue in Update on Terms.

Original post begins here.

Facebook published a new Terms of Service on February 4th 2009 which has a strong implication for how internet / cloud based data privacy is likely to be viewed. This was very well publicised here – Facebook’s New Terms Of Service: “We Can Do Anything We Want With Your Content. Forever.”. There was some consternation on the net especially on twitter about this change in facebook rules. While I did not use facebook much, I was sufficiently appalled at the change in rules to go and delete pretty much most of my data one line at a time. It is unclear to me if the old data is still available to facebook for sublicensing from a legal perspective (I know all the data will be there in their archives), but I decided it probably wouldn’t hurt to nevertheless to go delete most of it. I didn’t actually delete the account since Facebook still helps me keep in touch with my friends. But it is quite safe to assume that any interactions with them with an assumption or requirement of any data privacy will no longer be done on facebook.

Whats wrong with the new terms of service ?
Some people in forums argued that most of the data on internet is likely to be there forever. So one just needs to be careful and not worry about it. I don’t quite agree with that line of thinking. When I blog, tweet, post to usenet or forums, I am upfront aware of the fact that that data is going to be cached by google and other search engines and that once I press the publish button, there’s often no way to revoke it. However in case of Facebook, there is a general expectation that the data will be shared only within a network of friends, a network that I have control over. There is an expectation that that data will not get cached by search engines and short of an accidental data breach or some intentional malafide activities that data will not become public. What is unnerving with the new terms of service is that Facebook changed these rules at will without even sending me an email about the same.

Asymmetry of Privacy Expectations :

It is interesting to note how asymmetric some of the terms are. For example in the section User Content, the following is to be found.

By using or accessing the Facebook Service, you represent, warrant and agree that you will not Post:

* User Content that violates the law or anyone’s rights, including intellectual property (”IP”) rights or other proprietary rights (such as rights of publicity and privacy);
* any Contact Information or private information of any third party;

Further down in the section Licensing, it states,

You hereby grant Facebook an irrevocable, perpetual, non-exclusive, transferable, fully paid, worldwide license (with the right to sublicense) to (a) use, copy, publish, stream, store, retain, publicly perform or display, transmit, scan, reformat, modify, edit, frame, translate, excerpt, adapt, create derivative works and distribute (through multiple tiers), any User Content you (i) Post on or in connection with the Facebook Service or the promotion thereof subject only to your privacy settings or (ii) enable a user to Post, including by offering a Share Link on your website and (b) to use your name, likeness and image for any purpose, including commercial or advertising, each of (a) and (b) on or in connection with the Facebook Service or the promotion thereof.

As you can see, you undertake to not violate anyone else’s IP or other proprietary rights, but information about you will not be treated with the same level of respect by Facebook, though its done quite legally by documenting the same in the Terms of Service.

Moreover anything you post or any information on your stream is now sub-licensable by Facebook. Now why would I exactly want to sign away all rights on status updates, photographs etc. on content which I posted assuming that it was secure and private ?

But the earlier terms were also quite onerous. So how come you did not complain ?

Apparently under the earlier terms, facebook also had the rights on the content, so whats the big deal ? Two main issues.

1. The earlier TOS did not grant Facebook the right to sublicense the content : The possibility of sublicensing means you have no control or idea on who the eventual user of that data could be. I still get angry at so many commercial parties at having leaked my email and phone number data. The likelihood of a similar scenario where facebook sells that data for commercial purposes now cannot be ruled out, purposes on which I will have no control on that data.
2. The earlier TOS had an escape clause of deleting the account Basically Facebook did not have the right on the data once you deleted the account. This is important as can be seen by another case on Twitter Privacy Disaster At Twitter: Direct Messages Exposed (Update: GroupTweet Is Likely Culprit). In this case private messages were apparently accidentally made public due to confusing software usability. The person immediately responded by deleting the account. This is a useful kill switch to have in case one makes a terrible terrible mistake of putting out something accidentally. This kill switch is also no longer available.

Bait and Switch : By not informing users of the change in terms of service especially since these were so important, I think this creates an impression that the user is a victim of bait and switch (even though the real underlying causes of the change which I am unaware of could be different). Facebook should’ve informed the users about the change in rules, offered a button to delete all prior data / photographs / content or at least made clear that the earlier content will continue to be governed by the earlier TOS – something thats a little unclear in this situation.

Implications for Internet Web sites and users : I think sites should very clearly document how they will control and use the data that they gather. Many of them do by explicitly document the same. Moreover any substantial changes to the same should be communicated to the users. Finally users need to be now aware of potentially changes of Terms of Services on a number of web sites that they interact with. Data that they assume to be private may no longer stay so and the user may not be any wiser about the same if the Terms of Services are changed without him being explicitly informed.


Updates :

Why did I delete the data ? Seems some readers are thinking I deleted the data believing that that will get rid of it. Thats not why I deleted the data. I am fully aware the data is likely to live perpetually in facebook archives and be accessible to facebook. I deleted it because that data had been submitted and generated under the old Terms of Service. Letting it be around to me seemed like an implicit acceptance of the new Terms of Service around old data, which I was uncomfortable with. So I deleted the data at the first available opportunity on realising that the Terms of Service around that data had changed. Any new interactions I do with facebook will be under an awareness of and therefore an acceptance of new Terms of Service.

Response from Facebook : Mark zuckerberg attempts to address the issue on facebook blog : On Facebook, People Own and Control Their Information. I could not find any rationale to why Facebook needs the privilege to sublicense the content. I also thought the way the blog post was written and the way the Terms of Service are structured are very very different. In my opinion its the Terms of Service that count.

This topic has been also getting a lot of traction on other blogs. Am quoting some other interesting opinions on the topic on the internet along with link backs to the posts below

• cnet News.com : The Open Road : Facebook changes terms of service to control more user data :
  

  Google has had its own problems with user privacy, but this Facebook move calls into question the wisdom of clouds or, rather, storing one’s data in others’ Web services like Facebook. We need to come up with new licenses or new mandates for open data in the cloud. Facebook shouldn’t own our data.

• Mashable : Facebook: All Your Stuff is Ours, Even if You Quit :
  

  The possible implications of this TOS change go beyond these concerns. Sure, you can choose not to use Facebook at all, but that doesn’t mean a thing. Someone can still take your photo, slap it on Facebook, and now neither you nor the author of the photo can stop Facebook from using the photo in whichever way they please. Looking at it globally, millions of people are uploading bits of information on everyone and everything, to a huge online database, and by doing so they’re automatically giving away the rights to use or modify this information to a private corporation. And not only that; they now also waiver the right to ever take it back from it.

  Facebook should take a long, deep look into how it treats its users. Until now, users had options with regards to how the data they generated on Facebook was used. Now, they have no options whatsoever, rather than quit the service altogether. It’s a major difference; I’m not going to take it lightly, and neither should you.

• Wet Asphalt : The Facebook Freakout :
  

  You are only granting those rights “on or in connection with the Facebook Service or in the promotion thereof.” What does that mean? Well, it means that you are licensing the use on Facebook branded websites or any other media and the Facebook Platform, which is the legal name for the APIs that allow third parties to create Facebook applications. So if there was a Facebook TV show, they could use your stuff on that. Or if they launched a Facebook concert series or a Facebook magazine, they could use your stuff in that. Presumably, if there were a Facebook dogfood, they could use your content on that. Or if they wanted to make an advertisement FOR any of those things, they could use your stuff in that. Precisely WHY Facebook would want to do any of those things, I leave to the reader to speculate on. What they most emphatically CAN’T do is what Walters claims, that “We can do anything we want with your content forever.” They can do anything they want with your content ON Facebook or to Promote Facebook forever. But if they said that it probably wouldn’t cause the internet panic and generate hits for the consumerist and readers to stroke Walter’s ego with diggs and trackbacks and twitterposts either.

Related posts:

1. Data Sans Frontiers : Data on the cloud : Facebook and its new statements
2. REST is the DBMS of the Internet

Subsequently Jeff Atwood wrote Real Ultimate Programming Power and I posted a comment on it which compared the issue to that related to fitness (search for fitness to reach my comment). Thats what also made me realise that if one really looked at the entire discussion as one between Happiness and Fitness, it just seemed so much easier to understand and comprehend.

So if one goes back to the first podcast #38 where Joel appears to take on software quality, and if one takes up an analogy where software usability and customer satisfaction are treated as happiness (of the software) and the quality is considered as fitness (again of the software) then what Jeff and Joel seem to be primarily saying is that (words are entirely mine)

In the overall scheme of things happiness is more important than fitness.

That would make a lot of sense that most would readily agree to. However one more statement in there says

Fitness doesn't matter so much

And thats what probably triggered off a whole bunch of reactions. It also seemed to offer an explanation of why there was such a storm raised.

The way I perceive it, Jeff and Joel were making an argument for happiness which probably would’ve gone unnoticed but for the fact that the portrayal of fitness was (if I may say so) a bit incendiary. It wasn’t so wrong as that it simply seemed to be sending out a completely wrong message. And if this was an opinion on some small blog it would still have gone unnoticed. But Jeff and Joel being the influential voices that they are were less than likely to be ignored especially when the message they were sending out was considered “dangerous” by the fitness community. Dangerous in the sense that it could lead to a whole bunch of people treat fitness with even lesser importance (especially in the context where general fitness levels were quite suspect). As I revisited the podcasts and the blog posts, the happiness / fitness analogy seemed to generally hold up.

This paragraph is a little speculative in the sense I don’t know that this is what actually happened and am speculating at my end. So Joe n Jeff got a little flustered about the fact that they couldn’t understand why they had kicked up a storm in the first place since they believed in what they had said about happiness. So they got together with Uncle Bob to sort things out in a subsequent podcast. To me it seemed like a rather uneasy and tepid podcast where they didn’t disagree with each other’s points of view but still continued to be uncomfortable with them. The multiple axes that were being referred to could’ve been happiness and fitness. Jeff still continued to defend their stance in a manner which perhaps only made matters worse. The Ferengi Programmer seemed to suggest that fitness regimens were bureaucratic steps (which cast them in a negative light) which were rather expensive to deal with and hence negotiable. The Real Ultimate Programming Power seemed to suggest that since most people wouldn’t worry about reading up on or attempting for better fitness, probably a much simpler set of rules along with a continuous thought to be more fit was what was really important.

I am convinced when I look at things in this perspective Jeff and Joel’s arguments make sense as do Uncle Bob’s. In my mind many of the differences can also be explained reasonably well by this analogy. It also explains why some of the statements seemed to invite so much ire. The issue probably lay in packaging of the arguments. If the statements are re-presented in a manner which does not seem to reduce the importance or desirability of fitness per se while continuing to emphasise the primary goal of happiness, it could be possible to close the discussion and move beyond the debate back into the real issues related to happiness and fitness, oops customer satisfaction and code quality.

Related posts:

1. An experienced programmer doesn’t use SOLID as a checklist – he internalises it.

In the post the author treats SOLID principles by Bob Martin as a ruleset that programmers apply from time to time. Once you get yourself into that frame of mind it is difficult to then contest the rest of the post. I would want to re-present the same topic with a different frame of mind.

SOLID principles are principles that you learn in your early days as a designer. These are formative stages when you are honing your skills and attempting to review your designs in terms of specific checklists of items to go through as a mechanism of validating your design. But each time you apply them, you internalise a part of them, and soon in 3 or more years of regular application, their application becomes internalised and ingrained. At this stage you might even well forget that they exist, since you apply them subconsciously, day after day, time after time, and sometimes referring back to them only when debating or reviewing your designs with other designers.

The analogy to the painter is in the post referred to : Are You Following the Instructions on the Paint Can? is also quite instructive. The instructions on the paint can are for one time painters, hobbyists, amateurs etc. No experienced painter is likely to be reading them since he’s probably internalised them. But each seasoned painter would want every junior painter to learn the instructions and the costs of not following them before stepping up to deciding whether and when not to follow them. He is unlikely to teach a new painter in the making – follow the instructions that make sense.

Sure you do make tradeoffs at times in design. Most people tradeoff guidelines in all spheres. However the keyword is to understand that you are breaking a guideline and then do so explicitly knowing its costs fully well.

My big difficulty with the post is that it is an advice which may do more harm to junior programmers than good. It might encourage them to make tradeoffs before they learn the cost and implications of making the tradeoffs. And it might set themselves away from a path that requires careful and judicious application (which requires a lot of effort in the early days) and helps them internalise the principles.

I would not recommend the post I refer to to any junior and upcoming programmer. My advice is as follows. If you have grown to a stage where you are applying these rules implcitly – don’t worry, you have the experience on your side to generally make the right judgement calls and you are likely to anyway apply them under most of the cases. In such a situation, this post and the one it refers to are probably inconsequential to you. If you are at a stage where you still need to review your design with respect to the SOLID principles (or other appropriate design principles) – please take your time to apply the principles, learn if you are breaking them, understand the costs of doing so (I would recommend that involve a senior programmer / designer in the process) and then by all means make the best judgement. Principles distilled over time and experience should be adjusted preferably by those who understand the cost and implications of doing so, the rest should strive to reach that state first.

To clarify, the reason why I upfront made the statement “seems not grounded in sustained experience in applying the principles”, is that those who have internalised them hardly every feel the burden (if at all) of applying them, and are unlikely to ever ever treat it as an explicit checklist, and they seem like checklists to those who haven’t internalised them. Precisely the audience to whom you want to craft a more careful and nuanced message.

Related posts:

1. So were Jeff / Joel / Uncle Bob discussing happiness and fitness ?
2. Python from a Java perspective – Part 2 – How duck typing influences class design and design principles
3. Jerk programmers can’t be managed tactically

Ahh! That is a term we haven’t found being used in IT context too often. Have we ?

Abuse in the past

Well let us start with some terms that have been used, and in my opinion abused in the past :

Web 2.0 : If there was indeed a term that actually set itself up for abuse this was it. There was nothing in it for people to refer to and hold on to as a basic and essential premise around which it gets applied. Eventually Tim O’Reilly the author of the term had to come out with a clarification What is Web 2.0, in which he said it refers to the Design Patterns and Business Models for the Next Generation of Software. Amongst the many trends he pointed out were

• The Web as a Platform
• Harnessing Collective Intelligence
• Data is the next Intel Inside
• End of the software release cycle
• Lightweight programming models
• Rich User Experience

He also defined the core competencies of the Web 2.0 companies

• Services, not packaged software, with cost-effective scalability
• Control over unique, hard-to-recreate data sources that get richer as more people use them
• Trusting users as co-developers
• Harnessing collective intelligence
• Leveraging the long tail through customer self-service
• Software above the level of a single device
• Lightweight user interfaces, development models, AND business models

Did Web 2.0 get abused. You bet. In many ways it was too clearly defined. But inherent in the number of attributes that were associated with Web 2.0 was the risk that anything sharing but a fraction of the attributes would want to call itself Web 2.0 (given the popularity of the term). Moreover it was actually hard not to accidentally abuse it for the same reason. However people have got around that eventually and now treat Web 2.0 as simply a moniker enhancer, and then evaluate the application or company to figure out whether it indeed is Web 2.0. The overuse of the term is leading to it becoming less relevant today

Service Oriented Architecture (SOA) : While I’m pretty sure this term has been around since early 90’s and especially used in the CORBA and Tuxedo worlds, the earliest online web page I could find which described it is “Service Oriented” Architectures, Part 1” which describes it as A service-oriented architecture is a style of multitier computing that helps organizations share logic and data among multiple applications and usage modes.. A very ironic article was “BEA Fits Tuxedo With SOA Makeover” especially considering that CORBA and Tuxedo users were probably amongst the earliest proponents of the term. Instead of seeing SOAP-WS* as one more element in a continuum of possibilities, it became the “in thing” which got weighted down by so many promises, expectations, goals and so much scope creep to its perceived capabilities. Herein lay the seeds of the disappointment one sees today especially in situations when it is doing rather well.

Lifecycle of abuse :
The following are the stages through which a term that is subject to abuse goes through :

• Initial formulation of seed thought and early implementations or practices.
• Further refinement of thought into a notion or hypothesis that has been tested and refined through a few experiences.
• Success visibility by neighbours leading to greater chatter along with active promotion of the term and its benefits.
• A positive spiral of both field successes and good PR leading to a strong mindshare
• High mindshare attracts a large and diverse community of “second stage thought movers”
• Desire to brand or market oneself in terms of using that term to sound “in” or “leading edge”
• Attempts to reevaluate the terms meanings in one’s own context to achieve the best fitment leading to a term meaning modification (either newer attributes being introduced or existing attributes associated with the term getting diluted)

A term’s essential value proposition is that it manages to convey one sometimes composite and sometimes complex notion succinctly. Yet successful people, successful companies, and successful terms act as great magnets. Once a term is successful, everyone attempts to use it in their own context. This is when the term starts becoming different things to different people. Thats when it goes down the Blind men and an elephant path. Thats when the term loses its core value proposition, and people using it get steadily more confused over a period of time. One way to avoid this disintegration is to be aware of the process and attempt to slow it down by drawing attention to the core notion and value proposition of the term.

Similar to the terms above, I believe other terms are now getting ripe for being abused. However sometimes abuse leads to improper expectations and sometimes even to “<Term> is Dead” which is actually an indication of the terms abuse dying rather than its basic set of offerings which continue to do well. A couple of times I have blogged about things *not* dying are SOA ain’t dead but it certainly is transforming and Java : the perpetually undead language.

I do see many terms in Software development and IT that are in the stage where they are being abused or are starting to be abused. So before we have to bury them eventually for not meeting the expectations such abuse creates, can we rein in the abuse. I doubt it, but this post is an effort in that direction.

REST : This one theoretically should be the least likely to be abused. Simply because it is so well defined in Roy Fielding’s Dissertation. But even he had to eventually write in his post REST APIs must be hypertext-driven.

I am getting frustrated by the number of people calling any HTTP-based interface a REST API. Today’s example is the SocialSite REST API. That is RPC. It screams RPC. There is so much coupling on display that it should be given an X rating.

What needs to be done to make the REST architectural style clear on the notion that hypertext is a constraint? In other words, if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API. Period. Is there some broken manual somewhere that needs to be fixed?

Strong words indeed. And if you think if these were against some small application, the social site REST API is the OpenSocial REST API which is a rather prominent standard (and its kind of hard to imagine google and other participants wouldn’t realise the distinction). Yet the underlying shift is that for internet based applications people are increasingly relying on simple HTTP based API and since perhaps HTTP is too bland a name to term an API with, they go with the sexier sounding REST. Not really necessary. If I have an HTTP API, its all right to call it an HTTP API. Another term that did start getting used alternatively was POX/HTTP (Plain Old XML over HTTP) but the slight issue with this term is that JSON is starting to be another data payload vehicle so POX doesn’t necessarily cut it universally.

There is another issue as well with using REST as the moniker to describe HTTP based APIs. Roy Fielding eloquently points out that REST is so different from RPC. Yet in a logical sense, many requirements for HTTP APIs are very RPC like. It is possible to represent such an API using REST semantics (instead of user/delete/123 call it userdeletion/create/123 .. a name mangling trick). Moreover REST requires statelessness, a requirement that not all applications necessarily comply with. I am not suggesting they should, just that should they choose to not be completely stateless it would be difficult to term their APIs REST. These were both issues I had referred to in my earlier post Fomenting unREST : Is RESTfulness a semantics game ? Why does REST require statelessness ?.

Clearly REST is not going to cut it as a universal mechanism of describing simple HTTP based APIs. I would suggest call them HTTP APIs, or perhaps think of a better term which is not REST since it seems rather likely that many of the HTTP APIs we shall develop may simply not meet the requirements or expectations of REST. The more we use REST the way it should be, the less we are likely to abuse it. So may I request that we use REST only when the API adheres to all the expectations as specified by Roy Fielding ?

Cloud :

I am certain the term cloud began thanks to the abstraction of internet that got used in tons of network topology diagrams. So it perhaps referred to the segments of the network that formed the entire plumbing of the internet and all the external network segments which were not relevant to the topology under discussion, and to the applications and services that resided on these “external” network segments (primarily the internet). However further along the way additional attributes started getting implied. An example is the rapidly scalable infrastructure (the scale on demand attribute). This blog is hosted on a pre specified hardware by a web hosting company. Is it on the cloud ? Its a little unclear (for certainly it does not have the scale on demand attribute). Occasionally implied attributes are multi-tenancy. IMO thats stretching it too far, multi tenancy is an optional attribute of applications and services that can be hosted on the cloud, it isn’t the attribute of the cloud itself. It would be better to keep the scale on demand attribute away from the cloud term and use it as an additional qualifier (eg. elastic cloud). However one of the effects of that is that it is leading to a proliferation of cloud related terms which is if you pardon my pun, making the situation a little too cloudy. Some of the terms are for lack of a more civilised response quite amusing eg. “Internal Cloud”. I mean if cloud represents the external segments of the network and the application and services that are hosted on them, why would one call the internal network the internal cloud as if the external network would be the external cloud. But then whats the difference between the network and the cloud. Or does the network becomes the cloud or should it be that the cloud becomes the network. This highly avoidable tendency of clouding everything also has a cloudy term for itself cloudwashing – slapping the word “cloud” on products and services you already have. See the point ? Having said that, I still think it is preferred to have a proliferation of qualified terms rather than a proliferation of contextually implied attributes into a single term itself.

So can we keep cloud restricted to the “external network segments and the applications and services hosted on them” ? Thank You. And lets keep out an eye on all the additional terms that further qualify cloud. Many of them might not make sense, so just decloud them.

*****-as-a-service :

It all probably began with Software as a Service (Saas). Saas rocks and is a term which is catching on. But it is also now getting a ton of other “as a service” followers. Just google for this and the following words crop up as the first part of -as-a-service : platform, desktop, hardware, desktop, sql, data, ui, infrastructure, PC, research, os, malware, identity management, analytics, middleware … get the picture ? Now in this current global economy where more than a third of the economy is based on services, just imagine the sheer proliferation of terms that are likely to happen simply because someone doesn’t say I offer hosted middleware, no it is middleware as a service. This blog will recast itself as Gratis chronological opinion as a service. People are going nuts, and there’s no one to stop them. In the meanwhile whats happening to the one which started it all – SAAS ? Theres something called Level 1 SAAS where each customer has its own customized version of the hosted application and runs its own instance of the application on the host’s servers. I have read at least one blog post which refers to ISVs needing to customise Saas offerings for individual enterprise customers. So rather than newer implied attributes being added to the term Saas, existing ones are being removed in a manner where any outsourced software hosting will be Saas. Pretty soon, a business unit which outsources software development, maintenance and hosting to another intra corporate business unit will start calling it a Saas offering to keep up with the pressure to seem current. Pretty soon all software will tend to Saas.

Can we please keep Saas restricted to uncustomized offerings made by other service providers which essentially rent out the use of the software they’ve built or own on an as-is basis (with version upgrades from time to time of course) ?

Mashups :

Lets just for a moment look at how wikipedia describes Mashups.

a mashup is a web application that combines data from more than one source into a single integrated tool.

I see a number of enterprise vendors starting to offer Mashup products. That in itself is welcome. I however also anticipate a full wave of rebranding of Integration Services, SOA, Portal Servers into Mashups in addition to the few new mashup features being introduced. Also expect such buzzwords to drive newer budgets and customer excitement even though relatively little has changed under the covers. Eventually the term could become all encompassing where it covers all forms of system integration and service aggregation which will eventually make it useless as a term. In the meanwhile authors of existing Web 2.0 Mashups built on Google Maps, Flickr and Twitter will be left wondering, “Hmm.. was this was mashups was all about ?”

Agile : I don’t know that its terribly abused, but when I see presentations that combine elements of Agile with layered sequences (eg documented design followed by review followed by development followed by testing) and heavy management control procedures and call it the right way of doing agile – it smacks of abuse to me. Thats like taking a cheetah, cladding him with medieval knight armour, and saying ain’t it a very nice and strong fastest animal on earth ? I don’t think it is feasible to adapt a heavy process to agile or vice-versa to somehow take on the attribute of being agile. Agile requires a migration from processes which encourages repeatability through documentation, division of work and control into one which achieves high productivity and quality through capability enhancement. This requires a reduction in the control processes and an increase in the capabilities training in a manner where the quality and control is not maintained through processes built around a replaceable workforce, but instead through a set of (relatively) informal and frequent communication channels built around a increasingly capable team.

A lighter parting note.

Finally on a lighter note a friend suggested another term that is being abused. “Software engineering – a Term that should never have been applied to software and insults engineering.” I agree even though I know the comment probably stemmed from the usage of this word in this blog’s tagline. While Software Craftsmanship has been discussed for long, I find many software building efforts not worthy of being called craftsmanship. Therefore I contribute a new term to the lexicon – Software Construction And Maintenance. This is one term (or its acronym) I am not concerned of getting abused. In fact it is unlikely to get used.

Related posts:

1. Preannouncing a blog post : Top 5 software / IT terms ripe for abuse
2. Agility and/or Process Maturity : How do I perceive these terms and why these should not be confused

Abuse refers to the overuse of a word and implying it with too many different attributes in different contexts, so that eventually the same word can no longer be used without providing a lot of qualifications about “this is what I mean when I say …..”. Examples from past are n tier architecture, SOA etc.

Related posts:

1. Software / IT Terms in early stages of abuse or ripe for Abuse
2. Agility and/or Process Maturity : How do I perceive these terms and why these should not be confused

The definition of SOA is quite simple – the name is the definition – it is “Service Oriented Architecture”. Services are generally software things (I don’t want to call it modules or objects or components since that will set off another debate) which offer one small focused set of capabilities through a service API. As a generally accepted set of practices such services are often intended to be coarse grained, reusable in multiple contexts and loosely coupled from each other. When an architecture is oriented around building something that is built on services it is service oriented architecture. Period.

Now that we have looked at the broad outlines of what SOA is, lets consider what SOA “per se” is not (though these aspects are not antithetical to SOA itself – they are largely orthogonal)

1. SOA is not about business alone : SOA can be used for business services. But it can just as easily be used for building technical services. I can fully imagine a network management system which is not a critical offering of a business, yet can be built using SOA. Those who focus on SOA being for business only miss out on the big time benefits that can also be had by applying it to the technical infrastructure.
2. SOA does not require workflow management / BPM etc. :These are often relevant, but their existence is irrelevant to whether an architecture is SOA. What is critical is if it is built using and around services. There are a whole range of possibilities of building systems or even system sets without getting anywhere close to Workflow Management or BPM.
3. SOA need not even be distributed :While 99% of SOA systems are likely to used remotable services (ie. distributed, over the network) there is no fundamental mandatory requirement that this be so. One could potentially build a desktop based personal finance application which is built using SOA (though I do not necessarily see the economic prudence in going down such a path)
4. SOA does not prescribe messaging or event based processing :There is no such requirement, however many enterprises choose to have such services in their enterprise mix
5. SOA is not about web services, REST etc.: DCE, Tuxedo, CORBA, REST, XML, json etc. are all implementation protocols and/or transports and/or data formats for SOA (I have designed and built systems using each one of them, so at least I am not using their names loosely). SOA doesn’t dictate that a particular technology / protocol / transport / data format be used. It is the enterprise architecture which makes the choice. In fact, the most successful SOA transport till date is HTTP. Twitter API which offers data formatted using json or XML or HTML over HTTP is a great example of a “simple service” and to the best of my knowledge its method to post a message gets invoked more than many many million times on a busy day.

SOA is a classic case of a simple idea being force fitted into a grand panacea. If one goes by the premise that a twitter API is indeed a SOA service, how can one justify the expenditure on training, consulting, outsourcing, software stack purchasing, hardware etc. etc ? Its not that the more advanced things like BPM or WS-* or “business services” are not important. It is just that these ride on SOA – they are not SOA.

So why is there a grand confusion on this issue. IMHO it shouldn’t have ideally been about SOA at all. It stems from the following :

1. Service identification is hard : Identifying good, clean reusable services is a non trival task. Moreover when attempted at an enterprise level, there are too many stakeholders so agreement on what constitutes a service itself can be hard.
2. Reuse is powerful. It creates territorial issues : Once identified, the services do not necessarily lend themselves to neat classification in an organisational structure. Moreover a piece of software that is reusable is worth far more than a silo’ed piece of software. This can lead to turf and clout issues whose resolution falls in the domain of Human Oriented Architecture
3. The Hype :Thanks to the hype one now needs to get in an army of consultants, trainers, new software and hardware stacks, simply because it is supposed to be something really new and something big.

So you essentially have service identification and building which does not necessarily align itself with organisational structures and business functions by default within the context of territorial and ownership issues getting raised while absorbing a whole bunch of new consultants, trainers and vendors. And as a result we hear things like – SOA is difficult, SOA is complex, SOA sucks and most importantly we cannot agree on what SOA is. I am not saying that these problems are not important – its just that laying them at the doorstep of SOA is not being entirely honest.

So can I please put in a humble reminder – SOA defines itself quite nicely – it is Service Oriented Architecture, ie. an architecture built around services.

And finally, couldn’t help but include one of my tweets related to current debate on whether SOA is dead – “Knock, Knock, whos there ? SOA. SOA who ? So-out-of-fashion everyones calling me dead.”

Related posts:

1. Is a large corporate making money off open source or open standards an oxymoron ? In a Sun / Java Context
2. SOA ain’t dead but it certainly is transforming