/var/log/mind

Functional Programming with Python – Part 2 – Useful python constructs

Dhananjay Nene — Mon, 01 Mar 2010 16:48:05 +0000

In Functional Programming with Python – Part 1, I focused on providing an overview. In this post I shall focus on the core python language constructs supporting functional programming. If you are experienced pythonista, you may choose to skip this post (and wait for the next post in this series )

Sequences in python are not immutable :

When using sequences in python the thing to be noted is that sequences are not immutable. This provides you with the following options.

a) Use immutable sequence types : This is only possible by defining different types for sequences than the ones built into the language.
b) Ignore all methods on the sequences which modify them
c) Waive functional programming tenet of immutability

My preferred option when explicitly focusing on writing functional code is to use b)

Sequence types in python

The most commonly used sequence types in python are :

Tuple : Immutable, Ordered, Indexed collection represented as (val1, val2)
List : Ordered collection represented as [val1, val2]
Dictionary : A key indexed collection represented as { key1: val1, key2 : val2 }
Set : An unordered collection allowing no duplicates. Represented as set([val1,val2])
str and unicode : While primarily meant to serv as ascii and unicode strings, these data structures also act as sequences of characters.Represented as “abcd” or ‘abcd’

Of the above only tuples are immutable.

There are other sequences used less often as well. An example is frozenset which is an immutable set.

Simple iteration
The for construct allows you to loop through a sequence. eg.

1
2
3

one_to_five = [1,2,3,4,5]
for num in one_to_five :
print num

Iterators on dictionaries

Unlike tuples, lists and sets where iterators essentially traverse through the sequence constituents, there are a number of different iterators on dictionaries. These are :

1
2
3
4
5
6
7
8
9

d = {1:"One", 2: "Two", 3:"Three"}
# keys
for val in d : print val
# an alternative for keys
for val in d.keys() : print val
# values
for val in d.values() : print val
# key, value tuples
for key,val in d.items() : print key, val

Slices

Slices allow creation of a subset on a sequence. They take the syntax [start:stop:step] with the caveat that a negative value for start or stop indicates an index from the end of the sequence measured backwards, whereas a negative step indicates a step in the reverse direction. The following should quickly indicate the use of slices

1
2
3
4
5
6
7
8
9
10

seq=[0,1,2,3,4,5,6,7]

#first 3
print seq[:3]
# last 3
print seq[-3:]
# 2nd to second last
print seq[1:-1]
# reverse the sequence
print seq[::-1]

The iterator protocol
Since python is an object oriented language as well, it provides strong support for allowing iteration over an object internals. Any object in python can behave like a sequence by providing the following :

a. It must implement the __iter__() method which in turn supports the iterator protocol
b. The iterator protocol requires the returned object, an iterator, to support the following ie. an __iter__() method returning self. and a next() method which returns the next element in the sequence, or raise a StopIteration in case the end of iteration is reached.

I’ve tested iterators which do not themselves have an __iter__() method and it still works, but I still do not give in to the temptation of not defining the next() method alone since that would be inconsistent with the documented python specifications

Let us examine a simple class indicating a range. Note that this is just for demonstration since python itself has better constructs for a range.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# An iterator object to sequentially offer the next number
# in a sequence. Raises StopIteration on reaching the end
class RangeIterator(object):
def __init__(self,begin,end):
self._next = begin - 1
self.end = end
def next(self):
self._next += 1
if self._next < self.end :
return self._next
else :
raise StopIteration

# A class which allows sequential iteration over a range of
# values (begin inclusive, end exclusive)
class Range(object):
def __init__(self,begin,end):
self.begin = begin
self.end = end
def __iter__(self):
return RangeIterator(self.begin, self.end)

oneToFive = Range(1,5)

In the above example, __iter__() on Range returns an instance of a RangeIterator.

The way this capability is used is as follows :

1
2
3
4
5

for number in oneToFive :
print 'Next number is : %s' % number

print 'Is 2 in oneToFive', 2 in oneToFive
print 'Is 9 in oneToFive', 9 in oneToFive

The output you get with it is :

1
2
3
4
5
6

Next number is : 1
Next number is : 2
Next number is : 3
Next number is : 4
Is 2 in oneToFive True
Is 9 in oneToFive False

Generators

In the above situation, the state necessary for the iteration (ie.begin and end attributes) need to be stored. This was stored as a class member. Python also provides a function based construct called a generator. In case of a generator, the function should just do a yield instead of a return. Python implicitly provides a next() method which resumes where the last yield left off and implicitly raises a StopIteration when the function completes. So representing the above class as a function,

1
2
3
4
5
6
7
8
9
10
11

def my_range(begin, end):
current = begin
while current < end :
yield current
current += 1

for number in my_range(1,5) :
print 'Next number is : %s' % number

print 'Is 2 in oneToFive', 2 in my_range(1,5)
print 'Is 9 in oneToFive', 9 in my_range(1,5)

I do not document the results since these are identical to the earlier code using a RangeIterator.

Note: I used my_range() and not range() since there already exists another range() already provided by python

An interesting point to realise is that in both the above situations, the next element to be returned in the sequence was being computed dynamically. To put it in the terminology better consistent with functional programming, it was being lazily evaluated. While python has no mechanism of lazy evaluation of functions, the ability of functions or objects to lazily evaluate the return values are sufficiently adequate to get the benefits of lazy evaluation at least from the perspective of cpu utilisation only on demand, and minimal memory utilisation.

List comprehensions

List comprehensions are one of the most powerful functional programming constructs in python. To quote from python documentation (even as I leave out a very important part of the full quote .. to be covered later),

Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses. The result will be a list resulting from evaluating the expression in the context of the for and if clauses which follow it. If the expression would evaluate to a tuple, it must be parenthesized.

A sample list comprehension is one which returns a sequence of even numbers between 0 and 9 :

1
2
3
4

# A list comprehension that returns even numbers between 0 and 9
even_comprehension = (num for num in range(10) if num % 2 == 0)
print type(even_comprehension)
print tuple(even_comprehension)

The output one gets on running the code above is

1 2	type 'generator' (0, 2, 4, 6, 8)

Note that the list comprehension returns a generator which then returns a sequence containing 0, 2, 4, 6 and 8.

The for and if statements can be deeply nested.

lambda

Lambdas are anonymous functions with a constraint. Their body can only be a single expression which is also the return value of the function. I will demonstrate their usage in the next section.

map, filter and reduce

Three of the functional programming constructs which probably have aroused substantial discussions within the python community are map, filter and reduce.

map takes a sequence, applies a function of each of its value and returns a sequence of the result of the function. Thus if one were to use map with a function which computes the square on a sequence, the result would be a sequence of the squares. Thus

1 2	def square(num): return num * num print tuple(map(square,range(5)))

results into

1	(0, 1, 4, 9, 16)

I mentioned earlier I will demonstrate usage of a lambda. In this case I could use a lambda by defining the square function anonymously in place as follows :

1	print tuple(map(lambda num : num * num,range(5)))

filter takes a predicate and returns only the elements in the sequence for whom the predicate evaluates to true.

Thus

1	print filter(lambda x : x % 2 == 0, range(5))

results in the following output

[0, 2, 4]

Finally the most controversial of them all – reduce. Starting with an initial value, reduce reduces the sequence down to a single value by applying a function on each of the elements in the sequence along with the current manifestation of the reduced value. Thus if I wanted to compute the sum of squares of numbers 0 through 4, I could use the following

1	print reduce(lambda reduced,num : reduced + (num * num), range(5), 0)

In the above example, 0 as the right most argument is the initial value. range(5) is the sequence of numbers from 0 thru 4. Finally the anonymous lambda takes two parameters – the first is always the current value of the reduced value (or initial value in case it is being invoked for the very first time) and the second parameter is the element in the sequence. The return value of the function is the value which will get passed to the subsequent invocation of the reduce function with the next element in the sequence as the new reduced value. The reduced value as returned finally by the function is then the returned value from reduce

The functional programming folks are likely to find this an extremely natural expression. Yet reduce resulted in a substantial debate within the python community. With the result that reduce is now being removed from python 3.0. (Strictly speaking it is being removed from python core but will be just an import statement away as a part of the functools package). See The fate of reduce() in Python 3000. Why so controversial ? Simply because the usage of map, filter, reduce above could be rewritten as

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#map
seq = []
for num in range(5) :
seq = seq + [num * num]
print seq

#filter
seq = []
for num in range(5) :
if num % 2 == 0 :
seq = seq + [num]
print seq

#reduce
total = 0
for num in range(5) :
total = total + (num * num)
print total

Just look at the two blocks of code and figure out which is easier to read (especially look at reduce). One of the areas where a large number of pythonistas and lispists are likely to disagree is the tradeoffs between brevity and easy readability. Python code is often english like (well, at least to the extent that a programming language can be) and some pythonistas do not like the terse syntax of lisp thats hard to follow.

I earlier mentioned that I left out a part of the description of list comprehensions from python documentation. Here’s that part of the quote.

List comprehensions provide a concise way to create lists without resorting to use of map(), filter() and/or lambda. The resulting list definition tends often to be clearer than lists built using those constructs.

Having said that I do believe many including myself will continue to use map, filter, reduce

Other helpful functions in python core that are helpful for functional programming are :

all : Returns True if all elements in a sequence are true
any : Returns True if any element in a sequence is true
enumerate : Returns a sequence containing tuples of element index and element
max : Returns the maximum value in a sequence
min : Returns the minimum value in a sequence
range : Return a sequence for given start, end and step values
sorted : Returns a sorted form of the sequence. It is possible to specify comparator functions, or key value for sorting, or change direction of sort
xrange : Same as range except it generates the next sequence element only on demand (lazy evaluation) thus helping conserve memory or work with infinite sequences.

We’ve seen many of the constructs that are typically useful for functional programming. I left out one big part – the itertools package. This is not a part of python core (its a package which is available with a default python installation). Its a large library and substantially helps functional programming. That along with some more sample python programs shall be the focus of my next blog post in the series. At this point in time I anticipate at least a few more parts after that to focus on a) Immutability b) Concurrency and c) Sample usage.

Hope you found this useful and keep the feedback coming so that I can factor it into the subsequent posts.

Functional Programming with Python – Part 1

Dhananjay Nene — Tue, 23 Feb 2010 18:05:51 +0000

Lately there has been a substantial increase in interest and activity in Functional Programming. Functional Programming is sufficiently different from the conventional mainstream programming style called Imperative Programming to warrant some discussion on what it is, before we delve into the specifics of how it can be used in Python.

What is Functional Programming?

To quote from Wikipedia,

In computer science, functional programming is a programming paradigm that treats computation as the evaluation of mathematical functions and avoids state and mutable data. It emphasizes the application of functions, in contrast to the imperative programming style, which emphasizes changes in state.

For beginners, one of the most fluent starter pages I would recommend for the history and specifics of functional programming is Functional Programming For The Rest of Us. This is a must read article which provides the reader with a good overview without getting too much into the nitty gritties of functional programming.

Since a detailed discussion on functional programming (henceforth referred to FP) is beyond the scope of this post, I will just briefly summarise the most critical elements of FP.

Functions as the basic building blocks : Unsurprisingly FP requires the construction and usage of functions as the basic building units. In the simplest terms “int add(int x, int y) { return x + y; }” is a simple addition function written in ‘C’. It takes two parameters x and y, adds them, and returns the result. This is a rather obvious and simple case but I stated it since I would like to refer back to it subsequently in this post.
Functional Programming prefers functions without side effects : The add function above is a good example of a function without side effects. A function is said to be without side effects if the only changes it makes are those that are manifested in the return values. In other words such a function cannot change any global variables, write to the console, update the database etc. A fairly related term is referential transparency. A function is said to be referentially transparent if its invocation can be substituted by the return value in a program without impacting the program in any other way.
Immutability : Pure functional programming often requires you to deal with immutable data structures. Thus the value of any variable is not open to modification (Thus they are called values and not variables). This aspect complements the functions without side effects. Thus the way most changes to state are implemented are not by modifying an object in place (which is how imperative programming deals with it) but by cloning the data structure with some of the values getting modified and the modified data structure being returned by the function. Java programmers are aware of the immutability of the String instances wherein any modifications to the string result in a new String instance being created. Imagine the same happening to all the datatypes across the program.

Benefits of Functional Programming :

Some of the nice benefits (I am tempted to say side effects) of functional programming are :

Superior ability to deal with concurrency (multi threading) : Threaded programs are nasty to write. And nastier to debug. In an imperative environment, you not only have to deal with data structures being modified in place by some other parts of the program, in a threaded environment such modifications can happen using peer threads, even as your current thread whose logic you are focusing on is attempting to exercise that logic. Its an extremely unpredictable environment which has resulted in a number of how-to’s for safe threaded programming using constructs such as locks, mutexes etc. Functional programming deals with the issue far more elegantly. Instead of controlling and managing unpredictability, it takes it out completely. Because a data structure once constructed will not be modified and because the source of the modifications can be clearly located to the function which instantiated the datastructure, the unpredictability of data changing right under you is gone. This can be a little expensive to manage and FP does sometimes come up with some compromises (or cool features depending on how you view it) such as Software Transactional Memory but a discussion on that is completely beyond the scope of this post.
Easier testing and debugging : Because modifications to data are contained and because a function communicates with the context outside it only via its return values, testing and debugging become far easier. You essentially need to focus on testing each function individually. Similarly during debugging you need to be able to quickly locate the function likely to have the problem, after which you can easily focus on the function to be able to quickly resolve the issue. Mocking out functions can also help testing each function in isolation. In general because of fewer side effects, testing under functional programming is often a lot easier, and the importance of having to do “integration” testing and “module” testing is lesser since testing functions in isolation is likely to identify most issues, far more than in typical imperative programming.

Why Python?

Python is not the best functional programming language. But it was not meant to be. Python is a multi paradigm language. Want to write good old ‘C’ style procedural code? Python will do it for you. C++/Java style object oriented code? Python is at your service as well. Functional Programming ? As this series of posts is about to demonstrate – Python can do a decent job at it as well. Python is probably the most productive language I have worked with (across a variety of different types of programming requirements). Add to that the fact that python is a language thats extremely easy to learn, suffers from excellent readability, has fairly good web frameworks such as django, has excellent mathematical and statistical libraries such as numpy, and cool network oriented frameworks such as twisted. Python may not be the right choice if you want to write 100% FP. But if you want to learn more of FP or use FP techniques along with other paradigms Python’s capabilities are screaming to be heard.

Sample Program :

I debated whether I should introduce various elements of functional programming using python in detail and then put it all together in a sample program all in future blog posts of this series, or whether I should start with a sample program which cover various aspects of function programming in this post and then explain various aspects in much more detail in future posts. For better or for worse, I have chosen the latter option. That means I shall be explaining one sample program and shall leave it to future posts in this series to get into greater details.

The sample program I have chosen is that of a simple calculator. A typical calculator supports simple unary or binary mathematical operators and performs floating point operations. Without much ado we now get into the sample program.

Immutable Data :

1
2
3
4
5

from collections import namedtuple

Context = namedtuple('Context','stack, current, op')
def default_context():
return Context([],0.0,None)

Python is not particularly strong at immutable data. However one of the data structures, a tuple is immutable. A namedtuple is another data structure which supports both tuple like access through indices or through named elements in the tuple. For the calculator I shall need a Context which contains a stack for storing any incomplete operations, an attribute current reflecting the current value being shown on the screen and an op which might reflect a pending operation which is typically required for binary operators where the second value still needs to be provided. While namedtuple is a reasonable construct for simple tuple like objects, it would be helpful to have immutable objects as well – but thats to be covered in a future post.

Simple Functions

1
2
3
4
5
6

def add(x,y): return x + y
def sub(x,y): return x - y
def mult(x,y): return x * y
def div(x,y): return x/ y
def reverse_sign(x): return -1 * x
def pow(x,y): return x ** y

There’s not much to describe here. The functions should be self explanatory. For purpose of emphasis I would like to note that in the above code, “add” is now an entry in the namespace which is a reference to a function. This reference can be passed around, assigned to other entries. Thus the code below should work (though it does not form a part of the calculator program). This is to demonstrate how python treats attributes and functions virtually identically consistent with the Uniform access principle.

1
2
3
4

otheradd = add
add = sub
assert otheradd(7,3) == 10
assert add(7,3) == 4

Currying
Currying is a treatment afforded in functional programming which allows a function of n parameters to be treated as a sequence of n sequential functions each of one parameter.

1
2
3

from functools import partial

square = partial(pow,y=2)

Here partial is a function reference. square now refers to another function with its y parameter value being anchored to 2

Invoking functions dynamically

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

unary_functions = {'!' : reverse_sign, '@' : square }

def handle_unary_op(ctx,x):
return ctx._replace(current = unary_functions[x](ctx.current), op = None)

binary_functions = {'+' : add, '-' : sub, '*' : mult, '/' : div}

def handle_binary_op(ctx,x):
return ctx._replace(op = binary_functions[x])

def handle_float(ctx,x):
if not ctx.op :
return ctx._replace(current = x)
else :
return ctx._replace(current = ctx.op(ctx.current,x), op = None)

Note that I created a unary_functions dict (or dictionary or hashmap) where the key is the character which represents the function and the value is the reference to the function.

Also note that in the handle_unary_opfunction, I invoke ctx._replace method. On a named tuple it creates another tuple based on the existing namedtuple data, but with some of the values modified as specified in the keyword paramters passed to _replace. After looking up the appropriate unary function ie. unary_functions[x], I also invoke it on the current value ie. unary_functions[x](ctx.current). I also defined another dict for binary operators. The handle_binary_op method reflects how the op in the context is set to the appropriate binary function that should be triggered after the subsequent value is known.

Finally the handle_float function either sets the current value to the incoming value or in case the current operator is already set it applies the binary operator to the current value and the incoming value and replaces current with the computed value.

Additional Code
When I wrote the calculator program, I wrote the functionality to introduce braces. However that functionality is not particularly important in this explanation. So it is being listed here for completeness.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def start_brace(ctx):
newstack = ctx.stack
newstack.append((ctx.current,ctx.op))
return ctx._replace(
stack = newstack,
current = 0.0, op = None)

def end_brace(ctx):
stack = ctx.stack
current = ctx.current
oldcurrent, oldop = stack.pop()

oldctx = Context(stack,oldcurrent,oldop)
return process_key(oldctx,current)

tokens = { '(': start_brace, ')' : end_brace}

def handle_tokens(ctx,x):
return tokens[x](ctx)

Processing one key
I must confess I started off using key to represent the keystrokes, but along the way the key can also represent a complete floating point number (not just a single keystroke). Thus the key parameter can refer to a single character operator or a sequence of characters representing a floating point number

1
2
3
4
5
6
7
8
9
10
11
12
13

function_groups = {
tuple(unary_functions.keys()) : handle_unary_op,
tuple(binary_functions.keys()) : handle_binary_op,
tuple(tokens.keys()) : handle_tokens
}

def process_key(ctx,key):
if isinstance(key,(types.FloatType,types.IntType, types.LongType)) :
return handle_float(ctx,key)
elif isinstance(key,(types.StringType)) :
for function_class in function_groups :
if key in function_class : return function_groups[function_class](ctx,key)
return ctx

In this case I set up a dictionary where the key is a tuple of all the keys representing a particular class of a function. Note that the .keys() method is a method which returns a list of all the keys in a dictionary. However since list is mutable, it cannot get used as a key into the overall hashmap, hence I convert it into a tuple.

The process_key function takes the incoming key, passes it handle_float if it is a number, or treats it as an operator. If it is the latter it searches for it in all the keys of each operator groups, and if it finds a match, it locates the corresponding handler function from the map and invokes it. Finally in case no match is found it ignores the key.

Processing a sequence of keys

1 2	def process_keys(keys): return reduce(lambda ctx,key : process_key(ctx,key), keys, default_context())

Here you see a reduce function being invoked. This belongs to the family of map and filter functions which are used extensively in functional programming. I shall attempt to briefly explain it here, but this family of functions in addition to a number of others will again be dealt with in a future blog post.

To interpret the usage read the above reduce statement right to left. Thus we start with a default context, and for each key in the sequence of keys, we invoke a lambda (thats like an anonymous function), which calls process key with the context and the key. Note that the first parameter to the lambda is either the initial value (the default context) or the return value of the last process_key (which is also a context) and the key is each key in the keys sequence injected sequentially.

To further make it easy I re-represent the same function below differently which is much more readable and easier to understand. This shows one more strength of python. Because of its focus on readability, it actually can be used to write functional programs are much more readable by a large mass of programmers than most of the functional programming languages themselves (readability being subjectively interpreted by me as what is most natural for english or similar language speaking people).

1
2
3
4
5

def process_keys(keys):
ctx = default_context()
for key in keys :
ctx = process_key(ctx,key)
return ctx

Usage
As a mechanism to conduct some rudimentary tests on the code written so far, the following code is introduced. Here you can get an overall feel of the program.

1
2
3
4
5
6

if __name__ == "__main__" :
assert process_keys((2,'+', 3)).current == 5
assert process_keys((2, '!', '+', 5)).current == 3
assert process_keys((2, '@')).current == 4
assert process_keys((2,'+',3,'*',5)).current == 25
assert process_keys((2,'+','(',3,'*',5,')')).current == 17

Some more slightly advanced Functional Programming

Finally to tickle your interest even more, here’s a slightly more advanced usage of functional programming constructs. Here I shall add all numbers between 1 through 10.

1
2
3
4
5
6
7
8
9
10

from itertools import chain

def plus_num_seq(n):
count = 1
while count <= n :
yield '+', count
count += 1

keys = list(chain.from_iterable(plus_num_seq(10)))[1:]
assert process_keys(keys) == 55

The plus_num_seq is a generator. Note the usage of the yield statement. Thus it will continuously generate tuples with the first element of the tuple being the ‘+’ character and the second being the number with the number varying from values 1 through n. The chain.from_iterable flattens the generated list (it thus has 20 items for n = 10, each alternate one being the ‘+’ character starting with the first items). Since we do not need the very first ‘+’ character, I removed it using the [1:] slice operator.

Just like reduce this style of code is quite typical of functional programming. Thats something I shall detail upon much more in future posts.

Hope you enjoyed the post. Keep the feedback coming so I can better structure the subsequent posts based on the feedback.

Note: The full source for the calculator calculator.py can be accessed here.

Google Buzz Test Drive Report

Dhananjay Nene — Wed, 10 Feb 2010 00:05:40 +0000

Feature Notes :

Note: Google Buzz had not yet been enabled for me through normal GMail Desktop access. These notes are based on my experience with the same by switching my Firefox user agent to iPhone and accessing the iPhone Google Buzz UI.

Has a very facebook / friendfeed like interface.
- Comments : People can enter comments against a post which appear sequentially below the post.
- Like : Has the Like feature also supported by facebook / friendfeed
- Inline images and videos : Images and videos appear inline as a part of the stream.
- No 140 character limitation : Or at least none that I observed
Each conversation becomes a web page. eg. the comment stream against my very first google buzz post.
- The same conversation is also posted to your inbox. The conversation in the inbox gets dynamically updated as the conversation continues. A conversation that has been read earlier gets marked as unread when more comments are entered against it.
- The conversation web page becomes an alternative location for participation. If say the conversation link is forwarded to others, they can immediately comment on (or choose to “Like”) the conversation from the web page itself.
Private Channeled Messaging. You can mark posts as private but directed to a group of people. These groups are maintained and managed using Google Contacts. To the best of my understanding the future conversation is also constrained to the same group. Such messages are obviously not visible publicly. This is an excellent feature for closed group interactions.
Directed messaging is supported using the familiar twitter ‘@’ style. However in this case you use ‘@’ followed by the entire email address. Any posts made with ‘@’ addressing also get delivered to the user’s inbox.
Geolocation integration is supported. Thus you can view the public posts being made by people nearby you. Since I used firefox, it used firefox for geolocation support. I imagine and presume it would be using alternative superior mechanisms for mobile based conversations.

Comparison to Twitter :

One of the reasons I test drove Google Buzz was since I was wondering if it was a better Twitter. In many ways it is. But I don’t think thats likely to be universally true. There is the 140 character simplicity to twitter which is its identifying feature. In many ways Google Buzz is a Friendfeed clone – not a twitter clone (with added inbox integration). I always found Friendfeed much superior to twitter for conversations and I suspect those who shared that perception will find Google Buzz far superior to twitter too. However it will run into the same difficulty that Friendfeed did. If the twitter user doesn’t correlate his twitter account to the friendfeed account, it is very laborious in friendfeed to add a user you follow as an imaginary user in order to have his tweets appear in your friends stream.

I think where Buzz will take on Twitter .. and big time is the really large number of people who sign on to twitter, maybe make a few tweets and then say I don’t get it and go away. I suspect most of them will “get it” on Google Buzz – especially if they have any Facebook experience (which is a large large set of people). In that sense while Twitter appealed to only a small set of people, Buzz at least has the potential to appeal to the mainstream .. and in that sense go for the jugular – Facebook’s that is.

Comparison to Facebook :

Since facebook over a period of time has cloned its update stream to be like that of Friendfeed, it should come as no surprise that the Buzz stream is likely to look and feel quite similar. However two big differences stand out. The first is the way the social graph is managed. The social graph unlike facebook is asymmetric. ie. two persons don’t have to agree to be friends to follow each other. In case of Buzz, the social graph is managed through google contacts. You can follow people who have public google profiles or follow anyone in your contacts. Thus if a person doesn’t have a public google profile or his email address is not known to you, you may not be able to follow him/her. You don’t need the other person’s permission to follow. (At least thats the impression I formed, though I could stand corrected). The other big difference is the email integration. The entire stream and conversations are integrated into the gmail account. For most knowledge workers, I think these differences are likely to make for a more positive experience. In that sense – buzz is extremely well suited to be the Facebook of knowledge workers. Whether it can compete with the trivial, frivolous, chatty social networking capabilities of Twitter or Facebook is something I couldn’t form a clear opinion on.

On carving a new market :

Based on my earlier statements, it should be obvious that I believe buzz will carve new markets which did not use Facebook / Twitter / Friendfeed earlier. These markets are :

People who tried twitter but didn’t get it
People who primarily use email for communications instead of facebook / twitter (for both Personal and Business Networking)
People who would like to use Facebook / Twitter like capabilities from a knowledge working / business perspective.
People who are just too nervous to use anything new on the net may still find the email integration a continuum of their email exchange rather than a ‘new web site’.

On Privacy :

I think Buzz definitely ranks superior on privacy, given its capability to conduct private directed closed group conversations. As an example, I can imagine many google apps installations which could use buzz for intra company conversations even as they clearly restrict even the public streams not to be visible outside the domain. While I haven’t seen any comment from Google on whether buzz would be available for google apps, I imagine its only a matter of time since it is so particularly well suited for that environment.

On google wave :

Google wave in many ways was a damp squib. It imposed a new usability paradigm even as some of the capabilities such as near real time updates were clearly superb. Google buzz reverses the wave approach by offering a different conversation model which dovetails not only into similar usability expectations as defined by the competition but goes one step beyond by integrating into the killer web application – web mail. I wouldn’t be surprised if it came to light later that Google buzz is a google wave backend with a completely new usability experience.

A note on integration :

While its only a statement of intent at this stage, google has setup a site for the Google Buzz API. However the promise is awesome in terms of multi standard support for variety of standards and protocols support including RSS, Atom, PubSubHubbub, Social Graph API, OAuth, WebFinger and Salmon. It does seem that google will want to support a variety of other alternative and mashup use cases around Google Buzz which might allow for more interesting uses to emerge (as it happened for Facebook and Twitter). Clearly a space to watch with some interest in months to come.

In summary :

Google buzz makes the strongest case for appealing to the power email users and those particularly focused on privacy (eg. knowledge working for businesses). Its strong differentiator is its mail integration. It does offer adequate features to compete with Facebook / Twitter for their existing markets as well. And joins the party with a large number of people – who already have a google account.

No related posts.

Software development is about the middle and not just the end

Dhananjay Nene — Thu, 14 Jan 2010 13:45:07 +0000

Software development often gets similar treatment as other disciplines when it comes to matters of management. Often at a cost. And the cost comes from the fact that, the learnings and best practices gleaned from a number of disciplines are hard to apply in the field of software development, without understanding and dealing with some of the essential differences. And those differences and their implications are what this post explores.

There was an interesting post recently Why program with continuous time?.

Today I read a confusion that I’ve heard many times before about continuous time, which I’d like to bring up, in part because I love continuous time, and in part because there’s a much broader underlying principle of software design.

I don’t see why the lack of continuous streams leaves a “gap”. In the end all streams are discrete

“In the end”, yes. Just as in the end, numbers are displayed as ascii numerals. However, programming is more about the middle than the end, i.e., more about composition than about output. … Similarly, continuity in space and in time is better for composition/modularity, leaving discreteness to the output step.

The post above is completely unrelated to management. Its talking about a programming style called Functional Reactive Programming. But an idea it eloquently brings out is programming is more about the middle than the end, i.e., more about composition than about output. And thats a thought that I would like to expand upon.

While arguably in all disciplines the middle is also important, I would emphasise that it is particularly more so in software development. And that stems from a number of uncertainties that abound in software. The requirements are often not available in adequate detail, and to the extent available, are not stable. Moreover solutions to these problems are not all thought through upfront. Not because people don’t do their homework – its simply because it is actually extremely difficult to visualise all the moving parts, and then anticipate and resolve all the challenges. In fact many of the challenges don’t show up until you start programming. Thus programming often is an exercise in identifying surprises, resolving them, working out algorithms, discovering flaws with them, fixing these flaws only to find more surprises. Rinse and repeat. Basically its a significant exercise in uncertainty management. In such a situation, estimation results in underestimates since these are based on visible not actual challenges and surprises. The solution that does get applied with some success is to use historic estimation errors to compensate current estimates. (Note that thats not always obvious since the estimate could have the compensation already built in).

To add to the problem there is usually no Bridge 2.0, Lathe 2.0 or a Marketing Campaign 2.0 (and to the extent there might be some, these are far less dependent upon the quality of the work that went into the 1.0). Some programmers are continuously focused on 1.0, while many are thinking about the 10.0 even as they work on the 1.0. Optimising for 1.0 alone may not be the right way to approach an initiative. In the good old design methodology one attempted to anticipate and build for a lot of future changes. This requires a lot of extra complexity and development in terms of layering, parameterisation, open extensibility etc. In conventional agile methodologies the focus is much more on the 1.0, but there is a deliberate and definite cost introduced in terms of creation of a large number of automated test cases. These automated test cases is the assurance premium to be paid, in order to provide the comfort to undertake the necessary refactoring challenges when moving from 1.0 to 2.0. Thus whichever way one looks at it, the 1.0 is always suboptimised to allow for a more optimal 2.0, 3.0 etc. (note: I am referring to 1.0 figuratively as the current initiative and higher version numbers to future initiatives building on the current one). But selection of the necessary suboptimisations, with a view to optimise the overall sequence, requires a substantial understanding of the software development processes, the code, the tools etc.

Finally programming has a far far higher human element involved than other disciplines. After all there’s hardly any raw material dependency and the tooling is often a desktop, the cloud and set of open source libraries. Thus it is subject to the vagaries and frailties of human behaviour. Coordination, Communication, Productivity, are all held hostage by this fickle human element. While programming is not unique in this respect, achieving high team productivity is as much a function of morale, motivation, and communication as perhaps any other discipline. Thus progress is often non linearly proportional to such soft factors.

I believe there’s some benefit in evaluating software development management styles with how sports teams are managed. In sports, team performances are rarely consistent. Moreover even though team playing skills are critical, raw talent is also very important. But the analogy is most appropriate when one evaluates the game. The game is often a series of surprises reacted to by immediate on the fly action. Thats where adaptability and ability to counter attack surprises, can be effectively brought to bear. And in sports as in software development, player morale and motivation needs to be enhanced even as the player arrogance or dysfunctional behaviours are contained.

Since different sports are structured differently let me clarify the roles here. In some cases the manager and coach are different. In others they are vested in the same person. In my mind the manager’s role is resource management, broader strategy planning, impediment removal and stakeholder communication. The role of the coach is to enhance team skills, ensure fitness and training, and in case of many sports actively guide the team in refining the on field strategy when the game is in progress. Now lets consider the players. In most cases the players are living the game. They are in the middle. A substantial portion of their interesting professional time is spent in the middle. And when they are doing that, they are continuously dealing with a set of constant surprises, even as they broadly attempt to achieve the deliverables required for the end. A slight difference between sports and software development is that in case of sports the time is fixed and the score is variable. In case of software development at least in the non agile world, the score (feature set) is fixed and the time therefore becomes the variable. But the interesting takeaway from this analogy is that once the game starts, the ends become the background and the game dominates. The players and the developers are all living in the middle.

Why is this particularly relevant ? It is so because it helps us understand the implication of management styles in software development teams performance. If one wants to optimise – one needs to very well understand the middle and not just the ends. And (to come to the point) thats where I have a big issue with software developers, who end up imagining that as they rise up in the hierarchy, they somehow can stop worrying about coding and quality assurance, and instead start focusing only on managing people and deadlines. Thats a mistake. Since the game is played in the middle, you need to be able to help your team play the game. And you need to do it as a coach or a captain. For that, so long as you want to manage an ongoing game, you need to keep on playing and keep on being either sufficiently conversant about whats happening in the middle or be the captain and play the game with the team. Thinking that somehow you’ve grown in stature enough to not worry about the tactics, and focus on things increasingly strategic might actually make you unhelpful. The game requires skill and continuous adaptation – and you will need to be able to play, until you one day decide, you are no longer interested in managing the game but only the periods between.

The non playing managers who attempt to manage the game on the other hand display some interesting smells. They stop understanding the sport. They start believing that since they had played some particular sport many years ago, its easy for them to now manage any other sport. They are unable to appreciate the actual game play that happens. So when required to score a particular number of goals in 90 minutes (say in soccer), they start pushing in more team members (alas, this is a luxury not available to sports teams). Now on field coordination starts getting more difficult. The newer members being relatively junior are the ones who start getting intercepted more often (thus the team keeps on losing the ball more frequently). If the team luck is all run out, the same managers also attempt to run the tactics and gameplay selection. This is based on some fairly old set of experiences. The spectators (customers) start seeing far more confusion on the field. And the objectives change from playing the game well to just scoring the required number of goals. And in the end the spectators may get to see the number of goals promised, but only through a confused dissatisfying game, and that too sometimes only after many bouts of extra time. And soon enough once the non playing managers realise they are unable to effectively control the game play, they try to manage other variables. But really whats starting to happen is that they are hurting the gameplay, and sometimes instead of managing the game, they start managing the spectators with the game on autoplay. Suboptimisation maximised. Alas, sometimes the spectators also get used to this and start forgetting what a good game used to be like.

To summarise, ends are good to have, ends are important, ends provide the background urgency. But software development is not so much about managing the end goals as much as managing the middle. If you want to build good software and help your team build good software, make sure you continue to focus on how software is built not just the quantitative targets. Make sure you live and enjoy the middle. More often than not, the ends you achieve eventually could be amongst the best possible ends that were practical – but more importantly the customers will also go home satisfied. So long as you keep your eye on the ball and don’t lose sight of the middle. And if you want to view it a bit philosophically, in the end we are all dead. Only the middle matters.

Authors Note : This post has been substantially revised from its earlier version from a syntax, grammar and punctuation perspective. And the philosophical note (the last line) was added later.

Concise python code

Dhananjay Nene — Tue, 05 Jan 2010 20:19:59 +0000

I love python for a number of reasons. Brevity is one of them. Readability is another. However writing readable concise code can take some time to getting used to. This post discusses some code snippets especially in the context of some of the comparisons done between python and clojure – another language which excels at brevity and shares yet another feature with python, ie. people either love or hate its syntax, they are rarely indifferent to it.

Please note : The intent of this post is not to compare LOC of python with other languages. It is meant to demonstrate python code implementation which helps describe the capabilities of python in terms of writing clean, concise, readable code. This post continuously refers to other posts and may require you to read the referred posts in conjunction with this post to get maximum value.

Code and ceremony

The post Java vs Clojure – Lets talk ceremony! describes a simple problem which requires 28 LOC in Java and 1 or 9 in clojure. It is a simple logic which creates a list, populates it with four values (two of them identical) and creates a list with unique items. There are two alternative clojure implementations, one in 1 line of code and another in 9 – the former using a built in construct in the language which especially helps to write a more concise logic, and the latter by writing code similar to the Java code. Without any further ado, here are the two alternative python implementations

a. By using built in capabilities – In this case we use the set collection

1	print set(("foo","bar","baz","foo"))

b. Now here’s where we will not use a built in operator which automatically chooses the unique elements but again write code similar to the java code. This code is as follows :

1	print reduce(lambda x,y: x + [y] if y not in x else x ,("foo","bar","baz","foo"),[])

There – both of them are exactly one line long.

Project Euler solutions
The post Python vs Clojure – Evolving refers to many sample python implementations solving some of the problems documented on project euler site. We shall revisit these python implementations here.

a. Find all numbers dividable by 3 or 5 in 0 < x < 1000:
In this case the clojure code is 4 lines long where as the python implementation is 3 lines long

1	print sum(i for i in range(1,1000) if i%3 == 0 or i%5 == 0)

Again the implementation is exactly 1 line long – no lambdas, no filters etc.

b. Get the sum of all even Fibonaccis below 4 mill.

The suggested python implementation is about 17 lines long. The implementation below 7.

1
2
3
4
5
6
7
8
9

from itertools import takewhile

def fib():
x,y = 1,1
while True :
x,y = y,x+y
yield x

print sum(i for i in takewhile(lambda x : x < 4000000,fib()) if i%2 == 0)

c. Finding Palindromes
The suggested python solution uses about 23 lines of code, to 6 in clojure and takes 15 seconds almost 300% slower than the clojure code (emphasis from the post referred to).

Here’s another alternative implementation.

1	print max(x * y for x in xrange(100,1000) for y in xrange(100,1000) if str(xy) == str(xy)[::-1])

Again if you notice – exactly one line of code. And incidentally on my notebook it clocks in about 0.85 seconds which makes the clojure implementation about 470% slower than the python implementation. (emphasis mine)

Top Rank Per Group

In yet another post Python vs Clojure – Reloaded, the author discusses the Rosetta Top Rank Per Group implementation of python. Unsurprisingly the python implementation uses 11 lines of code (not including the initialisation) compared to clojure’s 6.

One of the commenters suggests an eminently readable and simple python solution in about 6 lines of code. The code snippet is as follows

1
2
3
4
5
6
7
8
9
10
11

from itertools import groupby
from operator import itemgetter

data = [dict(name="Tyler Bennett", id="E10297", salary=32000, department="D101"), ...]

department = itemgetter('department')
for dep, persons in groupby(sorted(data, key=department), department):
print "\nDepartment", dep
print " Employee Name Employee ID Salary Department"
for person in sorted(persons, key=itemgetter('salary'):
print "%(name)-15s %(id)-15s %(salary)-15s %(department)-15s"

An alternative far less readable solutions which shaves off one more line but certainly not preferred to the solution above (am just listing it since it shaves off the step of having to create an intermediate data structure) is

1
2
3
4
5
6
7
8
9
10

import operator
from itertools import groupby

data = [('Employee Name', 'Employee ID', 'Salary', 'Department'),....]

print reduce(lambda x,y : x + y,
(('\nDepartment %s\n Employee Name Employee ID Salary Department ' % dept +
reduce(lambda x,y : x + ("\n " + "%-15s " * len(data[0]) % y),
sorted(recs, key=lambda rec: -rec[-2])[:3],''))
for dept, recs in groupby(sorted(data[1:],key=operator.itemgetter(3)), lambda x : x[3])),'')

Some might wonder whether I am joining multiple lines of code into one just to reduce the line count. The code is consistent with the idiomatic usage of python and also with the way python list comprehensions are documented in the formal python proposal for list comprehensions PEP 202 : List Comprehensions. Besides python is a whitespace sensitive language and the compiler disallows multiple lines of code in one line without using the ‘;’ separator which I have most definitely not used in the suggested solutions.

PS: There’s of course an interesting image at the end of the post Python vs Clojure – Reloaded indicating the perception of how clojure code speaks for itself . The reason I mention it is that I believe if one is expressing a strong opinion as the one reflected by the picture, one better be far more careful and thorough with verifying the ~~veracity of the assertions~~ appropriateness of the samples compared to. Let me clearly state that I do not know that the clojure loc compared to is an appropriate target so do not infer this post to be a Python vs Clojure LOC as much as a post which emphasises that Python is a good vehicle to write concise readable code. FWIW I am learning clojure and look forward to building more competence with it.

Update: There is another nice post by Stephan Schmidt – Anatomy of a Flawed Clojure vs. Scala LOC Comparison which does reflect an opinion in the context of Scala and a post authored on the same blog as the ones I refer to above.

Post conference recap : The 4th Indicthreads.com conference on Java Technology

Dhananjay Nene — Mon, 14 Dec 2009 14:31:12 +0000

The annual indicthreads.com java technology conference is Pune’s best and possibly one of India’s finest conferences on matters related to Java technologies. I looked forward to attending the same and was not disappointed a bit. The last one was held about 3 days ago on Dec 11th and 12th, and this post reviews my experiences at the same.

As with any other conference usually something or the other isn’t quite working well in the morning, so I soon discovered we had a difficulty with the wireless network being swamped by the usage. There were some important downloads that needed to be completed, so my early morning was spent attempting to get these done .. which meant I missed most of Harshad Oak’s opening session on Java Today.

The next one i attended was Groovy & Grails as a modern scripting language for Web applications by Rohit Nayak. However I soon discovered that it (at least initially) seemed to be a small demo on how to build applications using grails. Since that was something I was familiar with, I moved to the alternative track in progress.

The one I switched to even as it was in progress was Java EE 6: Paving the path for the future by Arun Gupta. Arun had come down from Santa Clara to talk about the new Java EE6 spec and its implementation by Glassfish. Arun talked about a number of additional or changed features in Java EE6 in sufficient detail for anyone who got excited by them to go explore these in further detail. These included web fragments, web profile, EJB 3.1 lite, increased usage of annotations leading to web.xml now being optional, and a number of points on specific JSRs now a part of Java EE6. Some of the things that excited me more about Glassfish were, (a) OSGi modularisation and programmatic control of specific containers (eg Servlet, JRuby/Rails etc.), embeddability, lightweight monitoring. However the one that excited me the most was the support for hot deployment of web apps for development mode by allowing the IDEs to automatically notify the running web app which in turn automatically reloaded the modified classes (even as the sessions continued to be valid). The web app restart cycle in addition to the compile cycle was alway one of my biggest gripes with Java (second only to its verbosity) and that seemed to be going away.

I subsequently attended Getting started with Scala by Mushtaq Ahmed from Thoughtworks. Mushtaq is a business analyst and not a professional programmer, but has been keenly following the developments in Scala for a couple of years (and as I later learnt a bit with Clojure as well). Unlike a typical language capability survey, he talked only about using the language for specific use cases, a decision which I thought made the presentation extremely useful and interesting. The topics he picked up were (a) Functional Programming, (b) DSL building and (c) OOP only if time permitted. He started with an example of programming/modeling the Mars Rover movements and using functions and higher order functions to do the same. Looking back I think he spent lesser time on transitioning from the requirements into the code constructs and in terms of what he was specifically setting out to do in terms of higher order functions. However the demonstrated code was nevertheless interesting and showed some of the power of Scala when used to write primarily function oriented code. The next example he picked up was a Parking Lot attendant problem where he started with a Java code which was a typical implementation of the strategy pattern. He later took it through 7-8 alternative increasingly functional implementations using Scala. This one was much easier to understand and yet again demonstrated the power of Scala quite well in terms of functional programming. Onto DSLs, Mushtaq wrote a simple implementation of a “mywhile” which was a classical “while” loop as an example of using Scala for writing internal DSLs. Finally he demonstrated the awesome power of using the built in support for parser combinators for writing an external DSL, and also showed how a particular google code of summer problem could be solved using Scala (again for writing an external DSL). A very useful and thoroughly enjoyable talk.

The brave speaker for the post lunch session was Rajeev Palanki who dealt both with overall IBM directions on Java and a little about MyDeveloperworks site. In his opinion he thought Java was now (post JDK 1.4) on the plateau of productivity after all the early hype and IBM now focused on Scaling up, Scaling down (making it easier to use at the lower end), Open Innovation (allow for more community driven innovation) and Real Time Java. He emphasised IBMs support to make Java more predictable for real time apps and stated that Java was now usable for Mission Critical applications referring to the fact that Java was now used in a USS Destroyer. He referred to IBMs focus on investing in Java Tooling that worked across different JRE implementations. Tools such as GCMV, MAT, and Java Diagnostic Collector. Finally he talked about the IBM MyDeveloperWorks site at one stage referring to it as the Facebook for Geeks.

The next session was Overview of Scala Based Lift Web Framework by Vikas Hazarati, Director, Technology at Xebia. Another thoroughly enjoyable session. Vikas dealt with a lot of aspects related to the Lift web framework including various aspects related to the mapper, the snippets, usage of actors for comet support etc. I was especially intrigued by Snippets which act as a bridge between the UI and the business logic have a separate abstraction for themselves in the framework and how the construct and functionality in that layer is treated so differently from other frameworks.

I subsequently attended Concurrency: Best Practices by Pramod Nagaraja who works on the IBM JRE and owns the java.nio packages (I think I heard him say owns). He talked about various aspects and best practices related to concurrency and one of the better aspects of the talk was how seemingly safe code can also end up being unsafe. However he finished his session well in time for me to quickly run over and attend the latter half of the next presentation.

Arun Gupta conducted the session Dynamic Languages & Web Frameworks in GlassFish which referred to the support for various non java environments in Glassfish including those for Grails/Groovy, Rails/JRuby, Django/Python et. al. The impression I got was Glassfish is being extremely serious about support for the non java applications as well and is dedicating substantial efforts to make Glassfish the preferred platform for such applications as well. Arun’s blog Miles to go … is most informative for a variety of topics related to Glassfish for both Java and non Java related aspects.

The last talk I attended during the day was Experiences of Fully Distributed Scrum between San Francisco and Gurgaon by Narinder Kumar, again from Xebia. Since a few in the audience were still not aware of agile methodologies (Gasp!), Narinder gave a high level overview of the same before proceeding down the specific set of challenges his team had faced in implementing scrum in a scenario where one team was based in Gurgaon, India and another in San Fransciso, US. To be explicit, he wasn’t describing the typical scrum of scrum approaches but was instead describing a mechanism wherein the entire set of distributed teams would be treated as a single team with a single backlog and common ownership. This required some adjustments such as a meeting where only one person from one of the locations and all from another would take part in a scrum meeting in situations where there were no overlapping working hours. There were a few other such adjustments to the process also described. The presentation ended with some strong metrics which represented how productivity was maintained even as the activities moved from a single location to a distributed model. Both during the presentation and subsequently Narinder described some impressive associations with senior Scrum visionaries and also some serious interest in their modified approach from some important companies. However one limitation I could think of the model was, that it was probably better geared to work where you had developers only in one of the two locations (offshoring). I perceived the model as a little difficult to work if developers were located across all locations (though that could end up being just my view).

The second day started with a Panel Discussion on the topic Turning the Corner between Arun Gupta, Rohit Nayak, Dhananjay Nene (thats yours truly) and moderated by Harshad Oak. It was essentially a discussion about how we saw some of the java and even many non java related technologies evolving over the next few years. I think suffice to say one of the strong agreements clearly was the arrival of Java the polyglot platform as compared to Java the language.

The next session was Developing, deploying and monitoring Java applications using Google App Engine by Narinder Kumar. A very useful session describing the characteristics, opportunities and challenges with using Google App Engine as the deployment platform for Java based applications. One of the take away from the sessions was that subject to specific constraints, it was possible to use GAE as the deployment platform without creating substantial lockins since many of the Java APIs were supported by GAE. However there are a few gotchas along the way in terms of specific constraints eg. using Joins etc.

I must confess at having been a little disappointed with Automating the JEE deployment process by Vikas Hazrati. He went to great depths in terms of what all considerations a typical J2EE deployment monitoring tool should take care of, and clearly demonstrated having spent a lot of time in thinking through many of the issues. However the complexities he started addressing started to get into realms which only a professional J2EE deployment tool writer would get into. That made the talk a little less interesting for me. Besides there was another interesting talk going on simultaneously which I was keen on attending as well.

The other talk I switched to half way was Create Appealing Cross-device Applications for Mobile Devices with Java ME and LWUIT by Biswajit Sarkar (who’s also written a book on the same topic). While keeping things simple, Biswajit explained the capabilities of Java ME. He also described LWUIT which allowed creation of largely similar UI across different mobile platforms. He explained that while the default Java ME used native rendering leading to differing look and feel across mobile handsets just like Java AWT, using LWUIT allowed for a Java Swing like approach where the rendering was performed by the LWUIT library (did he say around 300kb??) thus allowing for a more uniform look and feel. He also showed sample programs and how they worked using LWUIT.

Allahbaksh Asadullah then conducted the session on Implementing Search Functionality With Lucene & Solr, where he talked about the characteristics and usage of Lucene and Solr. It was very explicitly addressed at the very beginners to the topic (an audience I could readily identify myself with) and walked us through the various characteristics of search, the different abstractions, how these abstractions are modeled through the API and how some of these could be overridden to implement custom logic.

How Android is different from other systems – An exploration of the design decisions in Android by Navin Kabra was a session I skipped. However I had attended a similar session by him earlier so hopefully I did not miss much.

However Navin did contribute occasionally into the next session Java For Mobile Devices – Building a client application for the Android platform by Rohit Nayak. Rohit demonstrated an application he is working on along with a lot of the code that forms the application using Eclipse and the Android plugin. A useful insight into how an Android application is constructed.

As the event drew to a close, the prizes were announced including those for the Indicthreads Go Green initiative. A thoroughly enjoyable event, leaving me even more convinced to make sure to attend the next years session making it a third in a row.

No related posts.

Rinse and Repeat with TDD

Dhananjay Nene — Wed, 25 Nov 2009 15:01:49 +0000

I must confess having had a awed love relationship with TDD (Test Driven Development). This is the style where you first write the test cases and then write the code to make them pass. I’ve always felt awed by it. I have always firmly believed that it is the right way to offer sustainable quality. As someone who hasn’t been particularly awed by separate QA teams, I have always imagined TDD to be the silver bullet to offer controlled, sustained, high quality. Well, the silver bullet is a bit of an exaggeration, but sounded nice while I was on a roll . A number of times I wrote automated test cases post facto. But when the rubber met the road – I didn’t ever implement TDD.

Its probably a function of style. I like to write a piece of code and then keep on testing it against different scenarios and often keep on remoulding it substantially. My focus is at this stage on the code structure – the design aspects. Trying to do it with TDD always created a mess. The amount of intense refactoring I do always meant I soon got tired of also simultaneously refactoring the test cases and gave up on them soon.

Anyways, I finally think I found a formula that works – at least for me. I do not write production code in the first pass. I just write prototype code. I keep on playing with it, shaping it, challenging it, cajoling it, recasting it. Until I’m satisfied with the overall structure and internal design. Now when thats done, I rewrite it. Completely. Umm, I copy/paste maybe 40-50% of the code. But this time I write it on a completely new fresh set of files. And I write the test cases before I write the code. I find I can now really write tons of test cases and code quite rapidly. And with the tremendous satisfaction that at the end of the day, I now have the control harness to be able to keep on adding new features without having to continuously worry if I broke something else. With substantially detailed test cases, now I can actually feel quite confident that if I did break something – I’ll get to know it. Not next month, not next week, not even tomorrow – in the next few minutes.

So this approach seems to be working for me – the initial efforts have delivered great results and I’m quite satisfied. Now just wondering what to call it – for lack of a better term I shall term it “Rinse and Repeat with TDD”.

So if TDD works for you – great. You have one of the most important coding disciplines nailed down. But if you’re wanting to try to do TDD but haven’t been able to successfully implement it – you may just want to check out the Rinse and Repeat with TDD style. Who knows, you might find it useful too.

No related posts.

Five important trends on the enterprise architect’s radar

Dhananjay Nene — Mon, 02 Nov 2009 09:19:02 +0000

It is no secret that the internet architectures are influencing enterprise architectures. This post attempts to summarise some of the recent trends in the internet space, which seem to be carrying some momentum sufficient enough to influence the enterprise. So without further ado, these trends are :

REST : The Representational State Transfer architecture style builds on the essential elements of those constructs which made the internet so globally scalable. A detailed explanation of the rationale and strengths of REST are completely beyond the scope of this article. If your job requires you to be continuously aware of emergent trends and whether they fit your enterprise architecture needs – this is the one must explore trend.
Impact : Web based architectures, Service Oriented Architectures, wide availability and immediate usability of data and processing requests (resources) through simple HTTP URIs and minimal integration effort
Interoperable Cloud : The interoperable cloud is the ability to create a private cloud and also leverage a public cloud. This has been made possible by offerings such as the Ubuntu Enterprise Cloud which allows you to build a private cloud or use a public cloud such as Amazon EC2 while being able to access them using the same set of APIs thanks to open source efforts such as Eucalyptus. This allows you the flexibility of initially using either a private or public cloud and then subsequently shifting to the other, or being able to use both simultaneously.
Impact : Large servers vs cluster of commodity servers, virtualisation, elastic deployments, flexible hardware procurement / provisioning, infrastructure management in organisational hierarchy.
NoSQL : While I am unhappy with the name, it has stuck. This refers to a set of options now available to store your data unconstrained by many RDBMS requirements (eg. flexible schema, key value pairs etc.). Some of the databases also allow you to store data in a distributed manner over a number of servers with an intent to support high availability in write intense scenarios even as they may require you to move towards eventual consistency. These options increase your manouverability / flexibility as an architect even as they require you to meet a different set of challenges.
Impact : Relational databases, data storage strategies, data distribution strategies, vertical vs. horizontal scalability, transactionalisation, consistency and availability
Polyglotism : Developer costs now occupy an increasing percentage of total costs, development time is being an increasingly dominant factor for time to market, and ability of software to change and adapt quickly to newer demands is now a critical success metric. One of the solutions is to write different parts of the software in a different languages most appropriately suited for concise and rapid coding as well as supporting quick reaction changes to each part appropriately. Thus it is conceivable to have some of the business rules written in a dsl written using jruby and some of the algorithms written in clojure in a software built on the JEE platform.
Impact :Development culture and processes, minimum developer skill and scalability, risk management for managing required vs. available skills.
Decentralised processing : Thanks to many developments which are leading to increasingly distributed processing including REST and NoSQL, applications will need to be a set of collaborating network based components (we’ve heard this before with distributed objects as well). However especially given some of the lesser guarantees that such architectures can provide around immediate guaranteed processing, latency issues, distributed control and asynchronous processing, a particular piece of business logic may get satisfied in a staggered fashion across a number of collaborating components. This may increase challenges in terms of currency of available data even as it helps actually deliver on the vision of distributed objects and simplifies individual component development. While asynchronous capabilities such as those supported by MQ series and the like have been used in the enterprise for ages, I do anticipate increasing use of lighter messaging constructs such as PubSubHubbub within the enterprise.
Impact : Application partitioning, network based components, difficulty in supporting fully synchronous workflows.

Service Oriented Architecture is primarily about business and not technology. Bollocks!

Dhananjay Nene — Mon, 26 Oct 2009 10:56:51 +0000

There’s quite a few times I’ve heard / read a gross oversimplification of architecture in reference to business and technology. And while I believe I understand the ‘essential cause’ which drives such a simplification, I’ve often felt quite frustrated at the resultant impression thats provided by such a simplification. In many ways and forms, it boils down to the statement (not exactly the same since I’m not quoting directly), quite similar to the one below :

Service Oriented Architecture is primarily about business and not technology

This also reflected in the recent SOA Manifesto which states as its very first described value :

Business value over technical strategy

Allow me to straight away start picking some holes into this :

Anything that a business does – whether it is soa, software architecture, building architecture or simple plant and machinery design, to the extent (which is exactly 100%), technology serves the business goals, all technology activities (and non-technology as well) are at the end of the day about achieving business objectives and therefore about business. So why single out architecture? And even more so why single out SOA?
Architecture is also about business. But its not the same as saying its primarily about business and not so much about technology. For a moment lets step away from Software/Hardware Architecture and look at Building Construction Architecture. The legendary creation of Ayn Rand – Howard Roark, for all his eccentricities and seemingly portrayed egocentric and egotistic behaviour did meet the test of business objectives to the extent of making the residents of his creations extremely satisfied. And at no point would you gather the impression that he in any manner put construction technology to any secondary position to his business context and objectives. At the end of the day thats what architecture is. It is not about making one of business or technology more important than or subservient to other. Its about effectively mapping the two to provide a strong technology solution appropriate to the business needs.

I suspect one of the important causes here is that people have forgotten that the A in SOA stands for architecture, and therefore shorn of architecture, business and technology can be seen to be competing in a non win-win form.

So if SOA stands for Service Oriented Architecture, then I must submit that architecture is the art of getting the two working together. And I am of the opinion that an exercise suggesting one is more important than other is an exercise in a field unrelated to architecture. I suspect many practicing software architects will agree with this. I suspect Ayn Rand wouldn’t disagree as well.

Stop calling me NoSQL

Dhananjay Nene — Thu, 22 Oct 2009 23:26:09 +0000

Dear Reader,

Apologies for sending this note to you completely unannounced and out of the blue. However I find myself in a peculiar situation of having a very weird name being dumped upon me. While I am indifferent to the name per se, I am greatly pained as I realise that it is a completely inappropriate name. What is even more confounding is the very bunch of people who have happily assigned me the name and continue to popularise it belong to that class of people some of whom actually are extremely particular about accurate nomenclature and have no hesitation in creating a 100 letter class or function name by concatenating 20 words just to make sure the name is unambiguous and conveys the intent clearly.

Ahh.. but I digress and impose upon you without introducing myself adequately first. I am a data storage style. I am not new, but lately far too many a software engineer have started taking a liking for me. Ever since I have been around, I have with great amounts of jealousy watched my cousin the RDBMS being courted by the finest of engineers (in all honesty there were some fine engineers interested in me too, but far too few compared to my cousin). But lately multiple concurrent developments have made a fair amount of attention come my way too.

You see unlike RDBMS, I don’t require that data be clearly split into tables, columns and rows. I can work with data the way it is most naturally represented. As a tree of individual data fields, lists, arrays, dictionaries etc. Also I do not require that you always clearly define each and every possible schema element before being able to store data corresponding to the schema. I can happily accept a schema dynamically or even work without a schema. Some of my early forms were based on key value pairs stored as B-Trees (eg. Berkeley DB). Over the years people have figured out ways to represent the data as a set of decomposed document elements, store data spread across a cluster, replicate it for better availability and fault tolerance, and even perform post storage processing tasks using map-reduce sequences. But really what separates me from my cousin and other storage systems is that I don’t make demands on the data – I take it in its naturally found form and then store it, replicate it, slice it, dice it and glean information out of it. And therein lies my true identity – I will work with data the way the data is best represented with all its arbitrary inconsistencies and inabilities to always clearly specify a constraining schema. And the engineers who’ve spent time with me seem to have enjoyed it quite a bit.

But the horror of it – they gave me a completely inappropriate moniker – ‘NoSQL’. First and foremost I exist to promote a storage style and thats what identifies me. I work with data in its natural and arbitrary forms. Therefore to make it seem like I represent a lack of something else is utterly missing the point. The SQL in NoSQL stands for Structured Query Language, which depends upon Fixed Structure Relational Data. Since I change the very nature of the data being stored, that SQL is not required or relevant is automatic and inconsequential.

Its like calling a under-the-ocean-mountain_range as NoIgloo. Its dead obvious igloos will not be found there. But calling that mountain range NoIgloo is a big disservice to visitors. You use that as a marketing term, attract people, then tell them that NoIgloo actually has nothing to do with Igloos – its got to do with mountains and oceans, and that they need to first unwind all the confusion they created in their minds due to NoIgloo and then go through a phase of reunderstanding mountains and oceans. And while they came prepared for a possibly warmer place given the name NoIgloo – it actually is a wet place so they need to again change their garments and equipment for the journey. A wholely avoidable situation.

Update: Brad Anderson pointed out this interesting post NoSQL: A Modest Proposal which traces the genesis of my name which leaves me very very disappointed. Almost seems to suggest that people are flocking together and naming me not based on something inherently powerful about me – but as a mechanism to demonise my cousin RDBMS. This is most unfortunate, since we actually end up being useful in very different situations and more often than not are likely to complement each other rather than compete with each other. I do hope a better moniker does prevail over time

What I would like is to see a better / more appropriate name for me. Hmm .. call me free form storage, natural persistence or flexi schema storage or perhaps something else even more appropriate (this blog owner prefers “natural persistence”). Each of these conveys far more about me far more accurately than NoSQL does. Basically please please call me something better than NoSQL. So can I request you to carry forward my plea by further forwarding and retweeting this to your friends and ask them how they can so callously call me by such a silly name when they take the utmost precautions in properly naming their classes and methods. Plead with them to stop doing this and please work with others to give me a better name. I think it will cause less confusion over the coming months and years, and the field of software shall recover its glorious tradition of maintaining precision in communication by using accurate naming.

Sincerely,
The one who doesn’t want to be called NoSQL

PS : As a background to this there was an interesting conversation earlier today between this blog owner dnene and Kent Beck on twitter, where Kent so kindly and graciously helped carry forward the thought process of helping identify my essential characteristics, and it is in no small part, thanks to this conversation that I was able to articulate myself and my grief. I reproduce that conversation below. (Update: though in all likelihood Kent’s intent was to help clarify the thought rather than contest the names. In hindsight, it makes sense to ask for permission to reproduce conversations .. even when such are on the public twitter stream – something that wasn’t done in this case. )

Twitter ID	Tweet / Message
dnene	NoSQL is such an inappropriate name. NoTables at least makes a little more sense.
KentBeck	@dnene but what would nosql be called if you wanted to say something positive about it?
dnene	@KentBeck Thats a great question .. still thinking .. best thought so far – FlexiStore (though not good enough yet )
KentBeck	@dnene what can you do with a nosql store that you can’t do with an sql database? why would you be excited to use one?
dnene	@KentBeck I see where u r going with this (a) unconstrained & composite storage (b) store resources not records (c) shard/scale horizontally
dnene	.@KentBeck I think there is a merit in attempting to define nosql in terms of what it is rather than what it isn’t
KentBeck	@dnene there are many more people confused about what datastore to use than who hate sql. the positive approach appeals to the former.
dnene	@KentBeck Agreed .. and I’m aware of many more who wonder why we need a different datastore than the RDBMSs. NoSQL as a name doesn’t help.
KentBeck	@dnene well, why do we need a different data store?
dnene	@KentBeck Primary Need : We need support for flexible/arbitrary schemas with complex depths – RDBMSs don’t dance well in this space.
	@KentBeck Secondary Need : Support for deferred processing required for analytics (eg. Map/Reduce). RDBMS don’t do too bad a job here
	@KentBeck Tertiary Need (not one that I’ve felt strongly yet) : Distributed and horizontally scalable storage on commoditized h/w.
KentBeck	@dnene it seems like you’re looking for realistically structured data, not data twisted to fit a formula convenient for mathematicians.
dnene	@KentBeck Yes.. thats it! I’m looking for realistically or naturally structured data storage / persistence. Rocks compared to the term nosql
dnene	@KentBeck Wonder if the term arbitrarily structured makes sense as well. This has been one heck of a conversation/Q&A so far +1:)
KentBeck	@dnene glad you found it helpful. you get bonus points if the opposite of the name you pick is unattractive, a la “structured programming”

NoSQL – A fluid architecture in transition

Dhananjay Nene — Wed, 21 Oct 2009 18:21:14 +0000

Lot of talk about NoSQL. Much of it well deserved. And while lot of the excitement around it is well understood by those in the know, some of it may actually be confusing to those who are relatively new to the matter. This post is actually for the latter group – not to argue for or against NoSQL – just to put it in perspective.

What is NoSQL : Some of the characteristics shared by most if not all the NoSQL engines are as follows :

Schemaless or Hierarchical Schema Storage NoSQL assumes at its very basis a schemaless or a hierarchichal schema storage system. In most cases this consists of a simple key value pair storage. While some storage engines excel at storing small values (LightCloud/Tokyo Cabinet), some are strong at storing large documents (CouchDB).
Distributed storage : This is one of the driving forces of NoSQL growth, though not a distinguishing characteristic of NoSQL. One of the areas these storage systems separate themselves from RDBMS’s is their ability to allow better horizontal scalability. This varies from the simple master-master replication for MongoDB, to multi node sharding using consistent hashing with LightCloud (a la memcached) to a multiple master eventual consistency model of Riak. The basic premise in using some of the NoSQL engines is that storage will scale horizontally.
Support for deferred processing :Many of these engines allow for some degree of deferred processing. Whether this be simple lua scripting in case of LightCloud or map-reduce scripts in case of CouchDB, the general assumption is that some amount of latency in computation times is acceptable, and some of the computations (especially related to analytics based views) will be performed post storage.
Eventual Consistency : This may seem like a necessary feature of all NoSQL storage systems but it isn’t. While clearly some such as CouchDB (in terms of its map-reduce views) and Riak are better placed for supporting and implementing eventual consistency, it is quite feasible to use others such as LightCloud or MongoDB to implement immediate consistency using a single master-master pair. Suffice to note that eventual consistency is not a necessary side effect of using a NoSQL storage system, though it wouldn’t be incompatible for the two to work together.

But the points I would really like to emphasize are :

NoSQL is not a direct competitor to RDBMS/SQL : It is actually a solution to many use cases where using RDBMS was perhaps a poor fit. Thus the decision for an architect is not which of the two competing options (RDBMS or NoSQL) Update: ~~one should select~~should be the preferred standard storage strategy, – it simply is which one is the more appropriate storage system for the application under consideration.
NoSQL is still at a fluid stage of its development : All the NoSQL storage systems (but for LightCloud/Tokyo Tyrant) are still being quite actively developed. These have not reached v 1.0 (Update: MongoDB is at v 1.1) and it is likely that some time will pass before any of these get beyond the beta and release candidate stage and get a 1.0 in-production stamp. While there is a lot of interest, there still is a substantial amount of experimentation in terms of the right feature sets leading to differently focused developments in different storage systems. To an architect this represents an interesting challenge. I think the way to approach this right now is to not use these in mission critical (eg. life or health impacting) systems, and to focus on reasonable expectation management in terms of ensuring the right kind of SLAs around their availability (simply because many of these haven’t yet been put to intense use in production the way say an Oracle or MySQL have been). This is not an attempt to spread “FUD” about NoSQL – far from it it is an exercise in setting appropriate expectations. i would also recommend that it would be appropriate to evaluate the available NoSQL choices only when reasonable SLAs can be worked out for their usage. It is certainly preferred to using NoSQLs rather than using RDBMS’s in an inappropriate manner (large objects serialised into BLOBs or into name-value pair tables). However, I would suggest that you do not deeply bind yourself into a particular NoSQL engine. The future development of most of the storage systems is still unknown to a certain extent, as is the future landscape including any shakeouts. Should one recommend usage of a NoSQL engine – it is important to have a clear plan for switching over to an alternative engine should a need arise in the future. While this is easier said than done, deciding the appropriate level of abstraction to use (ie. code to the API directly vs. use a layer of abstraction for engine independence) is best left to designer / architect to dwell upon.

Configuring a secure Ubuntu Linux Virtual Private Server

Dhananjay Nene — Mon, 05 Oct 2009 13:56:20 +0000

This post is based on my notes for an initial configuration for an Ubuntu 9.04 Virtual Private Server with a focus on security. At that time I searched for a number of references on security, and while I have not kept the note of all their URLs, most of what I write below is as a result of other documents even though I cannot specifically cite them (in other words, there is little originality except perhaps for attempting to cover the entire gamut of configuration activities into one article).

Keep in mind that these steps are based on my notes which might be a little incomplete especially around the part where acidbase is installed.

Initial Configuration : Basic configuration when getting started up.
Setting up basic security : Basic Security configuration
Setting up rootkit detection : Setting up rootkit detection
Setting up Bastille : Setting up Bastille
Setting up the lamp stack : Set up the LAMP stack including mysql and apache
Setting up Snort and acidbase : Configure Intrusion detection using Snort and Acidbase.
Setting up File Integrity using AIDE : Setting up File integrity checks using AIDE

Initial Configuration

These steps cover the initial setup of a server

Setup the hostname

Lets say the hostname we want to setup is vps.

$ echo “vps.mydomain.com” > /etc/hostname
$ hostname -F /etc/hostname

Now update the /etc/hosts file to reflect the hostname and the fully qualified domain name
Replace 12.34.56.78 with the IP address of your host

127.0.0.1 localhost.localdomain localhost
12.34.56.78 vps.mydomain.com vps

Updating the ubuntu repositories

You will need to update your ubuntu repositories to include jaunty-updates and universe repositories. This is so that you may install additional packages as required from these repositories as well. In my case, the earlier version of the file /etc/apt/sources.list was as follows.

However please note, that repository selection and its update strategy may be linked to your company or application strategy. Please make sure these steps are consistent with your policy. If not, kindly adapt consistent with your team / organisations policy. Also instead of us.archive.ubuntu.com, you may find other country specific server names. In that case you may want to continue to use the other server name as already listed in your file.

deb http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted

deb http://security.ubuntu.com/ubuntu jaunty-security main restricted
deb-src http://security.ubuntu.com/ubuntu jaunty-security main restricted

Upon adding jaunty-updates and the universe repositories, the resultant file is as follows.

deb http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted universe
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted universe

deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe
deb-src http://security.ubuntu.com/ubuntu jaunty-security main restricted universe

deb http://us.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe

Now update the sources. This will scan all the repositories

$ sudo apt-get update

Finally upgrade ie. replace any existing packages which have a newer upgrade

$ sudo apt-get upgrade

Download the language pack

To add the necessary for the preferred language of your choice add the appropriate language pack. In my case I add support for english (en)

$ sudo apt-get install language-pack-en

Set the timezone

Set the timezone of the server. You may choose to set it based on server location, or typical user location or to UTC.

$ dpkg-reconfigure tzdata

That will start a small app, from which you can select the timezone. I selected None of the Above which offered me a choice of timezones based on UTC offsets and subsequently selected UTC.

Setting up Mail sending

I do not need this VPS to act as a mail server. However I do need to have capabilities to send email from this machine. Many unix tools routinely assume the existence of sendmail or equivalent MTA. However that is an overkill in this context. So we shall not be installing sendmail or postfix or exim or any other equivalent. Instead we shall configure this server to be only able to send out mail using an SMTP account on another mail server. For this we shall install a tool called mailx. Note: If you have mailx already installed through another ubuntu package called mailutils, you may either continue with the same (in which case you will need to configure the remainder of the mail stack correspondingly eg. sendmail) or remove mailutils and add heirloom-mailx

$ sudo apt-get install heirloom-mailx

We shall also configure a global configuration for sending out mail. In my case its all right to always send mail using only one account irrespective of the process or user who is sending it.

$ sudo vi /etc/nail.rc

Note that in the above configuration, we shall be placing the mail account password in clear text. Make sure it is a mail account you do not use for any other purposes and that its password is not the same as used for any other purposes. Now enter the following as contents of the /etc/nail.rc file. Obviously change the relevant fields to appropriate values. Note that this file is configured for sending mail via gmail. You may need to configure it differently based on your own SMTP configurations.

set smtp-use-starttls
set from=my_user_id@gmail.com
set smtp=smtp.gmail.com:587
set smtp-auth-user=my_user_id@gmail.com
set auth-login=my_user_id@gmail.com
set smtp-auth-password=my_password

You can try testing whether this got set up successfully. Enter the following (replace youremailid@youremaildomain.com by the email id where you would like the mail to be sent to)

$ mail youremailid@youremaildomain.com
Subject: This is a test mail
Hello
.

Basic Security

In this section we shall make some basic configuration changes with a view to enhance the system security.

Mounting the shared memory as read only

Open and edit the file /etc/fstab to add an entry to mount shared memory in read only mode. The reason we do it is because many exploits use shared memory to attack other running services.

If you have a good reason to make shared memory writeable skip this step.

$ vi /etc/fstab

Now add the following line at the end of the file

tmpfs /dev/shm tmpfs defaults,ro 0 0

Tightening the passwords

One of the easiest exploits is to attempt a brute force login using dictionary based attacks. In order to ensure strong ie. non-guessable passwords we shall update the password checking policy so that it allows only strong passwords. A simple way to ensure that is to ensure a reasonable minimum length and to ensure multiple character classes.

First lets install a new pam authentication module pam_cracklib. To install the same run the following

$ sudo apt-get install libpam-cracklib

Answer ‘Y’ to the prompt it asks for regarding continuing.

Note: if you did not add the universe repository to your sources.list file, you will not be able to install libpam-cracklib. In that case you will need to skip this step.

This should’ve resulted in the file etc/pam.d/common-password having an entry for pam_cracklib.so and pam_unix.so. Update the pam_cracklib.so entry to add one more requirement ie. minclass=4.

In my case, the resultant two lines in /etc/pam.d/common-password are as follows. Note that I added the minclass=4 clause manually.

password requisite pam_cracklib.so retry=3 minlen=8 difok=3 minclass=4
password [success=1 default=ignore] pam_unix.so obscure use_authtok try_first_pass sha512

There. You now have a strong password scheme which will conduct a whole range of password checks in addition to ensuring that the password has a minimum length of 8 and each new password has at least one each of the four character classes. The four character classes are lower_case, upper_case, digit and special_characters (the last one being any non alpha-numeric character)

Creating the first user

Note: if you have already created at least one more non root user this step is not required. We are primarily creating the new user so that we shall eventually allow sudo and remote ssh login privileges to the user and disable remote ssh privileges for the root user.

Setup the first new user. One of the reasons you should create a new user is so that it will afford you the ability to allow him to perform root actions through sudo, and thus subsequently allow you to disable root access over ssh. By default when one creates a new user, another group gets created with the same name as well. In this case we shall create a new group “dev” and then create a new user associated with that group “someuser“. Use the groupname and the username as you would like to setup when executing the commands below. In the commands below we create a new home directory for the user, associate the /bin/bash shell with his account instead of the default /bin/sh, (I just prefer bash to the plain sh) and finally set the password for him.

$ groupadd dev
$ mkdir /home/someuser
$ useradd -d /home/someuser -s /bin/bash -g dev someuser
$ chown someuser.dev /home/someuser
$ passwd someuser

We shall also create the .ssh directory for the user which we shall be using later

$ mkdir /home/someuser/.ssh
$ chmod 700 /home/someuser/.ssh
$ touch /home/someuser/.ssh/authorized_keys
$ chmod 600 /home/someuser/.ssh/authorized_keys
$ chown -R someuser.dev /home/someuser

Now we shall create the keypair for the user to log in to the host remotely. Note that if you are going to do this for multiple users, then you might want to have each user run the next step locally and then copy over his public key onto the server before continuing to the ssh tightening operations described later.

The user should do the following on his local workstation from which he is most frequently likely to connect to the server (not the server that we are hardening).

Note: the part after -C in ssh-keygen is just a comment to identify the keys – enter something to identify the user and his machine.
Also make sure not to keep the passphrase blank though ssh-keygen will allow a blank passphrase. The reason is that if the user’s local machine is compromised the attacker can then get an easy access to the server being hardened.

change someuser and some.host.com below based on the user id and vps name correspondingly

$ mkdir ~/.ssh
$ ssh-keygen -t dsa -b 1024 -C “some user on his desktop”
$ scp ~/.ssh/id_dsa.pub someuser@some.host.com:/home/someuser/.ssh/someuser.pub

Now the user should himself ssh to the remote server and on the remote server move his public key into the authorized_keys file. So execute the following command after being connected to the VPS

$ cd .ssh
$ cat someuser.pub >> authorized_keys
$ rm someuser.pub

At this stage the user can disconnect from the VPS and attempt to reconnect using ssh. If all works well, he should get connected to the vps in a manner where it does not prompt him for a password but instead he does get prompted for the passphrase to his private key (assuming he did set one).

This stage of updating the authorized_key file can also be performed by an administrative user / root once we later reconfigure ssh to only allow public key based logins.

Enabling the user to perform sudo operations

We shall enable any group who belongs to the group ‘admin‘ to be able to conduct root operations through using sudo.

First create a group ‘admin‘. subsequently associate the user with that group as well. Note : For best security ensure you allow associate only a very small number of users with the ‘admin’ group since that will effectively allow them control over the whole machine (assuming you setup the privileges as I subsequently describe below).

$ groupadd admin
$ adduser someuser admin

Now we shall enable any user who belongs to the admin group to perform root actions by using sudo. To edit the sudo policy file do the following

$ sudo visudo

At the end of the file which is now opened up – add the following line

%admin ALL=(ALL) ALL

Note this grants all superuser privileges to the users who belongs to admin group when conducting operations using sudo. You can use the sudo policy configurations to set up far more fine grained set of privileges, but thats beyond the scope of this document.

To test whether the configuration worked successfully, you can login as someuser and execute the following command.

$ sudo cat /etc/shadow

Tightening up ssh

To create the group and associate the users with them perform the following command (use the appropriate username instead of someuser for each user who you would like to allow SSH access).

sudo addgroup sshlogin
sudo adduser someuser sshlogin

Now that we have at least one user id which can conduct root operations using sudo, its time to disable root login from ssh. Go open the file /etc/ssh/sshd_config. Make the following changes

This one disables root login for ssh.
PermitRootLogin yes to PermitRootLogin no

Adding the following line allows only users of a particular group to login. Note: if your user count is small, you could use AllowUsers instead. We shall be separately creating the group and associating the users with the group.
AllowGroups sshlogin

This change reduces the login grace time (time before a user needs to login after making the ssh login request). This is to allow better protection to sshd against DOS or brute force attacks.
LoginGraceTime 20 to LoginGraceTime 20

This changes the port number that sshd listens on from the default 22. Changing the default port takes away the ability of an attacker to easily guess the port on which to attempt to penetrate the system via ssh.
Port 22 to Port 9999 (or any other suitable number)

This change disables X11Forwarding over the SSH connection. It will result in you not being able to run X11 GUI applications remotely. If you need that flexibility, do not make this change
X11Forwarding yes to X11Forwarding no

The next change disables password authentication (thus allowing only users with their public key being stored in the corresponding authorized_keys folder to connect using ssh). Thus passwords, which need to be transferred to the server in clear text no longer are a valid authentication mechanism. The only available choice is public key based authentication.
PasswordAuthentication yes to PasswordAuthentication No

The following change of uncommenting the banner allows a welcome message (not really a welcome one ) to be displayed to SSH logins. It does not improve any aspect of the physical security but is more from a legal notice perspective.
#Banner /etc/issue.net to Banner /etc/issue.net

Now go update the contents of the file /etc/issue.net to something similar to following

***************************************************************************
NOTICE TO USERS

This computer system is the private property of its owner, whether
individual, corporate or government. It is for authorized use only.
Users (authorized or unauthorized) have no explicit or implicit
expectation of privacy.

Any or all uses of this system and all files on this system may be
intercepted, monitored, recorded, copied, audited, inspected, and
disclosed to your employer, to authorized site, government, and law
enforcement personnel, as well as authorized officials of government
agencies, both domestic and foreign.

By using this system, the user consents to such interception, monitoring,
recording, copying, auditing, inspection, and disclosure at the
discretion of such personnel or officials. Unauthorized or improper use
of this system may result in civil and criminal penalties and
administrative or disciplinary action, as appropriate. By continuing to
use this system you indicate your awareness of and consent to these terms
and conditions of use. LOG OFF IMMEDIATELY if you do not agree to the
conditions stated in this warning.
****************************************************************************

Restart the ssh daemon and you will no longer be able to connect as root over ssh directly.

$ sudo /etc/init.d/ssh restart

Disabling su

As per the policy being followed, only users belonging to the ‘admin‘ group will have any special privileges. For all other users, we shall attempt to lock down the system to the extent feasible. As an early step we shall disable the ‘su‘ (switch user) command for all but those users belonging to the admin group. Note : since this disables sudo for a brief duration (between the commands), first switch over as a root before executing these commands

$ sudo su -
$ chown root.admin /bin/su /usr/bin/sudo
$ chmod 04750 /bin/su
$ chmod 04550 /usr/bin/sudo

Install rootkit detection

Install chkrootkit

$ sudo apt-get install chkrootkit

Now create this file /etc/cron.daily/chkrootkit.sh and enter the following contents (replace sever.domain.com with the servername and emailid where you would like the report to be sent to).
Since the mail is rather verbose, and may lead to useful alerts getting ignored, I’ve removed the lines with the common messages. You may change or not use the grep commands as per your preference.

#!/bin/bash
/usr/sbin/chkrootkit | \
grep -v “not found” | \
grep -v “nothing found” | \
grep -v “not infected” | \
mail -s “Daily chkrootkit from server.domain.com” mailid@destination.com

Allow execute permissions on the file, and test once by running .. you should get a mail with the report.

$ sudo chmod 755 /etc/cron.daily/chkrootkit.sh
$ sudo /etc/cron.daily/chkrootkit.sh

Installing Bastille

Bastille is a comprehensive package for security configuration.

Bastille Bug? and workaround

You may need to perform this workaround if your /etc/debian_version contains 4.0 and the current ubuntu bastille package has not resolved the issue. Ideally one should just need to install bastille from the ubuntu repositories

On Ubuntu 9.0.4 (jaunty) as of the day this document was written, I received the following error
~& ERROR: ‘DB5.0′ is not a supported operating system.

I’ve had a difficulty to install Bastille on Ubuntu 9.0.4. This is due to the fact that Bastille does not support the debian version number in /etc/debian_version. As per Bastille bug reports this got fixed in 3.0.8, but did not work for me in Bastille 3.0.9. Hence the following is a work around to install bastille.

This workaround involves downloading lynx to access internet, downloading the debian package directly, and then installing the debian package. When you reach the web page it will show you a list of mirrors. Download the .deb package from one of the mirrors

$ sudo apt-get install lynx libcurses-perl
$ lynx http://packages.debian.org/squeeze/all/bastille/download
# Download the package using lynx
$ sudo dpkg –install bastille_3.0.9-12_all.deb

Install psad which we shall need later and perl-gtk which we shall need to configure bastille with the following command

$ sudo apt-get install psad

You will notice an error message :

To resolve the same execute the following :

$ echo -e ‘kern.info\t|/var/lib/psad/psadfifo’ | sudo tee -a /etc/syslog.conf
$ sudo /etc/init.d/sysklogd restart

Now run bastille in an interactive mode

$ sudo bastille -c

The resultant dialog with bastille is too long and I am just showing the final configuration file which is produced as a result of bastille configuration. I’ve used the port 22 as a proxy for the SSH port which I assume you’ve changed to some other value – use the other value instead of 22

# Q: Would you like to enforce password aging? [Y]
AccountSecurity.passwdage=”Y”
# Q: Should Bastille disable clear-text r-protocols that use IP-based authentication? [Y]
AccountSecurity.protectrhost=”Y”
# Q: Should we disallow root login on tty’s 1-6? [N]
AccountSecurity.rootttylogins=”N”
# Q: Would you like to deactivate the Apache web server? [Y]
Apache.apacheoff=”N”
# Q: Would you like to disable CTRL-ALT-DELETE rebooting? [N]
BootSecurity.secureinittab=”N”
# Q: Should we restrict console access to a small group of user accounts? [N]
ConfigureMiscPAM.consolelogin=”Y”
# Q: Which accounts should be able to login at console? [root]
ConfigureMiscPAM.consolelogin_accounts=”root”
# Q: Would you like to put limits on system resource usage? [N]
ConfigureMiscPAM.limitsconf=”N”
# Q: Would you like to set more restrictive permissions on the administration utilities? [N]
FilePermissions.generalperms_1_1=”Y”
# Q: Would you like to disable SUID status for mount/umount?
FilePermissions.suidmount=”Y”
# Q: Would you like to disable SUID status for ping? [Y]
FilePermissions.suidping=”Y”
# Q: Do you need the advanced networking options?
Firewall.ip_advnetwork=”N”
# Q: Interfaces for DHCP queries: [ ]
Firewall.ip_b_dhcpiface=” ”
# Q: DNS Servers: [0.0.0.0/0]
Firewall.ip_b_dns=”0.0.0.0/0″
# Q: ICMP allowed types: [destination-unreachable echo-reply time-exceeded]
Firewall.ip_b_icmpallowed=”destination-unreachable echo-reply time-exceeded”
# Q: ICMP services to audit: [ ]
Firewall.ip_b_icmpaudit=” ”
# Q: ICMP types to disallow outbound: [destination-unreachable time-exceeded]
Firewall.ip_b_icmpout=”destination-unreachable time-exceeded”
# Q: NTP servers to query: [ ]
Firewall.ip_b_ntpsrv=” ”
# Q: Force passive mode? [N]
Firewall.ip_b_passiveftp=”Y”
# Q: Public interfaces: [eth+ ppp+ slip+]
Firewall.ip_b_publiciface=”eth+”
# Q: TCP service names or port numbers to allow on public interfaces: [ ]
Firewall.ip_b_publictcp=”22 80 443″
# Q: UDP service names or port numbers to allow on public interfaces: [ ]
Firewall.ip_b_publicudp=” ”
# Q: Reject method: [DENY]
Firewall.ip_b_rejectmethod=”DENY”
# Q: Enable source address verification? [Y]
Firewall.ip_b_srcaddr=”Y”
# Q: TCP services to audit: [telnet ftp imap pop3 finger sunrpc exec login linuxconf ssh]
Firewall.ip_b_tcpaudit=”telnet ftp imap pop3 finger sunrpc exec login linuxconf ssh”
# Q: TCP services to block: [2049 2065:2090 6000:6020 7100]
Firewall.ip_b_tcpblock=”2049 2065:2090 6000:6020 7100″
# Q: UDP services to audit: [31337]
Firewall.ip_b_udpaudit=”31337″
# Q: UDP services to block: [2049 6770]
Firewall.ip_b_udpblock=”2049 6770″
# Q: Should Bastille run the firewall and enable it at boot time? [N]
Firewall.ip_enable_firewall=”N”
# Q: Would you like to run the packet filtering script? [N]
Firewall.ip_intro=”Y”
# Q: Would you like to set up process accounting? [N]
Logging.pacct=”N”
# Q: Would you like to deactivate NFS and Samba? [Y]
MiscellaneousDaemons.remotefs=”Y”
# Q: Alert on all new packets?
PSAD.psad_alert_all=”Y”
# Q: psad check interval: [15]
PSAD.psad_check_interval=”15″
# Q: Would you like to setup psad?
PSAD.psad_config=”Y”
# Q: Danger Levels: [5 50 1000 5000 10000]
PSAD.psad_danger_levels=”5 50 1000 5000 10000″
# Q: Email addresses: [root@localhost]
PSAD.psad_email_alert_addresses=”mailid@destination.com”
# Q: Email alert danger level: [1]
PSAD.psad_email_alert_danger_level=”1″
# Q: Should Bastille enable psad at boot time? [N]
PSAD.psad_enable_at_boot=”N”
# Q: Enable automatic blocking of scanning IPs?
PSAD.psad_enable_auto_ids=”N”
# Q: Enable scan persistence?
PSAD.psad_enable_persistence=”N”
# Q: Port range scan threshold: [1]
PSAD.psad_port_range_scan_threshold=”1″
# Q: Scan timeout: [3600]
PSAD.psad_scan_timeout=”3600″
# Q: Show all scan signatures?
PSAD.psad_show_all_signatures=”N”
# Q: Would you like to disable printing? [N]
Printing.printing=”Y”
# Q: Would you like to disable printing? [N]
Printing.printing_cups=”N”
# Q: Would you like to display “Authorized Use” messages at log-in time? [Y]
SecureInetd.banners=”Y”
# Q: Should Bastille ensure inetd’s FTP service does not run on this system? [y]
SecureInetd.deactivate_ftp=”Y”
# Q: Should Bastille ensure the telnet service does not run on this system? [y]
SecureInetd.deactivate_telnet=”Y”
# Q: Who is responsible for granting authorization to use this machine?
SecureInetd.owner=”Company Name”
# Q: Would you like to set a default-deny on TCP Wrappers and xinetd? [N]
SecureInetd.tcpd_default_deny=”N”
# Q: Do you want to stop sendmail from running in daemon mode? [Y]
Sendmail.sendmaildaemon=”Y”
# Q: Would you like to install TMPDIR/TMP scripts? [N]
TMPDIR.tmpdir=”N”

Installing the Lamp Stack

Let us use the not so well known tasksel command to download all the packages necessary for a lamp stack. Note that tasksel is simply a convenience tool around apt-get and allows one to install a whole bunch of packages based on a class of necessity – in this case a LAMP (Linux,Apache,MySQL,PHP) stack.

$ sudo tasksel install lamp-server

During the installation you will be required to create the password for the mysql root user. In interest of tight security do make sure to create a really strong password.

This actually sets up the basic LAMP stack and that should get you operational.

Securing Apache

Change the user and group apache runs as

You may be aware that installation of apache resulted in a new userid and group being created (both called www-data). I just find it a little bit more comforting to change the well known usernames to something that are less predictable. In the steps below I change www-data to apache but you may want to use something a little less predictable than apache.

$ sudo usermod -l apache www-data
$ sudo groupmod -n apache www-data

We shall need to change the same in the apache configuration file. Edit the file /etc/apache2/envvars and change the two lines as follows

export APACHE_RUN_USER=apache
export APACHE_RUN_GROUP=apache

Install and configure mod_security

Earlier when we configured, bastille and through it the iptables firewall, we allowed incoming public traffic on the SSH, HTTP and HTTPS ports. We’ve already covered the tightening of SSH security. However we still do not have a good way to control HTTP and HTTPS traffic. The solution to that is installing the apache module mod_security. It allows us abilities to inspect and if necessary reject or redirect HTTP/S traffic. The full description of mod_security is beyond the scope of this document, but a good example of how it can be used further to implement application level security as well can be had from the document Securing web services with mod_security. Another useful overview page is Secure your Apache2 with mod-security.

We shall start off with installing mod_security.

$ sudo apt-get install libapache-mod-security

You should notice a new file in /etc/apache2/conf.d called security. These are the global mod_security settings.

Change the following variables in the file to make server identification difficult. Note that while these settings do not enhance the security directly, they do make it harder for an intruder to easily identify the specific server and configuration, thus making it harder for him to attempt configuration specific exploits.

ServerTokens Prod
ServerSignature Off
TraceEnable Off

Also uncomment the following four lines. Note that this may require you to configure other web applications which are not configured appropriately.

AllowOverride None
Order Deny,Allow
Deny from all

While not related to mod_security, to minimise server information leakage, you may consider changing these two variables to the shown values in /etc/php5/apache2/php.ini

expose_php = Off
display_errors = Off

As mentioned above, there is a whole range of additional controls you can enforce using mod_security. Depending upon your specific requirements feel free to leverage mod_security much more.

Disable apache modules you do not need

In general it is better to turn off unrequired modules unless you really need them. The default configuration installs a number of modules which may not be required. The modules that have been enabled are in the /etc/apache2/mods-enabled directory which are all symbolic links to the modules that have been installed in the /etc/apache2/mods-available directory. Note that turning off some modules may affect some web applications you install, in which case you may choose to subsequently turn them on only if necessary. New modules can be enabled with sudo a2enmod module_name command and enabled modules can be disabled by sudo a2dismod module_name command.

Depending upon my requirements, I disabled the following modules

$ sudo a2dismod autoindex # used to build directory indexes like the ls command
$ sudo a2dismod cgi # used to run cgi scripts
$ sudo a2dismod env # used to set environment variables for CGI & SSI scripts

Chrooting apache

This is an advanced topic that I shall skip. Note that if you do setup chrooting, there’s a lot of additional considerations which will need to be observed, considerations especially around shared libraries and configuration files which will make the overall process of configuration extremely difficult.

Setting up snort and acidbase

Author’s Note: Careful – Very likely, there were a few things that I might have missed or incorrectly noted especially in the context of acidbase and following these instructions may not get acidbase running properly. You may have to do some steps outside these notes to get it all working properly. Unfortunately I am unable to rerun the process on a fresh server so am unable to immediately note the possible lacunae in the notes

Now that you have a mysql server going, create a new database for snort. In the following section I use snortdb as the database, snort as the user and snortpwd as the password for the database. However I do encourage you to replace the same with some non easily guessable names and passwords.

Here’s the commands to create a new snort database and user. In the following example ‘$‘ is the unix prompt while ‘>‘ is the mysql prompt.

$ mysql -u root -p # You will be prompted for the password for the mysql root account. Enter it
> create database snortdb;
> grant select, insert, update, delete, create, drop, index, alter, create temporary tables, lock tables on snortdb.* to ’snort’@'localhost’ identified by ’snortpwd’;
> flush privileges;
> quit;

Now we shall install snort.

$ sudo apt-get install snort-mysql

The installation process will require you to enter the network that you wish to protect. Since in this case we are only protecting a single machine, enter the IP_Address/32 eg. 12.34.56.78/32

The process will prompt you whether you wish to configure the database. Select “Yes” .. the script will proceed but terminate abnormally with the error “subprocess post-installation script returned error exit status 6″

Now execute the following commands to setup the tables in the database. At the end you shall open the snort configuration file to enter the database configuration parameters

$ cd /usr/share/doc/snort-mysql
# Enter the database password next when prompted for one
$ sudo zcat create_mysql.gz | mysql -u snort -p snortdb
$ sudo vi /etc/snort/snort.conf

Uncomment the line that starts with output database: log, mysql and add the configuration information at the end as shown

output database: log, mysql, user=snort password=snortpwd dbname=snortdb host=localhost

Now remove the file as shown below to indicate that the database configuration has been done and start snort. The subsequent command confirms that snort daemon is indeed up and running

$ sudo rm -rf /etc/snort/db-pending-config
$ sudo /etc/init.d/snort start
$ sudo /etc/init.d/snort status

Now, lets install acid which is a web based application to be able to view snort alerts

$ sudo apt-get install acidbase

Select No when it prompts you for configuring the database. By default it uses the database and userid ‘snort‘ and I prefer to keep these different from a predictable default, in which case acidbase database initialisation will fail.

Edit the file /etc/dbconfig-common/acidbase.conf.

Set the following values

dbc_dbuser=’snort’
dbc_dbpass=’snortpwd’
dbc_dbname=’snortdb’

The default configuration allows acid data to be accessed only from the machine where it is involved. Since this shall be a VPS which we shall be accessing remotely, it is time to relax the constraint. If you have a good idea of the network IPs that you will be accessing the acid web application from, update the same in the allow from directive in /etc/acidbase/apache.conf, else relax it fully to allow from all.

Also protect the ‘/acidbase‘ url by http basic authentication. The resultant section in the file /etc/acidbase/apache.conf looks like :

Options +FollowSymLinks
AuthType Basic
AuthName “Go Away World!
AuthUserFile /etc/acidbase/mypassword
Require valid-user
AllowOverride None
order deny,allow
deny from all
allow from all

php_flag magic_quotes_gpc Off
php_flag track_vars On
php_value include_path .:/usr/share/php

You will need to create a user id / password pair for the app or point AuthUserFile above to another file where you already have the ones set up for your server.

To do so you shall need to run the following command :

$ htpasswd -c /etc/acidbase/mypassword my_user_id

If you need additional user ids to be installed use the command as shown above again with different user ids – but make sure not to use the -c switch since thats only used the first time to create the password file.

Now start the web application by restarting apache

$ sudo /etc/init.d/apache2 restart

Goto the web page http:vps_fully_qualified_domain_name/acidbase/base_db_setup.php, click on ‘Create Base AG//’ and you are on your way.

File integrity checking using AIDE

Install AIDE

$ sudo apt-get install aide

Update the aide configuration file etc/default/aide to update your email id where you would prefer the integrity check reports sent (Look for the variable MAILTO)
Now initialise the aide configuration

$ sudo aideinit
$ cd /var/lib/aide
$ mv aide.db.new aide.db

Now run the process (which will be running once daily)

$ /etc/cron.daily/aide

This process takes a long time (about 5 mins for me) and will at the end email you a report if any files changed compared to the ones in the default database.

Opera Unite : A model for server disintermediation on the internet

Service oriented REST architecture is an oxymoron

Dhananjay Nene — Wed, 30 Sep 2009 21:34:41 +0000

It is infrequent for me to react with a level of consternation rather than agreement or disagreement that I felt upon reading [SOA] Boris on Service, Web and REST by Jean-Jacques Dubray. Not because I disagreed strongly with the arguments presented. It is that, I disagree substantially with the assumptions on which these arguments are made. And yet, as I recollect my own thoughts a year ago – a few months post my journey into REST, I realised that there was a time that I did actually believe some of these assumptions. I also realised that it is likely that many others who are dealing with a transition from SOA to REST are also likely to be perhaps sharing similar assumptions. Without much ado let me quickly get to the central assertion of this blog post.

Service orientation is neither essential for, nor is it the intention of REST.
Not only is REST not service oriented, service orientation is irrelevant for REST

There. But why was it so important to state that ? Allow me to quote from the blog post I referred to.

I say not surprisingly because RESTafarians have no clear position on “service”, they just say REST is the right way to build a Service Oriented Architecture. Yet, REST has no concept of “service” anywhere, just resources and their shiny uniform interface, links and bookmarks. Indeed there are no services in REST. Just read the thesis.

and it further goes on to state

But I digress, let’s go back to “services”. Even Bill, in this REST-* proposal is talking about creating a RESTful interface to non RESTful services. That certainly begs the question, how can a service be non RESTful since REST is all about SOA and replaces in its entirety WS-*.

The essential issue here is the flawed assumption that REST attempts to be service oriented or it is all about SOA. Its not. And why so ? Since it is resource oriented. And whats the difference ? Read on, because that’s what this post attempts to address.

Service

Wikipedia describes a service as follows :

the term service refers to a set of related software functionality, together with the policies that should control their usage.

OASIS (organization) defines service as “a mechanism to enable access to one or more capabilities, where the access is provided using a prescribed interface and is exercised consistent with constraints and policies as specified by the service description.”

Now lets attempt to understand a service in a little more dumbed down fashion. Lets hark back to the good old construct of flow charts and process charts. In these charts one basically divided an overall set of functionality into discrete set of functionalities and chained them together through some sequences and decision points. As an example if we were to consider a simplified retail outlet system, it would consist of steps that would support (a) ordering items, (b) receiving and reviewing items, (c) selling items. In a SOA world, these could be mapped into a Ordering Service, Receipt Service and Sales Service (you could of course come up with better names and further decomposition). But each service is essentially one of the decomposed tasks of a larger workflow. If the interface to such service could be standardised and documented it would help it to be reusable across multiple contexts. And to the extent such services are reusable across multiple workflows, the advantage of Service Orientation become obvious. And finally if such a service interfaces are exposed over the web – it is a web service. At the end of the day, each service is a reusable, composable task (or tasklet) performer.

Resources

But REST does not attempt to be service oriented. Thats because it does not view the process as a sequence of tasks to be performed. It views it as a sequence of resources under modification. To put it differently, it views the process as a set of actors who exchange resources (or documents for better visualisation) and carry out activities based upon receipt of such resources. Though not as equally apt as a process chart, the analogy here would be a data flow diagram. And what might such resources be ? Well in the above scenario, there’s a Purchase Order, a Goods Receipt and an Invoice. Those are the essential abstractions that REST focuses on. These are Resources. Just like Services where there’s no one valid set of abstraction of services, one could work out a different set of resources rather than those I listed. But the bottom line is that the essential abstractions are resources *not* services.

How are they different ?

You could build a system either way – as services or resources. In terms of being able to successfully build, deploy and maintain a piece of software, both REST and SOA are likely to be equally successful at building the software. But the essential vocabulary through which they decompose their various parts (and therefore describe their interface elements) will be different. And how is that different ?

Let us imagine the ordering service we talked about above. One way to build a SOA ordering service is to establish a interaction procedure which combines an overall protocol and a series of steps (Service API). To reduce potential errors, there is a document upfront which describes in adequate detail how such an interaction should be conducted, what are the data elements to be exchanged at each stage, and what are the necessary sequencing requirements between various steps for such interactions to be concluded successfully (WSDL). The focus here is the tasks being done and the protocol for the task instructions. In case of REST the essential construct will be exchange of one Purchase Order. The purchase order would have sufficient in band instructions about the fact that it is a purchase order and the attributes it has (in-band metadata), and formal documentation if any would be restricted to the structure of the purchase order and its data than than to the sequencing, flow or any protocol level activities. (Thats why sometimes REST looks deceptively simple to be treated as just another CRUD).

More often than not when called upon to describe a service, the description will describe what the service does, and the service interface will mirror the steps required to perform the activities. Resources on the other hand will simply describe themselves and anyone who looks at a resource description will be none the wiser about what processing exactly happens behind the scene.

Thats why I believe even if both REST and SOA can be used to build software effectively, the essential focus on resources as the central abstraction makes REST much easier to use for the clients. But thats just my opinion – you may form your own.

You cheated! REST meets the service orientation definitions you listed above

Yeah, kind of (in theory). There is one way where REST over HTTP is service oriented. Imagine a document service which could store, update, fetch or delete documents. Now replace document with resource in the earlier statement. Thats your typical HTTP service that REST works off of to implement a resource management service – but thats just a single service which is standardised for REST over HTTP. And all REST implementations will be service oriented to that extent. However the sheer simplicity and ubiquity of this service makes the associated service orientation of REST rather uninteresting and thus largely ignorable.

So next time one wants to debate the merits for REST and/or SOA – feel free to add to the tons of stuff thats already written. But don’t measure REST based on service orientation. Service orientation is largely irrelevant for REST. And that per se does not make one better than other – it just makes them different.

Note: There were many other points in the blog post I referred to that I would want to offer different opinions on. But in this case, I believed it was important to keep this post focused on an essential thought that I really wanted to emphasize.

The best amount of polyglotism is that you can manage successfully

Dhananjay Nene — Mon, 28 Sep 2009 13:05:20 +0000

Polyglotism was one topic I got into some discussions on twitter earlier this week. So this is to just pen down some of my thoughts on the same. Polyglotism is increasingly being talked about, ever more so since Java started moving from the realm of being a language into a runtime environment to host a number of languages. Yet polyglotism is not new. Most programmers have been mixing their favourite languages with SQL for ages. The advent of the web introduced HTML and javascript as additional languages that web programmers needed to know to be able to successfully write web programs. At the same time other tools started appearing which started to reduce your requirement to know these languages. Object Relationship Mappers could help you get out of writing SQLs (at least so they claimed ) in many cases. And one of the features of GWT is to ensure that you work primarily on only one language.

But the goal of this post is to neither encourage nor discourage the use of polyglotism. It is to suggest that at the end of the day even if polyglotism really stimulates the technogeek sensors in us, it needs to be evaluated not in technology terms, but in business and economic terms. A point that is also partially supported by The plague of polyglotism. Unless we use tools like GWT and hibernate to keep us monoglots, (which is likely to be a very small proportion), polyglotism is the default state that we shall operate in. So its not worthwhile to fight that trend per se. The important aspect is to ensure that we understand the costs of supporting each new language in the mix. And the reason to do so is to evaulate whether the benefits of adding the language outweigh the costs in the medium to long term. So here are some of the costs (in no specific order)

Syntax and libraries : This is probably one of the lowest costs even though it is the most apparent. For every language we add to the mix, generally for a long running project, at least three developers (if you have reasonable risk management objectives) need to learn the new language syntax and the libraries they work with. Along with this is the cost of books, training courses and materials etc. (yes for many enterprises, many developers but those purely self motivated will need to be formally provided all of these).
Idioms and Philosophies : Knowing the language is not adequate. You need to understand its philosophy, its design intent and its typical idioms. Being able to use a language to write code is not the same as being able to use it idiomatically. And while that may not constrain you in over small code bases in the short term, over a period of time for reasons such as performance, consistency and ability to understand other libraries that you may reuse, you will be forced to deal with these aspects of the language as well. The book Dreaming in Code talks about how Java programmers who started writing python could not write idiomatic python code, and thus a fair degree of code had to be redesigned or rewritten. And that is by no means the only such instance. I suspect similar trends are likely to be found if java programmers move to writing ruby or scala code. And its a trend you might have likely observed when people comfortable with writing relatively simpler languages such as Visual Basic or PL/SQL or even more complex languages like C++ moved to writing Java. Knowing the language is not the same as knowing it idiomatically – and that takes time. Which means – do not take on high risk activities when you absorb a new language, it will take a few iterations for your team to absorb the new language – far far more time than it will take to read a book or attend a training course.
Skills to manage multiple idioms in one head : Unless you are going to have different programmers for different languages, the programmers working on multiple languages at a time will not only have to deal with the multiple idioms, but will need to concurrently apply them. In my experience, this is not a trait we are born with, and it takes some time and experience with it. It will take some time for them to learn that the best practices are not identical across various languages. It will take even more time to figure out why they are different. So there is a good likelihood, you will not be in a situation to have all your developers work on all the languages simultaneously – especially some of the Junior ones.
Skill portfolio management will become more difficult : For typical enterprise or ISV teams, the managers need to ensure adequate skill availability across all teams. As the number of languages in use increase, this management process will become harder. Which means there is an automatic economic disincentive which rises is in magnitude, each time you add yet another language to the mix.

So is polyglotism good or bad ? I submit such a value attribute cannot be assigned to polyglotism per se. It needs to be really evaluated in your context, preferably beyond a project window into the medium or a long term. What is indeed obvious is that each new language will bring in some very strong capabilities and aspects where it is superior to the other languages in the mix. Equally obvious is that each new language will bring in associated costs. Just like there is no perfect language, there is no perfect number of languages that make sense in every context. Just like you do your homework in terms of language selection, make sure you do the same for your language portfolio not just in technology but in management and economic terms as well.

No related posts.

Mind your language: Pragmatic programmers should not be re-termed duct tape programmers

Dhananjay Nene — Thu, 24 Sep 2009 05:17:43 +0000

The Duct Tape Programmer by Joel Spolsky makes for an interesting reading. However it suffers from one issue. A big issue. It attempts to take terms with a particular well associated meaning with it, assign it to something else which already has a better available term and in the process creates one big ball of confusion.

In my understanding which hasn’t changed for many years – a duct tape programmer is a programmer who always takes the shortest route from point A to point B without giving regards to longetivity or maintainability of the solution. When he applies a duct tape – he is ticking off a job as “done”. He has not accounted for how long will the solution last. Is it even the right solution or is it just something that on the face of things seems right. He is only pragmatic enough to make sure he can get home early or be in his bosses good books. Thats where his pragmatism ends. Thats why the term has a negative meaning. Thats why it is not meant to be a compliment.

Now what happens when you associate it in the way Joel just did in his blog post ? Suddenly it no longer seems to be a negative term. It gives rise to confusion. If someone now calls me a duct tape programmer – I’ll be tempted to take it as a compliment. On the other hand – when someone writes a shitty crappy piece of code – I now have to figure out a new term – since I can no longer term it duct tape programming. I mean if I do and he responds a link to the post – what chance would I stand compared to someone as esteemed as Joel.

I understand as bloggers we want to make a point. I also understand that there are ways to make the point in a manner that really really stands out. But at times there are risks and negative consequences when we go down that path. These include confusing the juniors, the post getting misunderstood, misconstrued or even misquoted. These also include possible positive spin on essentially negative characteristics. Lets stick to calling pragmatic programmers as pragmatic programmers and leave duct tape programming to the realm of the essentially short sighted, fragile and often brittle programming that it really stands for.

Jerk programmers can’t be managed tactically

Why should I switch to Scala ?

Dhananjay Nene — Mon, 17 Aug 2009 15:33:44 +0000

This post is a role-play and does not reflect my individual opinion about scala accurately. I am convinced about the capabilities and features of Scala along with the fact that it deserves the mantle of a long term replacement for Java. However language adoption goes beyond technical capabilities, and this post is a speculation on what a typical manager might be dealing with when attempting to decide whether to switch to Scala.

So I have been reading a lot about Scala lately and even opinions about how it will be a long term replacement for Java. I’ve also read some interesting writeups about Scala adoption such as On Scala’s Future and A Tipping Point for Scala. While I used to code a lot, my responsibilities today require me to interact with and address a lot of issues including those faced by our customers, our development teams and also engage with my peers and superiors on many other difficulties bedeviling our organisation. This gives me little time to try out Scala. I know I should be doing that, but sincerely I do not have the time. So I rely on the feedback of my team, the trade journals and other influential architects within and outside my organisation.

I have heard about many developers switching from Java to Python / Ruby. However I have heard of relatively only a smaller number of large Java shops which have done the shift – most of the switch stories I’ve heard reflect a smaller sized teams. I can feel the excitement Scala has generated amongst the development teams – the brevity, the functional programming model introduction, the exciting stuff being done concurrently et al. I have no doubt that, given so much excitement it must really be a good language.

To introduce my organisation – it is one of those shops which service many projects concurrently. Given the tremendous business and growth, I must confess we do not always have the luxury of being able to hire the most top notch talent. We do have a lot of projects we use Java for, and thats a language our customers are comfortable with. I’ve had some of the senior people check out Scala to gain some feedback into the language. But at this stage I must say I am inclined to evaluate the shift but not convinced enough to do so. I am sure that I could if convinced drive the change to Scala incrementally. However my fear stems from the fact that if things don’t turn out well, despite all the great advice I’ve received – its going to be my rear end on the line. So here’s some of my concerns regarding evaluating the shift to Scala and there are many of them, so some of you might be able to help me through this thought process.

Functional Programming : I’m sure in many ways it rocks. But my guys tell me they are not sure how to use it in the typical bread and butter applications which read from database, do some processing and write back to the database. Does Functional Programming help me in this context ? Will my team scale into being able to write functions with no side effects assuming thats a desirable goal ? What if they tie themselves up in knots and my release to the customer is risked ? I can’t afford that. Is functional programming even desirable in such contexts ? So I am not sure if in these contexts I should just ditch functional programming and work with just normal imperative programming capabilities of Scala. I am so confused, and afraid.
Different Syntax : While Scala runs on the JRE, its syntax is very different from Java. From what I could gather, it is much easier for a Java programmer to read (make sense of) simple Python code than to read Scala code. Is it true ? So even if I do get compatibility in terms of the runtime environment, would I be picking up a language that is syntactically so different a language that it would involve a substantial relearning curve ? I remember when we had to learn Java and Javascript. For better or for worse these were indeed relatively minor modifications of the C/C++ syntax, compared to what I sense as the syntactic shift between Java and Scala. Am I wrong ? If so, could you help point me to resources which help me understand that Scala code is not much different than Java ?
Sample code : Guys, I need your help. I need to see some good sample code. Some code which reflects how a typical application is architected, designed and programmed in Scala. And I don’t need it for a complex multi threaded actor based processing – I just need to see simple J2EE server based departmental applications maybe a simple recruitment tracking or library maintenance application. If I find a good one, I’ll just take it and give it to my team and say – there, thats how we’re largely going to build it, and even if we make a few changes along the way we at least have a reasonable template that we can build from.
Dumbed down environment : I remember my great adventures with C and vi and make. But my team today is very different. They want great IDEs. They must have syntax highlighting, autocompletion and nice refactoring capabilities. If I ask them to move, some of them might be excited about the change and be willing to overcome these short term hurdles. But there are some of them who will not be keen to do so and may be disinclined to support such a shift. And at the end of the day my ability to conduct this shift is a function of my ability to carry a large proportion of them along with me. Even when I considered a shift from svn to git, the IDE support was a big issue even though quite obviously git capabilities were really exciting. I couldn’t push along that change, and in this case we are talking of changing the language.
Is this a good time to shift to Scala ? I remember the early adopters of Java from 1996 thru 2001. While they gained a lot of experience, JRE and J2EE really matured only post JRE 1.3. Scala seems to be coming out with so many enhancements so fast, I am not sure if it has stabilised. I am told there is a 2.8 coming out in a few months. So if I train my team and Scala continues to change rapidly will I have to keep on retraining my team regularly ? And what about the customers I take to production. Will the frequent upgrades mean I end up supporting multiple customers on multiple versions of Scala ? Maybe Scala is stable but it would be helpful for someone important enough to make a clear statement that there are no new major shifts anticipated anytime soon and that these version shifts are likely to be no faster than the JRE version upgrades (which were fast enough).
Support from peers and superiors : I remember the day I decided to shift to Java. What made the move easy for me was the sheer fact that Java was a big paradigm leap away from the then dominant C++. Not only was it cross platform with binary compatibility thrown in for good measure, Sun ensured that it made all the right noises to appeal to the enterprise architects and all the business managers. I see the senior developers in my team clamouring for the shift to Scala, but my peer managers and my superiors don’t display even the fraction of the enthusiasm they displayed during the Java shift. The implication for me is that the risk cover I get when I order the shift is far lesser than what I had when I made the move to Java. Which means if things don’t quite work out well, I’m really going to be screwed.
Business friendliness : I understand all the nice talk about the technical excellence of Scala. But I really need to translate all these great language features into a projected ROI that I can use to convince others about. So I would like to see actual case studies of applications that were moved to Scala and what impact it had on the time and cost so that I can use it to compute my ROI. And what scares me is that learning curve may risk the initial applications long enough to push my breakeven point of shifting to Scala well beyond a 12 month and perhaps even a 24 month period. I fear things might not be as difficult but in absence of known studies, I am likely to lean towards projecting a worst case scenario rather than an optimistic one.

So folks, I am asking for your help. And while a lot of you may think that people like us who balk at the thought of limited IDE support are wimps, please remember that 80% of us don’t fit into the top 20%. And if you would like Scala to be popular, you need us as much as we need you. And if you are not too sure, please remember Lisp and Smalltalk are great languages as well.

A brush with Functional Programming and Scala

CRUD is not only good for, but is the only consistent way to build REST over HTTP

Dhananjay Nene — Fri, 14 Aug 2009 08:35:45 +0000

This is to comment on a perception forming that REST encourages exposing basic data elements through CRUD and that it encourages development of dumb applications (applications with shallow business logic).

Apart from some tweets I saw on the topic and some twitter conversations, the blog posts which perhaps set off the thought were

CRUD is bad for REST by Arnon Rotem-Gal-Oz
A summarisation of the same on InfoQ : Is CRUD bad for REST?
Can We REST For A Minute? 6 Lessons From The CRUD vs. Hypermedia Debates by Dion Hinchcliffe

The underlying fear and rationale for these posts makes a lot of sense – the fear of creating real dumb passive and shallow applications. I submit, that the problem however is not CRUD – it is resource identification and scoping, and CRUD is not only good for but is the right way to build intelligent, active and deep applications.

CRUD supports Uniform Interface : The primary reason why CRUD gets used is because it supports a uniform interface. At the end of the day, a consistent Create/Read/Update/Delete or POST/GET/PUT/DELETE interface makes things easy. It makes things easy for the development team because of the consistency it introduces in their applications. It makes things easy for the clients who have a simple and consistent interface to deal with. At the interface level CRUD breeds consistency, and at the risk of broad generalisation, consistency is good.

So why do we end up creating shallow applications at times with REST ? CRUD in general works with simple forms built on simple tables. Quite often this style of programming gets elevated into simple forms over simple domain objects. Standardised CRUD helps a lot at the lower end of application development and most database driven application developers are likely to have at some stage in their early development life attempted to build a small CRUD library or framework to help themselves substantially. The reason why we are likely to be ending up creating shallow applications is not because we apply CRUD, but because we continue to apply CRUD on tables or simple domain objects. And therein lie the distinctions

REST is not about CRUD on tables – its about CRUD on resources
CRUD is the interface – not the implementation

I attempt to bring up the difference in the example that I detail below.

Simple Account Transfer Example

Lets say we want to build the software to transfer amount X from account A into account B. Lets further specify that a transfer is not effected immediately and requires one more explicit approval. Lets also specify that while a transfer is waiting to be approved, it could be amended. Thats the simple scenario that we shall deal with.

In order to implement this, we shall define a datastructure / table / object for Account which shall contain a field called balance. Further there shall also be Transfer table / object which shall contain the fields sourceAccount, destinationAccount, amount and status. The possible status values shall be Initiated and Completed.

In a simple service oriented application we shall perhaps have a transfer service. Ignoring error handling, SOA wrapping etc., the service interface will probably boil down to the following equivalent Java interface.

public interface TransferService
{
    public Long transfer (Long sourceAccountId,
                                Long destinationAccountId,
                                BigDecimal amount);
    public Transfer get(Long transferId);
    public void amend(Long transferId,
                             Long sourceAccountId,
                             Long destinationAccountId,
                             BigDecimal amount);
    public approve(Long transferId)
}

Lets think of these might get modeled in a REST environment. The important thing to remember is – don’t think about services or functions or methods – think about what are the resources you choose to expose using a simple CRUD interface.

# The following creates a new transfer. The returned data shall include
# the URI of the new transfer, and the URI to approve it
POST /transfer
# The following retrieves the status of a current transfer. If it has not
# been approved the returned data shall include the URI to approve it.
GET /transfer/${id}
# The following modifies the transfer. The returned data shall also
# include the URI to approve it
PUT /transfer/${id}
# The following approves and further processes the transfer. It shall
# return the URI for the transfer
POST /transfer/${id}/approve

While most of this seems all right – what sticks out like a sore thumb to me is the approve URI. Its just so SOAish / RPCish. Plus at least the way this particular interface has been implemented, there is no way to access the approval specific information, without actually accessing the transfer. Hence I suggest that we define a new resource TransferApproval to account for the same.

# The following creates a new approval. If successfully executed
# the transfer is complete and no future amendments or approvals
# are allowed. the returned data shall include the TransferApproval
# URI and the transfer URI
POST /transfer/${id}/approval
# The following gets an existing approval
GET /transfer/${id}/approval/${approvalId}

Please note that the “${approvalId}” at the approval URI simply wasn’t required – since there exists a 1-1 relationship with the transfer. I just included it for easier understanding. If I had to implement the functionality as is I would choose to skip it however if I knew I would very soon need to build in multi-stage approval (as in most banking systems), I would keep it so that each approval against a transfer can also be listed.

But the really interesting method above is the POST. This is a seemingly simple new (in RDBMS parlance) insert into TransferApproval table. But if you are building a REST service, you might be tempted to encourage your clients to not only create the new TransferApproval resource, but also go back and update the Transfer table to update a status to Approved. That would be a smell. Once the POST on the approval is processed, all side effects on other tables should be handled while servicing the POST request. In other words the POST request is not just an insert – its an insert with an associated trigger to conduct all the necessary downstream processing. And its essential one looks at request servicing in this manner so that CRUD can be used effectively. Servers should be designed this way, and clients should anticipate it and we should be on our way to build non-shallow applications.

So finally – CRUD is good. It makes things easy for the clients. Stick to CRUD. Just remember that it is CRUD on resources and not on tables, and the resources shall handle all the downstream changes necessary so that you don’t have to. And finally CRUD is the interface, not the implementation.

Note: Some might notice that this post is just a much more detailed elucidation of one of my earlier posts – REST is the DBMS of the internet.

REST is the DBMS of the Internet

The Microsoft Word injunction has nothing to do with XML

Dhananjay Nene — Wed, 12 Aug 2009 21:36:23 +0000

Obviously the development community is buzzing loudly about a injunction issued by a court against Microsoft disallowing it apparently from marketing Word. However what foxes me is that most articles give the impression that it is an issue with XML documents. As an example :

I also came across many other tweets bemoaning the verdict and expressing the opinion that it is a sad turn of events – it being related to storage of documents as XML. I did take a quick look at the said patent : Patent 5787449.

And therein I thought there’s seems to be a big misunderstanding (at least to my lay reading). The patent itself has nothing to do with storing data or documents or XML. Its got to do with a particular implementation of data storage which requires maintaining a ~~metadata~~metacode map. I shall use an example from the patent itself.

Lets say the XML is as follows :

The Secret Life of DataData is hostile. The End

The patent suggests a storage which would maintain a metacode map as follows :

Element Number	Element	Character Position
1		0
2	</td> <td>0</td> </tr> <tr> <td>3</td> <td>	23
4		23
5		39
6		46

The metacode map essentially stores a tag along with its position. You can also see the same clearly on page 15 of the patent document on google patents.

My reading suggests that the patent and alleged infringement if any has got nothing to do with storage of XML documents per se. In fact, that the tags being used are XML/SGML like is probably completely coincidental – these could very well be ${chapter} instead of . And XML documents are stored with the tags embedded along with the content – this patent actually refers to maintaining a map of these tags and the positions they should be inserted into.

Could I be wrong in my interpretations ? Perhaps, since I haven’t seen anyone else point this out and would prefer to be corrected. But the fact remains that at this point in time as I write this post, I believe that XML and storage of XML documents are completely orthogonal to the patent and the case around it – that centers around a metacode map, and metacode maps are not a characteristic of typical XML storage at all. So there’s probably one big misunderstanding about what this case is about and if the injunction upsets you because you are against patents in principle, thats fair. But if you are disappointed about the possibility that this somehow substantially impacts XML storage, the way I interpret it – there’s no such implication.

I shall keep my fingers crossed and hope no one points out a seemingly obvious flaw in my interpretation.

No related posts.

Presentation : ReST explained

Dhananjay Nene — Tue, 07 Jul 2009 09:05:13 +0000

My presentation as a part of TechWeekend, Pune last Saturday. Note that it is a very long presentation. It was designed for a 3 hour slot and attempted to introduce ReST to users who had relatively little exposure to ReST.

ReST (Representational State Transfer) Explained

View more presentations from Dhananjay Nene.

No related posts.

Most american graduates are unemployable because …

Dhananjay Nene — Tue, 23 Jun 2009 15:25:09 +0000

Came across this post Top Indian CEO: Most American Grads Are ‘Unemployable’ leading to some substantial discussion (not all of it impassionate). I would first like to clarify that I am assuming the content in the post to be accurate based on the fact that other Indian sites are reporting that “No response or clarification from Nayar has as yet been issued”. Moreover it is not targeted against Mr. Nayar or HCL but to the suggested thought process which attempts to make process orientation so critically important overriding many other qualities.

What perhaps hasn’t been clearly stated but is important is that that the standards for employability here are set by the employing organisation based on its context and preferences which may be different from standards required by many other organisations and I would submit that the statement ought to be looked at in that context as well.

To quote from the article :

The official wanted to know why HCL, a $2.5 billion (revenue) company with more than 3,000 people across 21 offices in 15 states, wasn’t hiring more people in his state. Vineet’s short answer: because most American college grads are “unemployable.” (In fairness to HCL, the company recently announced plans to open a delivery center in another state, North Carolina, and invest $3.2 million and hire more than 500 employees there over the next five years under a Job Development Investment Grant.)

Many American grads looking to enter the tech field are preoccupied with getting rich, Vineet said. They’re far less inclined than students from developing countries like India, China, Brazil, South Africa, and Ireland to spend their time learning the “boring” details of tech process, methodology, and tools–ITIL, Six Sigma, and the like.

As a result, Vineet said, most Americans are just too expensive to train–despite the Indian IT industry’s reputation for having the most exhaustive boot camps in the world. To some extent, he said, students from other highly developed countries fall into the same rut.

So why are some graduates are unemployable ?

Some of these are folks who are working on creating tons of new languages (Java, C#, Scala, Closure, Erlang, Python etc.), Operating Systems (Linux, Mac and Windows), frameworks for web applications, clustering, fault tolerance and scalability, schemaless and distributed databases to ensure availability and fault tolerance at a wide scale, competing messaging architectures that each service a particular problem differently etc. etc. Would you believe it some have set up clusters of hundreds of thousands of machines and service search requests very rapidly. And some others are working on creating innovative and disruptive models in social networking, application integration, peer to peer networking etc. Many are working out next generation mobile technologies including building (not building on) android, iphone OS and the palm (whatever OS it has). And so many in their spare time are spending a whole bunch of time creating open source software, blogging and micro blogging about all the work they have done and sharing it with the wide world so that they can learn off it. And if these are consumer and technology stories, there are a whole bunch of people building critical business infrastructure architectures and frameworks and solutions as well. Even after you complete reading this post and come back to it a month from now, I shall still be busy trying to figure out what exactly India has been able to deliver that can challenge this. Of course let’s not miss out on the fact that most of these activities are conducted by fairly small sized teams. (I am aware some of the examples I quote could have non-american heritage, but that per se is not likely to detract from the issue)

Let’s call a spade a spade. When you have large or very large software construction and maintenance contracts, there are multiple ways to deal with it. In my experience, many Indian companiess have honed to a fine art the process of recruiting, deploying and juggling large armies of programmers to service such expectations (not all Indian companies are based on the same model). Many companies have indeed managed to acquire, retain and expand customer engagements through a combination of technology and business innovation. And they have done a good job of it in the context they’ve defined for themselves. In my limited understanding and experience it is not technology innovation, creativity or extraordinary technical prowess that is at the top of the list of skills that get deployed (though these are indeed found in sufficient levels across the board and some of the prowess can be impressive) – it’s the clockwork project management and business methodologies, techniques and innovation (and even these do result in projects with delays) with their reliance on “another brick in the wall” that gets the customer serviced.

So, if true, the “employable” was perhaps in that context. If so based on the earlier two paragraphs, I submit it reflects positively and negatively on the employing organisation. Perhaps even more so than the graduates themselves. I do wonder if “incompatible” would have been a better choice of word. And I wonder what it reflected more poorly upon, as well who should be feeling more sad about it.

Disclaimer : These views are mine, mine alone and should not be puported to be shared by any other people or companies I’ve been associated with in the past, present or the future.

Opera Unite : A model for server disintermediation on the internet

Dhananjay Nene — Tue, 16 Jun 2009 09:22:14 +0000

Since Opera Unite got introduced a couple of hours ago. I had a chance to do a quick review of the functionality. As I begin to play with it, couldn’t but help write a short post on what it seems to be doing.

Here’s what my preliminary look into Opera Unite suggested :

It has a web server bundled into the browser.
It allows you to write (what would be in current parlance be called) serverside apps using javascript which are hosted by the web server.
Sample applications include photosharing apps to share photos on your desktop or fridge which allows internet users to post notes to you.
It allows these apps to be accessed across a router on a dynamic IP using a subdomain on a centralised operaunite service ..operaunite.com (not sure yet whether across a firewall as well but seems so)
It allows these applications to be shared / published using the config.xml file (similar to google widgets ?). There’s also a central opera directory for the same, but I don’t think sharing is restricted to that service alone (apple are you listening ?)
These applications can be further installed by other users of Opera Unite (which is the web server service running inside the Opera browser).

So why is this such a big deal ? Without going into any further elaboration let us just imagine user’s used it for following (for desktop-to-desktop or peer-to-peer) communication :

Send mails to each other – disintermediates email services
Send short burst messages to each other – potentially disintermediates twitter
Share files with each other – disintermediates ftp servers and shares characteristics with gnutella, kazaa etc.
Share resume, product profiles etc. – disintermediates traditional static web hosts
Build networks of other interested users – disintermediates linkedin, facebook

The constraint it introduces is that there is no global list of user ids to search from – you need to know the URL upfront (like the facebook vanity URL).

The possibilities are many. As are the interesting uses this model could be put to. And the characteristics of improved privacy, data ownership and control (including fine grained access control or selectivity). And there does remain a potential for malicious actors and virus writers to ride on popular apps to exploit vulnerabilities to tend to their nefarious needs.

As I put it differently in another tweet – this opens up the possibility for a google wave without the google in it.

References :

Taking the web into our hands one computer at a time : An introductory writeup
An introduction to Opera Unite : A get started guide
Opera Unite developer’s primer : A primer for writing server side applications

Design Characteristics of REST / Resource Oriented Server Frameworks and Clients

Improve your web based software development and maintenance ROI with dynamic programming languages

Dhananjay Nene — Mon, 15 Jun 2009 07:43:53 +0000

This is a cross post of my article which appeared in PuneTech in March 2009 here. The article is reproduced verbatim including the editor’s notes (in italics). I had already posted the slides referred to in the talk that was proposed in this article in the blog postTalk Slides : Programming Language Selection.

After we carried a few quick articles on why you should learn more about Ruby and Ruby on Rails (take 1, take 2) last month, we decided that we wanted to give people a much deeper article on why these new languages (Ruby, Python, PHP) and frameworks (Rails, Django) are setting the web world on fire. We invited Dhananjay Nene to write an article with an in depth discussion of the technical reasons how these new languages differ from the older ones and when to choose one over the other. He responded with this article which, as an added bonus, also includes the business reasons for your decisions. At the request of the community, Dhananjay is also giving a talk on the relative strengths and weaknesses of different programming languages on Saturday, 28th March, 4pm, at SICSR. All those who found this article interesting should definitely attend.

Introduction

Programing language selection is often a topic that elicits a lot of excitement, debate and often a bit of acrimony as well. There is no universally superior programming language that one can recommend, so I tend to generally disregard most language opinions which say ‘X language is the best’, without specifying the context under which it is superior. Finally most language debates often deal with the technical issues and not the ROI issues. Hopefully I shall be able to address this topic without being guilty of any of these problems.

So what languages are we referring to here ?

: Image via Wikipedia

The range of languages that fall under Dynamic Programming Languages category is rather extensive. My experience is primarily limited to Python and to a lesser extent PHP, Ruby, Javascript, and Groovy. For the rest of this article, I shall be primarily referring to Python or Ruby when I use the word dynamic languages, though many of the references may continue to be applicable and relevant for a number of other dynamic programming languages.

As I describe the technical characteristics, I shall also continue to attempt to address the business aspects as well, so you might find this article at a little techno-business level. Assuming I am able to excite their interest, the tech guys would not find sufficient technical details and would be hungry to hunt for more, and while the business guys would get a little teased with the possibilities, they will not quite get the ROI served in the traditionally formatted excel spreadsheets. Being aware of that, I continue down this path with a feeling that this perhaps will be the most appropriate level for me to abstract this article to.

Characteristics of Dynamic Programming Languages.

Let us quickly review some of the characteristics :

: Image via Wikipedia

Object Oriented : Many dynamic languages support full object orientation. There are many who don’t necessarily buy the benefits of Object Orientation, but it is my strong belief, that once a piece of software grows beyond a certain threshold of complexity and / or size, Object Orientation starts delivering very strong dividends. There are a few areas such as highly complex, algorithmic processing which might be better suited for functional programming. However a majority of the medium-to-large sized web applications are better served by OO. The empirical evidence at least bears out the fact that most of the most popular languages today (except C) are Object Oriented. However this still is a very very large class of languages which in them include C++, Java, PHP, Python, Ruby etc. The one area where some dynamic languages separate themselves from the others is in the notion of “everything is an object”, ie. primitives such as numbers, functions are all objects by themselves.

Business implications: OO code well designed and implemented allows for a substantial reduction in maintenance costs. When working with a team which is up the curve on OO, it is likely to lead to lower costs and time on inital coding as well. On the other hand, both training costs and skill requirements are higher for fully OO languages. If you are already using partialy OO / hybrid languages such as PHP, C++ or Java, and are convinced about OO, using fully OO languages such as Python or Ruby will help you leverage the OO capabilities even further.

Duck Typing : In very loose terms, duck typed languages do not require you to declare an explicit interface. You send an object a message (ie. invoke a function or access an attribute) and if it can respond to it, it will, and if it can’t it will result in an error. Duck typing is a specific typing system which is a subset of a broader system called Dynamic Typing, which often makes for an interesting debate with its counterpart – Static typing : Static and Dynamic Type checking in practice. For people well grounded in static typing alone, this can sometimes seem to be sacrilegious. I am convinced that duck typing makes writing code much much faster for two reasons – a) You now require to write fewer lines of code and b) You often don’t have to keep on regularly waiting for the compiler to do its work. There is also a substantial capability enhancement that dynamic typing makes to the language type system, which allow the frameworks to build dynamic types on the fly. This in turn offers the framework users many more capabilities than frameworks written in other languages. That is why it is nearly impossible to write frameworks like Rails or Django in Java (You can modify the class loaders and use byte code generation to generate the new types, but the compiler can’t see them so you cant use them). That is also why there is a lot of anticipation of using JRuby, Jython and Grails on the JVM since the languages underlying them (Ruby, Python and Groovy respectively) bring the dynamic typing capabilities to the JVM platform.

Business Implications :Writing code is much much faster. Maintenance depending upon the situation can sometimes be more or less difficult in case of dynamic typed languages. Refactoring is usually a lot more difficult in case of dynamically typed languages since the underlying type system is not able to infer sufficiently about the code to help the refactoring tools, as is possible in case of statically typed languages. It is my opinion that a skilled and trained development team using dynamic languages can generally substantially outperform another equally capable team using static languages. Insufficiently or poorly skilled development teams however can lead to very very different kind of pitfalls in these class of languages. In both cases the code becomes difficult to change or maintain due to a) cryptic code in case of dynamically typed languages and b) extremely large code bases in case of statically typed languages. Both are undesirable situations to be in but if I had to choose between one of the two, I would go for being in the cryptic mess since it is at least manageable by bringing in external skilled help.

Metaprogramming : Metaprogramming is in loose terms the ability of programs to write programs. A large proportion of developers may not use this capability too frequently. Specifically in web application development it gets used as a mechanism to transform one set of datastructures which a programmer specifies into code at runtime. As I point out later in this article, it in fact is a very important element in designing common frameworks and libraries which in turn offer substantial capabilities including small code and easier maintenance. A quick note to state that metaprogramming is not code generation. In case of code generation, one uses the generator to generate code which is then compiled. A big limitation with this is the fact that often people modify the generated code leading to really tough maintenance nightmares and the fact that it is a two stage process which is prone to more errors. Metaprogramming results in new code “coming to life” so to speak while your program is running.

Business Implications : Read on, they will get covered in the final roundup. They are large and they are positive.

Function blocks/objects, iterators, closures, continuations, generators: I will not go into any substantial details of this issue except to say that small pieces of code logic can be handled in a much much more concise way than if these weren’t supported. While many situations may not need closures support, you will be glad to have them on your side when needed.

Business Implications : Helps having shorter, cleaner code leading to lesser development and maintenance costs. Another significant positive is that your developers are just likely to be so much happier since they get some truly nice building blocks for concise and elegant expression of their logic. Can’t think of any significant negatives.

There are a full range of other capabilities, but none come to mind immediately as something that have strong business implications as well.

The role of frameworks

: Image via Wikipedia

When did these languages say Ruby and Python originate ? Most people are likely to be a little surprised if the answer is in the last millenium. Yet Guido von Rossum started working on Python in 1986 and Ruby was released in 1992. Python has been rather well known within the scientific community and perhaps a bit within the systems / OS utility programming communities for quite some time. However both languages grabbed a large mindshare only post 2005. A big reason for their popularity (especially in case of Ruby’s case) came from the popularity the frameworks which used them. Ruby on Rails for ruby and Django (to the best of my knowledge) for python. These frameworks combined the language capabilities with the learnings of good design practices for internet applications (eg MVC, declarative validations, simple ORM etc) into a simple usable package, which developers could take and build web applications quickly. There are examples of people having built simple web apps within a day and medium complexity apps in 1-3 weeks using these frameworks. The languages are the ingredients, the frameworks are the cooks – a great combination for serving great meals. Now you will find many such frameworks in these languages, including some which have better capabilities for building more sophisticated / complex applications eg. Merb and Pylons.

I am not too sure of how many people are exactly aware of the role of metaprogramming in the frameworks’ successes. I am willing to believe that but for metaprogramming, these frameworks simply would not have achieved anywhere close to the success they achieved. It is metaprogramming which takes the datastructures as defined by a developer and converts it into runtime code implicitly, saving the developer lots of time and effort. So even if most developers don’t actively write metaprograms, their lives are so much easier. Metaprogramming capabilities are also the reason why it is virtually impossible to write similar frameworks in Java. However if you are on the .NET or JVM environments, things are definitely looking encouraging with the possibilities to use IronPython or IronRuby on .NET or JRuby or Jython or Groovy+Grails on the JVM.

Business implications : If you are focused on scientific or desktop or highly algorithmic applications, where python especially is used extensively, you are likely to get benefits from these languages on their own merit alone. For web applications you will see the maximum benefits by using the web MVC frameworks along with the languages. I submit that on the whole you are likely to see very substantial reduction in development, enhancement and maintenance times – sweet music for any end user, investor or project manager.

Increased Business Agility

There is one more reason why I believe these languages are especially helpful. They help by increasing development agility to an extent where it now allows for the business to be more agile. You can get a first prototype version up in weeks, take it around to potential users, and gather feedback on the same. Incorporate elements of this feedback into the next release of working code quickly. The business benefits of such a scenario are tremendous. You might wonder that this is a process issue, so what does it have to do with a language selection. I would submit, that languages which allow changes to be made faster, help support this process in a far superior way. Another equally important facet is the superior risk management. Since you are able to build features with lower investments, you are able to get a series of customer feedbacks into your decision making process much faster. This helps being able to come up with a product that really meets the customer expectations much earlier. This happens by allowing the better features to come in earlier and also by allowing the lesser important or lesser relevant features to be decided to be deferred earlier. That’s precisely the reason why the dynamic languages have found a strong acceptance in the startup world. I believe the increasing agility which is often required in the startup world, is and will continue to be increasingly required of established enterprises. Precisely the reason why I believe these languages will continue to do better in the enterprise space as well. Finally, these languages make it relatively easier to tell your business sponsor – We will work with you on imprecise requirements rather than spending months on nailing down requirements which anyways are likely to change later. This has both a pro and a con especially for outsourcing situations. It is likely to allow for tremendous customer delight in terms of a vendor that works with him in such a flexible manner, yet it does introduce challenges in terms of how the commercials and management of the project are handled.

The reason I would like to especially point out increased business agility is because programmers don’t often visualise or evangelise it much, but when I wear a manager’s hat, it is perhaps the most compelling benefit of these languages.

Concluding

As I said earlier, there is no single universal language which is the best for all scenarios. There are some scenarios where using dynamic languages will not be helpful

A Treemap view of sales of programming language books by O’Reilly Media in 4Q2008. The size of a box represents the total sales of a book. The color represents the increase or decrease in sales compared to same quarter in 2007. Green = increase, bright green = big increase, red = decrease, bright red = large decrease. See full article at O’Reilly Radar for lots of interesting details.

When not to use these languages

You are building a simple / small application and don’t have the available skill sets. One exception to this is where you decide to use it in a simple application to allow yourself a non risky mechanism of building these skillsets.

Extremely High performance requirements. However please make sure that you really need the high performance capabilities of say a C, C++ or Java. In my experience 80% of developers like to believe that they are building highly performant applications where the maximum speed is a must have. Yet the top 10% of them are facing far far more critical performance requirements than the remainder. Unless you are convinced you are in the top 10%, you should certainly consider dynamic languages as an option. Moreover in case of most high performance requirements, these can sometimes be boiled down to a few inner loops / algorithms. Consider implementing the same in C, / Java or other .NET languages (depending upon the choice of your dynamic language interpreter implementation)

You have an architecture standard in place which does not allow using these languages. If you are convinced your applications are better served by using dynamic languages both from your individual application and an overall enterprise perspective, consider taking the feedback to your standards setting body to see if you can pilot a different approach. Also evaluate if the .NET or JVM versions can help you comply with the architecture guidelines.

You are unable to commit to the retraining requirements. While these languages are easy and powerful to use, leveraging that power can require some amount of retraining. If that does not fit your business plans, since the retraining effort could impact immediate and urgent requirements, that could be a reason to not use these languages. However in such situations do consider investing in building this skill sets before you get to another similar decision point.

You need a very high levels of multithreadinging as opposed to multi processing support. While this is not a typical situation for web applications, you should be aware that most dynamic languages have some limitations in terms of multi threading support. This actually is not necessarily an issue with the language as with the implementation eg. the C implementation of python has the notorious Global Interpreter Lock which constrains you from being able to use more than a handful of threads per processes efficiently. However the same restriction is not present in Jython (the jvm implementation of python). This is likely to be an issue for a miniscule percentage of the web applications market for the primary reason that multi process / shared nothing architecture styles often work quite well for many web applications and they don’t really need multi threading.

So where’s my return on investment ?

First of all lets talk of the investment part. If you get into it in a paced approach, the investment may not be that great. Start with a team size of anywhere between 2-6 people (depending upon your organisation and project size). Think of 15 days of intensive training followed by a 2-6 months coming up the curve effort (more likely 2 than 6). Make sure your first project is not a critical one under tremendous business pressure. This can be subsequently followed by more people getting retrained as necessary. In the longer term it might actually help reduce your incremental investment, since it might be much easier to ramp up new programmers in Ruby or Python than say Java or C#.

Secondly lets look at the incrementally higher costs. You are likely to need people who are a little bit more capable in terms of understanding and debugging the same logic expressed in fewer lines of code (that sometimes can be a challenge) and then be able to modify and enhance the same. This may increase your testing and fixing costs in the earlier days. Finally while the fewer lines of code can make refactoring easier, you could find that your total refactoring costs are a little higher.

Now the returns part. I am convinced that the increased business agility is the strongest return in business terms. Immediately after that is the substantial reduction in development, enhancement and maintenance times. If neither of these benefits are appealing, when contrasted with some other issues that you might perceive, maybe considering dynamic languages in your context is not such a great idea.

One more factor that I would of course encourage you to evaluate from a business perspective are the implications for you if your competition (assuming it is not already using them) started using these languages. The implications would vary from case to case, but it could also help you decide how important this issue is for you.

About the author – Dhananjay Nene

Dhananjay is a Software Engineer with around 17 years of experience in the field. He is passionate about software engineering, programming, design and architecture. He did his post graduation from Indian Institute of Management, Ahmedabad, and has been involved in Senior Management positions and has managed team sizes in excess of 120 persons. His tech blog, and twitter stream are a must read for anybody interested in programming languages or development methodologies. Those interested in the person behind the tech can check out his general blog, and personal twitter stream. For more details, check out Dhananjay’s PuneTech wiki profile.

Musings on REST

Dhananjay Nene — Fri, 12 Jun 2009 00:16:41 +0000

This is a summarisation of a four part series of posts I wrote on REST over the past week. This post lists each of them along with a very high level summary and a small snippet from each hopefully sufficient enough to tickle your thoughts and interests.

Why REST ?This is a rather long post which provides a narrative of the history of web and service architectures eventually coming together into web services. It refers to many of the strengths that made web architectures so omnipresent, and to the uneasy coming together of the two architectures as web services. It details many REST characteristics, describes how REST provided a style by which the strengths of the web architectures could be retained even as the processing aspects of service architectures could be supported, and finally enumerates some of the benefits of REST.
On the coming together of Web and Service oriented architectures.

Clearly as WWW started getting used far more, people were only too keen to use it for much more than storing or retrieving documents. This led to the development of CGI and subsequently other dynamic web application technologies (eg. LAMP, J2EE etc.) which would allow us to use the web to ‘do something’. Since these were clearly offshoots of the SOA world, being mapped onto the WWW infrastructure, the characteristics of such dynamic applications often had a lot in common with SOA, and they started dropping many characteristics of the traditional static WWW. Thus was born the child of the world wide web and distributed service oriented architectures – web services. This led to newer SOA technologies such as WS-* and SOAP.

Like the typical scenarios after the discovery of any highly profitable opportunity, the early rush was to leverage the opportunity and it was only a little later when the dust died down, that people started wondering if they had sacrificed something in the heat and dust of the moment. That stock taking resulted in the realisation, that some of the very basic characteristics of the extraordinarily successful internet technologies (FTP / SMTP / WWW) had been diluted, and even if such dilution still allowed immediate progress to have occurred, some of them would need to be corrected to be able to continue the explosive growth that had been seen so far. One such exercise in my opinion is the laying down of the REST architecture style.

On the aspect that even though REST is not as feature rich as SOA, its strength is the simpler abstractions it employs

I have generally found that simpler abstractions even though harder to deal with initially, often win in the long run. Notice the fact that the bare bones rendering functionality of HTML/WWW completely trounced the rich UI and application integration capabilities then available (eg. Windows/Java and DCOM/CORBA/RMI). This is not to suggest that the extra capabilities are not required. That is why Rich User Interfaces on WWW continue to be a dominant part of the internet technology wishlist. However the simpler, cleaner and minimalistic abstractions often are far more important than feature richness. A point I would want to make in favour of REST even as I admit that conventional SOA technologies are far more feature rich than REST.
REST is the DBMS of the InternetThis post reflects the thought that since REST effective allows one to GET, PUT, POST and DELETE resources, it is similar to being a database which exposes its tables to many applications to SELECT, INSERT, UPDATE and DELETE from. In this analogy, each media type is effectively a new table and the REST interface is primarily of the nature of allowing basic operations on a set of tables (resources).

To summarise the exchange differently

“If WS-* is the RPC of the Internet, REST is the DBMS of the internet“

To expand on it a bit more :

Traditional SOA based integration visualises different software artifacts being able to interact with each other through procedures or methods. REST effectively allows each software artifact to behave as a set of tables, and these artifacts talk to each other using SELECT, INSERT, UPDATE and DELETE. (or if you wish GET, PUT, POST, DELETE). And where exactly is is the business logic ? Is it in the stored procedures ? Not Quite. Its in the triggers.
Design Characteristics of REST / Resource Oriented Server Frameworks and ClientsThis post dwells into the many aspects of design of a REST or Resource Oriented Serverside Framework and attempts to enumerate a large number of their characteristics. One of the aspects it brings up is the role of the Controller. Since the a resource oriented interface primarily consists of basic primitive operations on resources, it suggests that the controller could either be merged with the Resource or support only basic operations and have a one to one relationship with a resource.

This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

Again to extend the analogy of a DBMS, it argues that instead of a lot of logic being in the controllers which are the entry points of the interface (stored procedures), the interface now changes to support basic operations on the resources (tables) and that the logic could perhaps be modeled in a separate class of handler functions (triggers).

Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.
ReST : SOA, WOA or ROA ?This post dwells on how consistent REST is with Service, Web and Resource oriented architectures. It argues that REST could perhaps be argued to be SOA only in a most specific form, and that for all practical purposes REST should not be expressed as SOA.

And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

It brings out some potential inconsistencies in how WOA is currently not only portrayed to be a set of architectural elements in addition to REST but also as a extension of / future of SOA.

My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

…

So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying.

Finally it refers to ROA as the architecture (which actually is defined with REST as the basis) with which it is most consistent with.

Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

ReST : SOA, WOA or ROA ?

Dhananjay Nene — Wed, 10 Jun 2009 19:52:35 +0000

This is part 4 of a continuing series of posts on ReST. So you might want to read up the earlier ones as well (chronologically they are the three posts before this)

Nice alphabet soup indeed. But what style of architecture does ReST (Representational State Transfer) correspond to.

Before we get into any definitional issues which are hugely referred to in case of such debates – I shall be referring to the currently available definitions and descriptions as available on Wikipedia viz. Service Oriented Architecture, Web Oriented Architecture and Resource Oriented Architecture.

Service Oriented Architecture

As per the definition on Wikipedia (emphasis is mine):

In computing, service-oriented architecture (SOA) provides methods for systems development and integration where systems package functionality as interoperable services. A SOA infrastructure allows different applications to exchange data with one another.

Service-orientation aims at a loose coupling of services with operating systems, programming languages and other technologies that underlie applications. SOA separates functions into distinct units, or services, which developers make accessible over a network in order that users can combine and reuse them in the production of applications. These services communicate with each other by passing data from one service to another, or by coordinating an activity between two or more services.

On the topic of Service Orientation, it further goes on to state

Service-orientation is a design paradigm that specifies the creation of automation logic in the form of services. It is applied as a strategic goal in developing a service-oriented architecture (SOA). Like other design paradigms, service-orientation provides a means of achieving a separation of concerns.

While REST does attempt to solve many similar goals as SOA, I believe there lies an important distinction. The essential focus of SOA is to separate functions or automation services. An example here is to separate an authentication service from authorisation, monitoring, logging etc. A SOA architecture that consists of a number of SOA services assembles such “functionalities” into one feature consistent whole. But is that how REST works ? Only in a very vague sense even if arguably so. REST standardises the functions ie. GET, PUT, POST and DELETE in case of HTTP connectors. It is arguable that REST could be used with a different set of functions, but even in that case the function set is likely to remain consistent. This is further supported by Roy Fielding’s post “It is Okay to use POST” in which he argues

Some people think that REST suggests not to use POST for updates. Search my dissertation and you won’t find any mention of CRUD or POST. The only mention of PUT is in regard to HTTP’s lack of write-back caching. The main reason for my lack of specificity is because the methods defined by HTTP are part of the Web’s architecture definition, not the REST architectural style. Specific method definitions (aside from the retrieval:resource duality of GET) simply don’t matter to the REST architectural style, so it is difficult to have a style discussion about them. The only thing REST requires of methods is that they be uniformly defined for all resources (i.e., so that intermediaries don’t have to know the resource type in order to understand the meaning of the request). As long as the method is being used according to its own definition, REST doesn’t have much to say about it.

So across the board the functions remain the same and the data types (or media types) change. Sounds familiar ? Thats like supporting SELECT, INSERT, UPDATE, DELETE on a broad range of tables each having a different schema. An argument I make in an earlier post REST is the DBMS of the Internet. Moreover Roy further goes on to elaborate that in “REST intermediaries don’t have to know the resource type in order to understand the meaning of the request” and that “method is being used according to its own definition”. Each of this further introduces constraints into traditional service orientation. And these constraints make the field of use so narrow, that even though REST could be argued to be a teeny weeny specific use case of SOA, it could be argued to be Service Oriented to the same extent that a Database could be argued to be Procedure Oriented (since all tables support the procedures SELECT, INSERT, UPDATE, DELETE). In other words for all practical purposes REST is not Service Oriented.

Web Oriented Architecture

This is not yet a particularly widely used term yet, but I did come across a reference to it on a recent article in InfoQ “REST is a Style – WOA is the architecture“. Looking it up on Wikipedia leads us to the same sources as referred to by InfoQ – articles written by Dion Hinchcliffe. Wikipedia states it as follows

Web Oriented Architecture (WOA) is a style of software architecture that extends service-oriented architecture (SOA) to web based applications, and is sometimes considered to be a light-weight version of SOA. WOA is also aimed at maximizing the browser and server interactions by use of technologies such as REST and POX.

But I just argued in the earlier section that REST is only a very very specific use case of SOA, whereas the statement above says WOA extends SOA to web based applications. If we were dealing with objects and not architectural styles here, an argument that X is a specific (constrained) sub type of Y and X extends Y would instantaneously be flagged off as a violation of Liskov’s Substitution Principle (LSP). So something isn’t quite right here and the whole situation does not add up to a consistent whole for me. I will leave it to the reader to decide whether there exists a flaw in my reasoning here or elsewhere. My assessment is that if WOA is a collection of Web related architecture elements in addition to REST, then the only way to successfully and consistently resolve it is by saying if WOA builds on REST then it cannot be simultaneously extending SOA.

To be fair I couldn’t quite find the same worlds (WOA extends SOA) in Hinchcliffe’s writings. Referring to one of them “What is WOA ? Its the future of Service Oriented Architecture (SOA)“, he refers to another of his post “Beating a Dead Horse: What’s a SOA Again? All About Service-Orientation…” which in turn states

It’s here that John Reynolds’ well-known SOA Elevator pitch comes tantalizingly close to capturing the essence:

SOA is an architectural style that encourages the creation of loosely coupled business services. Loosely coupled services that are interoperable and technology-agnostic enable business flexibility. An SOA solution consists of a composite set of business services that realize an end-to-end business process. Each service provides an interface-based service description to support flexible and dynamically re-configurable processes.

This business view is right on, and doesn’t mean business in a traditional, white-collar way. In this context, “business” means the actual functionality of the system, apart from technical details.

There we go again on services. But Business Services cannot be GET, PUT, POST, DELETE. I would emphasise again that REST does not expose business services – it exposes some very basic CRUD services. So even if in this case there is no violation of LSP, the essential inconsistency still remains. WOA cannot be REST and SOA at the same time. This inconsistency is a bit worrying. But it is likely that Hinchcliffe meant that WOA is built on REST and is similar to SOA in terms of the goals when he says WOA is the future of SOA. But honestly I could not quite figure out how exactly he would want to describe the relationship between WOA and SOA.

Resource Oriented Architecture

The wikipedia page states :

Resource Oriented Architecture (or, ROA) is a specific set of guidelines of an implementation of the REST architecture.

REST, or Representational State Transfer (see Roy Thomas Fielding’s Doctoral Thesis “Architectural Styles and the Design of Network-based Software Architectures”), describes a series of architectural constraints that exemplify how the web’s design emerged. Various concrete implementations of these ideas have been created throughout time, but it has been difficult to discuss the REST architecture without blurring the lines between actual software, or the architectural principals behind them.

Since ROA is a set of guidelines of an implementation of a REST architecture, I think its a slam dunk conclusion that REST is consistent with ROA (for the silly reason that ROA seems to be defined using REST ).

Per Wikipedia, Leonard Richardson and Sam Ruby further provide the guidelines for ROA in “RESTful Web Services“, but again since the evolution of ROA stems from REST, it is unsurprising that REST is consistent with ROA.

Design Characteristics of REST / Resource Oriented Server Frameworks and Clients

Dhananjay Nene — Wed, 10 Jun 2009 14:04:51 +0000

This post is the third part of continuing series of articles on REST. The first one was Why REST ? and the next one was REST is the DBMS of the internet with hopefully some more to follow in the coming weeks.

Struts, Django, Ruby on Rails. We’ve worked with these and many other similar frameworks. Some time back I started thinking of what would a completely new ground up REST / Resource oriented framework would look like (ground up to ensure it had no legacy design to deal with). Would such frameworks be similar to the ones dominantly used today ? What about the ecosystem that surrounds and interacts with them (client libraries) ? And finally what about the implications on the fine grained object model (assuming there is one) and its relationship with the resource model ? This post deals with some of the thoughts.

There are some specifics the post does not address and is agnostic about :

Language : I shall be avoiding language issues as much as possible. Wherever I do bring in code constructs these may be assumed to be in Java (or pseudo-Java)
Convention or Configuration : I think both are valid choices in their appropriate contexts, and I don’t specifically emphasise one over the other in this post

The frameworks mentioned above are not the only ones out there. There are many, and some actually are very REST specific eg. Apache CXF JAX-RS or Restlet. It would certainly be interesting to contrast my thoughts with these, but for reasons of insufficiently detailed knowledge about them, I shall choose to skip it (better to not make any statements than making incorrect ones).

I shall be assuming a HTTP connector with GET, PUT, POST and DELETE as the constant set of operations. These four operations shall be collectively referred to as Resource Operations.

We shall first start with the server side characteristics, and the term ROF shall refer to a Resource Oriented (server side) Framework

A ROF will have a resource oriented interface : Certainly not a profound statement, but it was important to lay that down upfront. So what is a Resource Oriented Interface. Given a particular resource, a Resource Oriented Software will support or consume end points which allow you GET, PUT, POST or DELETE the resource. There is one reason why this particular constraint is relaxed just a little bit. Modern browsers do not support all the four methods easily eg DELETE and make it just slightly hard to use the PUT method. Hence these methods can also be invoked by using a URI segment containing the method name eg. delete.

A ROF will have an abstraction to represent a resource as an end point : Again, that seems to be pretty obvious. But there is a reason why I make it explicitly. In many situations we see controllers acting as end points. To the extent a controller acts as an abstraction for a resource end point which essentially only has the resource operations as public methods, it would fit this requirement. However if I was using an Order as a resource and if I introduced an approve method on the OrderController that would not be consistent with this requirement. That would need to be modelled as an OrderApproval resource which may on successful completion, effect a state change on the Order resource to the status ‘approved’.

This is where a potential differences with conventional frameworks arise. If I was to think of it from an EJB like perspective, I would model a OrderController as a Session bean and a Order as an entity bean. In case of lightweight POJO based model, I would have an OrderController as the endpoint exposed by say using Struts and model the Order as a entity POJO and map it to the database using Hibernate. In other non java frameworks, I would have a class to represent an OrderController and another one to represent the order along ActiveRecord pattern. But I would argue this separation is not entirely necessary, since what we want is something that implements a single abstraction mapping onto a Resource which also support the primarily lifecycle methods or resource operations of GET, PUT, POST and DELETE. But there is an issue to be worked through here. These resource operations are actually class level and not object level methods. Thus if we have an abstraction to represent the resource instance, the class level methods cannot be defined in the same class except as class level (static) methods. This is a tricky problem, and I would submit the designer may make one of two choices (a) Implement the resource operations as class level methods on the Resource abstraction (ie. they will get or return the resource references as method parameters and not rely on the ‘this’ or ’self’ qualifier for getting access to the resource variables or (b) Implement the resource operations as methods on a separate one-to-one mapped class on the resource abstraction (eg. an OrderHome in case of an EJB like analogy)

Given consistent expectations of the Resource Operations these will actually be auto-magically implemented : Thats a bit of a turnaround from what I was just describing in the earlier paragraph. What I mean to suggest is that the class level methods I just referred to will be implemented within the framework. What the framework will allow are plugins to provide extended functionality at specific points. Thus a “public static Order Order.put(Order order)” method will be implicitly implemented by the framework. But before a put can be processed it needs to be validated. Thus the framework will allow the developer to plug in / override his own implementation for an Order.validate(Order order). There are multiple ways such plug-ins could be implemented. Depending upon the nature of abstraction, it could be an overridden method as I just described, or it could be a standalone method that is registered into the overall workflow (either through convention or configuration). The latter might be especially useful in case one wants to implement the functionality as stand alone methods or in case of functional programming languages. The plugin points provided could be framework specific. eg, One may want to validate a resource for consistency even at it is being read from the database. For the rest of the post I shall refer to these as plugins. In addition, framework will most certainly provide methods for for downstream handling of impact of PUT, POST or DELETE. This is covered in the next point. In case the framework chooses to not deal with persistence, it may choose to allow capabilities for integration with other persistence frameworks.

A ROF will provide capabilities to a developer to override or register methods to handle downstream impact of PUT, POST and DELETE : Before I get into the details of this, I encourage you to take a look at my earlier post REST is the DBMS of the internet in case you have not already done so. To summarise it quickly, I have drawn the analogy that a REST based system is like a DBMS where client applications can perform direct SQL such as SELECT, INSERT, UPDATE, DELETE (GET, PUT, POST, DELETE in case of HTTP/REST) on the Tables (Resources in case of REST), and the business logic is implemented as triggers. Thus the framework will need to allow the developer to define such triggers. Such methods will need to support ability to reject the request (in case of downstream validation failures), and update the resource state (to reflect the appropriate resource state after the completion of the downstream processing). It is also feasible to imagine scenarios where such methods are triggered asynchronously. Much of the logic of the traditional controllers which controlled interactions across multiple objects etc. is likely to now be shifted into these methods. I have no particularly good name for such methods. They could be referred to as triggers, event or message handlers, glue methods, extension points etc. For the rest of this post I shall refer to these methods specifically as ‘handlers’.

Note that the actual invocations to select, insert, update, delete the resource are *NOT* to be programmed by the developer. These are automatically handled by the framework. The developer essentially fills in the necessary logic to the plugin methods (eg. Order.validate) or handlers (eg. Order.onCreate)

A ROF will provide a mechanism to describe or map a resource abstraction to to the actual programming constructs : There are a number of ways this could be achieved. XML, YAML, DSL, Annotation – take your pick. Also the actual class could be defined (as in case of a POJO) and the resource characteristics mapped onto it, or the class may manifest itself at runtime based on metaprogramming around the metadata. Sample possibilities here are Hibernate like Resource-to-Object-to-Relation mapping (using either Annotations or XML) or a a completely metaprogrammed ActiveResource. One important aspect that the framework will need to cover is the situations where a Resource is a composite of many or partial underlying business objects. eg. an Order resource instance could theoretically span one Order instance and many OrderItem instances. Thus a one to one relationship between a resource and underlying business objects (or datastructures) is not assumed. What is assumed is that the framework will allow such relationships to be described or introspected.

A ROF will allow resources to be mapped onto URI or URI segments : This is too obvious an requirement to be explained and is mentioned here only for completeness.

A ROF will allow foreign keys across resources which manifest themselves as URIs to be mapped onto the underlying business object references : Resources refer to each other through URIs. The underlying business objects refer to each other through object references. Given the resource descriptions and URI mappings, the framework should allow for a transparent referencing/dereferencing between such URIs and the object references.

A ROF will allow factory methods for locating or allow injection of other resources / business objects : Within the handler functions, developers will need references to the associated resources or business objects. I say resources or business objects, since the developer may choose to interact with these at a coarse grained (resource) or fine grained (business object) level. The framework should allow the necessary support for such activities.

A ROF may provide additional support for typical aspects of lifecycle (eg. validation) : While I mentioned validate as a possible plugin function. However given the omnipresence of validations, the framework may provide additional support for such activities. Thus the framework may choose to automatically implement such capabilities using the resource descriptions.

A ROF may provide capabilities for domain specific extension of resource capabilities : Certain domains have standardised mechanisms of working with resources. As an example most banking systems based on the four eyes principle require approval activities. While this particular aspect is much tougher than it seems, a ROF may choose to allow extension of such capabilities using template like functions or mix ins. As an example in this situation, once an Order resource is defined, an OrderApproval resource will be automatically made available as will the GET and PUT methods on it (POST and DELETE in this particular case may not be relevant), as will the necessary and appropriate handler functions on OrderApproval.

A ROF will provide capabilities for automatically generating the resource representation from the resource and vice-versa : Resources manifest themselves in multiple possible formats eg. XML, JSON etc. An ROF will allow such transformations between the representation and the resource/business object instance automatically.

A ROF will provide capabilities for assembling more complex representations using templates : In many situations, especially when the representations are being composed for manual (browser based) consumption, additional resources may need to be pulled into a view. A ROF will allow for such assembly of resources to be composed into a final view using templates.

A ROF will allow for introduction of appropriate additional URIs in views using templates : Thanks to HATEOAS (I’ve really avoided it thus far ), the framework will need to allow some mechanism of describing what are the additional context specific URIs to be included in the final representation. The template logic should allow the developer to specify such URIs.

A ROF should allow for the resource / media-type descriptions to be shipped in band with the resource representation : Since REST allows media types to be auto discovered and auto described, the framework should allow for the metadata for such media types to be also presented to the client. While I think it is essential that such in band information should be conveyed on demand, the framework may also optionally support upfront interrogation for media types and their details, which will require such information to be shipped out of band as well. I am not aware of any specific standards around such interrogation APIs so the framework could implement a custom API for the same. The actual metadata could be represented using any of the typical appropriate standards such as RDFa, XML Schema snippets etc.

A ROF should optionally allow support for auto generation of bindings for clients : I really really cringe as I write this. I cringe because to me the great attraction of REST is the simplicity and the ease of introducing incremental integration. The client binding generation (especially if it is statically generated) flies in the face of many accepted lightweight design scenarios. However I think there are likely to be some situation where availability of such client side bindings would be helpful. When possible (eg. with dynamically typed, metaprogramming capable languages like Python or Ruby), such bindings should be dynamic. In such cases the client can automatically introspect the server side media types and make available the necessary client side objects on the fly. In cases where statically typed languages such as Java or Scala are used, the client side may choose to expose everything as generic datastructures (e.g trees of name value pairs) or may allow for generation and compilation of client side bindings. I have no specific thoughts around the API support needed on the client side, except that quite obviously this would include support for the resource construction, resource operations etc. and that they would allow the client to interact with the server using the underlying language constructs rather having to work at a raw HTTP level.

In addition to the characteristics described above, I suspect frameworks will have many other optional characteristics such as support for monitoring, auditing / logging, transaction management, object pooling etc. etc. However these are unlikely to be particularly interesting when focusing on the framework aspects especially from a resource oriented perspective, which is indeed the focus of this post.

Update : InfoQ covered this blog post here : Design Characteristics Of Resource Oriented Server Frameworks