The conference was spread over 3 tracks. So the best I can do is talk about the talks I attended.

The first talk I attended was Node.js, HTML5 and Phonegap for high performant content site app by Prasoon Kumar. I learnt some stuff about phonegap and espresso. Good starting point, but hardly got to see any actual code.

Next one was Synchronized models using Backbone, Sockets and Node by Ruben Stolk. He started up a demo and suggested that people in the room connect to it. It basically had a canvas with a image and amongst many things, one could turn the light on the light-stand on and off .. and any actions propagated to the backend and then to all the connected clients. Similarly with post-it kind of notes that one added on the canvas. Was a good example of using backbone for defining models both on the client and the server and using socket.io to stream events between the two. He went through a fair amount of code too and it was very understandable. I came back with a distinct impression that the model maintenance and ability to synchronise the model changes between a client and server and then between the server and all connected clients (thus in effect across all connected clients) was pretty cool. However I still felt there needed a fair degree of wiring up of the event related code, though I am not sure about it and need to experiment with it more.

The next one was How to apply BDD and TDD practices, using Jasmine library? by Anil Tarte. Anil and I have been colleagues in the past and I have a high respect for his skills. In the brief period he was able to convey how to use jasmine to write BDD Test cases for the client side code. His server was over web sockets, and he was essentially intercepting calls into specific methods that eventually communicated with the backend. He also mentioned it would be possible to intercept the over the wire HTTP traffic instead of javascript functions, though it would create more difficulties if one was not able to precisely control the server’s outputs. I quite frankly had not imagined that BDD could be used so effectively and especially localised strictly to the client. So it was an insightful talk, that definitely will have me thinking about javascript BDD the next time. However he did seem rushed and under pressure and perhaps the talk lengths could be extended from the 45 mins to 55 mins or so.

Next was Building real-time web applications … (Introduction to Websockets / Socket.IO) by Aditya Y. He talked a bit about evolution of websockets, various libraries that were developed along the way and the current state of websockets support. There was an interesting demo at the end where changes in one browser session were almost being reflected in real time into another browser session via websockets connections back to the server. One of the important points noted was that Websockets continues to ride over port 80 so can work across many firewalls, and while its initial handshake is HTTP like, the subsequent traffic is essentially TCP like over that HTTP connection.

A thoroughly enjoyable talk was Advanced JavaScript Techniques by Rajasekharan Vengalil. He looked like he had enough stuff to talk about for the whole day, and delved into some of the OO related aspects of Javascript and the additions to ECMAScript 5. He spent some good amount of time in explaining how prototype based languages work. There are just so many intricacies to the language and so many really strange behaviours (like silently ignoring assignments etc.) that if I hated the language I would’ve just continuously muttered WTFs and if I happened to passionately like the language, I just might’ve facepalm’ed my way through the talk. Thankfully my outlook towards javascript is quite neutral, so I just enjoyed it . A lot.

Node.js Patterns and How we build ActiveNode was the next talk by Sreekanth Vadagiri. He talked about some of his experiences with node.js, his preference for using CoffeeScript rather than JS (despite it being harder to debug), many of the patterns he liked, some of the libraries he used and deployment matters. He was just as candid about some of the gotchas as well. A useful insight into the node.js system.

The last one of the day was Amplify your stack by Sunil Pai. He talked about a lot of libraries that could be used for client side development and deployment, about using javascript for templating, ensuring rigorous unit test coverage at all stages. Gave me the feeling there’s a lot I simply do not know about whats happening on the client side JS related libraries and frameworks. Couple of remarks I recollect – “You (the JS developer) own the browser” and “When you have a strong test case coverage, you can CODE BOLDLY”.

Summary: There’s a lot I need to learn about whats happening on the client side. Thats true about the server side as well. And the day helped me understand just a little bit better what I don’t know. Yet I got the distinct feeling of discomfort with node.js. Not with the tool. But with the assumptions that seem to go with its usage. There was poor articulation about what kind of use cases it is good for. Except that it is really good for high thruput/no. of client connections. Either there was a misplaced understanding of it being the only good way to get this kind of thruput or there was inability to clearly articulate what other benefits developers could expect out of using node.js in situations where thruput/concurrent connections is not particularly important for them. Perhaps I was just at the wrong places when someone was offering a more insightful articulation of this. But I really did not hear it.

On the whole, I enjoyed the sessions, and this was a day very well spent. Hats off to Kiran and his team for organising a good event. Makes me look forward to the next one they organise.

Related posts:

1. Why you should register to attend Python Conference Pune (Sept 2011) right now

Recently there was a useful discussion triggered off by the post True Scala complexity by Yang Zhang. Much has been debated about it in the comments on the post and other forums, most of it both articulate and constructive.

Martin Odersky proposed an idea on the ycombinator news thread as follows (next paragraph)

So, I believe here is what we need do: Truly advanced, and dangerously powerful, features such as implicit conversions and higher-kinded types will in the future be enabled only under a special compiler flag. The flag will come with documentation that if you enable it, you take the responsibility. Even with the flag disabled, Scala will be a more powerful language than any of the alternatives I can think of. And enabling the flag to do advanced stuff is in any case much easier than hacking your own compiler.

The basis of the idea wasn’t new. Martin himself had earlier suggested Scala Levels, though the suggested enforcement of two levels via a compiler flag was. While I thought it was a useful idea, strangely enough I found many (especially in the twitter streams I follow) less than enthusiastic. I think there exists a perspective where one can look at this idea with more enthusiasm, and this post details that.

Learning scala is non trivial. For a person who is well versed with Java programming and with little else, this will require learning at least the following (I am listing only a few items)

• Learning traits and objects (and unlearning statics). There’s a lot of stuff here including multiple inheritance, member overriding, whether the members should be defs or vals etc. which takes some times to get used to.
• Learning to deal with immutability and occasionally lazy evaluation
• Learning collections constructs such as for comprehensions or, map, flatMap, filter etc.
• Understanding the capabilities that scala offers in terms of generics including being able to define co/contravariant collections, and then learning to use these capabilities effectively
• Being able to understand and leverage many aspects of type theory

Typically when a person makes a sustained effort driven by passion, he can cover a lot of ground very quickly. Thus some might be able to deal with the scala learning curve quickly. While this ability to learn could be achievable for many especially smaller companies, it is likely to be quite difficult for many others. Especially when the team sizes are large and/or developer capabilities might be modest, this is a real issue. One could form an analogy with a Cheetah and an Elephant. Some animals will be able to run fast individually, and others will trod slowly as a herd. One could wish elephants could run as fast as cheetahs but thats unhelpful, since the jungle is defined, and one best deal with it the way one finds it.

An ideal way to introduce scala would be to start with small groups of 3-5. But thats not practical in all cases. Besides for large teams, that means, most will need to wait for a much longer time before being able to use scala and profit from its substantial benefits. So lets say an organisation attempts to introduce scala in a modest team of say 20. Of which 2 are passionate, self driven souls, who tend to work much harder the rest to be able to progress much faster. Because learning to use scala the way scala wished to be used, is not a sprint but is a marathon (or at least a long distance jog), this creates issues. Soon enough the 2 are likely to run far ahead of the rest of the team and will start writing code which the others can’t grok. Even as some of the remaining may actually be struggling hard but are likely to be losing enthusiasm simply because they can’t find themselves being able to keep pace. An even bigger risk is the 2 continuing to make rapid progress even as they realise so much more remains to be learnt, even as the remainder are unable to start leveraging scala, since they still think there’s a lot of distance yet to cover and they become far more focused on catching up rather than leveraging what they have learnt. So while learning proceeds, actual development with direct economic benefit stalls.

Lets say hypothetically this team decided that they would only focus on reducing java boiler plate (which is a big deal) and continue to use scala in a primarily OO paradigm (since that is what they are used to) and defer learning remainder of scala by at least six months. There’s likely tons of benefits to be had by shifting to scala even with this limited scope. Yet all it takes to destroy harmony is one very enthusiastic developer. Who starts inserting (as yet) alien notions such as writing highly functional code, or implementing advanced type theory concepts (eg. scalaz) or starting to use a largely un-understood symbol soup (scalaz or the dispatch periodic table). This would make a mockery of the phased learning and force every one else to catch up (even if just to be able to read and maintain somebody else’s code).

We’ve long known hills are not conducive to agriculture. If these hills are our learning curves, fields could be considered to be directly impactful deployment of our skills. Agriculture generally happens flatlands and plateaus but not hillslopes. And yet we’ve learnt to tame the tall hills using terraces. Instead of climbing for a long time, climb a little & grow a little. If teams can learn for a month, and deploy these skills for the next 6-9 months with the cycle repeating itself, it could help in the following ways :

• Organisations do not need to invest in long training times where the opportunity cost of lost development time is one large fixed cost.
• Intra team disparity is likely to be contained since the deployment period will allow many to catch up with those in the lead
• Instead of scala being viewed as one big leap, it could be viewed as a incremental leaps with managed effort. I saw a remark which said people anyways deal with complexity – its called OO. And Java programmers deal with OO. So they should be able to come to terms with scala quickly. But there is a difference. Java wasn’t as big a learning curve leap over C++ as Scala is over Java. And as importantly, a learning curve one has triumphed over already, somehow seems far less onerous than an identically sized one, one still needs to deal with.

In short learning scala needs terraces. Terraces, where some developers will not run far ahead of the pack during the learning stages. Terraces where developers won’t have to feel too scared of alien’ish code suddenly appearing on their monitors. Terraces where developers will learn a few things and then deploy these skills over the next few months. These terraces could be made to work in different ways. It could just be a honour system using a set of levels identical or similar to the levels Martin Odersky proposed (though it would be extremely hard to ensure the same without tooling support). Or they could implemented using (as yet unwritten) tool like a scala-lint which will flag off advanced usages as warnings. Or as compiler switches the way Martin proposed.

There could be other ways to look at it too. Scala could be considered as a sharp weapon. Lethal in the hands of the trained. Self damaging in the hands of the untrained. So compiler switches could simply be a way to graduate from using sticks, to wooden swords, to blunt steel swords to the really sharp ones.

I saw a remark which said a compiler switch would divide the programmers. I seem to think such steps which help create terraces will divide the learning challenges. I also saw another remark which said, implementing compiler switches would play into the hands of languages like Kotlin or Ceylon or Fantom. I believe exactly to the contrary. The lower terraces can compete with Kotlin or Ceylon, and the upper terraces can provide a growth path that many other languages will not have. Not being able to plan a phased and an economically practical learning curve for large teams will make the case for other languages stronger.

Related posts:

1. Why Java folks should look forward to Scala
2. Contrasting Performance : Languages, styles and VMs – Java, Scala, Python, Erlang, Clojure, Ruby, Groovy, Javascript

Yet one point that keeps on bothering me time and again. That the lens used to visualise the many issues around these help clarify, magnify but lose the bigger picture. This post suggests a broader view to be applied.

Every argument refers to either improved benefits, lower costs or better contained risks. And in fact much of the arguments focusing proving the correctness of the code, or compiler ensured verification and thus the resultant guarantees, or the extent of unit test required to cover a wide array of differently typed inputs all boil down to risk.

The risk of code not behaving precisely like the way the author intended it to behave.

Yet there are other risks.

* The risk of spending too much time building something you eventually realise is either not required.
* The risk of not being able to provide rapid feedback to the product team, OR The risk of the customer not being able to partake and participate in your newest features to provide you critical inputs
* The risk of building a fantastic software, but a software that in hindsight needed to be very different.

When rewriting a legacy system, you often know exactly how you would like things to be rebuilt. When having adequate cash flows from alternate revenue streams, which can absorb product development costs, the life of your company may not depend upon the particular development in progress. When a part of a very large enterprise where anyways tons of internecine political empires rule the roost, a particular product / project development may hardly influence overall corporate risk.

In each of these situations – the risk of the software being developed is largely the same as the risk of the software getting delivered. Thats when risk is measured in terms of the software not meeting its feature, performance or delivery schedule goals. Thats when risk management becomes very important. Thats when provability, robustness etc. become important metrics for predictively containing risk.

Yet, not all environments have that luxury. Startups especially. Those that are dependent strongly on a piece of software – bet the farm on that software. Many actually have only a blur of an idea – but require tremendous market feedback to refine the vision. Yet others are operating on fixed constraints in terms of budgets before they decide whether the investments they made are worthy of further investments or whether to cut the losses. The risks here are very different.

In such situations – the risk to manage is not the risk of software correctly meeting its specifications. Its the risk of the software specifications not correctly meeting the market expectations. The real risk here is agility. Can you create the minimal functionality needed for feedback in the quickest manner. Can you act on the received feedback to adapt in the most competitive manner. In short can you iterate. Again. And again. And again. Fast enough before the market throws you out, or your competitor eats your lunch or your budget runs out. Work backwards from these goals. And you’ll find the right type system. And programming language. For you.

Programming languages and type systems are but vehicles to goals. Work backwards from your customer satisfaction perspective, your particular organisational constraints (or lack of them), your imperatives to iterate, and your time to market pressures. Work backwards from these to understand what exactly your company’s life depends upon. And of course factor in your current strengths (languages known / experience levels etc.). The right type systems and programming language should be relatively easy to decide.

And yes, provability is important. So long as you know exactly what you wish to prove first. The software or your business model.

Oh, lest you feel confused which type systems I prefer, I am ambivalent. They exist because they all have a role to play. But I do believe each context has one right choice. Just make the right choice for your context. And yes the programming languages I program in currently are Python and Scala. And yes, I have found in my experience dynamically typed languages help very substantially when fast TTM or iteratibility is needed (a view not everyone shares)

No related posts.

Disclaimer : Both I and Navin are on the organising team of PyCon India 2011. However we act in a volunteer capacity to help further encourage python and software development activities. We do not gain financially from this association. PuneTech is a blog specifically focused on encouraging technology developments in Pune.

PyCon India, the International Python Conference that happens in India every year, will be in Pune this year on September 16-18, 2011. Early Bird Registration – Rs. 300 (includes lunch, 3 days) is open until the end of the week. Register now! If you need convincing as to why you should attend Pycon, here are some reasons:

• Raymond Hettinger, one of the top pythonistas in the world is the keynote speaker. Raymond (@raymondh on twitter) is a Python core developer. He is the author of the itertools and set modules and most of the collections modules in the standard library, the peephole optimizer for Python, and dozens of ASPN cookbook recipes. It will literally be many years before you get a chance to hear a technologist of this calibre.
• Learn Python: This is your chance to learn Python. Start learning Python right now, and by September, you’ll be ready to get maximum value out of the tutorials in the conference (including Twitter/Facebook/Linked-in/Google Data hacking, web scraping, image processing, and functional programming using Python). If you need arguments on why everybody must learn python check here, here and here.
• Excellent Talks: There are 24 high quality talks, on all kinds of interesting topics including Data Analysis and Business Intelligence, Python-to-Javascript cross-compliation, Telephony apps, Robotics, Web Apps, Python in Biology and Life-Sciences, Cloud Computing, Android, testing, GIS, and much more. There is also one talk on using Python to do your homework.
• Meet Smart People: Even if you don’t agree that people who choose to work with Python are smarter than most others, you will have to agree that this will be one pretty darn interesting bunch of 500+ developers from all over India and outside. Rs. 300 to get a chance for that kind of networking is nothing.
• Hire Smart People: If you are having trouble hiring top quality technology talent for your company, you definitely need to be at PyCon, handing out your card, and telling everybody what a cool company you work for. Far better use of your time than going through resumes sent to you by your recruiter.
• Just Rs. 300: Early Bird Registration closes on 1st August, so act now. That’s only Rs. 300 for a high quality conference and it includes lunch and snacks for the 3 days of the conference. That’s right, you’ll be paying less than the cost of the food! And, unlike the other, regular tech events that happen in Pune, this is not a cheapo event – there will be swag – T-shirts and other stuff being given away. Did you realize that PyCon sponsors are paying for the privilege of giving you free stuff?
• Make PyCon Pune the biggest PyCon: Pune now has a reputation to keep up – whenever any tech event that happens in different cities, invariably, the biggest turn-out is for the Pune instance. PHPCamp with 1000+ registrations and 700+ actual attendance is probably the biggest ever tech unconference/barcamp style event in the country. DocType HTML5 in Pune had far more registrations than other places and the organizers had to close registrations. Recently GizmoMeet had their biggest turnout in Pune. The Python community in Pune is far younger than the Python community in Bangalore, so it will be tough for Pune PyCon to beat the Bangalore PyCon, but we definitely need to give them a at least a tough fight.

What are you waiting for? Register now

(We’d like to mention here that amongst the various sponsors of PyCon (including Google and GitHub), are these cool Pune companies/institutions: Venue sponsor: Symbiosis, Gold: Vayana, Silver: Druva and GSLab)

No related posts.

I think the cloud just became stronger. Even as many services go impacted, some didn’t. As an example Netflix was able to continue with its services. See Slide 33 in a recently conducted presentation of its architecture. I am certain there were many others who also were able to ride the storm. Since they had not just planned for redundancy of equipments but the architecture had accounted for disaster recovery (redundancy of locations). And many others who got impacted and put their customers through undeserved anguish still perhaps learnt a lot. A lot that they could’ve done differently.

What is interesting about “a lot that they could’ve done differently” is that cloud infrastructure as a service opens up a few more options to ensure high availability. Amazon AWS apparently has 5 regions and the US East region which got affected has at least four availability zones. It was one of that availability zone which got substantially impacted. Though in fairness the impact seems to have spread into other availability zones, something that shouldn’t ideally have happened. I am sure there probably were a lot of things AWS probably learnt and perhaps would do differently. But the same applies now to AWS customers as well. Some may choose to lose faith in the cloud. But many others might choose to realise that cloud infrastructures have the potential to offer so much more redundancy and options to high availability. Its just that one needs to realise that cloud data centers are not infallible, and after being aware of all the redundancy options, one just needs to design the right way to leverage them.

And how does one leverage them ? AWS has multiple availability zones. An application should ideally leverage at least two. If you read the Netflix presentation I referred to, Netflix apparently uses three. Do not assume the servers will not go down. Assume it is possible that at least one availability zone could go down. Make sure you have the systems to quickly activate, systems in the alternative availability zone. For that you will need to find ways to keep data current across availability zones. Also find ways to ensure you have the ability to quickly switch to and fro between availability zones. More advanced options could include concurrently active systems across availability zones or those spread across AWS regions or even between AWS and other vendors.

There’s little if any learnings here that are specific to AWS. They are indeed applicable in general to cloud based infrastructure as a service providers. But unless you are a part of a Fortune 500 or equivalent it is very unlikely that your internal infrastructure will offer as many options for redundancy as at least some of the larger infrastructure as a service providers could. Which is why I believe even if some may choose to switch back to the seeming comfort of private infrastructure, many currently using private infrastructure are likely to look at today’s events and realise that the public cloud actually offers so many better options for building highly available systems. The issues are not yet fully resolved. And many customers still perhaps are not being served. And while one hopes not, it could so happen that there could be further disruptions. But I find them really aspects of dealing with a learning curve as one transitions across class and scale of infrastructures. Ignoring the short term pain, and looking at it a little bit in the longer term, whichever way one really looks at it, I think the cloud just became stronger.

No related posts.

There are blog posts which take an essential thought or a message and by putting together various arguments bring it to some logical conclusion. This post is not one of them. It just describes the results of my thought process which started off contesting the assertion, to accepting it and then realising that either way, it really didn’t matter a whole lot. I don’t claim it to be sufficiently accurate except to state that it is accurate only to the best of my belief and understanding. Pointers to any inaccuracies or alternate interpretations in the comments shall be gratefully received. And yes, this post in the end reaches a somewhat boring conclusiion.

Having got that little expectation management out of the way, lets get back to monads, jQuery and Object Orientation. Writing a tutorial on Monads is a rite of passage for many. Thats something I’ve never done. And I don’t wish to, but I cannot escape the fact that I’m going to have to introduce monads to those who are as challenged in their understanding as I am.

What are Monads ?

I’ll give you the loose idea. Whenever a function completes any computation, the results of the computations are often returned as the return value of the function. But every once in a while you need a container around that return value. A container to store the state of a computation, sometimes with additional metadata or historical annotations for use later. At its very basis, a monad is a container for storing computations and being the container to be carried between functions. Think of it as a shipping container if you will, which moves materials between factories – the factories being various functions which operate on the contents of that container.

But it takes more for a monad to be a monad than just being a container. Given any content, the monad should be able to construct itself by wrapping itself around the content. Thats what the source factory invokes when it fills up the container with goods. Some prefer to call it a type constructor – the type here being the container, or the monad (which may internally contain goods of various types, but I get ahead of myself).

In addition once the container reaches the destination factory, unlike typical container, this one does not just allow its contents to be unloaded. Nay, that would make it too simple. In this case, the container has to be taken to a factory where it allows a robotic arm of the factory to plug into a receptacle it provides, which the factory can use to extract the underlying values, often one at a time. The factory further processes these goods, which again come out at the other end of the factory as a different set of containers. And it is quite likely that the type of the goods in the new containers could’ve changed.

That in essence is a monad. Easy, right ? I’m sure not and I’m now only going to make it a bit harder.

The wikipedia, definition of monad Monad

I’m just going to let you read the definition as provided by wikipedia.

A monad is a construction that, given an underlying type system, embeds a corresponding type system (called the monadic type system) into it (that is, each monadic type acts as the underlying type). This monadic type system preserves all significant aspects of the underlying type system, while adding features particular to the monad.
The usual formulation of a monad for programming is known as a Kleisli triple, and has the following components:

1. A type construction that defines, for every underlying type, how to obtain a corresponding monadic type. In Haskell’s notation, the name of the monad represents the type constructor. If M is the name of the monad and t is a data type, then “M t” is the corresponding type in the monad.
2. A unit function that maps a value in an underlying type to a value in the corresponding monadic type. The result is the “simplest” value in the corresponding type that completely preserves the original value (simplicity being understood appropriately to the monad). In Haskell, this function is called return due to the way it is used in the do-notation described later. The unit function has the polymorphic type t→M t.
3. A binding operation of polymorphic type (M t)→(t→M u)→(M u), which Haskell represents by the infix operator >>=. Its first argument is a value in a monadic type, its second argument is a function that maps from the underlying type of the first argument to another monadic type, and its result is in that other monadic type. The binding operation can be understood as having four stages:
   1. The monad-related structure on the first argument is “pierced” to expose any number of values in the underlying type t.
   2. The given function is applied to all of those values to obtain values of type (M u).
   3. The monad-related structure on those values is also pierced, exposing values of type u.
   4. Finally, the monad-related structure is reassembled over all of the results, giving a single value of type (M u).

In object-oriented programming terms, the type construction would correspond to the declaration of the monadic type, the unit function takes the role of a constructor method, and the binding operation contains the logic necessary to execute its registered callbacks (the monadic functions).

In practical terms, a monad (seen as special result values carried throughout the pipeline) stores function results and side-effect representations. This allows side effects to be propagated through the return values of functions without breaking the pure functional model.

The bind operation

So coming back to the container analogy, the container is the monad, the goods are the underlying types, and the factory is a function which takes a underlying type and spews out another monad which contains the newly manufactured goods. There is a difference to be noted, the container doesn’t go into the factory, its the factory thats plugged into the monad through the bind operation, which triggers the processing. Also note that the output of each such factory is not just goods, but containers of such goods it manufactures.

There, you can now imagine a series of containers and factories, or monads and functions which are sequenced to assemble a pipeline to produce the desired goods / computations.

Before we go on to how this works in OO, I would like to spend some more time on the function thats provided to the binding operation. This function accepts the individual contents of the incoming container and returns a container with the newly manufactured goods. A somewhat similar function is used when using map operations on containers eg. Lists. So if I had a list of integers, and I wrote a function which doubled their values, then the map operation would be written in python as :

This is a good example where the container (tuple) is unbundled inside the map operation, each constituent value of the list is passed to the twice function, and all the results of the twice function are reassembled into yet another container (list). The type of the constituents also changed from ints to str (string) along the way. To be a strictly monadic construct, the twice function should’ve returned not just the double of the value, but a double contained in some other container (either a tuple or a list), and the map function should’ve extract the individual strings from each of the individual tuples/lists and constructed a final tuple / list out of the same.

While unlikely to be confused as so, let me restate for clarity, the map function is not the monad. The tuple and the lists are the monads here. The map function is an example of the contextual capabilities that monads expect from their environment (eg. via and around the bind operator in haskell). And twice is the operation thats performed on the monad. Again to restate, the monadic chain is like a set of factories that spew out containers, which allow other factories to plug into them and in turn extract the contents and then spew out even more containers.

Monads and Object Orientation

So how does this work in an object oriented context ?

Lets take rules 1 and 2 of a monad. If I was to declare a class which had a constructor which took in a value and then wrapped it, then I would satisfy these two rules for being then be able to suggest that the class is a monad. However the rule 3 gets a little bit more interesting. Object oriented languages have the “.” operator which is somewhat analogous to the do block which can chain operations. So if I was to write o.foo() that would be equivalent of suggesting that I invoke the method foo() on the object which being a member method of the class has access to all the object internals and thus is able to access the wrapped value and do the necessary computations. Now if foo() were to return any object again of a class which satisfies these very rules, then I would be able to say that this class along with its member function foo() is a reasonable object oriented analogy of the monad. And I would be able to start chaining the methods as in o.foo().bar().baz()

Whoa! let me restate that again. If a class, has a constructor which takes a value and wraps that value, and has a number of additional member functions which each operate on these underlying values and return an instance of either the same object or another object of a class which follows the same rules, then I can say that that particular class is a monad. Incidentally thats exactly what fluent interfaces do, except that they do not have any specific expectation of wrapping. And a large number of classes may incidentally fit this description. And if jQuery is a monad, so are all of them.

Well, we did relax a few constraints along the way. First of all the functions are not stand alone functions. They are member methods which have direct access to the underlying wrapped value. Secondly they themselves don’t return a monad around an individual item in a collection. They return a monad around the entire collection of values. Thus the complexities and capabilities of the do block and the bind function are substantially simplified when using the “.” operator. And finally going back to the container analogy, the class defines and consists of both the containers and the factories. Seems like the threshold for stating a particular class is a monad is actually quite low. Turns out we reached a boring end. And it seems we are not much wiser at least in terms of any specific conclusions or insights. But every once in a while sometime the journey is more exciting than its end. For me this did seem like one.

Why are objects simple and monads complex ?

At least for me the above was true. Turns out objects collate related data and functions together into one class. Also the do block and bind operator on an object is very simple. So for many especially coming from the OO school, objects are well understood. On the other hand understanding the requirements of monadic constructs takes quite some time. So there’s a lot of gray cells that need to be exercised to start understanding what a monad is and how the various requirements for a monadic construct can be satisfied. And when mapping between monads, and objects which can also be seen to be monads, a lot of that complexity is either partially waived (eg. functions returning monads, the infrastructure around bind operators unbundling and bundling monads) or just simplified (functions and underlying values being colocated in the same class, the “.” operator being a far more simplified version of do block). But after some substantial headaches, it does start to seem that perhaps, just perhaps, monads aren’t so complicated after all .

So is jQuery a monad ? I believe one could choose to be very pedantic and point out minor issues with that assertion. Or one could accept that the intent of monadic sequences are well represented in jQuery chains and accept it as one. I started with the former and ended at the latter position. Would I express that jQuery is a monad ? To the monadically challenged – No. That obfuscates far more than it enlightens. To them I would say jQuery is a fluent interface which allows continuous chaining of operations on an underlying set of dom objects. And is there a big “Aha moment” when one realises that jQuery is a monad ? I couldn’t find one. While the journey of trying to understand monads and correlate it with monads was very exciting, at least the OO practitioner in me is unlikely to have missed much had I not known that. But I got to understand monads better – and thats well worth the time, and all the headspinning.

No related posts.

That experience taught me a good lesson. In attempting to deal with complexity, I was attempting to come up with the appropriate abstractions in a very bottom up way. So, strategic closure (of the Open Closed Principle), dependency inversion principle, et. al. all found their way into these abstractions I was modeling. These abstractions represented themselves via the various programmatic interfaces in the software design and their roles. At the same time there was a different perspective of the software. The way the end users would’ve preferred to see that software. The abstractions the way the end users saw the system were sometimes different than the underlying abstractions I modeled.

In a good design, the two abstractions should line up, and if there is an inconsistency, there is probably an issue with the design. Certainly this could be an issue in some cases. However in many cases the underlying backend might be built to offer far more capabilities than what are being exposed through the early version of the user interfaces. Probably there are multiple intents for which the software is being designed, and the particular user interface on table is just one of them. Probably, the underlying complexity of the back end is way too high which requires a very different nature of bottom up abstractions, which are different from the top down ones. Frankly, the reason doesn’t matter. Even after so many years, I look back and am comfortable with the thought that there needed to be two abstractions – one top down and one bottom up, one front end and one backend.

Fast forward many years. Another example helped me further attain some insight into the matter. We had a bunch of mobile smartphones floating around with the traditional PC UI abstractions being carried over into their design. Along came an iPhone – which rethought the interface the way users would’ve preferred to see the interface. Now iPhone was built on a traditional operating system. So it had the same abstractions that these operating systems have in the backend. However it decided to change some of the front end abstractions – in tune with the target market and the device / form factor peculiarities. Did the iPhone change the backend abstractions baked into its software – most likely not. However it changed the frontend abstractions, just enough to make the experience really simple and easy for its users.

As an engineer, I have lived with the regret of allowing backend abstractions to leak into the front end. But I have learnt something along the way.

• Don’t let the inapplicable frontend abstractions leak into the backend. This is especially true for most reasonably complex software. Strictly top down design can lead to a lot of brittleness in the long run, requiring very substantial surgery eventually.
• Don’t let the inapplicable backend abstractions leak into the frontend. In most cases this is a usability nightmare. ’nuff said.
• Realise that you have to simultaneously service independent expectations ie. robustness of usability and robustness of modeling. Work with both the abstractions to mould each of them well: Work your frontend abstractions well enough to mould them to reflect the use cases of the software in the most usable manner. Work with the backend abstractions to let them emerge from the problem space you are trying to resolve at the backend. Now make sure you work with both of them to map them into each in a reasonably smooth fashion. This is easier said than done. But doing it is whats necessary.

PS: I am not referring to what is conventionally referred to as leaky abstractions, where the implementation leaks into an abstraction. I am referring to a set of frontend and backend abstractions leaking into each other. However if all the backend abstractions were to be treated as an implementation, then this would be a case of leaky abstractions as well.

No related posts.

Why ?

Programming should be simple. And it should be accessible. And when I mean accessible I mean people with IQ of approximately 100 should be able to write programs. I am disappointed that many of the trends seem to raise either the minimum IQ or the training time required to gain competency. And while that helps a community of the super brilliant, it does not make a substantial difference to programming in general. It remains esoteric and does not stoop to touch everybody.

What ?

So what are the features of my preferred programming language :

• JRE support: Should run on the JRE with Java interop. Thats the dominant well engineered platform that runs across all classes of desktops, servers and devices. Additional support for CLR is a bonus but not mandatory.
• Simplicity: Should be fairly simple to read, learn and understand. Python is a good example. PHP is a great example (at a simpler class of problems). C++ and Scala not good examples.
• Multi paradigm : Should support both OO and FP constructs. eg. Scala and Python. Half hearted support to functional programming as with python discouraged. Ditto with passionate support for objects with second class treatment for functions as in Java/Ruby.
• Multi core compatible : Should have good constructs for leveraging multiple cores eg. erlang, scala, clojure and many others.
• Type inferencing : (and I am a python programmer ). Good type inferencing coupled with on the fly non intrusive/disruptive compilations as with say the eclipse on the fly compiler or play framework. Three cheers for Scala. One reason I prefer type inferencing to dynamic typing is the much superior performance even while maintaining brevity and removing boilerplate.
• Constructs that are natural to humans not mathematics : This is actually a sub point to Simplicity. The constructs should be consistent with the normal average non mathematically trained brains. 2 + 3 is much simpler to understand than (+ 2 3). Python rocks. Lisp / Clojure or for that matter brainfuck dont.
• Closures and code blocks : Love ruby for this.

Is this a pipe dream ?

For the moment seems so. Do you know of a language which helps meet these requirements ?

And to be very clear (because there is a substantial risk of the same) – this is no flame bait or an opportunity to trigger language wars. It is meant to highlight two things

1. There is no ideal language out there, and
2. When designing languages – make them simple to learn and use. ie. for a given problem statement a good language is one which requires the minimum talent or training to solve the problem

No related posts.

No related posts.

Sequences in python are not immutable :

When using sequences in python the thing to be noted is that sequences are not immutable. This provides you with the following options.

a) Use immutable sequence types : This is only possible by defining different types for sequences than the ones built into the language.
b) Ignore all methods on the sequences which modify them
c) Waive functional programming tenet of immutability

My preferred option when explicitly focusing on writing functional code is to use b)

Sequence types in python

The most commonly used sequence types in python are :

• Tuple : Immutable, Ordered, Indexed collection represented as (val1, val2)
• List : Ordered collection represented as [val1, val2]
• Dictionary : A key indexed collection represented as { key1: val1, key2 : val2 }
• Set : An unordered collection allowing no duplicates. Represented as set([val1,val2])
• str and unicode : While primarily meant to serv as ascii and unicode strings, these data structures also act as sequences of characters.Represented as “abcd” or ‘abcd’

Of the above only tuples are immutable.

There are other sequences used less often as well. An example is frozenset which is an immutable set.

Simple iteration
The for construct allows you to loop through a sequence. eg.

Iterators on dictionaries

Unlike tuples, lists and sets where iterators essentially traverse through the sequence constituents, there are a number of different iterators on dictionaries. These are :

Slices

Slices allow creation of a subset on a sequence. They take the syntax [start:stop:step] with the caveat that a negative value for start or stop indicates an index from the end of the sequence measured backwards, whereas a negative step indicates a step in the reverse direction. The following should quickly indicate the use of slices

The iterator protocol
Since python is an object oriented language as well, it provides strong support for allowing iteration over an object internals. Any object in python can behave like a sequence by providing the following :

a. It must implement the __iter__() method which in turn supports the iterator protocol
b. The iterator protocol requires the returned object, an iterator, to support the following ie. an __iter__() method returning self. and a next() method which returns the next element in the sequence, or raise a StopIteration in case the end of iteration is reached.

I’ve tested iterators which do not themselves have an __iter__() method and it still works, but I still do not give in to the temptation of not defining the next() method alone since that would be inconsistent with the documented python specifications

Let us examine a simple class indicating a range. Note that this is just for demonstration since python itself has better constructs for a range.

In the above example, __iter__() on Range returns an instance of a RangeIterator.

The way this capability is used is as follows :

The output you get with it is :

Generators

In the above situation, the state necessary for the iteration (ie.begin and end attributes) need to be stored. This was stored as a class member. Python also provides a function based construct called a generator. In case of a generator, the function should just do a yield instead of a return. Python implicitly provides a next() method which resumes where the last yield left off and implicitly raises a StopIteration when the function completes. So representing the above class as a function,

I do not document the results since these are identical to the earlier code using a RangeIterator.

Note: I used my_range() and not range() since there already exists another range() already provided by python

An interesting point to realise is that in both the above situations, the next element to be returned in the sequence was being computed dynamically. To put it in the terminology better consistent with functional programming, it was being lazily evaluated. While python has no mechanism of lazy evaluation of functions, the ability of functions or objects to lazily evaluate the return values are sufficiently adequate to get the benefits of lazy evaluation at least from the perspective of cpu utilisation only on demand, and minimal memory utilisation.

List comprehensions

List comprehensions are one of the most powerful functional programming constructs in python. To quote from python documentation (even as I leave out a very important part of the full quote .. to be covered later),

Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses. The result will be a list resulting from evaluating the expression in the context of the for and if clauses which follow it. If the expression would evaluate to a tuple, it must be parenthesized.

A sample list comprehension is one which returns a sequence of even numbers between 0 and 9 :

The output one gets on running the code above is

Note that the list comprehension returns a generator which then returns a sequence containing 0, 2, 4, 6 and 8.

The for and if statements can be deeply nested.

lambda

Lambdas are anonymous functions with a constraint. Their body can only be a single expression which is also the return value of the function. I will demonstrate their usage in the next section.

map, filter and reduce

Three of the functional programming constructs which probably have aroused substantial discussions within the python community are map, filter and reduce.

map takes a sequence, applies a function of each of its value and returns a sequence of the result of the function. Thus if one were to use map with a function which computes the square on a sequence, the result would be a sequence of the squares. Thus

results into

I mentioned earlier I will demonstrate usage of a lambda. In this case I could use a lambda by defining the square function anonymously in place as follows :

filter takes a predicate and returns only the elements in the sequence for whom the predicate evaluates to true.

Thus

results in the following output

Finally the most controversial of them all – reduce. Starting with an initial value, reduce reduces the sequence down to a single value by applying a function on each of the elements in the sequence along with the current manifestation of the reduced value. Thus if I wanted to compute the sum of squares of numbers 0 through 4, I could use the following

In the above example, 0 as the right most argument is the initial value. range(5) is the sequence of numbers from 0 thru 4. Finally the anonymous lambda takes two parameters – the first is always the current value of the reduced value (or initial value in case it is being invoked for the very first time) and the second parameter is the element in the sequence. The return value of the function is the value which will get passed to the subsequent invocation of the reduce function with the next element in the sequence as the new reduced value. The reduced value as returned finally by the function is then the returned value from reduce

The functional programming folks are likely to find this an extremely natural expression. Yet reduce resulted in a substantial debate within the python community. With the result that reduce is now being removed from python 3.0. (Strictly speaking it is being removed from python core but will be just an import statement away as a part of the functools package). See The fate of reduce() in Python 3000. Why so controversial ? Simply because the usage of map, filter, reduce above could be rewritten as

Just look at the two blocks of code and figure out which is easier to read (especially look at reduce). One of the areas where a large number of pythonistas and lispists are likely to disagree is the tradeoffs between brevity and easy readability. Python code is often english like (well, at least to the extent that a programming language can be) and some pythonistas do not like the terse syntax of lisp thats hard to follow.

I earlier mentioned that I left out a part of the description of list comprehensions from python documentation. Here’s that part of the quote.

List comprehensions provide a concise way to create lists without resorting to use of map(), filter() and/or lambda. The resulting list definition tends often to be clearer than lists built using those constructs.

Having said that I do believe many including myself will continue to use map, filter, reduce

Other helpful functions in python core that are helpful for functional programming are :

• all : Returns True if all elements in a sequence are true
• any : Returns True if any element in a sequence is true
• enumerate : Returns a sequence containing tuples of element index and element
• max : Returns the maximum value in a sequence
• min : Returns the minimum value in a sequence
• range : Return a sequence for given start, end and step values
• sorted : Returns a sorted form of the sequence. It is possible to specify comparator functions, or key value for sorting, or change direction of sort
• xrange : Same as range except it generates the next sequence element only on demand (lazy evaluation) thus helping conserve memory or work with infinite sequences.

We’ve seen many of the constructs that are typically useful for functional programming. I left out one big part – the itertools package. This is not a part of python core (its a package which is available with a default python installation). Its a large library and substantially helps functional programming. That along with some more sample python programs shall be the focus of my next blog post in the series. At this point in time I anticipate at least a few more parts after that to focus on a) Immutability b) Concurrency and c) Sample usage.

Hope you found this useful and keep the feedback coming so that I can factor it into the subsequent posts.

No related posts.

Note: Google Buzz had not yet been enabled for me through normal GMail Desktop access. These notes are based on my experience with the same by switching my Firefox user agent to iPhone and accessing the iPhone Google Buzz UI.

• Has a very facebook / friendfeed like interface.
  • Comments : People can enter comments against a post which appear sequentially below the post.
  • Like : Has the Like feature also supported by facebook / friendfeed
  • Inline images and videos : Images and videos appear inline as a part of the stream.
  • No 140 character limitation : Or at least none that I observed
• Each conversation becomes a web page. eg. the comment stream against my very first google buzz post.
  • The same conversation is also posted to your inbox. The conversation in the inbox gets dynamically updated as the conversation continues. A conversation that has been read earlier gets marked as unread when more comments are entered against it.
  • The conversation web page becomes an alternative location for participation. If say the conversation link is forwarded to others, they can immediately comment on (or choose to “Like”) the conversation from the web page itself.
• Private Channeled Messaging. You can mark posts as private but directed to a group of people. These groups are maintained and managed using Google Contacts. To the best of my understanding the future conversation is also constrained to the same group. Such messages are obviously not visible publicly. This is an excellent feature for closed group interactions.
• Directed messaging is supported using the familiar twitter ‘@’ style. However in this case you use ‘@’ followed by the entire email address. Any posts made with ‘@’ addressing also get delivered to the user’s inbox.
• Geolocation integration is supported. Thus you can view the public posts being made by people nearby you. Since I used firefox, it used firefox for geolocation support. I imagine and presume it would be using alternative superior mechanisms for mobile based conversations.

Comparison to Twitter :

One of the reasons I test drove Google Buzz was since I was wondering if it was a better Twitter. In many ways it is. But I don’t think thats likely to be universally true. There is the 140 character simplicity to twitter which is its identifying feature. In many ways Google Buzz is a Friendfeed clone – not a twitter clone (with added inbox integration). I always found Friendfeed much superior to twitter for conversations and I suspect those who shared that perception will find Google Buzz far superior to twitter too. However it will run into the same difficulty that Friendfeed did. If the twitter user doesn’t correlate his twitter account to the friendfeed account, it is very laborious in friendfeed to add a user you follow as an imaginary user in order to have his tweets appear in your friends stream.

I think where Buzz will take on Twitter .. and big time is the really large number of people who sign on to twitter, maybe make a few tweets and then say I don’t get it and go away. I suspect most of them will “get it” on Google Buzz – especially if they have any Facebook experience (which is a large large set of people). In that sense while Twitter appealed to only a small set of people, Buzz at least has the potential to appeal to the mainstream .. and in that sense go for the jugular – Facebook’s that is.

Comparison to Facebook :

Since facebook over a period of time has cloned its update stream to be like that of Friendfeed, it should come as no surprise that the Buzz stream is likely to look and feel quite similar. However two big differences stand out. The first is the way the social graph is managed. The social graph unlike facebook is asymmetric. ie. two persons don’t have to agree to be friends to follow each other. In case of Buzz, the social graph is managed through google contacts. You can follow people who have public google profiles or follow anyone in your contacts. Thus if a person doesn’t have a public google profile or his email address is not known to you, you may not be able to follow him/her. You don’t need the other person’s permission to follow. (At least thats the impression I formed, though I could stand corrected). The other big difference is the email integration. The entire stream and conversations are integrated into the gmail account. For most knowledge workers, I think these differences are likely to make for a more positive experience. In that sense – buzz is extremely well suited to be the Facebook of knowledge workers. Whether it can compete with the trivial, frivolous, chatty social networking capabilities of Twitter or Facebook is something I couldn’t form a clear opinion on.

On carving a new market :

Based on my earlier statements, it should be obvious that I believe buzz will carve new markets which did not use Facebook / Twitter / Friendfeed earlier. These markets are :

• People who tried twitter but didn’t get it
• People who primarily use email for communications instead of facebook / twitter (for both Personal and Business Networking)
• People who would like to use Facebook / Twitter like capabilities from a knowledge working / business perspective.
• People who are just too nervous to use anything new on the net may still find the email integration a continuum of their email exchange rather than a ‘new web site’.

On Privacy :

I think Buzz definitely ranks superior on privacy, given its capability to conduct private directed closed group conversations. As an example, I can imagine many google apps installations which could use buzz for intra company conversations even as they clearly restrict even the public streams not to be visible outside the domain. While I haven’t seen any comment from Google on whether buzz would be available for google apps, I imagine its only a matter of time since it is so particularly well suited for that environment.

On google wave :

Google wave in many ways was a damp squib. It imposed a new usability paradigm even as some of the capabilities such as near real time updates were clearly superb. Google buzz reverses the wave approach by offering a different conversation model which dovetails not only into similar usability expectations as defined by the competition but goes one step beyond by integrating into the killer web application – web mail. I wouldn’t be surprised if it came to light later that Google buzz is a google wave backend with a completely new usability experience.

A note on integration :

While its only a statement of intent at this stage, google has setup a site for the Google Buzz API. However the promise is awesome in terms of multi standard support for variety of standards and protocols support including RSS, Atom, PubSubHubbub, Social Graph API, OAuth, WebFinger and Salmon. It does seem that google will want to support a variety of other alternative and mashup use cases around Google Buzz which might allow for more interesting uses to emerge (as it happened for Facebook and Twitter). Clearly a space to watch with some interest in months to come.

In summary :

Google buzz makes the strongest case for appealing to the power email users and those particularly focused on privacy (eg. knowledge working for businesses). Its strong differentiator is its mail integration. It does offer adequate features to compete with Facebook / Twitter for their existing markets as well. And joins the party with a large number of people – who already have a google account.

No related posts.

Please note : The intent of this post is not to compare LOC of python with other languages. It is meant to demonstrate python code implementation which helps describe the capabilities of python in terms of writing clean, concise, readable code. This post continuously refers to other posts and may require you to read the referred posts in conjunction with this post to get maximum value.

Code and ceremony

The post Java vs Clojure – Lets talk ceremony! describes a simple problem which requires 28 LOC in Java and 1 or 9 in clojure. It is a simple logic which creates a list, populates it with four values (two of them identical) and creates a list with unique items. There are two alternative clojure implementations, one in 1 line of code and another in 9 – the former using a built in construct in the language which especially helps to write a more concise logic, and the latter by writing code similar to the Java code. Without any further ado, here are the two alternative python implementations

a. By using built in capabilities – In this case we use the set collection

b. Now here’s where we will not use a built in operator which automatically chooses the unique elements but again write code similar to the java code. This code is as follows :

There – both of them are exactly one line long.

Project Euler solutions
The post Python vs Clojure – Evolving refers to many sample python implementations solving some of the problems documented on project euler site. We shall revisit these python implementations here.

a. Find all numbers dividable by 3 or 5 in 0 < x < 1000:
In this case the clojure code is 4 lines long where as the python implementation is 3 lines long

Again the implementation is exactly 1 line long – no lambdas, no filters etc.

b. Get the sum of all even Fibonaccis below 4 mill.

The suggested python implementation is about 17 lines long. The implementation below 7.

c. Finding Palindromes
The suggested python solution uses about 23 lines of code, to 6 in clojure and takes 15 seconds almost 300% slower than the clojure code (emphasis from the post referred to).

Here’s another alternative implementation.

Again if you notice – exactly one line of code. And incidentally on my notebook it clocks in about 0.85 seconds which makes the clojure implementation about 470% slower than the python implementation. (emphasis mine)

Top Rank Per Group

In yet another post Python vs Clojure – Reloaded, the author discusses the Rosetta Top Rank Per Group implementation of python. Unsurprisingly the python implementation uses 11 lines of code (not including the initialisation) compared to clojure’s 6.

One of the commenters suggests an eminently readable and simple python solution in about 6 lines of code. The code snippet is as follows

An alternative far less readable solutions which shaves off one more line but certainly not preferred to the solution above (am just listing it since it shaves off the step of having to create an intermediate data structure) is

Some might wonder whether I am joining multiple lines of code into one just to reduce the line count. The code is consistent with the idiomatic usage of python and also with the way python list comprehensions are documented in the formal python proposal for list comprehensions PEP 202 : List Comprehensions. Besides python is a whitespace sensitive language and the compiler disallows multiple lines of code in one line without using the ‘;’ separator which I have most definitely not used in the suggested solutions.

PS: There’s of course an interesting image at the end of the post Python vs Clojure – Reloaded indicating the perception of how clojure code speaks for itself . The reason I mention it is that I believe if one is expressing a strong opinion as the one reflected by the picture, one better be far more careful and thorough with verifying the veracity of the assertions appropriateness of the samples compared to. Let me clearly state that I do not know that the clojure loc compared to is an appropriate target so do not infer this post to be a Python vs Clojure LOC as much as a post which emphasises that Python is a good vehicle to write concise readable code. FWIW I am learning clojure and look forward to building more competence with it.

Update: There is another nice post by Stephan Schmidt – Anatomy of a Flawed Clojure vs. Scala LOC Comparison which does reflect an opinion in the context of Scala and a post authored on the same blog as the ones I refer to above.

No related posts.

Its probably a function of style. I like to write a piece of code and then keep on testing it against different scenarios and often keep on remoulding it substantially. My focus is at this stage on the code structure – the design aspects. Trying to do it with TDD always created a mess. The amount of intense refactoring I do always meant I soon got tired of also simultaneously refactoring the test cases and gave up on them soon.

Anyways, I finally think I found a formula that works – at least for me. I do not write production code in the first pass. I just write prototype code. I keep on playing with it, shaping it, challenging it, cajoling it, recasting it. Until I’m satisfied with the overall structure and internal design. Now when thats done, I rewrite it. Completely. Umm, I copy/paste maybe 40-50% of the code. But this time I write it on a completely new fresh set of files. And I write the test cases before I write the code. I find I can now really write tons of test cases and code quite rapidly. And with the tremendous satisfaction that at the end of the day, I now have the control harness to be able to keep on adding new features without having to continuously worry if I broke something else. With substantially detailed test cases, now I can actually feel quite confident that if I did break something – I’ll get to know it. Not next month, not next week, not even tomorrow – in the next few minutes.

So this approach seems to be working for me – the initial efforts have delivered great results and I’m quite satisfied. Now just wondering what to call it – for lack of a better term I shall term it “Rinse and Repeat with TDD”.

So if TDD works for you – great. You have one of the most important coding disciplines nailed down. But if you’re wanting to try to do TDD but haven’t been able to successfully implement it – you may just want to check out the Rinse and Repeat with TDD style. Who knows, you might find it useful too.

No related posts.

1. REST : The Representational State Transfer architecture style builds on the essential elements of those constructs which made the internet so globally scalable. A detailed explanation of the rationale and strengths of REST are completely beyond the scope of this article. If your job requires you to be continuously aware of emergent trends and whether they fit your enterprise architecture needs – this is the one must explore trend.

   Impact : Web based architectures, Service Oriented Architectures, wide availability and immediate usability of data and processing requests (resources) through simple HTTP URIs and minimal integration effort

2. Interoperable Cloud : The interoperable cloud is the ability to create a private cloud and also leverage a public cloud. This has been made possible by offerings such as the Ubuntu Enterprise Cloud which allows you to build a private cloud or use a public cloud such as Amazon EC2 while being able to access them using the same set of APIs thanks to open source efforts such as Eucalyptus. This allows you the flexibility of initially using either a private or public cloud and then subsequently shifting to the other, or being able to use both simultaneously.

   Impact : Large servers vs cluster of commodity servers, virtualisation, elastic deployments, flexible hardware procurement / provisioning, infrastructure management in organisational hierarchy.

3. NoSQL : While I am unhappy with the name, it has stuck. This refers to a set of options now available to store your data unconstrained by many RDBMS requirements (eg. flexible schema, key value pairs etc.). Some of the databases also allow you to store data in a distributed manner over a number of servers with an intent to support high availability in write intense scenarios even as they may require you to move towards eventual consistency. These options increase your manouverability / flexibility as an architect even as they require you to meet a different set of challenges.

   Impact : Relational databases, data storage strategies, data distribution strategies, vertical vs. horizontal scalability, transactionalisation, consistency and availability

4. Polyglotism : Developer costs now occupy an increasing percentage of total costs, development time is being an increasingly dominant factor for time to market, and ability of software to change and adapt quickly to newer demands is now a critical success metric. One of the solutions is to write different parts of the software in a different languages most appropriately suited for concise and rapid coding as well as supporting quick reaction changes to each part appropriately. Thus it is conceivable to have some of the business rules written in a dsl written using jruby and some of the algorithms written in clojure in a software built on the JEE platform.

   Impact :Development culture and processes, minimum developer skill and scalability, risk management for managing required vs. available skills.

5. Decentralised processing : Thanks to many developments which are leading to increasingly distributed processing including REST and NoSQL, applications will need to be a set of collaborating network based components (we’ve heard this before with distributed objects as well). However especially given some of the lesser guarantees that such architectures can provide around immediate guaranteed processing, latency issues, distributed control and asynchronous processing, a particular piece of business logic may get satisfied in a staggered fashion across a number of collaborating components. This may increase challenges in terms of currency of available data even as it helps actually deliver on the vision of distributed objects and simplifies individual component development. While asynchronous capabilities such as those supported by MQ series and the like have been used in the enterprise for ages, I do anticipate increasing use of lighter messaging constructs such as PubSubHubbub within the enterprise.

   Impact : Application partitioning, network based components, difficulty in supporting fully synchronous workflows.

No related posts.


]]> http://blog.dhananjaynene.com/2009/11/five-important-trends-on-the-enterprise-architects-radar/feed/ 1 http://blog.dhananjaynene.com/2009/11/five-important-trends-on-the-enterprise-architects-radar/?utm_source=rss&utm_medium=rss&utm_campaign=five-important-trends-on-the-enterprise-architects-radar http://feedproxy.google.com/~r/var/log/mind/~3/d_mJprNs8Ak/ http://blog.dhananjaynene.com/2009/10/service-oriented-architecture-is-primarily-about-business-and-not-technology.-bollocks/#comments Mon, 26 Oct 2009 10:56:51 +0000 Dhananjay Nene http://blog.dhananjaynene.com/?p=902 There’s quite a few times I’ve heard / read a gross oversimplification of architecture in reference to business and technology. And while I believe I understand the ‘essential cause’ which drives such a simplification, I’ve often felt quite frustrated at the resultant impression thats provided by such a simplification. In many ways and forms, it boils down to the statement (not exactly the same since I’m not quoting directly), quite similar to the one below :

Service Oriented Architecture is primarily about business and not technology

This also reflected in the recent SOA Manifesto which states as its very first described value :

Business value over technical strategy

Allow me to straight away start picking some holes into this :

1. Anything that a business does – whether it is soa, software architecture, building architecture or simple plant and machinery design, to the extent (which is exactly 100%), technology serves the business goals, all technology activities (and non-technology as well) are at the end of the day about achieving business objectives and therefore about business. So why single out architecture? And even more so why single out SOA?
2. Architecture is also about business. But its not the same as saying its primarily about business and not so much about technology. For a moment lets step away from Software/Hardware Architecture and look at Building Construction Architecture. The legendary creation of Ayn Rand – Howard Roark, for all his eccentricities and seemingly portrayed egocentric and egotistic behaviour did meet the test of business objectives to the extent of making the residents of his creations extremely satisfied. And at no point would you gather the impression that he in any manner put construction technology to any secondary position to his business context and objectives. At the end of the day thats what architecture is. It is not about making one of business or technology more important than or subservient to other. Its about effectively mapping the two to provide a strong technology solution appropriate to the business needs.

I suspect one of the important causes here is that people have forgotten that the A in SOA stands for architecture, and therefore shorn of architecture, business and technology can be seen to be competing in a non win-win form.

So if SOA stands for Service Oriented Architecture, then I must submit that architecture is the art of getting the two working together. And I am of the opinion that an exercise suggesting one is more important than other is an exercise in a field unrelated to architecture. I suspect many practicing software architects will agree with this. I suspect Ayn Rand wouldn’t disagree as well.

No related posts.


]]> http://blog.dhananjaynene.com/2009/10/service-oriented-architecture-is-primarily-about-business-and-not-technology.-bollocks/feed/ 5 http://blog.dhananjaynene.com/2009/10/service-oriented-architecture-is-primarily-about-business-and-not-technology.-bollocks/?utm_source=rss&utm_medium=rss&utm_campaign=service-oriented-architecture-is-primarily-about-business-and-not-technology.-bollocks http://feedproxy.google.com/~r/var/log/mind/~3/lIqzzGZmSrI/ http://blog.dhananjaynene.com/2009/10/stop-calling-me-nosql/#comments Thu, 22 Oct 2009 23:26:09 +0000 Dhananjay Nene http://blog.dhananjaynene.com/?p=884 Dear Reader,

Apologies for sending this note to you completely unannounced and out of the blue. However I find myself in a peculiar situation of having a very weird name being dumped upon me. While I am indifferent to the name per se, I am greatly pained as I realise that it is a completely inappropriate name. What is even more confounding is the very bunch of people who have happily assigned me the name and continue to popularise it belong to that class of people some of whom actually are extremely particular about accurate nomenclature and have no hesitation in creating a 100 letter class or function name by concatenating 20 words just to make sure the name is unambiguous and conveys the intent clearly.

Ahh.. but I digress and impose upon you without introducing myself adequately first. I am a data storage style. I am not new, but lately far too many a software engineer have started taking a liking for me. Ever since I have been around, I have with great amounts of jealousy watched my cousin the RDBMS being courted by the finest of engineers (in all honesty there were some fine engineers interested in me too, but far too few compared to my cousin). But lately multiple concurrent developments have made a fair amount of attention come my way too.

You see unlike RDBMS, I don’t require that data be clearly split into tables, columns and rows. I can work with data the way it is most naturally represented. As a tree of individual data fields, lists, arrays, dictionaries etc. Also I do not require that you always clearly define each and every possible schema element before being able to store data corresponding to the schema. I can happily accept a schema dynamically or even work without a schema. Some of my early forms were based on key value pairs stored as B-Trees (eg. Berkeley DB). Over the years people have figured out ways to represent the data as a set of decomposed document elements, store data spread across a cluster, replicate it for better availability and fault tolerance, and even perform post storage processing tasks using map-reduce sequences. But really what separates me from my cousin and other storage systems is that I don’t make demands on the data – I take it in its naturally found form and then store it, replicate it, slice it, dice it and glean information out of it. And therein lies my true identity – I will work with data the way the data is best represented with all its arbitrary inconsistencies and inabilities to always clearly specify a constraining schema. And the engineers who’ve spent time with me seem to have enjoyed it quite a bit.

But the horror of it – they gave me a completely inappropriate moniker – ‘NoSQL’. First and foremost I exist to promote a storage style and thats what identifies me. I work with data in its natural and arbitrary forms. Therefore to make it seem like I represent a lack of something else is utterly missing the point. The SQL in NoSQL stands for Structured Query Language, which depends upon Fixed Structure Relational Data. Since I change the very nature of the data being stored, that SQL is not required or relevant is automatic and inconsequential.

Its like calling a under-the-ocean-mountain_range as NoIgloo. Its dead obvious igloos will not be found there. But calling that mountain range NoIgloo is a big disservice to visitors. You use that as a marketing term, attract people, then tell them that NoIgloo actually has nothing to do with Igloos – its got to do with mountains and oceans, and that they need to first unwind all the confusion they created in their minds due to NoIgloo and then go through a phase of reunderstanding mountains and oceans. And while they came prepared for a possibly warmer place given the name NoIgloo – it actually is a wet place so they need to again change their garments and equipment for the journey. A wholely avoidable situation.

Update: Brad Anderson pointed out this interesting post NoSQL: A Modest Proposal which traces the genesis of my name which leaves me very very disappointed. Almost seems to suggest that people are flocking together and naming me not based on something inherently powerful about me – but as a mechanism to demonise my cousin RDBMS. This is most unfortunate, since we actually end up being useful in very different situations and more often than not are likely to complement each other rather than compete with each other. I do hope a better moniker does prevail over time

What I would like is to see a better / more appropriate name for me. Hmm .. call me free form storage, natural persistence or flexi schema storage or perhaps something else even more appropriate (this blog owner prefers “natural persistence”). Each of these conveys far more about me far more accurately than NoSQL does. Basically please please call me something better than NoSQL. So can I request you to carry forward my plea by further forwarding and retweeting this to your friends and ask them how they can so callously call me by such a silly name when they take the utmost precautions in properly naming their classes and methods. Plead with them to stop doing this and please work with others to give me a better name. I think it will cause less confusion over the coming months and years, and the field of software shall recover its glorious tradition of maintaining precision in communication by using accurate naming.

Sincerely,
The one who doesn’t want to be called NoSQL

PS : As a background to this there was an interesting conversation earlier today between this blog owner dnene and Kent Beck on twitter, where Kent so kindly and graciously helped carry forward the thought process of helping identify my essential characteristics, and it is in no small part, thanks to this conversation that I was able to articulate myself and my grief. I reproduce that conversation below. (Update: though in all likelihood Kent’s intent was to help clarify the thought rather than contest the names. In hindsight, it makes sense to ask for permission to reproduce conversations .. even when such are on the public twitter stream – something that wasn’t done in this case. )

Twitter ID	Tweet / Message
dnene	NoSQL is such an inappropriate name. NoTables at least makes a little more sense.
KentBeck	@dnene but what would nosql be called if you wanted to say something positive about it?
dnene	@KentBeck Thats a great question .. still thinking .. best thought so far – FlexiStore (though not good enough yet )
KentBeck	@dnene what can you do with a nosql store that you can’t do with an sql database? why would you be excited to use one?
dnene	@KentBeck I see where u r going with this (a) unconstrained & composite storage (b) store resources not records (c) shard/scale horizontally
	.@KentBeck I think there is a merit in attempting to define nosql in terms of what it is rather than what it isn’t
KentBeck	@dnene there are many more people confused about what datastore to use than who hate sql. the positive approach appeals to the former.
dnene	@KentBeck Agreed .. and I’m aware of many more who wonder why we need a different datastore than the RDBMSs. NoSQL as a name doesn’t help.
KentBeck	@dnene well, why *do* we need a different data store?
dnene	@KentBeck Primary Need : We need support for flexible/arbitrary schemas with complex depths – RDBMSs don’t dance well in this space.
	@KentBeck Secondary Need : Support for deferred processing required for analytics (eg. Map/Reduce). RDBMS don’t do too bad a job here
	@KentBeck Tertiary Need (not one that I’ve felt strongly yet) : Distributed and horizontally scalable storage on commoditized h/w.
KentBeck	@dnene it seems like you’re looking for realistically structured data, not data twisted to fit a formula convenient for mathematicians.
dnene	@KentBeck Yes.. thats it! I’m looking for realistically or naturally structured data storage / persistence. Rocks compared to the term nosql
	@KentBeck Wonder if the term arbitrarily structured makes sense as well. This has been one heck of a conversation/Q&A so far +1:)
KentBeck	@dnene glad you found it helpful. you get bonus points if the opposite of the name you pick is unattractive, a la “structured programming”

No related posts.


]]> http://blog.dhananjaynene.com/2009/10/stop-calling-me-nosql/feed/ 18 http://blog.dhananjaynene.com/2009/10/stop-calling-me-nosql/?utm_source=rss&utm_medium=rss&utm_campaign=stop-calling-me-nosql http://feedproxy.google.com/~r/var/log/mind/~3/7R4sW1986Xo/ http://blog.dhananjaynene.com/2009/10/nosql-a-fluid-architecture-in-transition/#comments Wed, 21 Oct 2009 18:21:14 +0000 Dhananjay Nene http://blog.dhananjaynene.com/?p=867 Lot of talk about NoSQL. Much of it well deserved. And while lot of the excitement around it is well understood by those in the know, some of it may actually be confusing to those who are relatively new to the matter. This post is actually for the latter group – not to argue for or against NoSQL – just to put it in perspective.

What is NoSQL : Some of the characteristics shared by most if not all the NoSQL engines are as follows :

1. Schemaless or Hierarchical Schema Storage NoSQL assumes at its very basis a schemaless or a hierarchichal schema storage system. In most cases this consists of a simple key value pair storage. While some storage engines excel at storing small values (LightCloud/Tokyo Cabinet), some are strong at storing large documents (CouchDB).
2. Distributed storage : This is one of the driving forces of NoSQL growth, though not a distinguishing characteristic of NoSQL. One of the areas these storage systems separate themselves from RDBMS’s is their ability to allow better horizontal scalability. This varies from the simple master-master replication for MongoDB, to multi node sharding using consistent hashing with LightCloud (a la memcached) to a multiple master eventual consistency model of Riak. The basic premise in using some of the NoSQL engines is that storage will scale horizontally.
3. Support for deferred processing :Many of these engines allow for some degree of deferred processing. Whether this be simple lua scripting in case of LightCloud or map-reduce scripts in case of CouchDB, the general assumption is that some amount of latency in computation times is acceptable, and some of the computations (especially related to analytics based views) will be performed post storage.
4. Eventual Consistency : This may seem like a necessary feature of all NoSQL storage systems but it isn’t. While clearly some such as CouchDB (in terms of its map-reduce views) and Riak are better placed for supporting and implementing eventual consistency, it is quite feasible to use others such as LightCloud or MongoDB to implement immediate consistency using a single master-master pair. Suffice to note that eventual consistency is not a necessary side effect of using a NoSQL storage system, though it wouldn’t be incompatible for the two to work together.

But the points I would really like to emphasize are :

• NoSQL is not a direct competitor to RDBMS/SQL : It is actually a solution to many use cases where using RDBMS was perhaps a poor fit. Thus the decision for an architect is not which of the two competing options (RDBMS or NoSQL) Update: one should selectshould be the preferred standard storage strategy, – it simply is which one is the more appropriate storage system for the application under consideration.
• NoSQL is still at a fluid stage of its development : All the NoSQL storage systems (but for LightCloud/Tokyo Tyrant) are still being quite actively developed. These have not reached v 1.0 (Update: MongoDB is at v 1.1) and it is likely that some time will pass before any of these get beyond the beta and release candidate stage and get a 1.0 in-production stamp. While there is a lot of interest, there still is a substantial amount of experimentation in terms of the right feature sets leading to differently focused developments in different storage systems. To an architect this represents an interesting challenge. I think the way to approach this right now is to not use these in mission critical (eg. life or health impacting) systems, and to focus on reasonable expectation management in terms of ensuring the right kind of SLAs around their availability (simply because many of these haven’t yet been put to intense use in production the way say an Oracle or MySQL have been). This is not an attempt to spread “FUD” about NoSQL – far from it it is an exercise in setting appropriate expectations. i would also recommend that it would be appropriate to evaluate the available NoSQL choices only when reasonable SLAs can be worked out for their usage. It is certainly preferred to using NoSQLs rather than using RDBMS’s in an inappropriate manner (large objects serialised into BLOBs or into name-value pair tables). However, I would suggest that you do not deeply bind yourself into a particular NoSQL engine. The future development of most of the storage systems is still unknown to a certain extent, as is the future landscape including any shakeouts. Should one recommend usage of a NoSQL engine – it is important to have a clear plan for switching over to an alternative engine should a need arise in the future. While this is easier said than done, deciding the appropriate level of abstraction to use (ie. code to the API directly vs. use a layer of abstraction for engine independence) is best left to designer / architect to dwell upon.

No related posts.


]]> http://blog.dhananjaynene.com/2009/10/nosql-a-fluid-architecture-in-transition/feed/ 0 http://blog.dhananjaynene.com/2009/10/nosql-a-fluid-architecture-in-transition/?utm_source=rss&utm_medium=rss&utm_campaign=nosql-a-fluid-architecture-in-transition http://feedproxy.google.com/~r/var/log/mind/~3/rDQBoHSlFSg/ http://blog.dhananjaynene.com/2009/10/configuring-a-secure-ubuntu-linux-virtual-private-server/#comments Mon, 05 Oct 2009 13:56:20 +0000 Dhananjay Nene http://blog.dhananjaynene.com/?p=852 This post is based on my notes for an initial configuration for an Ubuntu 9.04 Virtual Private Server with a focus on security. At that time I searched for a number of references on security, and while I have not kept the note of all their URLs, most of what I write below is as a result of other documents even though I cannot specifically cite them (in other words, there is little originality except perhaps for attempting to cover the entire gamut of configuration activities into one article).

Keep in mind that these steps are based on my notes which might be a little incomplete especially around the part where acidbase is installed.

• Initial Configuration : Basic configuration when getting started up.
• Setting up basic security : Basic Security configuration
• Setting up rootkit detection : Setting up rootkit detection
• Setting up Bastille : Setting up Bastille
• Setting up the lamp stack : Set up the LAMP stack including mysql and apache
• Setting up Snort and acidbase : Configure Intrusion detection using Snort and Acidbase.
• Setting up File Integrity using AIDE : Setting up File integrity checks using AIDE

Initial Configuration

These steps cover the initial setup of a server

Setup the hostname

Lets say the hostname we want to setup is vps.

$ echo “vps.mydomain.com” > /etc/hostname
$ hostname -F /etc/hostname

Now update the /etc/hosts file to reflect the hostname and the fully qualified domain name
Replace 12.34.56.78 with the IP address of your host

127.0.0.1 localhost.localdomain localhost
12.34.56.78 vps.mydomain.com vps

Updating the ubuntu repositories

You will need to update your ubuntu repositories to include jaunty-updates and universe repositories. This is so that you may install additional packages as required from these repositories as well. In my case, the earlier version of the file /etc/apt/sources.list was as follows.

However please note, that repository selection and its update strategy may be linked to your company or application strategy. Please make sure these steps are consistent with your policy. If not, kindly adapt consistent with your team / organisations policy. Also instead of us.archive.ubuntu.com, you may find other country specific server names. In that case you may want to continue to use the other server name as already listed in your file.

deb http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted

deb http://security.ubuntu.com/ubuntu jaunty-security main restricted
deb-src http://security.ubuntu.com/ubuntu jaunty-security main restricted

Upon adding jaunty-updates and the universe repositories, the resultant file is as follows.

deb http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted universe
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty main restricted universe

deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe
deb-src http://security.ubuntu.com/ubuntu jaunty-security main restricted universe

deb http://us.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe
deb-src http://us.archive.ubuntu.com/ubuntu/ jaunty-updates main restricted universe

Now update the sources. This will scan all the repositories

$ sudo apt-get update

Finally upgrade ie. replace any existing packages which have a newer upgrade

$ sudo apt-get upgrade

Download the language pack

To add the necessary for the preferred language of your choice add the appropriate language pack. In my case I add support for english (en)

$ sudo apt-get install language-pack-en

Set the timezone

Set the timezone of the server. You may choose to set it based on server location, or typical user location or to UTC.

$ dpkg-reconfigure tzdata

That will start a small app, from which you can select the timezone. I selected None of the Above which offered me a choice of timezones based on UTC offsets and subsequently selected UTC.

Setting up Mail sending

I do not need this VPS to act as a mail server. However I do need to have capabilities to send email from this machine. Many unix tools routinely assume the existence of sendmail or equivalent MTA. However that is an overkill in this context. So we shall not be installing sendmail or postfix or exim or any other equivalent. Instead we shall configure this server to be only able to send out mail using an SMTP account on another mail server. For this we shall install a tool called mailx. Note: If you have mailx already installed through another ubuntu package called mailutils, you may either continue with the same (in which case you will need to configure the remainder of the mail stack correspondingly eg. sendmail) or remove mailutils and add heirloom-mailx

$ sudo apt-get install heirloom-mailx

We shall also configure a global configuration for sending out mail. In my case its all right to always send mail using only one account irrespective of the process or user who is sending it.

$ sudo vi /etc/nail.rc

Note that in the above configuration, we shall be placing the mail account password in clear text. Make sure it is a mail account you do not use for any other purposes and that its password is not the same as used for any other purposes. Now enter the following as contents of the /etc/nail.rc file. Obviously change the relevant fields to appropriate values. Note that this file is configured for sending mail via gmail. You may need to configure it differently based on your own SMTP configurations.

set smtp-use-starttls
set from=my_user_id@gmail.com
set smtp=smtp.gmail.com:587
set smtp-auth-user=my_user_id@gmail.com
set auth-login=my_user_id@gmail.com
set smtp-auth-password=my_password

You can try testing whether this got set up successfully. Enter the following (replace youremailid@youremaildomain.com by the email id where you would like the mail to be sent to)

$ mail youremailid@youremaildomain.com
Subject: This is a test mail
Hello
.

Basic Security

In this section we shall make some basic configuration changes with a view to enhance the system security.

Mounting the shared memory as read only

Open and edit the file /etc/fstab to add an entry to mount shared memory in read only mode. The reason we do it is because many exploits use shared memory to attack other running services.

If you have a good reason to make shared memory writeable skip this step.

$ vi /etc/fstab

Now add the following line at the end of the file

tmpfs /dev/shm tmpfs defaults,ro 0 0

Tightening the passwords

One of the easiest exploits is to attempt a brute force login using dictionary based attacks. In order to ensure strong ie. non-guessable passwords we shall update the password checking policy so that it allows only strong passwords. A simple way to ensure that is to ensure a reasonable minimum length and to ensure multiple character classes.

First lets install a new pam authentication module pam_cracklib. To install the same run the following

$ sudo apt-get install libpam-cracklib

Answer ‘Y’ to the prompt it asks for regarding continuing.

Note: if you did not add the universe repository to your sources.list file, you will not be able to install libpam-cracklib. In that case you will need to skip this step.

This should’ve resulted in the file etc/pam.d/common-password having an entry for pam_cracklib.so and pam_unix.so. Update the pam_cracklib.so entry to add one more requirement ie. minclass=4.

In my case, the resultant two lines in /etc/pam.d/common-password are as follows. Note that I added the minclass=4 clause manually.

password requisite pam_cracklib.so retry=3 minlen=8 difok=3 minclass=4
password [success=1 default=ignore] pam_unix.so obscure use_authtok try_first_pass sha512

There. You now have a strong password scheme which will conduct a whole range of password checks in addition to ensuring that the password has a minimum length of 8 and each new password has at least one each of the four character classes. The four character classes are lower_case, upper_case, digit and special_characters (the last one being any non alpha-numeric character)

Creating the first user

Note: if you have already created at least one more non root user this step is not required. We are primarily creating the new user so that we shall eventually allow sudo and remote ssh login privileges to the user and disable remote ssh privileges for the root user.

Setup the first new user. One of the reasons you should create a new user is so that it will afford you the ability to allow him to perform root actions through sudo, and thus subsequently allow you to disable root access over ssh. By default when one creates a new user, another group gets created with the same name as well. In this case we shall create a new group “dev” and then create a new user associated with that group “someuser“. Use the groupname and the username as you would like to setup when executing the commands below. In the commands below we create a new home directory for the user, associate the /bin/bash shell with his account instead of the default /bin/sh, (I just prefer bash to the plain sh) and finally set the password for him.

$ groupadd dev
$ mkdir /home/someuser
$ useradd -d /home/someuser -s /bin/bash -g dev someuser
$ chown someuser.dev /home/someuser
$ passwd someuser

We shall also create the .ssh directory for the user which we shall be using later

$ mkdir /home/someuser/.ssh
$ chmod 700 /home/someuser/.ssh
$ touch /home/someuser/.ssh/authorized_keys
$ chmod 600 /home/someuser/.ssh/authorized_keys
$ chown -R someuser.dev /home/someuser

Now we shall create the keypair for the user to log in to the host remotely. Note that if you are going to do this for multiple users, then you might want to have each user run the next step locally and then copy over his public key onto the server before continuing to the ssh tightening operations described later.

The user should do the following on his local workstation from which he is most frequently likely to connect to the server (not the server that we are hardening).

Note: the part after -C in ssh-keygen is just a comment to identify the keys – enter something to identify the user and his machine.
Also make sure not to keep the passphrase blank though ssh-keygen will allow a blank passphrase. The reason is that if the user’s local machine is compromised the attacker can then get an easy access to the server being hardened.

change someuser and some.host.com below based on the user id and vps name correspondingly

$ mkdir ~/.ssh
$ ssh-keygen -t dsa -b 1024 -C “some user on his desktop”
$ scp ~/.ssh/id_dsa.pub someuser@some.host.com:/home/someuser/.ssh/someuser.pub

Now the user should himself ssh to the remote server and on the remote server move his public key into the authorized_keys file. So execute the following command after being connected to the VPS

$ cd .ssh
$ cat someuser.pub >> authorized_keys
$ rm someuser.pub

At this stage the user can disconnect from the VPS and attempt to reconnect using ssh. If all works well, he should get connected to the vps in a manner where it does not prompt him for a password but instead he does get prompted for the passphrase to his private key (assuming he did set one).

This stage of updating the authorized_key file can also be performed by an administrative user / root once we later reconfigure ssh to only allow public key based logins.

Enabling the user to perform sudo operations

We shall enable any group who belongs to the group ‘admin‘ to be able to conduct root operations through using sudo.

First create a group ‘admin‘. subsequently associate the user with that group as well. Note : For best security ensure you allow associate only a very small number of users with the ‘admin’ group since that will effectively allow them control over the whole machine (assuming you setup the privileges as I subsequently describe below).

$ groupadd admin
$ adduser someuser admin

Now we shall enable any user who belongs to the admin group to perform root actions by using sudo. To edit the sudo policy file do the following

$ sudo visudo

At the end of the file which is now opened up – add the following line

%admin ALL=(ALL) ALL

Note this grants all superuser privileges to the users who belongs to admin group when conducting operations using sudo. You can use the sudo policy configurations to set up far more fine grained set of privileges, but thats beyond the scope of this document.

To test whether the configuration worked successfully, you can login as someuser and execute the following command.

$ sudo cat /etc/shadow

Tightening up ssh

To create the group and associate the users with them perform the following command (use the appropriate username instead of someuser for each user who you would like to allow SSH access).

sudo addgroup sshlogin
sudo adduser someuser sshlogin


Now that we have at least one user id which can conduct root operations using sudo, its time to disable root login from ssh. Go open the file /etc/ssh/sshd_config. Make the following changes

This one disables root login for ssh.
PermitRootLogin yes to PermitRootLogin no

Adding the following line allows only users of a particular group to login. Note: if your user count is small, you could use AllowUsers instead. We shall be separately creating the group and associating the users with the group.
AllowGroups sshlogin

This change reduces the login grace time (time before a user needs to login after making the ssh login request). This is to allow better protection to sshd against DOS or brute force attacks.
LoginGraceTime 20 to LoginGraceTime 20

This changes the port number that sshd listens on from the default 22. Changing the default port takes away the ability of an attacker to easily guess the port on which to attempt to penetrate the system via ssh.
Port 22 to Port 9999 (or any other suitable number)

This change disables X11Forwarding over the SSH connection. It will result in you not being able to run X11 GUI applications remotely. If you need that flexibility, do not make this change
X11Forwarding yes to X11Forwarding no

The next change disables password authentication (thus allowing only users with their public key being stored in the corresponding authorized_keys folder to connect using ssh). Thus passwords, which need to be transferred to the server in clear text no longer are a valid authentication mechanism. The only available choice is public key based authentication.
PasswordAuthentication yes to PasswordAuthentication No

The next change disables password authentication (thus allowing only users with their public key being stored in the corresponding authorized_keys folder to connect using ssh). Thus passwords, which need to be transferred to the server in clear text no longer are a valid authentication mechanism. The only available choice is public key based authentication.
PasswordAuthentication yes to PasswordAuthentication No
UsePAM yes to UsePAM no

The following change of uncommenting the banner allows a welcome message (not really a welcome one ) to be displayed to SSH logins. It does not improve any aspect of the physical security but is more from a legal notice perspective.
#Banner /etc/issue.net to Banner /etc/issue.net

Now go update the contents of the file /etc/issue.net to something similar to following

***************************************************************************
NOTICE TO USERS

This computer system is the private property of its owner, whether
individual, corporate or government. It is for authorized use only.
Users (authorized or unauthorized) have no explicit or implicit
expectation of privacy.

Any or all uses of this system and all files on this system may be
intercepted, monitored, recorded, copied, audited, inspected, and
disclosed to your employer, to authorized site, government, and law
enforcement personnel, as well as authorized officials of government
agencies, both domestic and foreign.

By using this system, the user consents to such interception, monitoring,
recording, copying, auditing, inspection, and disclosure at the
discretion of such personnel or officials. Unauthorized or improper use
of this system may result in civil and criminal penalties and
administrative or disciplinary action, as appropriate. By continuing to
use this system you indicate your awareness of and consent to these terms
and conditions of use. LOG OFF IMMEDIATELY if you do not agree to the
conditions stated in this warning.
****************************************************************************

Restart the ssh daemon and you will no longer be able to connect as root over ssh directly.

$ sudo /etc/init.d/ssh restart

Disabling su

As per the policy being followed, only users belonging to the ‘admin‘ group will have any special privileges. For all other users, we shall attempt to lock down the system to the extent feasible. As an early step we shall disable the ‘su‘ (switch user) command for all but those users belonging to the admin group. Note : since this disables sudo for a brief duration (between the commands), first switch over as a root before executing these commands

$ sudo su -
$ chown root.admin /bin/su /usr/bin/sudo
$ chmod 04750 /bin/su
$ chmod 04550 /usr/bin/sudo

Install rootkit detection

Install chkrootkit

$ sudo apt-get install chkrootkit

Now create this file /etc/cron.daily/chkrootkit.sh and enter the following contents (replace sever.domain.com with the servername and emailid where you would like the report to be sent to).
Since the mail is rather verbose, and may lead to useful alerts getting ignored, I’ve removed the lines with the common messages. You may change or not use the grep commands as per your preference.

#!/bin/bash
/usr/sbin/chkrootkit | \
grep -v “not found” | \
grep -v “nothing found” | \
grep -v “not infected” | \
mail -s “Daily chkrootkit from server.domain.com” mailid@destination.com

Allow execute permissions on the file, and test once by running .. you should get a mail with the report.

$ sudo chmod 755 /etc/cron.daily/chkrootkit.sh
$ sudo /etc/cron.daily/chkrootkit.sh

Installing Bastille

Bastille is a comprehensive package for security configuration.

Bastille Bug? and workaround

You may need to perform this workaround if your /etc/debian_version contains 4.0 and the current ubuntu bastille package has not resolved the issue. Ideally one should just need to install bastille from the ubuntu repositories

On Ubuntu 9.0.4 (jaunty) as of the day this document was written, I received the following error
~& ERROR: ‘DB5.0′ is not a supported operating system.

I’ve had a difficulty to install Bastille on Ubuntu 9.0.4. This is due to the fact that Bastille does not support the debian version number in /etc/debian_version. As per Bastille bug reports this got fixed in 3.0.8, but did not work for me in Bastille 3.0.9. Hence the following is a work around to install bastille.

This workaround involves downloading lynx to access internet, downloading the debian package directly, and then installing the debian package. When you reach the web page it will show you a list of mirrors. Download the .deb package from one of the mirrors

$ sudo apt-get install lynx libcurses-perl
$ lynx http://packages.debian.org/squeeze/all/bastille/download
# Download the package using lynx
$ sudo dpkg –install bastille_3.0.9-12_all.deb

Install psad which we shall need later and perl-gtk which we shall need to configure bastille with the following command

$ sudo apt-get install psad

You will notice an error message :

To resolve the same execute the following :

$ echo -e ‘kern.info\t|/var/lib/psad/psadfifo’ | sudo tee -a /etc/syslog.conf
$ sudo /etc/init.d/sysklogd restart

Now run bastille in an interactive mode

$ sudo bastille -c

The resultant dialog with bastille is too long and I am just showing the final configuration file which is produced as a result of bastille configuration. I’ve used the port 22 as a proxy for the SSH port which I assume you’ve changed to some other value – use the other value instead of 22

# Q: Would you like to enforce password aging? [Y]
AccountSecurity.passwdage=”Y”
# Q: Should Bastille disable clear-text r-protocols that use IP-based authentication? [Y]
AccountSecurity.protectrhost=”Y”
# Q: Should we disallow root login on tty’s 1-6? [N]
AccountSecurity.rootttylogins=”N”
# Q: Would you like to deactivate the Apache web server? [Y]
Apache.apacheoff=”N”
# Q: Would you like to disable CTRL-ALT-DELETE rebooting? [N]
BootSecurity.secureinittab=”N”
# Q: Should we restrict console access to a small group of user accounts? [N]
ConfigureMiscPAM.consolelogin=”Y”
# Q: Which accounts should be able to login at console? [root]
ConfigureMiscPAM.consolelogin_accounts=”root”
# Q: Would you like to put limits on system resource usage? [N]
ConfigureMiscPAM.limitsconf=”N”
# Q: Would you like to set more restrictive permissions on the administration utilities? [N]
FilePermissions.generalperms_1_1=”Y”
# Q: Would you like to disable SUID status for mount/umount?
FilePermissions.suidmount=”Y”
# Q: Would you like to disable SUID status for ping? [Y]
FilePermissions.suidping=”Y”
# Q: Do you need the advanced networking options?
Firewall.ip_advnetwork=”N”
# Q: Interfaces for DHCP queries: [ ]
Firewall.ip_b_dhcpiface=” ”
# Q: DNS Servers: [0.0.0.0/0]
Firewall.ip_b_dns=”0.0.0.0/0″
# Q: ICMP allowed types: [destination-unreachable echo-reply time-exceeded]
Firewall.ip_b_icmpallowed=”destination-unreachable echo-reply time-exceeded”
# Q: ICMP services to audit: [ ]
Firewall.ip_b_icmpaudit=” ”
# Q: ICMP types to disallow outbound: [destination-unreachable time-exceeded]
Firewall.ip_b_icmpout=”destination-unreachable time-exceeded”
# Q: NTP servers to query: [ ]
Firewall.ip_b_ntpsrv=” ”
# Q: Force passive mode? [N]
Firewall.ip_b_passiveftp=”Y”
# Q: Public interfaces: [eth+ ppp+ slip+]
Firewall.ip_b_publiciface=”eth+”
# Q: TCP service names or port numbers to allow on public interfaces: [ ]
Firewall.ip_b_publictcp=”22 80 443″
# Q: UDP service names or port numbers to allow on public interfaces: [ ]
Firewall.ip_b_publicudp=” ”
# Q: Reject method: [DENY]
Firewall.ip_b_rejectmethod=”DENY”
# Q: Enable source address verification? [Y]
Firewall.ip_b_srcaddr=”Y”
# Q: TCP services to audit: [telnet ftp imap pop3 finger sunrpc exec login linuxconf ssh]
Firewall.ip_b_tcpaudit=”telnet ftp imap pop3 finger sunrpc exec login linuxconf ssh”
# Q: TCP services to block: [2049 2065:2090 6000:6020 7100]
Firewall.ip_b_tcpblock=”2049 2065:2090 6000:6020 7100″
# Q: UDP services to audit: [31337]
Firewall.ip_b_udpaudit=”31337″
# Q: UDP services to block: [2049 6770]
Firewall.ip_b_udpblock=”2049 6770″
# Q: Should Bastille run the firewall and enable it at boot time? [N]
Firewall.ip_enable_firewall=”N”
# Q: Would you like to run the packet filtering script? [N]
Firewall.ip_intro=”Y”
# Q: Would you like to set up process accounting? [N]
Logging.pacct=”N”
# Q: Would you like to deactivate NFS and Samba? [Y]
MiscellaneousDaemons.remotefs=”Y”
# Q: Alert on all new packets?
PSAD.psad_alert_all=”Y”
# Q: psad check interval: [15]
PSAD.psad_check_interval=”15″
# Q: Would you like to setup psad?
PSAD.psad_config=”Y”
# Q: Danger Levels: [5 50 1000 5000 10000]
PSAD.psad_danger_levels=”5 50 1000 5000 10000″
# Q: Email addresses: [root@localhost]
PSAD.psad_email_alert_addresses=”mailid@destination.com”
# Q: Email alert danger level: [1]
PSAD.psad_email_alert_danger_level=”1″
# Q: Should Bastille enable psad at boot time? [N]
PSAD.psad_enable_at_boot=”N”
# Q: Enable automatic blocking of scanning IPs?
PSAD.psad_enable_auto_ids=”N”
# Q: Enable scan persistence?
PSAD.psad_enable_persistence=”N”
# Q: Port range scan threshold: [1]
PSAD.psad_port_range_scan_threshold=”1″
# Q: Scan timeout: [3600]
PSAD.psad_scan_timeout=”3600″
# Q: Show all scan signatures?
PSAD.psad_show_all_signatures=”N”
# Q: Would you like to disable printing? [N]
Printing.printing=”Y”
# Q: Would you like to disable printing? [N]
Printing.printing_cups=”N”
# Q: Would you like to display “Authorized Use” messages at log-in time? [Y]
SecureInetd.banners=”Y”
# Q: Should Bastille ensure inetd’s FTP service does not run on this system? [y]
SecureInetd.deactivate_ftp=”Y”
# Q: Should Bastille ensure the telnet service does not run on this system? [y]
SecureInetd.deactivate_telnet=”Y”
# Q: Who is responsible for granting authorization to use this machine?
SecureInetd.owner=”Company Name”
# Q: Would you like to set a default-deny on TCP Wrappers and xinetd? [N]
SecureInetd.tcpd_default_deny=”N”
# Q: Do you want to stop sendmail from running in daemon mode? [Y]
Sendmail.sendmaildaemon=”Y”
# Q: Would you like to install TMPDIR/TMP scripts? [N]
TMPDIR.tmpdir=”N”

Installing the Lamp Stack

Let us use the not so well known tasksel command to download all the packages necessary for a lamp stack. Note that tasksel is simply a convenience tool around apt-get and allows one to install a whole bunch of packages based on a class of necessity – in this case a LAMP (Linux,Apache,MySQL,PHP) stack.

$ sudo tasksel install lamp-server

During the installation you will be required to create the password for the mysql root user. In interest of tight security do make sure to create a really strong password.

This actually sets up the basic LAMP stack and that should get you operational.

Securing Apache

Change the user and group apache runs as

You may be aware that installation of apache resulted in a new userid and group being created (both called www-data). I just find it a little bit more comforting to change the well known usernames to something that are less predictable. In the steps below I change www-data to apache but you may want to use something a little less predictable than apache.

$ sudo usermod -l apache www-data
$ sudo groupmod -n apache www-data

We shall need to change the same in the apache configuration file. Edit the file /etc/apache2/envvars and change the two lines as follows

export APACHE_RUN_USER=apache
export APACHE_RUN_GROUP=apache

Install and configure mod_security

Earlier when we configured, bastille and through it the iptables firewall, we allowed incoming public traffic on the SSH, HTTP and HTTPS ports. We’ve already covered the tightening of SSH security. However we still do not have a good way to control HTTP and HTTPS traffic. The solution to that is installing the apache module mod_security. It allows us abilities to inspect and if necessary reject or redirect HTTP/S traffic. The full description of mod_security is beyond the scope of this document, but a good example of how it can be used further to implement application level security as well can be had from the document Securing web services with mod_security. Another useful overview page is Secure your Apache2 with mod-security.

We shall start off with installing mod_security.

$ sudo apt-get install libapache-mod-security

You should notice a new file in /etc/apache2/conf.d called security. These are the global mod_security settings.

Change the following variables in the file to make server identification difficult. Note that while these settings do not enhance the security directly, they do make it harder for an intruder to easily identify the specific server and configuration, thus making it harder for him to attempt configuration specific exploits.

ServerTokens Prod
ServerSignature Off
TraceEnable Off

Also uncomment the following four lines. Note that this may require you to configure other web applications which are not configured appropriately.


AllowOverride None
Order Deny,Allow
Deny from all


While not related to mod_security, to minimise server information leakage, you may consider changing these two variables to the shown values in /etc/php5/apache2/php.ini

expose_php = Off
display_errors = Off

As mentioned above, there is a whole range of additional controls you can enforce using mod_security. Depending upon your specific requirements feel free to leverage mod_security much more.

Disable apache modules you do not need

In general it is better to turn off unrequired modules unless you really need them. The default configuration installs a number of modules which may not be required. The modules that have been enabled are in the /etc/apache2/mods-enabled directory which are all symbolic links to the modules that have been installed in the /etc/apache2/mods-available directory. Note that turning off some modules may affect some web applications you install, in which case you may choose to subsequently turn them on only if necessary. New modules can be enabled with sudo a2enmod module_name command and enabled modules can be disabled by sudo a2dismod module_name command.

Depending upon my requirements, I disabled the following modules

$ sudo a2dismod autoindex # used to build directory indexes like the ls command
$ sudo a2dismod cgi # used to run cgi scripts
$ sudo a2dismod env # used to set environment variables for CGI & SSI scripts

Chrooting apache

This is an advanced topic that I shall skip. Note that if you do setup chrooting, there’s a lot of additional considerations which will need to be observed, considerations especially around shared libraries and configuration files which will make the overall process of configuration extremely difficult.

Setting up snort and acidbase

Author’s Note: Careful – Very likely, there were a few things that I might have missed or incorrectly noted especially in the context of acidbase and following these instructions may not get acidbase running properly. You may have to do some steps outside these notes to get it all working properly. Unfortunately I am unable to rerun the process on a fresh server so am unable to immediately note the possible lacunae in the notes

Now that you have a mysql server going, create a new database for snort. In the following section I use snortdb as the database, snort as the user and snortpwd as the password for the database. However I do encourage you to replace the same with some non easily guessable names and passwords.

Here’s the commands to create a new snort database and user. In the following example ‘$‘ is the unix prompt while ‘>‘ is the mysql prompt.

$ mysql -u root -p # You will be prompted for the password for the mysql root account. Enter it
> create database snortdb;
> grant select, insert, update, delete, create, drop, index, alter, create temporary tables, lock tables on snortdb.* to ‘snort’@'localhost’ identified by ‘snortpwd’;
> flush privileges;
> quit;

Now we shall install snort.

$ sudo apt-get install snort-mysql

The installation process will require you to enter the network that you wish to protect. Since in this case we are only protecting a single machine, enter the IP_Address/32 eg. 12.34.56.78/32

The process will prompt you whether you wish to configure the database. Select “Yes” .. the script will proceed but terminate abnormally with the error “subprocess post-installation script returned error exit status 6″

Now execute the following commands to setup the tables in the database. At the end you shall open the snort configuration file to enter the database configuration parameters

$ cd /usr/share/doc/snort-mysql
# Enter the database password next when prompted for one
$ sudo zcat create_mysql.gz | mysql -u snort -p snortdb
$ sudo vi /etc/snort/snort.conf

Uncomment the line that starts with output database: log, mysql and add the configuration information at the end as shown

output database: log, mysql, user=snort password=snortpwd dbname=snortdb host=localhost

Now remove the file as shown below to indicate that the database configuration has been done and start snort. The subsequent command confirms that snort daemon is indeed up and running

$ sudo rm -rf /etc/snort/db-pending-config
$ sudo /etc/init.d/snort start
$ sudo /etc/init.d/snort status

Now, lets install acid which is a web based application to be able to view snort alerts

$ sudo apt-get install acidbase

Select No when it prompts you for configuring the database. By default it uses the database and userid ‘snort‘ and I prefer to keep these different from a predictable default, in which case acidbase database initialisation will fail.

Edit the file /etc/dbconfig-common/acidbase.conf.

Set the following values

dbc_dbuser=’snort’
dbc_dbpass=’snortpwd’
dbc_dbname=’snortdb’

The default configuration allows acid data to be accessed only from the machine where it is involved. Since this shall be a VPS which we shall be accessing remotely, it is time to relax the constraint. If you have a good idea of the network IPs that you will be accessing the acid web application from, update the same in the allow from directive in /etc/acidbase/apache.conf, else relax it fully to allow from all.

Also protect the ‘/acidbase‘ url by http basic authentication. The resultant section in the file /etc/acidbase/apache.conf looks like :


Options +FollowSymLinks
AuthType Basic
AuthName “Go Away World!
AuthUserFile /etc/acidbase/mypassword
Require valid-user
AllowOverride None
order deny,allow
deny from all
allow from all

php_flag magic_quotes_gpc Off
php_flag track_vars On
php_value include_path .:/usr/share/php



You will need to create a user id / password pair for the app or point AuthUserFile above to another file where you already have the ones set up for your server.

To do so you shall need to run the following command :

$ htpasswd -c /etc/acidbase/mypassword my_user_id

If you need additional user ids to be installed use the command as shown above again with different user ids – but make sure not to use the -c switch since thats only used the first time to create the password file.

Now start the web application by restarting apache

$ sudo /etc/init.d/apache2 restart

Goto the web page http:vps_fully_qualified_domain_name/acidbase/base_db_setup.php, click on ‘Create Base AG//’ and you are on your way.

File integrity checking using AIDE

Install AIDE

$ sudo apt-get install aide

Update the aide configuration file etc/default/aide to update your email id where you would prefer the integrity check reports sent (Look for the variable MAILTO)
Now initialise the aide configuration

$ sudo aideinit
$ cd /var/lib/aide
$ mv aide.db.new aide.db

Now run the process (which will be running once daily)

$ /etc/cron.daily/aide

This process takes a long time (about 5 mins for me) and will at the end email you a report if any files changed compared to the ones in the default database.

No related posts.


]]> http://blog.dhananjaynene.com/2009/10/configuring-a-secure-ubuntu-linux-virtual-private-server/feed/ 3 http://blog.dhananjaynene.com/2009/10/configuring-a-secure-ubuntu-linux-virtual-private-server/?utm_source=rss&utm_medium=rss&utm_campaign=configuring-a-secure-ubuntu-linux-virtual-private-server