Now we can do this through an automated tool like junit/testNG/nunit/flexunit etc by creating stubs and even more often now mock objects which are not real objects but still mimic their behavior making it possible to unit-test functionality of a single class without actually calling underlying or collaborating classes. How is it possible manually?

The answer is NO, it is not and thus any form of manual unit testing is actually integration testing.

As per my earlier post we need to scale out our data store when the application data requirements outgrow a single instance capacity. This is in most cases achieved through sharding (a.k.a. partition of data).

With traditional RDBMS systems the application data model has to be modeled such from the very start to enable equalized partitioning of data or it may require a considerable change if we want to do it as an afterthought to scale up the application for using multiple database instances where each instance keep a balanced subset of the big data for an application. So in a way we have to plan for scaling out an application in typical RDBMS world but then with MongoDB it can actually occur as an afterthought as it provides automatic balancing and fail over of the partioned data

MongoDB as on 1.6 onwards supports production ready automated sharding architecture where we can convert a single instance into a cluster of shard whereby MongoDB manages the distribution of load of data across the shards (redistributing if the data on one node goes out of proportion with the rest) and also provides fail over support with each shard data being replicated in a replica set and all this with very minimal or no code change. The application needs to connect to a mongos process (which sits in front of the sharded cluster) which is responsible for managing the reads and writes across these shards giving a similar impression as we are working with a single node database systems.

Each shard as part of the cluster in itself consists of multiple servers where data is replicated either as a master slave configuration or as a replica set (v1.6 onwards). The data set or the collection of data is distributed through shard key pattern where we use one or more fields to define this key. These fields are to be choosen carefully and has to be granular enough to allow even distribution of data across the nodes and thus in most cases we end up using compound key with multiple fields.

Apart from the shard cluster we also have config servers and routing processes where config servers are the one with the metadata information about each node and the chunk of data each node has and the routing process being responsible for intercepting the client requests and processing them in orchestration with the different nodes of the cluster giving an impression that the request is being processed through a single server.

At the base minimum we can have just two or more shards, one config server and a mongos routing process to create a shard cluster. You can read more about configuring the shards here

If we ignore this fact at the very start concentrating more on the business aspects of the application (something which is misunderstood in Agile) we are doomed to fail even at the slightest hint of making the application scalable. Do not get me wrong, there are measures which can be taken a little later in the development cycle which can improve the scalability of the application but then they have their limitations. For example something like what to choose as data storage is something you need to decide upfront. Agreed having designed the application with a layer of data access objects abstracts the rest of the application from the data storage dependency but then why do the rework of rewriting the data access implementation. I am very sure we cannot regard such a change as trivial.

Have a strategy to allow the application to scale from the very start will definitely help and it is more a matter of choosing the right technologies and right architecture to accomplish it upfront.

N-tiered architecture with highly decoupled layers – Something quite basic but make sure you do not miss out on this. Decoupling can be achieved through use of interfaces and now something very common through dependency injection. What this does is allow us to separate out the different layers from the restriction for them to be co-resident on the same box and let them scale out as independent units may be to the extent of being implemented as web service

Scale the data store – Either choose something more traditional like MySQL or something which is gaining a lot of popularity due to its ability to allow the applications to scale : NoSQL databases like MongoDB or Cassandra but the idea is to allow cutting down the bottleneck of having a single instance of the database being accessed by large number of users. Few approaches to keep in mind

1. Use replication of data to provide scalability and more so high availability. Split down the read and writes with a single master server to write to and multiple slaves to read from using the replication of data from master to slaves but then this has a limitation with still a single master server to write to. Too many writes and replication takes too long

2. Database Partitioning – It allows us to split the datastore among multiple instances so that we can distribute both read and writes. We can choose a technology which has a built in support for it like MongoDB, Cassandra etc or we can implement something of our own as application logic and use MySQL for partitioning the data.

Introduce the Caching of data – Introduce a caching layer on top of the data access layer either using something like Ehcache as a second level cache on top of hibernate + MySQL or moving the cache onto a distributed solution like Memcached. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) which can be as a result of database calls, API calls, or page rendering

Go for a stateless application – Try to keep the application as stateless as possible cutting down the cost of session replication or if not possible going stateless move towards centralized sessions rather then local

So I have done my design, written my implementation as per the design and now I am writing the unit test cases to test out every possible scenario and then I realize I am not able to unit test a scenario until I modify the code to get it more suitable and enabled for testing. I modify my code to make it available for testing but then we know if our code is not testable, it is a flaw in the design and the design itself should have been such to allow for it to be tested. So I need to modify the design.

So in a way my tests are driving the way the code is written and eventually the design hence why not start with the unit tests at the very first place rather then going through an unnecessary loop and that is what is TDD (or more aptly TDDAD – Test driven development and design).

But then can we just stretch it even further and have my test cases drive the architecture of the system as well. Not really

1. Architecture by definition has much wider scope and relevance then may be just the design of few classes. It is a decision which is taken at a team and solution level and not merely confined to the reasoning being done at the pair programming level.

2. We need to get some basic architecture in place as part of zero iteration which takes in account just enough skeleton in place from which we can start building. We need to ensure we are pretty confident of this basic skeleton as it will be one of the hardest stuff to alter later in the future iterations and will have quite a strong consequence whenever we plan to do it.

3. Architecture decisions are very hard to postpone as something like going for a typical n-tier architecture or may be for SOA or in some case opting for a third party off the shelf solution or the very latest GAE as PaaS solution has to be taken upfront and cannot be delayed for a point in future

Not to say that we need to do big upfront architecture but still the bare bones or the minimal set has to be in place and then let the design evolve

1. Earn their trust and trust the ones you are managing as a team lead – Give them the space, enable them to be effective, keep them out of the politics and unnecessary management value and vision dialogues, do not over inform them of the goals and objectives as they very well know the importance of the business and how much impact they are already doing through high quality deliverable. Make them comfortable and protect them to gain their trust.

2. Remain connected always but not micromanage – Make sure they feel valued and even more their work by asking smart, relevant queries even if you the answer to many. Never micromanage and leave them to manage themselves just being sure they are very clear on the goal and timelines. Development and coding background especially ability to help with the design and architecture decisions really help and help you being closer to the so called technical crowd of the project

3. Let them own and feel responsible for their work – Empower them and let them handle it giving right support and guidance throughout. Do not make them agree to you all the time but challenge them asking for ways to improve and optimize the work and velocity

4. Do your bit for their morale – Nothing is good for the team as much as highly motivated and energetic team working with you. Inspire them , reason it out and keep them informed about all the important decisions no matter of it effects them directly or indirectly. Not just 1/10th but atleast 8/10th picture of the importance organization changes happening and why

5. Be there to say “No” for the team – Never leave it for the team to come back and say no to you. Be the first to take the shot if you really believe if it may cause distraction to the team to meet their goal and commitments, the team will not be able to complete it, we are not going to sacrifice on the quality, the testing team doesn’t have enough to get started with the testing.

I know there are a lot more and may be I will update the post as I evolve more as a manager and learn this craft which actually doesn’t have that many hard ground principles or rules as such

1. Behave, act and communicate what you are doing as path breaking, out of the world, big value add for the organization
2. Your sole aim of existence is to impress your boss, be on his side explaining your achievements and how much support you have got from him/her to achieve them
3. Always lunch with you boss and never with your team.
4. Try to keep things confidential as much as possible to enjoy being in the elite group
5. Keep the time which you need to spend with your team as low priority over everything else
6. Reorganize everything as per your convenience once you move into the new role to demonstrate your value and the impact of your role and hierarchy
7. Delegate the responsibility not the credit or recognition of the good work done
8. You be the single point of failure for any ideas or suggestions coming from the team you are leading
9. Never challenge your boss or even you have to, to demonstrate such a behavior do it where it impacts the least
10. Have few favorites within the team. Trust them blindly, they are always right

I soon realized NoSQL encompasses much more then I would liked (just because I find myself a little lost with so many options) with so much already out there challenging the SQL databases. My first stop Cassandra.

Why Cassandra? – For me at least to start with the need for a non RDBMS has been quite straight forward, a data store which can provide me high availability with complete fault tolerance, highly scalable, very minimal downtime if any. This is mostly achieved through highly replicated and db clustered environment. Cassandra comes out as a natural choice to explore.

Features –

Fault Tolerant : Data is automatically replicated to multiple nodes. Loosing a node doesn’t bring down the cluster

Flexible Schema : We are talking in terms of columns, supercolumns and columnfamilies instead of rows and tables. BigTable datamodel

Symmetric : No single point of failure, Every node within the cluster is identical and there are no network bottlenecks

Scalable : Linear with addition of new machines with no downtime or interruption to applications. Read and write throughput increase linearly as new machines are added

Support for Large Data : The ability to scale to many hundreds of gigabytes of data

Written in Java : Originally built for facebook and then made open source

Data Model – Very confusing to start with and will need some time to get a hang off.

Column – Simplest of all, a tuple with binary no-fixed length name and value along with the timestamp. To keep it simple ignore the timestamp for the moment.

Super Column – Essentially a container for one or more columns. It is again a tuple with a binary name and a map of where key is the same as the name of the column

Column Family – A structure which keeps an infinite number of rows just like a traditional table. Each row in itself has a binary key and a map of where again a key is the same as the name of the column

Super Column Family – Same as column family with the exception that each row has a map of super columns instead of columns. The map is keyed with the name of each SuperColumn and the value is the SuperColumn itself

Keyspace – It is like schema containing the column families

Sorting – The data is sorted as soon as we put the data within the cluster and it remains that way as there is no way to do it while fetching the data which makes it all the most necessary to plan it right as per the access path. I am not going to go all in detail here but I guess you got the idea.

The data model is not schema bound with no specific number of type of columns for each row, no relationship between the column families and thus no way to join them.

More on the data model is here and here

So no SQL then how to Query – Uses Thrift framework as its client API. Essentially a communication protocol used not just by Cassandra but by many others. Available for a large number of development languages, ranging from Java and C++ to Python, PHP and many other software development languages. It is an internally developed cross-language framework that ties all of these different languages together, making it possible for them to talk to each other.