sUnit blog space: moved to http://www.sunitparekh.in/home

Blog moved to http://www.sunitparekh.in/home

2014-09-08T15:32:00.004+05:30

Blog moved to http://www.sunitparekh.in/home

Zero downtime during deployment using modified blue-green deployment strategy

2013-10-23T21:35:00.002+05:30

Zero downtime during application deployment is one of the key requirements for continuos delivery. And no business would like their site to be down and showing maintenance page every few days/weeks during deployment.

To achieve this we decide to go for blue-green deployment. However, we were challenged with how to do this in legacy style infrastructure where we are,

Not able to spin new machines dynamically and throw away stacks
Don' have ability to add/remove instances dynamically using scripts from load-balancer
Network level configurations are done manually like firewall setting

Also keeping full in-active stack didn't sound good idea. Now we had to made some modifications to achieve zero downtime using blue-green deployment.

Decision 1: We took out database from the application deployment stack. Since we were using NoSQL MongoDB database, it didn't require any schema migration etc. However, we required to follow design principle, always write code which is backward compatible with the data models. And if there is any database migration required it should be run after deployment.

Decision 2: Use RPM to package apps and run like a service to start and stop. This enabled us to deploy application easily with the standard chef recipes. All application components are packaged as RPMs and published to repository from CI pipeline. Standardise deployment process across all components/application. We used FPM gem to build RPMs.

Decision 3: All application/component should provide 2 state heartbeat, live or standby, with ability to change the current state of the heartbeat at runtime using api call. Configure one load-balance to listen to live heartbeat and another to standby. Allows to add/remove instances from load-balancer by changing heartbeat state.

And with above changes the deployment steps are,

Step 1: Change heartbeat of the green stack to state 'standby'. This will remove green stack from LIVE load-balancer pool and no new request send to green stack.

Step 2: Deploy latest version of the application to green stack. Wait sometime to complete the inflight request on green stack before deployment.

Step 3: Sanity test green stack using standby load-balancer. Ideally automated, this will make sure deployment is good.

Step 4: Revert heartbeat of the green stack to state 'live'. This will put the green stack back to LIVE load-balancer pool.

If you notice the blue stack running on v1 and green stack running on v2. And both the stacks are connected to same database. Which means v2 codebase should work with old data models. And if there is any database migration should be carried only after all stack upgraded to latest version.

Step 5: Repeat above steps for blue stack,

And finally v2 deployed fully,

Step 6: Optional, run database migration (independent of the deployment)

What we achieved is, at any give point of time either blue or green stack is actively servicing the requests without downtime.

However, the one point to note here is that during deployment only half capacity available, which means deployment should be avoided at peak load time.

Building highly scalable and performant application using non blocking architecture

2013-05-07T15:30:00.002+05:30

I have been working on web application development since last 12+ years and had privileged to work on more than 20+ different project. Now days the expectations from web apps are totally different than it was few years back. End users are provided with rich content on single page (Amazon, CNN, ...). On a single page, lots of data needs to be mashed up and data may come from different sources.

We know how to put together the N-tier architecture.

And the request workflow will be as following,

Now if we notice the above workflow, it is sequential and most of the time thread on web server and app server spend is waiting for response form app server and database respectively rather than doing some meaningful work (processing). This is what is called BLOCKING. Lets take same example and try to draw web server thread timeline.

RED is time spend by thread waiting for response/result from app server. And GREEN is the time spend doing some meaningful processing by web server. In above example it was total 10 sec response time and 1 thread involved in processing. Lets assume that above 3 calls to app server can be parallelise to optimise response time to end user.

In parallel processing the response time was reduced to 7 sec. However, total 4 threads participated in processing and in total 13 sec of thread time it took. Again here also the processing was BLOCKING. Which means parallel processing provides better performance in terms of response time, but required more resources to process the request.

What we need is parallel processing with non-blocking threads. Means when the thread is waiting for the response from other systems, it should be made available to do some other meaningful work. Something like following,

If we can achieve non-blocking architecture than the response time goes down to as low as 5 sec. And also consumes less thread resources. Now the question is how I can achieve this? In today's polygot programming world, we have plenty of alternatives available.

Non-blocking is not new, Nginx is event based and uses similar principle, Messsage based architecture is another classical example to solve similar problems, however that brings in asynchronous complexity. MongoDB and similar polygot persistence works on same principles and provides eventual consistency.

I recommend trying out following technology/framework choices to achieve non-blocking architecture,

NodeJS, this is the most popular choice so far in today's technology choices. And Passanger can provide NodeJS cluster running multiple process on one machine. However, programming in NodeJS is all about callbacks. And lots of OO programmers don't like it. There are plenty of frameworks in NodeJS for web application, the popular ones are is express.js, meteor ...
Play Framework 2.1.1, this is Scala based web application framework. Framework is very mature and easy to adpat for Java shop organisations. It uses internally Netty, Scala and Akka and provides very nice abstracts to avoids callbacks hell. Based on Scala means love for OO programmers as well. Akka can also be used independently.

Disclaimer: Consider non-blocking architecture when your application have blocking calls and wait times. If your application has more about computation/processing and less or negligible blocking/waiting time the above architecture pattern may not provide better or favorable results.

Identifying Cross-Functional and Non-Functional requirement during agile project inception

2013-05-04T20:50:00.002+05:30

It's always been difficult to identify and easy to miss-out on creating stories for cross functional requirement (XFR) a.k.a NonFunctional requirements (NFR) during inception. Inception is a short, time boxed, 2-3 weeks initial requirement gathering phase of the software development project lifecycle. I have been to multiple inception and figured out a nice and collaborating way of capturing and converting cross functional requirements into stories. We run it like a workshops, collaborating with client to identify and capture XFRs.

After understanding and going to functional requirement, plan for XFR session with all the members of the team including stakeholders. We have list of XFRs printed on the cards at ThoughtWorks similar to the one shown below.

There are around 35 such "bility" that need to be discussed. During the session we take one 'bility' at a time and not down on stickies what are the requirements related to that 'bility'. It is important to discuss each and every 'bility' and there is a possibility few are not application to the project we are working on. Mark them as NA.

Once all the 'bility' is discussed break out in small team, discuss each XFR requirements and convert them into stories.

After creating the story list of XFRs, treat them as other stories. Combine the backlog of functional and XFR stories, estimate and prioritise stories together.

As documentation, I like to use mind map since it fits that format nicely. Please have a look at the one displayed below for your reference.

I like more calling NFRs as cross functional requirements rather than non-functional, since most of the non-functional requirements are kind of indirect functional requirements. e.g. page load time for end user is performance requirement which indirectly related to better user experience.

Why Responsive Web Design?

2013-02-12T11:16:00.001+05:30

Lots of time when I discuss Responsive Web Design (RWD), I realise that we don't understand the problem we try to solve with RWD. And the objective of this blog post is look into the history and learn what problem we try to solve with RWD.

The multiple device/screen size/resolution problem is not new, even in old days the monitors had different screen resolution. The problem exist from old days. In first attempt we solved multiple screen resolution problem with Fixed Width web page design (Fixed Width Layout Approach). 960px based fixed width design were very common for websites. One of the most popular CSS framework with fixed width was blueprint. The problem with fixed-width layout is white (empty) space on side of the screens. Designer also used that space by having creative backgrounds for the page.

The multiple screen resolution problem didn't go away, after smartphones and tablets the problem became more prominent. In second attempt to solve the problem, used multiple templates for different devices on server side (Seperate Mobile Website Approach). The downside of using the server side device specific templating is that, HTTP caching can't be done. Which is non-negociable for scalability, and to overcome caching issues, on first request the application server recognises the browser agent and redirects the site to device specific page like www.cricinfo.com or m.cricinfo.com By using different sites/urls for different devices HTTP caching problem was solved.

Try visiting page: www.guardian.com and www.cricinfo.com on mobile device and notice the redirect.

Now the problem started with maintaining multiple sites/applications. Use experience was not consistant across devices. It was very difficult to keep the feature parity across the device specific sites. Application development, maintenance & production cost & effort increased significantly.

During same time period people realised that we are wasting so much screen real estate by doing the fixed grid layout, and the fluid grid layout design started becoming popular for desktops.

However, fluid grid system depends heavily on browser support and performance. Making it old browser compatible was a big task.

The latest attempt of providing best viewing experience on all devices & screen resolution at client side (browsers) is known as Responsive Web Design. With frameworks like Twitter Bootstrap, it has become easy to get started learning and using Responsive Web Design concept.

Notice menu, text, links and buttons in above two screenshots. Same site visited in desktop browser and mobile browser.

To achieve HTTP caching it was very important that all this dynamic behaviour to be kept on client side, so server has to render the same page for all devices irrespective of the screen sizes and resolutions. With this approach HTTP caching CDN/Reverse proxy can be leveraged agin to achieve high scalability.

Be aware when using Responsive Web Design,

Works best with latest browsers and high end smartphones, Since RWD relies heavily on client side optimisation and adoption, client needs be powerful.
Slow internet connection speed will give bad user experience since it might take very long to load page. Since all content irrespective of the device is transferred to client. Sometime multiple variations are kept like Menu and the overall web page size increases. Specially for mobile the page size might be very large. Checkout the size of twitter bootstrap.

Next I will talk more about different solutions for achieving responsive web design.

Data Anonymization approaches Blacklist or Whitelist?

2012-09-04T12:08:00.002+05:30

Continuation of my previous post about the need for anonymized production data dump, here is more details on two anonymization approaches blacklist and whitelist. Lets take one simple example and understand both the approaches. Consider two tables of database: Customers and Config.

Blacklist

This approach essentially leaves all fields unchanged with the exception of those specified by the user, which are scrambled/anonymized (hence the name Blacklist!).

For Blacklist, create a copy of the prod database and choose the fields to be anonymized e.g. username, password, email, name, geo location etc. Fields are anonymized based on user-defined rules. Most of the fields have different rules e.g. password should be set to same value for all users, email needs to be valid.

Considering above example. Lets anonymize data using blacklist approach. In above we want to anonymize Customers.Name and Customers.Email, so that we can not identify user. After anonymization the data will look like following.

Look at the Age field and config table, they remained as is. That is, apart from data specified in anonymization rules, all other data remains as is after anonymization. This could be a data security issue. Such as when new fields are added they will not be anonymized by default. Human error in missing any users personal data could be damaging.

Whitelist

This approach, by default scrambles/anonymizes all fields except a list of fields which are allowed to copied as is. Hence the name whitelist! By default all data needs to be anonymized. So from production database data is sanitized record by record and inserted as anonymized data into destination database. Source database needs to be read-only. All fields would be anonymized using default anonymization strategy which is based on the datatype, unless an anonymization strategy is specified. For instance, special strategies could be used for emails, passwords, usernames etc. A whitelisted field implies that it's okay to copy the data as is and anonymization isn't required.

Using whitelist approach and applying similar rules to anonymize only Name and Email for above example will produce output like following.

Age field value is anonymised. Which means if u haven't specified any rule for anonymization the system should anonymize it using default anonymization rules for the data type. To get Age field as-is it is required to mention it as whitelist. Also, the config table didn't show up at all, means if I don't ask for the table explicitly it should not be copied at all.

This way any new field will be anonymized by default and if we need them as is add it to the whitelist explicitly. This prevents any human error and protects sensitive information.

Data Anonymization tool supports both blacklist and whitelist approach for anonymization. Read more here about the tool and how easy to use it is .

Data Anonymization home page: http://sunitparekh.github.com/data-anonymization

Data Anonymization, need for every site in production

2012-09-04T12:08:00.000+05:30

On one of my previous projects, we wrote a jMeter performance test suite, which runs periodically on performance environment. Once the application was in production, we enhanced our performance test suite based on actual user behaviours from Apache access logs and Omniture analytics. That provided us a great level of confidence in development for scaling. Now the next step was to get the production dataset so our performance testing becomes almost like production peak load.

Also we had few bugs manifesting themselves only in production and we were not able to reproduce the same on our local environment due to the dataset. In production the data has evolved over a period of time and might have some bad/inconsistent data leading to edge cases or defects not happening in other environments. To fix such issues it was required to have a production dataset to enable our team to debug and fix such issues with confidence rather than guess-work.

One other time, we had issues with our migration scripts, when the migration script failed during production release. The reason was again the kind of data that had evolved in the production system. It could have been avoided if we had a production dump to rehearse the production deployment.

Considering all the above hurdles, we got together our Product Owner, and the Dev ops and Security team mates for a production dump. The initial reaction was a NO: the primary reason it being 'Users' Personal Data' and according to Data Protection Act "No one should have access to users personal data". Now what? the answer was, we can get production dataset if we sanitize/mask/anonaymize users personal data, we can get the data from production. We quickly started googling around to findout is there a quick and easy tool available to achieve anonymization of the database. Hmm.... no luck.

Considering distributed teams spanning out of country, concern of security with data is increased multiple fold.

We thought lets write some scripts and anonymize production data. However, it has some serious security and data protection issues. Such as there is possiblity of missing out on certain attributes at all and those data slips into non production environments. Also it has issues with new content gets passed into non production as is without anonymization by default. There are two approaches for anonymization blacklist and whitelist.

As this is quite a common requirement across projects, I started working on this idea with a few colleagues at ThoughtWorks to build a simple tool in Ruby based on ActiverRecord to support multiple databases.

Data Anonymization tool supports both blacklist and whitelist approach for anonymization. Read more here about the tool and how easy to use it is .

Data Anonymization home page: http://sunitparekh.github.com/data-anonymization

In my next post I describe blacklist and whitelist anonymization approaches in more detail. Write to me (Twitter: @sunitparekh) or comment on this blog for any questions, feedback and suggestions.

Get notified with push notification

2012-07-13T10:32:00.000+05:30

Now days it's kind of defacto that every website requires some sort of mechanism to update changes dynamically. On my last project we had a specific need to keep updating the count of messages like Facebook does for unread messages. The website is a dot com site with reasonable user base online. It was challenging to come up with a solution so that we can get this developed quickly and scale well with the load.

We discussed multiple approach and chose one for development.

Approach 1) Poll for changes

This is kind of classical pull approach where every 'n' second poll for changes and update UI. The problem with polling every 30 sec were,

Unnecessary extra load on the server means fake http requests, 1 will have new message out of 100+ requests. Major hurdle in scaling app, suddenly the no of requests on server will increase multiple fold.

Delay in data refresh based on poll frequency and it won't be realtime.

We can't http cache the request using reverse proxy since it is user's personal data

Approach may be okay for sites like live cricket scores where requests can be http cached and served to multiple users and data changes frequently.

Approach 2) Push for changes

Second approach we came up with was push messages using websockets and libraries like socket.io etc. Using push the user experience is really awesome, user gets notified instantaneously about the new messages. No extra fake requests on the server and helps in scaling easily.

However, there are challenges

Setting up the Notification Server. We were using Apache server and it doesn't support websocket connections. Needs upgrade to our infrastructure.
Implementing server side component for Notification Server with security and authentication was significant effort.
More effort as compared to poll for maintaining state logic on Notification server & Javascript, requires resilience in building edge cases around data loss

To overcome over challenges we though of going to PaaS for Push Notification like PubNub and Pusher. However the challenge with PaaS is data security. Is it okay to send user personal data to 3rd party? The answer in our case of NO.

Approach 3) Push notification for changes

To overcome the problem of data security we came with the tweaked approach as shown above. We use PaaS for Push Notification only to notify end user about there is new message and no data. When the notification is received we do the poll request on main server to refresh the UI.

Key advantages of using this approach:

No effort required for building & maintaining Notification Server. Leverage the PaaS.
Just send the change notification, so issues with data security.
Less development effort since most of the PaaS provider has their own library to support all browsers.
Realtime updates to end user. Better user experience.
High availability and scalability using PaaS

We are using PubNub as PaaS and experience so far is really good.

Distributed team communication plan with help of anatomy of an iteration during project inception

2012-05-13T17:58:00.002+05:30

Agile software development is getting more and more attention now a days. One of the variations of it is called ‘Distributed Agile Development’. And a key ingredient for distributed teams to work effectively is “communication”. Quote from Kent Back on communication "Problems with projects can invariably be traced back to somebody not talking to somebody else about something important."

Currently we are doing an inception with a client and their past experience with distributed team has left them with some bitterness. In addition, the client has just started adopting agile practices in their software development cycle, so along with delivery we are also doing enablement for them. Also making sure business and new team member understand the different buzzwords of agile software development is very important.

In my experience every team has variations in mechanics of implementing agile, keeping all the principles intact. For example, story prioritization for some teams is Iteration Planning Meeting (aka Iteration Kick Off) and some teams go for more lean like method of story prioritization and adopts an iteration-less Kanban style pull mechanism. For distributed teams, the variations are important to know and everyone should understand the why’s of it. Daily standup need to be little different when working in distributed model, Iteration showcase and iteration length varies too from team to team. In my opinion all such mechanics are decided and derived based different parameters. However, the important part is to discuss, decide and communicate the mechanics & process of project’s agile development with all team members including business.

And its very important to plan for some session during inception to address need around all of the above.
Taking “Story Lifecycle” and “Anatomy of Iteration” our team (Sunit Parekh, Birinder Singh, Sarbashrestha Panda & Amit Dhakad) did the following session in little different settings and format. This blog is all about sharing the mechanics of running the session. I am calling it “Communication plan using anatomy of iteration lifecycle” and some of following is for the distributed team structure, where the business, SMEs, Product Owner and Product Sponsor are in UK and the development team is back in India. Hence few of the mechanics differ from co-located team structure.

Topics to broadly cover during session are,

What is a story? & Story lifecycle
What is an iteration? & Anatomy of iteration
Definition of Done (Story)
Testing approach (automated and manual)

We created a timeline on the board to start the session with first talking about what is story and story lifecycle.

As we kept discussing about the iteration, we started moving story stages onto iteration lifecycle. Also started building the events as shown in the picture below. Session took an hour to run.

Keep most of the stickies written before the session. This helps to remember all points to cover and running the session quickly. Have discussions during the session and make it as interactive as possible. As a team decides most of the stuff (e.g. definition of story done, iteration length, meetings/calls, Skype or Google Hangout) rather than dictating to others, this is very important as that makes it team's plan.

Here are a few things, that is different for a distributed agile team:

Standup, can be little different format. We use mingle card wall to just talk about cards updates if there are any queries or updates, achievements of the team during day, along with any blockers and show-stoppers for team. Keep it short. 15 min.
BA Catchup, business is busy most of the day with their job and so it is important to block SMEs calendar for certain time of the day so that can be utilized for any questions, queries and demo’s for feedback. Daily 1 hr. We do it just followed by distributed team standup.
Story prioritization, continuos process like Kanban. Regularly and BA Catchup call is used for story prioritization.
Iteration Kickoff and Iteration Showcase, we like Kanban style prioritization however it is good to get everyone together and share the project status, updates and showcase key features build since last meet and what’s the plan ahead for team. We do this as one meeting scheduled at the end of iteration as iteration showcase.
Tech Huddles, during initial stage of the project couple of times a week and later it’s kind of need basis once or twice a week.

Please share your comments, suggestions & experiences about this kind of sessions. What more we should cover, what we should watch out for...

Special thanks to Sarbashrestha Panda for reviewing the post and recommending valuable amendments.

Inception Board: Process timeline map using stickies

2012-05-07T21:13:00.001+05:30

One of the key challenges in an inception is to learn about the business processes and its workflow in a fair amount of detail in a relatively quick amount of time. The idea is, not to just learn about the client's business processes, but also to ensure that various stakeholders within the client actually share that understanding instead of having varying ideas in their own heads.

There are plenty of methods to study business processes but they are time consuming and heavy weight and may be not as collaborative. A classical way is to create process map diagram & value stream mapping. Such methods are valuable in different settings. In our context, an inception, we try to uncover information in enough detail to be able to kickstart delivery without spending loads of time and making it a formal inception phase.

During a recent inception, we (Sunit Parekh, Sarbashetha Panda, Birinder Singh & Amit Dhakad) combined ideas from different process analysis methods and came up with this hybrid light weight approach. It is not full blown value stream mapping but this approach does have elements of it and highlights the processes that are most time consuming and potentially wasteful.

The process map was built on a large canvas/wall, so that it would constantly be visible to all the participant in the inception. We started building the process map by plotting a timeline on x axis and noting different steps in the process flow on the y axis. Also, the actors/roles involved in the process were indicated for each process. A legends for denoting the actors was also put up on the wall for the ease of identification. Building this full as-is process map of the as-is process was done over multiple sessions spanning a couple of days.

Stage 1: After couple of session with business first cut of the as-is process timeline map looked something like this

Stage 2: After another iteration, the as-is process shaped up to something like this. It was also clearer in the client stakeholders' minds. It also had functional area clearly called out.

Advantage of having large visible canvas for the current as-is model is that it provided us a reference point at all times in inception. It became one of the living document for the team.

Stage 3: We then took this forward and mapped the future to-be process on top of it by putting magic chart in-between.

Being able to superimpose the potential 'to-be' process on top of the 'as-is' process allowed everyone to be able to 'see' how it was lighter weight in comparison. It also allowed us to ensure and double check that we were indeed covering the key activities / steps from the current process and losing / getting rid of only the non value adding ones.

At the end, we digitalized this using Gliffy. This structure / format / way of creating a process map was very well received by the business teams as well. They, in fact, suggested that the maps should be left as is in the room for anyone new to the business to use as a starting point for understanding their business processed. That feedback was indeed awesome. :) To me, this is called innovation and collaborative effort was what made it possible.

Stationary used for building such map:
Magic Charts with Grid
Mulitcolor Stickies (pink was used for timeline event/instances, yellow for activities in business process)
Arrow Stickies (cut sticky in arrow shape or get 3M arrow tags)
Small Dot Stickers (used for legends such as actors for activities)
Sharpies

Post written collaboratively by Sunit Parekh & Birinder Singh

Inception Game: Product Map

2012-04-29T16:40:00.001+05:30

One of the key factors for our project is that the client is almost 80+ yrs old in business. There are multiple products with different variations of it in the market. Now one of the task/session was to understand their product space, their groupings and how they differ from each other.

For understanding product and its family and how they differ from each other, we (Inception team - Sunit, Amit, Panda and Birinder) came up with a simple idea of creating a "Product Map".

Game: Product Map
Objective: To understand the breadth and depth of the products and their variations.
Duration: 30 min to 1 hr (depends on the # of products and product families)
Participants: SMEs and Business Stakeholders
Preparation: Create large board with matrix/grid having products on rows and comparison factors as columns. Write down all the known products and comparison factors on stickies to save time.

How to Play:

First steps to ask business to complete the product list, and then group them based similarities
Rearrange product list, keep the most popular product family and product on top
Discuss comparison factors, against which the products need to be compared or studied. Few options to consider are,

User base for the product
Process steps/work-flow the product goes through
Numbers related to product, such as revenue, time factors etc.

Start with most popular product first and considering it as the base, discuss the variations in other products in the same family. Put notes on small stickies for each variation discovered.

Variation:

Can be used for technical debt matrix e.g. put different modules/areas of codebase as rows, technical parameter such as complexity, test coverage, clean code etc as columns and start populating matrix. Can be very useful for creating visuals for legacy codebase.

How we did it?

In our case we put products as rows grouped together with same product family (yellow stickies in picture) and different processes steps followed for the product as columns (pink and blue stickies in picture - blue stickies are sub steps). .
We used different colors and size of stickies to show products (yellow), process steps (pink & blue small) and how they differ from each other (orange small).

Inception Board: Term Cloud

2012-04-28T22:45:00.001+05:30

During inception one of the job of the inception team is to learn the terminology used by the business. Generally the board is put up with all the business buzzwords, terms that they use in their day to day job. We started doing that on one board called 'Term Cloud'. Generally this board gets populated as side effect of all the games & exercise we do during inception also know as glossary, domain terms.

Our team came with an idea to play 'term cloud' as one of the exercise during the inception in initial days.

Exercise: Build Term Cloud
Objective: Everyone speaks the same business terms including development team, SMEs and stakeholders. Helps with more collaborative communication during inception. Better domain understanding. Less confusions with buzzwords having same & different meanings.
When to play: After a few sessions (preferably on the second day after you get a critical mass of terms)
Duration: 15-20 min
Participants: Business SME's & Stakeholders
Preparation: Prepare board having 'Term Cloud' written on it with few samples.

How to Play: very simple & easy to facilitate.

Distributes stickets to write buzzwords, jargons, terms.
Remember one term one sticky.
Give 5 min to write all the terms.
Ask participant to go in round robin fashion, explain the term and put it up on the board.
Put small sticky if that term requires explanation.
If different terms has same meaning, club them together.
If same terms has multiple meaning, come up with new term for less used meaning.

Variations:

Instead of everyone writing stickies, go in round robin and facilitator writes it on the board.
Create Term Cloud board and keep populating as and when new terms pops up. (old style)

How we did it?

Following is the sample from the inception that we are going through. Yellow sticky is the term and Orange is used as note to explain the term.

Sample Term Cloud Board

Ruby vs Java vs .NET - How much does it matter on web projects?

2012-04-25T18:44:00.000+05:30

I was looking into some statistics on my multiple web project and found one very interesting stats on simple Lines of Code (LOC).

* numbers are pure production code and does not include test code

After looking at above metrics, I asked myself: during initial stage of the project, are we thinking enough about every part of the software and doing enough research around which language, technology & framework to use for project?

On most of the projects we just talk and talk about which language is the right choice (Java vs Ruby vs .NET vs ...) for your web application development. We are in so much rush to start that we don't even have time to do enough study/spike on each technology decision and provide right technical solution to client.

So what should I be thinking about, apart from core language and framework while deciding on architecture in initial stage of the project?

Understand project and its components/parts during inception, discuss & make initial decision about choosing framework & library of different technologies. Make sure all components fit well with each other.
Web 2.0: JavaScript, HTML and CSS is big part of your web application code. Do not ignore that.

Is my application AJAX heavy? (page reload is kind of old fashioned now days). Choosing framework like angularjs, backbone.js, knockoutjs
Does my application has image cropping or chart need. Should I do it in browser using JavaScript such as raphaeljs.
Check out templating engine and its support e.g. mustache, haml and its port for specific language.
how about using less or saas for CSS. This can help reducing your css effort significantly and more beautiful, concise, clean code.

Data Store: Which data store is the right choice for kind of data processed and maintained by application?

Choices are plenty now days, starting from traditional relational database, NoSQL databases (mongodb, riak, couchdb ...), graph database (neo4j), fast in-memory like databases (redis & memcached).
Can I leverage client side data store (browser datastore) for features like storing user preferences, session information etc.

Evaluate Product as a Service: Can I leverage on PaaS for any feature in application?

Do I need push notification service to serve dynamic content on my pages? checkout pubnub
Is image processing such as resize and cropping is required for website? checkout services like lightspun and boxresizer

Go Could: Discuss and make initial thoughts on production environment setup and deployment. This helps on setting up local env which can be as close as production env and other non production setup.

Leverage could as much as possible.
Do I need to provision env using Chef or Puppet? Can I use same for local env setting and using tools like vagrant?
Can I use S3 with ColudFront to serve my static content?

Automated testing: how much and at which level? Decide on unit tests, integration tests and functional tests distribution. Make sure not to repeat same tests. Follow software testing pyramid structure.

Unit can be at different levels

core programming level unit tests
javascript unit tests are important since most of the sites now days are javascript heavy. Check out Jasmine for JavaScript unit testing.
template level unit tests

Service level integration tests using your choice of unit testing framework.
end2end functional tests, their are plenty of tools now days for browser functional testing. checkout phantomjs, tobi, cucumber, capybara and last but most popular selenium webdriver.

Rest all that should continue such as choosing right technology with framework for server side development such as RoR or Java/Spring or Python/django etc...

Happy inception.

Amazon S3 learning with HTTPS serving very slow

2012-02-25T23:01:00.001+05:30

On our site we have lots of images thumbnail getting served on HTTPS by Amazon S3. We were seeing very high load time in browsers upto 2-3 sec for approx 40-50KB size.

We tried following load test on S3 and the results helped us making decision.

Conclusion:

Serving S3 content over HTTPS is slow
S3 performance is not effected (does not degred by # of objects in bucket
Using CloudFront to serve S3 content over HTTPS gives significant improvement in time

NoSQL learnings & gotchas based on MongoDB experience

2012-02-14T09:13:00.002+05:30

I have been working on NoSQL database since last 6 months and had some interesting learning on when not to use NoSQL database.

All following learning are based on my 6+ months NoSQL experience with mongoDB. This is an ongoing document which I am planning to keep updating as I keep learning.

One primary gotcha of new technology is, when to use it by knowing when not to use it. Here is list of reasons, when you have to be careful choosing NoSQL,

If your data is structured. Means your data follows predefined structure for all records like online web store, ticketing system etc. [Added: 10 Feb 2012]
Your data has lots of relationships, including bidirectional like many to many etc. You need to carry references around in the dataset like customer has orders and order belongs to customer. [Added: 10 Feb 2012]
Need lots of indexes on a document. You need search your document by multiple search criteria, means either your data is structured or multiple document stored as single document in database. Need to revisit your document design. [Added: 10 Feb 2012]
Transaction is important. need to update multiple dataset and it should either all succeed or fail like financial transaction of transfer from one account to another and required multiple data updates. [Added: 10 Feb 2012]

Once you choose NoSQL don't fall into trap of designing and applying SQL best practices to NoSQL. Here is few example where SQL and NoSQL database differs.

Do not assume your query is successful if it returns successfully. MongoDB query returns immediately and queues up queries on server for execution. [Added: 10 Feb 2012]
It's not good practice to let your document grow over time. MongoDB stores document in sequential manner without much buffer, assuming the document doesn't grow over a period of time. And if you add more content in your document it has to move your whole document in new place. Trick is to create document with full content and default values OR adding some temp content while creating and removing it immediately. [Added: 10 Feb 2012]
Indexes are more costly in NoSQL than standard SQL database. [Added: 10 Feb 2012]

CMS usage pattern - choose your CMS

2012-02-07T14:37:00.000+05:30

Now days every business wants CMS capability in their app, at minimum for home page to show promotions, alerts, news & messages to end users. On each and every project I have been on in last 5 odd years, CMS was part of application in one or the other way. At the start of the project I always had to think and decide which CMS system should I go for. And working with different domain, some patterns started emerging in my thoughts and helped me making decision.

I have worked with multiple feature rich CMS products, which provides business critical features such as versioning, publishing work flow, content promotion... . However, one of the key pain area in all of this feature rich CMS products is that development of remaining custom components either as plugins or customization modules if very difficult. Developing these ~20% of the custom components takes lot of time & effort with specialized skills. Most of the time we need to hack to get custom component build in existing CMS.

And another pattern of CMS stated building in my mind, 1) Obtrusive and 2) Unobtrusive CMS. What I mean by obtrusive is, CMS doesn't integrate seamlessly in your application. Application needs to be developed within CMS. So now while choosing CMS I started looking into how easy or difficult it is to develop custom features/components within CMS.

And I started looking into following criteria while choosing CMS:

Support for digital asset management (image repository with resizing) are very important for CMS. Look for Amazon S3 support too.
Content promotion from non-production env to production env is one of the most used feature. e.g. Create all the required content in stage and promote it to production before launch. Gives more flexibility to push release early to stage and get the content ready and later deploy and publish content in one go to production.
Free text search has become defacto requirement now days for any website. And hence more critical feature for CMS to have. Don't forget to try and see how good SEO support and inbuilt search is. This could become make and break deal for some business while choosing CMS.
Be aware of oversold CMS features which are hardly used in production. Work flow management within CMS. Sounds good but practically what end user wants is just the option to preview and publish (as review mode) .

Which CMS to choose?

Type 1: Full CMS - Publishing & media websites

If you have capacity Build your own CMS. Here are some thoughts I have on the CMS system architecture.
Or
Based on the requirement and skills available within the team decide on the full featured CMS system such as Liferay, Drupal etc.

Type 2: Partial CMS - Online business websites

Take any of-the-shelf lightweight CMS, which integrates seamlessly (unobtrusively) in your application.

e.g. taking open source CMS as rails 3 mounted application (www.locomotivecms.com or http://refinerycms.com/ or http://www.browsercms.org) and build you application as normal rails 3 application. hint: opensource CMS can be forked if required.

Build your own simple unobtrusive CMS. Checkout this article and build your own core component.

Type 3: CMS for the namesake - few pages in application

Build your own wiki style CMS with lightweight admin interface to update markups. e.g. Markdown is very popular and available in all languages.

My take? just build what you need. initial investment will be more but it will save on customization, learning & lot with maintenance cost later. Spend time in development rather than learning, researching and hacking into CMS product.

Hope this article helps you in deciding your next CMS system.

Build your own CMS

2012-02-07T08:51:00.000+05:30

Unobtrusive CMS - that integrate with your web application seamlessly.

Yet Another CMS (YACMS)

Why build your own CMS?

CMS can’t be integrated within application, the application needs to be written in CMS. Drupal is written in PHP and it is full stack end2end. To develop application which is half & half, we need developer who can code in PHP Drupal way.
Some CMS are bound to frameworks - Radiant is written in RAILS and it ties down to specific version of rails and on top of that it changes rails default behavior so now I have to code according to radiant standard and not rails way.
Its quite difficult o migrate data from one env to another (test -> stage -> prod), take dumps from production to test/dev env, migration on version upgrade . One of the reason for this problem is use of RDBMS, which doesn’t suite the kind of data.
Editors friendly - users need to learn CMS way to create content. And its quick difficult for business users to create content in CMS due to its complexity. Should be able to see preview of the page before publish. Easy versioning and restore feature.
CMS doesn’t provide effective asset management within CMS. Integrating with other systems makes it really difficult for such tasks.

Goals:

Unobtrusive & Light Weight- I should be able to use CMS anyway I want to. Means I already have an application, I want to add CMS to it and not other way round.
Easy migration - be it from one env to another or upgrades.
Platform & Framework agnostic - I should be able to integrate & deploy on any platform.
Easy to extend - easy to create, integrate & extend features by means of plugins.
Integrated Asset Management - with base feature such as upload, resize, crop images within CMS.

Architecture:

CMS build as 3 independent components,
core (yacms-core) - engine that builds pages using modules (goal: Unobstrusive, Platform & Framework agnostic). this is the component this can be plugged into existing application unobtrusively.
admin web app (yacms-admin) - cool drag & drop UI for new page creation. allows editor to edit pages inline on actual site. (goal: Editor friendly), draft preview of the page.
web (yacms-web) - web app provides front end for end users to view pages. Allows different front ends such as for Sinatra, Spring MVC based Java, Mobile App.

plug-in architecture for modules (goal: easy to extend and add new features to CMS): page is made of modules. allows user to create their own module and integrate easily.
type of modules

page modules - page modules define page data e.g. paragraph, video, commenting etc.
application modules - modules that apply across pages such as navigation, tagging, google analytics etc.

modules/plugins has directory structure similar to rails applications, (this is just the draft to understand modules)

….
   modules
       <namespace>
           <modulename>
               views
               static
               controllers
               ...

dream goal is to make modules to be added/removed/upgraded using rake task and we need to provide module market place which will allow user to easily find modules.

Technologies stack

Ruby as development language for most of the components. This allows to run core on any platform e.g. using JRuby to run in container with Java web app and also planning to expose RESTful services if this needs to be out of container. Rails for admin web app component and Sinatra, Rails as sample web front end for CMS.
NoSQL - to store all page data as one record with page content, other data like tagging, comments, etc. also store all the customization such as layouts, snippets etc (goal: easy migration, versioning)
Domain Specific Language - to define page with layout, areas, block & modules (goal: Easy migration, Easy to extend, Framework agnostic)
Markup Language - to create page content
RMagick or ImageScience - for image processing

Design for Feature, Code for Story

2011-04-15T19:15:00.016+05:30

I have been practicing agile development engineering practices for almost more than 4 yrs now. One of practice about agile development is it talks about ‘code for story’, which means you just do simple enough implementation required for the story. Why? because you don’t know what’s required in future and will remain as it is or change.

Now considering ‘code for story’ practice, this is what happens on the ground. Pair picks up a first story of a feature and ‘just did what was required for the story to implement’, after few days another pair picks up a story on same feature and says ‘Ohh… I have to first refactor the code so that it can be extended to implement next story’, which means the work done by first story is partially or fully refactored. Fine, but similar approach continues as we keep picking up subsequent stories and Project Manager freaks out saying ‘Refactoring is good, but when it required for every story that we implement is not just right OR something wrong with my understanding of refactoring.’

Let me take a very trivial example to explain above scenario. For the first story, I just used simple member variable since that story deals with only one item,

public class Employee {
    String department;
}

next story extends the feature by changing simple member variable to List of objects which means changing all signatures and yes changing all tests as part of refactoring.

public class Employee {
    List<Department> departments;
}

Now next story requires extending the feature again with changing List to Map and again same refactoring but more tedious.

public class Employee {
    Map<string, Department> departments;
}

And we end up refactoring (throwing away) work done in previous stories. This scenario happens quite lot of time on projects.

We can avoid such scenarios on project by redefining a practice as ‘Design for Feature and Code for Story’. Which means when I am working on any story I keep in mind the all feature stories and design my application to cater for feature while making sure I just code what is required for the story in play.

To achieve this

developers and business analyst needs to make sure developers implementing the story has enough context about feature and understands the story scope within feature.
leads on the project to participate in discussion during first few stories of the feature development, call huddles initially to gather different ideas in designing for feature.
it might be that your first few stories takes little extra time that just doing simple way, however make sure to weight that extra time & effort vs. benefits in future story implementation.

Doing high level design early in cycle helps in faster development due to reduced rework and less throw away on project.

Watch Out and avoid following when you ‘Design for Feature and Code for Story’,

Doing low-level design and putting lots of effort in just design. Going into mode of ‘upfront low level design for feature’.
Do not implement or start writing code for future stories. It’s okay if your future stories may take just few hours, don’t do it now.
Design for Feature doesn’t mean you freeze it once decided. Let design still evolve as you keep implementing stories of the feature.