Tech Spot

Post Agile - State of Software Development

noreply@blogger.com (Aravind Ajad Yarra) — Thu, 21 Feb 2013 08:21:00 +0000

There is a significant change in how software is developed over the last decade. Agile had been the buzzword over the last decade and probably is most significant of changes in software development to date. Now that the hype around Agile has been subsided, it is worth an effort to understand what it is like post agile. Often we see Agile with a narrow perspective, limiting it to a few programming practices and nothing more. We also fail to notice the indirect influence and the larger impact of Agile over the last decade.

State of post Agile Software Development will have to be seen in multiple dimensions in order to really understand its impact:

Software Development & Life cycle tooling landscape has completely changed
Software Project management methods (apart from development method) have changed
Focus on Software craftsmanship
Re-imagined Technology stack

TOOLING

It is very interesting to see the way tooling has been changed over the decade. I must say this is the area where most innovation has happened. Almost all the aspects of software tooling has been changed; from version control systems to functional testing software. Several new categories of tools have born, which we couldn’t have imagined in 90s. Monolithic IDEs and tightly integrated eco systems are replaced with more specialized tools integrated via loose interfaces. Tools like GIT (distributed version control systems), Hudson (Continuous Integration Systems) are few examples of this change.

METHODS
While agile methods initially focused on changes related to software programming practices; over the time they went beyond. New ways of project management, new ways of testing have extended the focus much beyond. According to “State of Agile Development” survey in 2011 (http://www.versionone.com/state_of_agile_development_survey/11/), 80% of the organizations are using agile practices. That number sure tells a little about the adoption of agile methods and its impact.

CRAFTSMANSHIP
Thinking about Software craftsmanship is not very surprising given the agile method’s focus on people. In order to be successful, almost all the agile methods needs skill maturity in people, hence the stress on craftsmanship. Craftsmanship needs a learning organization & people and the focus on quality of work. Mentoring, a standardized toolbox, familiarity with tools is essential for a developer/tester to be a craftsman.

STACK
Last but not least, Technology stack also has been changing to fit the agile practices. Languages like Ruby, Groovy, frameworks like Rails and Grails and NoSQL databases have brought-in fundamental changes to reflect agile practices. Test cases integrated to the programming frameworks is not something one could have thought of a decade ago. Conventions over configuration, method missing and other such features have evolved to keep up with the expectations of agile practices.

PREPARING FOR POST AGILE
Most software development shops look at the impact of Agile in silos. For example, technology groups focus on agile tools and stack, but not methods. Process & Quality groups focus on methods but have neither have clue on tools nor the technology stacks. Craftsmanship is ignored altogether. Looking at impact together in all dimensions (across tools, stack, craftsmanship and methods) is half the work done. Aligning the team to fully benefit from the agile is the other half, which is probably the most difficult part.

Akka Essentials - book published

noreply@blogger.com (Munish K Gupta) — Fri, 21 Dec 2012 05:50:00 +0000

My book on Akka titled - Akka Essentials published by Packt was released on 26th Oct 2012. The book is available from PacktPub site as well on amazon.com.

"Akka Essentials" will show you the current challenges with Java Scalability and concurrency model and how Akka’s Actor Model can help you design and build applications that are inherently scalable and fault-tolerant. Whether you are building new applications or want to refactor an existing application, you will learn the right techniques to build and scale up in no time.

This book is aimed at developers, architects who are building large distributed concurrent and scalable applications using Java/Scala. The book assumes knowledge of Java/JEE concepts but no knowledge of Actor model is assumed.

I dedicate this book to my father who passed away on 30th Oct 2012. All earnings from this book will be donated to my father's favorite charity.

This post originally appeared at www.techspot.co.in

Application Scalability: Still elusive for Enterprises Apps

noreply@blogger.com (Munish K Gupta) — Tue, 21 Aug 2012 04:46:00 +0000

The advent of the Consumer Business application like Facebook, Twitter has changed the definition of Application Scalability. Decade back 10 million+ was a large user base, Facebook will touch 1Billion+ users by end of this year. There are hordes of applications in 100+ million user range. The techniques and approaches employed by these large Consumer Business Applications are different from the traditional enterprise application design and architecture techniques.

If we study the architectures principles employed by these large Consumer Business Applications, then we can conclude to the following

All the large consumer business applications are build and make use of Cloud Computing
Applications are build using a combination of open source products and platforms
Create solutions where the current set of solution do not meet requirement or scale up (e.g. HipHop, Hadoop, ChaosMonkey etc)
Constant knowledge sharing within the community (FB/Twitter/Google open sources lot of their internal products)

From the Application scalability perspective, the following key architecture patterns have emerged that are being used to scale the application

Stateless Applications - Modern Web applications maintain state, which means that they remember what you were did as part of your last request, and they remember all this data as part of the Session. The Stateful nature of the application means if the server holding that state goes down, the entire user state is lost. Traditional techniques like sync user session data across server nodes or persisting in data store do not scale very well. To overcome this shortcoming, the whole concept of creating stateless applications where the servers do not maintain state. The application uses cookies and makes use of RESTful API’s to craft the user experience. Newer frameworks like Play Framework, Node.js, Vertx.io all promote this stateless application development that scales with the increasing user load very well. Another option used is to offload session state is high throughput memory based key value store, which is common across stateless nodes.
Data Sharding – Another issue encountered with scale is the increasing load on DB servers. DB Servers do employ techniques like Master/Slave or Clustering techniques coupled with large powerful boxes but beyond 100+ million users, these techniques also fail. Further, the costly hardware and software licenses make the traditional DB options very costly. So, companies have used the MySQL as the base and created lots of solutions/topologies around the same. One of those techniques is data sharding. A Data shard is a horizontal partition of the data, meaning rows of the tables are held separately. There are numerous advantages to this partitioning approach. Since the tables are divided and distributed into multiple servers, the total number of rows in each table in each database is reduced. This reduces index size, which generally improves search performance. A database shard can be placed on separate hardware, and multiple shards can be placed on multiple machines. This enables a distribution of the database over a large number of machines, allowing database performance to be spread over multiple machines, greatly improving overall performance.Many new non-relational databases (commonly known as NoSQL databases) such as Cassandra, MongoDb and Hbase have been created with database sharding as a key feature.
Bring Data closer to user (Caching) – Caching has been adopted and used very well by most of the Consumer Business Apps. Open source products like memcache provide caching options across the tiers (web tier/app tier and data tier). Memcache provides a reliable alternative to the traditional caching solutions – coherence, terracotta. Further, the NoSQL solutions employ the caching engine to speed up the database read and write operations. Couchdb uses the memcache to provide an in memory read/write solution.
Service provider/consumer model – Another pattern that has emerged is the Service provider and consumer. It is derivative of the SOA model but leaner and simpler. The business functionality is exposed as a set of services via RESTful API with JSON data format. The presentation layers are the consumers that use these services to build up the user experience. The technologies to build the service and provider can be completely separate. Technologies are chosen based on what is best for the use case. Contracts are enforced by Service version and definition only. Front ends are typically built using PHP or Ruby over rails. Services are built using Scala, Akka, C++, Java etc.

Why Enterprises are looking at these patterns?

Enterprises have started adopting cloud (public/private) by and large. Elastic scaling means responding to the changing load pattern on real time basis. In the absence of application ability to scale, all that investment into cloud is waste
The traditional CPU/Core based licensing models are proving to be costly in the cloud world. Enterprise need to adopt the OSS frameworks/solutions that do not come with licensing baggage
The user expectations have gone up by few notches. Enterprise applications are compared and expected to have same resiliency & performance as any other Internet Scale application
Scale up model(adding more CPUs) for enterprise applications comes with its own headache of additional capital expenditure. Scale out models(add more smaller servers) that rely on commodity servers or can run over a hybrid cloud are becoming more cost effective solutions
Enterprises are going global which requires the systems to be up and available 24X7. Currently tightly coupled application designs do not lend very well to the scalability pattern

This post originally appeared at www.techspot.co.in

Top 5 Picks of Google IO 2012

noreply@blogger.com (Aravind Ajad Yarra) — Mon, 02 Jul 2012 14:39:00 +0000

Google IO 2012 developer conference has just concluded last week amid lot of fanfare. I for one think that Google has lot more influence on Enterprise technology than it seems. Some of what Enterprises see as latest and greatest of technology (such as Map Reduce) has been pioneered in Google a while ago. This is one the main reasons why I followed this event closely.

Over the 3 days Google has several technology and product announcements across the product lines. Here are the top 5 picks at Techspot.

#1 Google Compute Engine

With Compute Engine, Google made it intentions clear as a Cloud Service provider. Google entering Infrastructure as a Service (IaaS) market brings more innovation and cheaper compute resources; enhancing its portfolio beyond PaaS. How exciting!! It is interesting to see the reverse trend of PaaS offerings to IaaS offerings (Microsoft also recently started offering Linux VMs as a service). Does this mean uptake of PaaS is very low? Compute Engine offers Linux VMs running Ubuntu 12.04 and CentOS 6.2. Most of the concepts that Amazon EC2 uses such as zones, ephemeral versus persistent disks are present in Compute Engine tool. That said, this is a reasonable start for Google but a long way to be a threat to Amazon EC2. It is interesting to see Enterprises like Bestbuy listed as their beta customers.

#2 Jelly Bean, Android 4.1

Jelly bean, Android 4.1 is the newest version of the popular mobile operating System announced in IO. Apart from the several performance optimizations and various user level features, Jelly bean packs several interesting technology innovations. Prime among them is “Google Now”. Gartner has been talking about Context Delivery Architecture for few years, Google Now is probably first main stream execution of the same. Google now combines user location, calendar details, past searches, traffic, weather and time in interesting ways and provides useful information before even asking. One of the use case shown is, when you have an appointment at a certain location, it checks traffic and tells how long it'll take to get there. It notifies when you should leave, so that you can reach the destination on time.There are many other interesting features including Project Butter, Systrace tool, peer-to-peer service discovery, cloud messaging, smart app updates.

#3 Packaged Apps & Chrome Everywhere

Google chrome is the new generally available on Android 4.1.With good HTML 5 support, mobile web applications on Chrome in Android are very powerful. Chrome is also available on iOS. Packaged apps are one of the interesting features that are announced for Chrome. Packaged apps allow access to Chrome API, but are written in HTML 5, JavaScript and CSS. Packaged Apps are loaded locally and supports offline mode.In a way these are like Adobe AIR applications, but are written using standard web technologies. This is probably another category defining feature; developing cross platform hybrid applications running locally.

Some interesting stats on chrome are revealed. Chrome now has 310M active users across the world.

#4 Project Glass

Lot has been written and said about Project Glass from Google. Wearable computing is about to start. Project glass probably can make a new way of how we use and interact with computers. Technically not much has been revealed on project glass, other than the fact that it has camera, microphone, gyroscope and wi-fi connectivity. Key-note demo by Sergey Brin that involved blimp, sky divers, project glass and Google+ hang-outs is an epic.

#5 Nexus 7 Tablet

Technically Nexus 7 is just an Android Jelly Bean device. There is no reason to pick this other than for the fact that it is the first official Tablet from Google.At a price of $199 and with Quad Core Tegra 3 architecture and 12 GPUs, this tablet is a real power house. Wi-fi, gyroscope, GPS, accelerometer, Gorilla Glass, front facing camera, this is a sure main stream device and probably puts android tablets into millions of hands, another great revenue opportunity for Mobile developers. Interestingly Nexus 7 ships with Chrome as the default browser instead of Android’s native web-kit based browser. I guess Google is trying to avoid an IE 6 scenario here by separating browser from OS.

There are many more interesting announcements that didn’t make it to the top 5 list but I found them interesting enough to mention here:

Offline maps and offline voice recognition for Android
Enhanced Gesture support in Android
Apps Script for Good Apps Automation
Google Docs Offline

What are your top 5 picks?

This post originally appeared at www.techspot.co.in

Cloud Connect 2012, Bangalore

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 25 May 2012 15:59:00 +0000

Cloud Connect is a global Cloud Computing event that has come to Bangalore, India this year. The event is organized by UBM, a reputed media organization that publishes Information week magazine. Other popular events organized by UBM includes Interop. Bringing Cloud Connect to India, shows the importance of India as a Cloud market. Event happened this week 5/24 to 5/25 at Nimhans Convention Centre.
IBM is a platinum sponsor and the event started with key note by Vamsi from IBM.
IBM's Vamsicharan Mudiam will keynote@cloudconnect_in on May 24 in Bengaluru, #India. View his profile: bit.ly/KnahAi #CloudConnect

IBMCloud

4 days ago

Reply Retweet
IBM's session had several quotes form their recent C-Study report. Some of the interesting findings include the changing CEO priorities. IBM says cloud adoption is increased from 13% to 26% this year.
Ibm c-study: Technology factors and People skills are at the top. Glad to see that. #cloudconnect

Aravind Ajad

2 days ago

Reply Retweet
Like lot of cloud conferences earlier, this conference has lot of focus on Infrastructure Cloud. Most of the sponsors and speakers are focused on the Infrastructure aspects of the Cloud.

Some of the key observations I made were:

* There is quite a bit of talk on private clouds from the attending vendors. There are quite a few sessions covering this. There are sessions from VMware, L&T infotech, Citrix, Cisco, Brocade and so on. This is not surprising, given Infrastructure is the focus for vendors. There are however, few refreshing sessions that took a different take. Director from Citrix cloud.com talked about cloudstack and how it can be used to build amazon like clouds.
Architecting aws style clouds - Ram Chinta. cloud.com #CloudConnect

Aravind Ajad

9 hours ago

Reply Retweet
* There is significant focus on security as well. In fact, one of the conferences rooms almost dedicated to security. And there seems to be good response to these sessions as well. Probably shows cloud security still is a major concern. There is also some pragmatism on how the trade-offs can be made.
Common thread in al presn.@ #cloudconnect on#cloudsecurity -impln.depends largely on orgn.biz goals & risk appetite thereof!

Runa Desai Dalal

5 hours ago

Reply Retweet
1 mre good talk by Akhil Behl of @CiscoSecurity busting myths of #cloudsecurity!Cloud consumer just as responsible as a CSP!#cloudconnect

Runa Desai Dalal

4 hours ago

Reply Retweet
* Another interesting point is the focus on Applications. This is probably a departure from many cloud conferences. There are several talks on this topic.
Krishnan, editor at CloudAvenue and a popular name in Cloud circles spoke on the future of PaaS platforms. He says PaaS is the future of Cloud computing. He backs his arguments by what is happening and the trends he is seeing. He also thinks future of PaaS is really different from what it is now. Future PaaS platforms will be platforms beyond the current capabilities, with lot of Intelligence. They will have several services and capabilities built-into them such as Data Markets, Mobile Support and Social Connectivity. Data plays a big role in the future PaaS. These services makes building intelligent applications on Cloud very easy.
.@krishnan on stage - Intelligent platforms, future of pass. #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
Big data is at the core of next Gen of PaaS@krishnan #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
He gave some examples of which platforms that shows this potential of what future PaaS can be.
Candidates for future of pass: flow.net, brroklyn, pervasive #cloudconnect @krishnan

Aravind Ajad

10 hours ago

Reply Retweet
There is also a session on Cloud platform comparison from Jankiraman. He says Amazon Bean Stalk compared to CloudFoundry, GAE a good compromise between control and flexibility.
Good introduction to PaaS platforms and their current state. Like the PaaS comparison by @janakiramm at#cloudconnect

vivekjuneja

10 hours ago

Reply Retweet
Jason Bloomberg of Zapthink talked about how to used RESTful hypermedia state to make applications Elastic. He says adopting applications to the cloud is not lift and shift, but a change in the architectural thinking. He says RESTful architecture can help build elastic applications that leverage the potential of Cloud. By keeping the state on in the hypermedia, the applications can be more elastic. Jason also shared free poster on Zapthink's vision of future of IT. It is very interesting and full of Cloud, Mobile, Social, Democratization of technology. In a way pointing to demise of Enterprise IT :-).
RESTful Hypermedia state to make applications elastic.@TheEBizWizard #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
Free poster 'Vision for Enterprise IT 2020'@TheEbizwizard #cloudconnect

Aravind Ajad

11 hours ago

Reply Retweet
There are also couple of sessions focused on Data. One session from Nokia architect on Big Data adoption. There is a session form CTS architect on NoSQL database constructs.
* Popularity of AWS is evident through sessions. Being a proto cloud provider, there are references to AWS in almost every session. There are quite a few talks on how to deliver highly scalable applications on AWS. One such interesting session by Harish, explained how to achieve a peak tps 80000 requests using elasticity of AWS.
Good session on infrastructure elasticity using AWS.#cloudconnect

Aravind Ajad

a day ago

Reply Retweet
Sessions from me and my Wipro colleagues (not officially sponsored :-) ):

Munish Gupta spoke on Designing Performance of Cloud Applications, while I spoke on Delivering Enterprise Scale Apps on Cloud.
RT @write2munish: Speaking at the CloudConnect Conference in Bangalore on May 24 bit.ly/JFFXkB #cloud #cloudconnect @cloudconnect_in

Cloud Connect India

2 days ago

Reply Retweet
RT @aravindajad: All set for my speaking session at#cloudconnect conference tomorrow. goo.gl/V5Nsf @cloudconnect_in #in

Cloud Connect India

a day ago

Reply Retweet
Srinivas Adhyapak another Wipro colleague spoke about the Impact of cloud on Telcos and how they are in a good position to become public computing providers.
Sessions from my collegue Munish and me are in cloud (Slideshare) for your reference. Thanks to all of those who attended our sessions.
Cloud connect - Delivering Enterprise Scale Applications on Cloud

aravindajju
Thanks to UBM for bringing this event to India. UBM team has put a fantastic show and my appreciations to Srikanth and team.
Cloud Connect Bengaluru draws to a close! Thank you all for making it a success! We will keep in touch herebit.ly/KxNrEy

Cloud C

This post originally appeared at www.techspot.co.in

Blackberry 10 – Putting Developers First

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 04 May 2012 15:37:00 +0000

Blackberry 10 OS is unveiled this week at Blackberry World. In many ways Blackberry 10 is probably the last chance for Blackberry to gain back the market share and be a formidable competition to other Smartphones. If the announcements at the conference are any indication, Blackberry is going all out this time to make their next generation of OS as the best smartphone platform.

Some of the interesting features of BB10 mentioned in the key note:

UI Overhaul – More modern user interface based on Panels (similar to WP7) and the flow interface that switching between applications smooth and seamless. At last, BB user interface is on par with the competition.
Intelligent Keyboard – Finally BB is betting on touch interface. BB is trying to bring its legendary physical keyboard expertise to touch interfaces. Keyboard with gesture support and intelligent prediction keeping the usability a key priority.
Camera with timeline - Camera has a timeline history, so if there is an issue with a photo captured (such as somebody eyes are closed), one can rewind and capture another moment which is more perfect.

The most notable of all, is the focus on developers. There has been significant focus on Developers in the CEO keynote. One of the BB’s problems has been to attract developers to build quality apps to be able to compete with other smartphone platforms. To address this problem, BB has spent significant time and effort and put the developers first in BB10.

Here is how -

For starters, Blackberry 10 is not based on the older platform BB7, but is based on QNX (acquired by BB in 2010). QNX is a RTOS that has been existence for a while and is used in telematics and other mission critical applications used in medical and telecom industries. QNX is also the OS used in Playbook tablet released last year. QNX uses micro-kernel architecture and has great tooling support, making it a solid platform, attractive enough for developers.

Keeping developers first, BB has taken quite a different approach for applications (on BB 10) compared to the competition. All of the Blackberry competition has a single deeply integrated application development framework and uses a single programming language. Apple uses Objective C, Google with Java and Microsoft with C#. BB however, took a multi-language multi-framework approach, to keep a low entry barrier for all kinds of application developers. Following graphic illustrates how much BB10 is doing to include all kinds of developers, leaving nobody behind.

Blackberry WebWorks SDK:

Webworks SDK supports BB application development using HTML 5 and CSS. This is really the lowest barrier for application developers. Millions of web developers can potentially become blackberry developers with very little training. Unlike other smartphone platforms where HTML 5 applications are secondary citizens, BB10 has embraced the web development paradigm.

Ripple mobile environment emulator for testing the applications for different form factors
Local storage of HTML 5 and CSS applications, does not server access
Javascript extensions providing access to Blackberry messenger, PIM and hardware
bbUI.js for native look and feel

Blackberry Android Runtime:

Android has pioneered the use of virtual machine as the first class development platform on smartphones. Dalvik on android provides a bytecode compiler and a runtime allowing applications developed in Java language, to run on Android. Given there are thousands of android applications already running Dalvik, BB has made a smart move in transitioning these applications onto BB 10, tapping into large existing Android developer base. While there are quite a few API compatibility limitations, nevertheless BB Android runtime provides an easy option for porting existing Android applications.

Supports Android 2.3.3 (Gingerbread) applications
Online repackaging tool to convert APK files (android application package format) to BAR files
Support for multiple windows and in app payment

Blackberry Adobe AIR SDK:

Adobe AIR is a popular UI application development platform based on Action Script and Flash runtime. Action script is a variant of Java Script hence majority of the web application developers uses AIR to develop cross platform UI applications. AIR already supports Android, iOS and even desktop OSes and thousands of applications already written. Given AIR is support, all the AIR applications can be ported to BB10 easily. What is interesting to note is that, AIR applications are developed using Flex, which is a quite popular with Enterprise to develop in-house rich UI applications.

Blackberry Native SDK:

BlackBerry 10 Native SDK is targeted at building high-performance native applications. Supporting C++ as the language, it is not really meant for developing business applications, but meant for applications that need good response time, such as games.

Scoreloop social gaming SDK provides support for user profiles and global leaderboards, awards
Push Management APIs allow developers to create Push-based apps
Visual Studio IDE plug-in will now make it easier for VS developers
Cascades SDK – New UI framework for developing native UI applications supporting C++ and Declarative Qt Markup Language (QML)

As it happened with HP webOS and to an extent with Microsoft WP7, applications can make or break the smartphone platform. With BB 10, Blackberry is taking all the right steps to get the developer support, which is crucial to their success. BB is also pulling all the right strings to make it easy for developers to develop their applications on BB OS. That said, not supporting legacy Java applications (developed using BB 7 java SDK) is a significant issue.

It will be interesting to see how successful BB 10 will be and whether it can bring back the lost glory for the Blackberry.

Further reading:

Blackberry 10 announcement - http://www.blackberryworld.com/event-info/about
Blackberry developer blog - http://devblog.blackberry.com

This post originally appeared at www.techspot.co.in

Developing Scalable Apps for deploying on IaaS cloud

noreply@blogger.com (Munish K Gupta) — Tue, 01 May 2012 03:03:00 +0000

My presentation on Developing Scalable Apps for deploying on IaaS cloud, which i presented at Cloud Developer Conference 2012
Developing Scalable Apps for deploying on IaaS Cloud CDC April 2012

This post originally appeared at www.techspot.co.in

Plumbing the Web

noreply@blogger.com (Aravind Ajad Yarra) — Mon, 30 Apr 2012 14:22:00 +0000

Cloud based services are so ubiquitous that it is almost impossible to imagine our daily life without these. These services range from web e-mail services like Gmail to productivity apps like Evernote to Social Networks like Twitter to Information services like Weather.com. I can count at least twenty different services that I use on a daily basis and probably hard to live without. I get tremendous value from these services, both personally and professionally. However, there is an ugly side to these services. Each of these services live in their own island and it is hard to go on to each service to keep up with it.

Isn’t it great if I can push my favorite Google Reader article to Instapaper for a later reading?
Isn’t it great if I can get an SMS alert, if the weather.com forecasts rain in Bangalore today evening?
Isn’t it great if I can move my personal tweets to my Gmail account for later reference?
Isn’t it great if I can get an email alert if my twitter follower mentions me?

What we need is a way to plumb all these discrete services together to create more valued experience.

Ifttt.com
Ifttt.com is a web plumbing service that connects multiple web services together to create services that are much more useful than those individually are. Ifttt.com stands for If This Then That. It essentially creates a macro environment for web (à la macros in MS Office), enabling knowledge web workers to automate tasks across web. Given the workforce in Enterprises are increasingly populated with Gen Yers, such web plumbing services have an important place in the future of Enterprise IT.

It provides basic constructs to monitor and evaluate an event and then take an action on the same. To show how this service works, please see the screen grab below. This graphic shows a Task which sends an SMS alert if there is Rain predicted tomorrow on Weather.com.

As can be seen in the graphic there are two simple constructs Trigger and Action. There are predefined channels which can be used to define the triggers. In this case Weather.com is used as the channel and the trigger is “Tomorrow forecast calls for Rain”. Action is defined as an SMS message, which another pre-defined channel with my phone number attached to it.

Ifttt.com runs this task every 15 minutes and checks if there rain is forecasted tomorrow and then sends an SMS alert. That is how simple using this service. The user interface is barebones and is very simple and easy to use. One doesn’t need to have any Programming background to use. But don’t let the simplicity undermine its power. As Douglas Adams quotes in ‘The Salmon of Doubt: Hitchhiking the Galaxy One Last Time’, “The future of computer power is pure simplicity”.

Under the hood of Ifttt, REST APIs and RSS Feeds are hard at work. These technologies enable easy programmable access to various web services. Another important, but often overlooked technology that forms the technological foundation of Ifttt is OAuth. OAuth enables token based end user authentication and authorization between multiple web services.

Ifttt.com is not the first service to attempt web plumbing. There is a great service called Yahoo! Pipes, which probably started the idea of web plumbing. Yahoo! Pipes is very visual with a graphical interface reminiscent of popular WISYWIG image editors. Yahoo has done a great job in abstracting the complexity of REST APIs, RSS feeds as visual widgets that can be dragged and dropped. Unfortunately the service couldn’t get the traction with masses, however is quite popular among web geeks and programmers.

Extending the Idea to Enterprise:
It doesn’t take much imagination to see how useful web plumbing (in specific ifttt.com) can be in Enterprises. Enterprises have been trying hard for a long time to seamless integration of internal IT services. Years ago Enterprises have embarked the journey of Service Orientation Architecture, but are yet to reach the distant dream of seamless integration. Adoption of cloud SaaS services have only made reaching the dream harder.

Given how ifttt.com has brought web plumbing to masses, there is lot of speculation that ifttt.com is pursuing Enterprise Interests. Recently ifttt got a funding of $1.5 million, not bad for a two person company. Can’t wait for an Enterprise edition of ifttt.com.
Cloud Integration Platforms are already rage in Enterprises. To name a few, IBM’s Cast Iron and Dell’s Boomi are already heavy lifting the integration between on-premise services and SaaS service hosted in the cloud. SnapLogic (www.snaplogic.com) is another interesting Cloud Integration Platform. It has pioneered the use of App Store for its cloud integration platform. However, all these platforms are targeted towards the IT developers and not the Business Users. One day, I hope these services learn something from ifttt.com.

This post originally appeared at www.techspot.co.in

Social Networks Going Visual

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 13 Apr 2012 16:13:00 +0000

Social Networks are traditionally focused on sharing of status messages and links. The focus of these networks was about sharing of all kinds, but not as much as about the Visual aspects. This focus has been shifting recently to Visuals. This can be seen in the recent growth of the Visual Social Networks such as Instagram and Pinterest. These networks put the Visual aspects in front and center of the sharing.

Some of the facts that support this observation:

This week Facebook announced to buy Instagram for $1bn. (Instagram has just 13 employees but has about 50 million plus users. Instagram works by providing several photo filters that can make ordinary photos / pictures visually appealing. After applying filters these pictures can be share.)

Pinterest has 12 million users and has grown 145% from January to March. (Pinterest operates like an online pin board. Pinterest allows to bookmark (called pinning) any pictures that we see online and share them on a board. Anybody who likes your picture can also pin it. Popularity in terms of Pins most likely to get noticed and drive traffic)

The reasons for such focus on Visual aspects could be attributed to:

High quality Cameras on Mobile Devices:

With Smartphones, People now have access to high quality cameras all the time, leading to sharing pictures more and more. Some of the smartphones such as iPhone 4S have high quality cameras that can give many dedicated Point and Shoot cameras a run for their money.

Signal to Noise ratio on Traditional Social networks:

Traditional social networks such as Facebook, Orkut, MySpace, Twitter and Google+ can do so many things; so much so that it is hard to align with People’s interest. People refer this as Information Overload. I refer to this phenomenon as Social Pollution and Social Fatigue. This exact phenomenon is what is leading to Interest Based Social Networks. Visual Social Networks are a subset of Interest based Social Networks. Other examples of Interest based Social networks are – Quora.com (Q&A based Social Network based on Topics interesting to a User), Foodspotting (Recipe based Interest Networks) and Endomondo and Fitocracy (Fitness based Interest networks)

Visual Appeal:

Humans are inherently visual. Visually appealing object stimulates our brains and makes us enjoy and share them :-)

While the reasons can be many, the impact of Visual social networks especially is astounding. Especially Pinterest has an astounding effect on online commerce. On-line businesses are flocking to have their presence felt on Pinterest. Here are some interest facts on how Pinterest is contributing to Social Commerce.

Impact on On-line / Social Commerce:

Impact on On-line Commerce can be measured by the amount of referral traffic a Network brings. As can be seen in the graphic below Pinterest referral traffic has increased significantly within a month. Amount of referral traffic Pinterest brings is more than Twitter and Google plus. Given the size of Pinterest this is really significant. This can explain why on-line commerce world is eager to have their presence felt on Pinterest.

In order to capitalize on this trend, on-line businesses are now focusing on having high quality and artistic product images on their sites. This is one of the many ways how On-line businesses are leveraging the Social networks generate more leads and better sales.

References & Further Reading:

Everything you need to know about Pinterest http://techcrunch.com/2012/03/14/this-is-everything-you-need-to-know-about-pinterest-infographic/
Pinterest drives more referral traffic than LinkedIn http://blog.shareaholic.com/2012/03/pinterest-referral-traffic-2/
Why Instagram is worth #1bn http://mashable.com/2012/04/12/instagram-worth-1-billion/

This post originally appeared at www.techspot.co.in

Consumerization of IT – what it means for the Architect

noreply@blogger.com (Munish K Gupta) — Tue, 24 Jan 2012 07:30:00 +0000

Consumerization is described as the trend for IT to first emerge in the Consumer space and subsequently makes its way into the enterprise world. But what exactly in the consumer world, that is making the users, demand the similar things from the enterprise IT. To understand the underlying need, we need to first examine the basic requirements of the user.

Kathy Sierra, co-creator of the Head First series of books and founder of javaranch.com, describes the hierarchy of needs from the user(s) perspective. The needs are stacked in the order of increasing engagement from the user. Starting with the basic needs of a defined functionality and its correctness, moving on to the learnability, efficiency & usability and finally culminating in intuitiveness and enchantment. Merely provision of correctly working functionality is not guarantee of the success of the application(s). The idea is to hook the user; the application needs to do something extra.

Using the hierarchy of needs as the background, we will see how the applications in the consumer and enterprise world stack up.
Now, in the consumer world, the advent of Consumerization started with the proliferation of the mobile devices. In the connected world, the device (smartphone or tablet) imposed certain constraints

Screen size is small and limited
Processing power of the device is limited
Interface is touch enabled
Connectivity is not constant (read patchy) and speed/data is limited (no unlimited data plans)
User audience demography not known

Now, the Apps needed to be designed keeping in mind the constraints, which meant that the Apps are

Focused on providing only one functionality
Have simple and intuitive messages/steps in the absence of any help/guides
Providing simple interfaces (cater to worldwide audience)
Having uncluttered screens as real estate is limited
Using UI Controls that are big and usable (touch interface)
Having intuitive workflows
Engaging to stand apart among the millions of other Apps

This meant that the functionality does not require the user to be a geek. The average individual user can understand and use the App functionality without any external help. Majority of the Consumer Apps are heavily focused on the Efficiency and Usability part. Few of the Apps have been able to enchant and engage the user (e.g. Angry Birds).

In the enterprise world, the device is most likely to be powerful desktop/laptop with a large screen. The connectivity is via LAN or broadband. As a result the Enterprise Applications are more industrial with high focus on providing long list of functionality and ensuring the correctness of the same. Consumers facing business applications usually focused on usability, Specialized enterprise applications (e.g. Call Center Applications) focus on the overall workflow efficiency, but majority of the applications are resource intensive, lack efficiency and have high learnability curves.

The enterprise user experiences the simple, intuitive consumer apps on their mobile devices. With the advent of BYOD (Bring Your Own Device), enterprise user’s started bringing their smart devices into the enterprise. Soon the users are comparing the experience of consumer apps with the enterprise business apps; they wonder why the enterprise apps can’t provide a similar experience. Why are the enterprise apps so low on learnability, efficiency, intuitiveness factors?

The whole idea behind the Consumerization of IT is not just measures like BYOD but to bring the missing intuitiveness, usability and efficiency into the enterprise applications.

So, what does Consumerization means for the Architect?

When designing and architecting enterprise business applications, following considerations need to be kept in mind

Connected users means the applications need to be available 24 X 7 ( Always On, Always Available), 24 X 7 availability requires application solution(s) to be elastic – expand or shrink based on the load
Intuitiveness and Usability have to be high on the agenda when designing the interface and workflows
Solution needs offer APIs for additional applications to be build
Integration with systems and other applications (including SaaS) need to be simple, straight forward and well documented
User experience is the key to successful business application (it was always the key) but need to be (re)designed for the connected & mobile devices
Product evaluation(s) will now include SaaS applications that can provide the functionality
Saas adoption will increase leading to newer challenges in integration, data security and privacy measures
Applications need to be designed/tested keeping in mind various devices/OS combinations (days of designing apps that worked on IE6 only are over)

The impact of consumerization on the Enterprise IT will be felt in the years to comes, BYOD is just the harbinger of things to come.

This post originally appeared at www.techspot.co.in

Bigdata Trends in 2012

noreply@blogger.com (Munish K Gupta) — Tue, 17 Jan 2012 07:30:00 +0000

Bigdata has caught the fancy of each and everyone. Consumer and enterprise world is looking to adopt the Bigdata. Analysts are talking about it, proclaiming it to the next big thing!

This is my take on how the Bigdata market space will evolve in 2012 and years to come.

Data Analytics Service – We will start seeing SaaS vendors that will start offering data analytics as a service (e.g. 1010data). These vendors will be able to pull in your unstructured data and provide you standardized reports and data services. This is very similar to the Web analytics (like Omniture) but access lot more data to provide patterns. 70-80% of the data analysis pattern will be similar across industries. These vendors will be able to provide these reports as off the shelf capabilities. For the other 20%, the web interface will allow the user to create patterns to analyze the data

Data Visualization Service – Analysis is only one part of the data, the other big thing will be able to visualize the data (more like infographics) that help make sense of the scale and data points. Business will need help to make sense of all this analysis and again 3rd party vendors (via products or Saas) will be fulfilling this space

Crowdsourcing Model – enterprise will start using the crowdsourcing model to help analyze and find patterns in the data. Goldcorp was one the original pioneers that used the crowdsourcing model to analyze the geological data. This model will be seen more when dealing large geospatial data in the areas of mining, drilling, oil ad gas

RDBMS vendors will make a comeback – Bigdata has been scoring high on account of their ability to analyze and process unstructured data. The NoSQL vendors and products are all claiming to be the holy grail of data storage, fast processing and analytics. I believe, the Big Vendors –Oracle, IBM and to some extent Microsoft will release their version of relational databases that can store (using automated data sharding) and process large amount of data (with a SQL MapReduce layer). The vendors will incorporate techniques from the NoSQL world coupled with in-memory storage (think Coherence, eXtreme Scale) to provide sub second read write response times. The market is too big for the existing vendors to not claim supremacy

NoSQL market will shake up – The current crop of NoSQL solutions has emerged as need to fulfill certain use cases. Their base features and functionality have been decided by the patronizing user client/project co-sponsors (e.g. Cassandra – Facebook, Hadoop-Yahoo, membase-Zynga). These NoSQL solutions work very well for the specific use case. Attempts to broad base the use case, mean a lot of effort needs to put in. Along the way, some products will merge, some will lose community support and other will get acquired by product vendor companies

Data Analysis Libraries - To help the developers analyze data, the common patterns and designs will be available as off the shelf libraries. This means, analyzing data for most common patterns will be easier and will become no-brainer. The projected need for Data scientists will not materialize to the same extent with the proliferation of these libraries. E.g. MS Cloud Numerics

Bigdata will follow the Consumerization model, where the core infrastructure either comes as a service or an appliance. The data analytics, data visualization will be provided as a standard set of services on top of the raw data with an ability to create your own model.

Do let me know, how do you see the Bigdata market moving forward and do you agree or disagree with my predictions?

Application Performance and Antipatterns

noreply@blogger.com (Munish K Gupta) — Wed, 04 Jan 2012 13:35:00 +0000

Any application you pick up, there are some issues – big or small. There will be copy-paste code, mistakes, algorithms which could have better thought through. But what distinguishes an antipattern from these normal errors is that like patterns these antipatterns are recurring throughout the code base. In my recent experience in dealing with performance issues, I had observed certain recurrent themes that are undermining the overall application performance. Most of these antipatterns are well documented but it seems we do not learn from others mistakes. We need to make our own mistakes. I am recounting some of the common patterns that I observed in the recent months.

Excessive Layering - Most of the underlying performance starts with the excessive layering antipattern. The application design has grown over the usage of controllers, commands and facades. In order to decouple each layer, the designers are adding facades at each of the tiers. Now, for every request at the web tier, the request call goes through multiple layers just to fetch the results. Imagine doing this for thousands of requests coming in and the load the JVM need to handle to process these requests. The number of objects that get created and destroyed when making these calls add to the memory overhead. This further limits the amount of requests that can be handled by each server node.

Based on the size of the application, deployment model, the number of user’s, appropriate decision need to be taken to reduce the number of layers. E.g. if the entire application gets deployed in the same container, there is no need to create multiple layers of process beans, service beans(business beans), data access objects etc. Similarly, when developing an internet scale application, large number of layers start adding overheads to the request processing.

Remember, large number of layers means large number of classes which effectively start impacting the overall application maintainability.
Round Tripping- With the advent of ORM mappings, Session/DAO objects, the programmer starts making calls to beans for every data. This leading to excessive calls between the layers. Another side issue is the number of method calls each layer start having to support this model. Worse case is, when the beans are web service based. Client tier making multiple web service calls within a single user request have a direct impact on the application performance.

To reduce the round tripping, the application needs to handle or combine multiple requests at the business tier.
Overstuffed Session- Session object is a feature provided by the JEE container to track user session during the web site visit. The application start with the promise of putting very minimal information in the session but over a period of time, the session object keeps on growing. Too much of data or wrong kind of data is stuffed into the session object. Large data objects will mean that the objects placed in the session will linger on till the session object is destroyed. This impacts the number of user’s that can be served by the application server node. Further, I have seen, application using session clustering to support availability requirements but adding significant overheads to the network traffic and ability of application to handle higher number of users.

To unstuff the session object, take an inventory of what all goes there, see what is necessary, what objects can be defaulted to request scope. For others, remove the objects from session when their usage is over.
Golden Hammer (Everything is a Service) – With the advent of SOA, there is tendency to expose the business services, which can be orchestrated into process services. In the older applications, one can observe similar pattern being implemented with EJBs. This pattern coupled with the bottom up design approach at times, means exposing each and every data entity as a business service. This kind of design might be working correctly functionally, but from the performance and maintenance point of view, it soon becomes a night mare. Every web service call adds overhead in terms of data serialization and deserialization. At times, the data(XML) being passed with web service calls is also huge leading to performance issues.

The usage of services or ejb’s should to be evaluated from application usage perspective. Attention needs to be paid on the contract design.
Chatty Services – Another pattern observed is the way the service is implemented via multiple web service calls each of which is communicating a small piece of data. This results in explosion of web services and which leads to degradation of performance and unmaintainable code. Also, from the deployment perspective, the application starts running into problems. I have come across projects which have hundred plus services all getting crammed into a single deployment unit. When the application comes up, the base heap requirement is already in 2Gb range leaving not much space for application to run.

If the application is having too many fine grained services, then it an indication towards the application of this antipattern.

The above mentioned antipatterns are frequent causes of application performance issues. The teams usually start with the right intentions but over a period of time, things will start slipping. Some of the common reasons

Lack of design standards and reviews processes – even if these exists, the delivery pressure is leading to skipping these processes
Team members inexperience or narrow view leads to every programmer only looking at their module and nobody is looking at the overall application performance
Continuous Integration(CI) tools not integrated with compliance check tools like PMD, Checkstyle, FindBugs etc
No focus on profiling of the application on regular basis during the code construction phase
Not evaluating the results from the Load tests to decipher and fix the underlying issue (blaming the poor infrastructure setup)

What are the other antipatterns you have observed that have contributed to the degradation in the application performance. Do share!

Deciphering an Outage

noreply@blogger.com (Munish K Gupta) — Mon, 19 Dec 2011 13:40:00 +0000

Recently, there was a major fire outage reported. The building maintenance team informed the fire brigade department. The fire brigade came in, helped douse the fire and tried to salvage whatever they could. The owner of the building demanded an independent assessment of the reasons behind the fire, what triggered it and how to make sure this does not happen again. The building maintenance team called in an independent assessment team and asked them to comb the carnage and find out what might have caused this!

The independent assessment team went through the motions of accumulating the evidence, looking for telltale signs of what might have gone wrong and then eventually did an event reconstruction on how events would have unfolded on that day. Subsequently, the team came out with the recommendations that need to be implemented to make sure that such an outage does not happen again.

Typical reasons for most of fire outages are –not conducting regular fire safety drills, violation of the building codes, not designing the fire exit paths, not keeping common area’s clean, storing hazards materials in the basement and building unauthorized extensions. This was the same case here and the recommendations were mostly in same line. Do regular fire safety drills, design fire exit paths, adhere to the building codes, demolish unauthorized extensions, stop storing hazards materials in the basement.

Can you guess the reasons for?

why the building was built in violation of so many norms
how did all the inspectors certified the building
why did the client not got an independent assessment of the building done before possession
why did the maintenance team accepted the building without looking into the cost of maintenance in the absence of basic safety
why should the people using the building not lose confidence in the safety of the building

The above scenario is analogous to what I encountered recently. Only difference is that it was a customer facing web application that faced the outage.

Managing Enterprise Group Mail Content

noreply@blogger.com (Munish K Gupta) — Mon, 07 Nov 2011 05:56:00 +0000

How many times, we have seen that one of the easiest way of cleaning up your mail box is to delete all the enterprise announcements and group e-mails? Are we getting bombarded with too many policy change mails, group emails and announcements? After a few days of the email bombardment, are our brain gets conditioned to just ignore them ?

All this leads to a point where we start becoming clueless about

Happenings in the Enterprise
Policy/Process Changes that might be affecting us

which means we might be working harder and not smarter.

Imagine, the enterprise announces some template for making presentations. The same is announced with enterprise wide email. We ignore the email contents and continuing using the older templates. One fine day, the boss looks at the presentation and says why are you using the older template and there is no body else to blame except self.

Imagine, you did saw the email and know about the new template. But you deleted the email. When a fort night later, there was a requirement to make a presentation, you are searching high and dry for the link from where you can download the template. There is no direct link available on your intranet that points to the template. There is no way to retrieve the email also.

Why do enterprise not make available all those emails available on the intranet? Is there an easier way to manage this flood of important information? How can the individual employee choose to subscribe/unsubscribe to such emails? Is there a way the employee can refer to these emails later on - where the content is available tagged and ready to be searched?

The answer to these question is simple tool - Blogs.

Enterprise can start publishing all the announcements as blog entries. This simple tool alleviates all the pains and provide a fantastic way of organizing all the content that is properly tagged and available to be searched, subscribed and even allow debates on some of those emails. How many times we wanted to comment on some policy change but could not do so because it is a group email and there is no way to provide feedback.

Having a blog does not mean, the enterprise stops sending out emails but the blog can become the one stop content repository for all such announcements. Today, publishing an entry is as easy as sending an email.

Modern Web Social Collaboration tools provide blog as a feature out of box. Enterprises can make use of the Blog as a means of storing enterprise group email content.

Most of the internet based business already make use of the blogs to connect and disseminate information about their products and services. It is just a matter of picking up the same best practice within the enterprise.

Building Scalable Enterprise Systems

noreply@blogger.com (Munish K Gupta) — Mon, 17 Oct 2011 11:04:00 +0000

Successful enterprise are all about business agility and able to introduce new products and services in the market. All this business agility coupled with reduced IT overheads means, the enterprise need to find better ways to improve and transform their enterprise systems.

The advent of Cloud, Social, Mobile and Consumerization of IT means enterprises applications need to be adapt to the changing environment. Today, every enterprise is looking to fulfill the following demands

Adopt cloud for their enterprise applications (whether private / public / hybrid is a matter of enterprise priorities)
Replace or retire in-house enterprise applications ( where applicable) in favor of the equivalent SaaS applications
Expose enterprise data for third party consumption
Make the enterprise functionality available over variety of channels (web, mobile) - Consumerization
Make the systems available 24X7 to meet the ever growing business demands

The Enterprise application patterns for creating applications – Portals, CMS, SOA, Centralized DB’s are not functionally capable or geared to meet the new business demands. The enterprise needs to adopt the newer application patterns that are coming out of the consumer web world. Some of the patterns emerging out of new generation consumer web applications that can be applied for enterprise applications

Partition by Function – Divide the functionality by usage patterns or resource consumption. The functionality can be exposed using RESTful or Web services. This functional grouping will help in achieving optimization at the resource level. This helps to scale individual functional groups based on their resource characteristics (High CPU, More RAM, High I/O). This pattern organizes the enterprise platform around services which in turn provides agility. Things can be done in parallel because the output is a service. This allows fast time to market. Helps create an infrastructure that allows services to be built very fast.
Segregate the providers and Consumers – Instead of building large monolithic applications, segregate the functionality providers versus consumers. The enterprise should have separate teams – one that deal with the functionality provider and other that consume the functionality. The functionality is exposed via simple API’s having standard versioned interfaces
Public/private cloud integration – Enterprises need to have standardized integration strategy as the need on board multiple applications from multiple vendors start becoming common.
- Enterprise need to start integrating with SaaS vendors that can provide the required apps for the consumption of the enterprise users. At times, the integration might need to be handled at the API levels specifically for authentication/authorization.
- Enterprise applications specially that are consumer facing can also be hosted in public clouds.
Enterprise need to integrate services that handle data coming in or going out from public cloud / SaaS vendors to their data centers.

Why focus on Performance Engineering ?

noreply@blogger.com (Munish K Gupta) — Fri, 14 Oct 2011 10:55:00 +0000

Time and again, we witness, when a program goes into acceptance testing phase, the client and teams suddenly realize that the application is not meeting the Non-Functional requirements. Usually the application is very slow, or it is frequently going down or not scaling up as expected. I am not even talking of requirements mismatch here.

When client suddenly raises all these questions, all hell breaks loose within the development team. The team tries to find the hot spots and try to fix them. Invariably, I have seen the teams end only fixing the symptoms of the problem but not able to correct the problem. E.g if the application is making too many calls to the database. The quick fix at times is to cache the data. But invariably, this leads to problem in the future of cache invalidations and cache synchronization if the application is clustered. Remember, when we are fixing the symptom of the problem, we are only postponing the problem to a future stage (Technical Debt). All this means, with gating criteria's not getting met, cost and schedule over run becomes a norm. This becomes the most crucial phase of the program as the chances of program getting abandoned starts increasing exponentially. All this forms the long tail, as the program just keeps dragging on.

Focus on the Non-functional requirements right from day one is very important. Getting the architecture right at the beginning is very important, as the cost of fixing the problems keep on increasing exponentially as the application develops further and in the later stages it takes a heavy toll on every one. Techniques like Performance Driven Development is a good step in the direction. Other way to analyse each aspect of the requirement and design using the lens of non-functional requirement. At times, we as architects will be able to push requirements that do not meet the NFRs.

Why are Enterprises looking at BigData?

noreply@blogger.com (Munish K Gupta) — Mon, 10 Oct 2011 07:01:00 +0000

The advent of the Social Collaboration, Online Selling, Digital Goods, Mobile means every enterprise wants to process the transactional and analytical data, that is being collected at multiple customer touch points. All this data need to be processed so that the enterprise can better understand the customer, his social network, his buying patterns and other things.

This has led to ever increasing amount of data, which is leading to the following issues within the enterprise

Business do not have access to the real time data feeds
Queries are running in hrs and minutes and not in seconds
The batch processes, ETL, data loads are taking too much of time
Ability to construct new models based on the data is time consuming
Scalability of systems with ever increasing data is becoming a problem
Data Storage/Redundancy is another problem
Increasing License cost of software/hardware is another issue

When Business looks and reads about how Facebook, Yahoo, Google etc are managing large amounts of data (BigData) and are able to process the same at real time, they want to adopt some of their systems and techniques.

The new systems/technologies that have been open sourced and are getting adopted rapidly by the enterprises are Hadoop and its various commercial versions (Cloudera, Hortonworks, GreenPlum). In addition, other commercial vendors have also jumped in with their specific BigData offerings - IBM has BigInsights, Oracle has Exalytics In-memory machine. The commercial vendors are trying to sell big machines - with more RAM and more CPU to be able to process more data.

But, the question is - are enterprises looking at to buy more hardware, software, acquire more licenses to process data or they want to solve the issues.

I believe, the fundamental problem is speed of access to the data (FastData) which is a paramount requirement for the enterprise. BigData only promises to help solve the problem of large amounts of data but it still has a long way to go before it can fulfill rest of the enterprise needs.

Grappling with Requirement Complexities

noreply@blogger.com (Munish K Gupta) — Fri, 30 Sep 2011 05:01:00 +0000

For an Architect, as if deciphering the requirements between Functional and Non-Functional requirements was not enough, DevOps and Technical Debt has opened another head for bunching requirements.

For the uninitiated, DevOps is a movement that is meant to break down the silo approach between the Dev, QA and IT teams, where the code moved in a batch mode from Dev->QA->IT Operations teams. This silo'ed approach meant delay, disruption and duplication of efforts, which ultimately leads to delays in pushing out business functionality.
DevOps promote having a integrated team of application development, QA and operations people that shortens the delivery cycles and work within the 2-3 weeks sprint cycle. This means people with multiple skills (read polyglot programming) - either coders with sysadmin skills or vice-verse. Advent of cloud has accelerated this methodology as the team developing the feature is also responsible for its deployment.

With clients insisting on shorted delivery cycles, multiple releases in a day at times means teams need be agile and staffed with appropriate skills to be able to deliver the business functionality.

Damon Edwards explained DevOps very beautifully in a blog post last year:

The most fundamental business process in any company is getting an idea from inception to where it is making you money … The whole point of DevOps is to enable your business to react to market forces as quickly, efficiently, and reliably as possible.

Traditional models of SDLC like waterfall model can not cope with the newer demands of the business. Business wants to try out new features / functionality to guage the client reaction means more releases, more agility expected from the teams.

All this agility, means the code going out is not optimal and can be messy at times. This means maintaining code or adding new features become harder and harder as we progress. All this slows down the sprint cycles.

Technical Debt is a metaphor developed by Ward Cunningham, who drew the comparison between technical complexity and debt and said

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt.

If the business insists on reduction of technical debt along with the new features being developed, it adds another layer of complexity to the whole engagements.

Imagine, with every sprint, the team is adding new features , refactoring / rewriting the old code (to reduce/payback the Technical Debt) , ensure and account for any new Technical Debt getting added, the job of the architect to co-ordinate and co-relate all these varied requirements become lot more harder.

PaaS - is it right for me ?

noreply@blogger.com (Munish K Gupta) — Thu, 29 Sep 2011 12:31:00 +0000

An organisation embarking on a cloud journey invariably ends up looking for PaaS solutions. PaaS sounds like something that will help me take away all my pains. Creating your application, using drag and drop controls in a browser and everything else taken care (from deployment to running to scalability to back up of data and what not) sounds something exciting and very hard to believe.

Anyway, who wants to deal with the IT folks and deal with all the tantrums they throw, what is possible and what is not.

Bernard Golden, in his article "What You Need to Know About PaaS" talks about the various issues that need to be looked at, when selecting your PaaS vendor.

Lock-in - The productivity gains you get by using vendor API's also mean you get locked on to the platform and application migration becomes all the more difficult.
Complexity - Writing PaaS application is not the same as writing regular applications. Depending on the vendor, the complexity of writing application increases as you try to add more functionality
Cloud Service Provider Differentiation - The whole premise of PaaS platform is to make the writing of applications easy. With all the abstraction the platform provide, you end up getting locked at operation level
New Skills - PaaS platforms require new set of skills which means either re-train your existing employee or hire new people
Mapping existing practices to new frameworks - Every enterprise will have standard processes, tools and methodologies of working and developing applications. All these need to be re-looked based on the PaaS platform

All in all, PaaS platforms need to be evaluated based on your requirements and fitment as per your organisation needs.

Read the complete article here.

Practical approach to Capacity Planning

noreply@blogger.com (Munish K Gupta) — Wed, 28 Sep 2011 06:03:00 +0000

Capacity Planning is all about managing you resources better. Resources are finite, resources need to be procured, resources come at a cost, resources get consumed, as a result you need to do some capacity planning.

Capacity planning is an exercise undertaken in all the industries. There are plenty of models on how to perform capacity planning. But somehow application of these models in the software industry is too cumbersome, tedious and at times completely useless. These models work best when you have a standardized product and process. In software, every release changes the dynamics of the software product. The code base changes, the code performance changes, the usage pattern might change leading to the failure of the previous capacity planning exercises.

Amazon.com released a one day promotion- Lady Gaga album for 99 cents and the site went down. Lots of customers could not buy the offered album songs. Leaving lot of customers angry and loosing potential business

Looking at the problem, following question comes to mind, when to do capacity planning :

Should I wait for disaster to stuck?
Should I start capacity planning from day one?

Next question comes

How to go about doing capacity planning?

There is a simple process for doing capacity planning. It is not different from any other planning process. There are primarily four steps in the process.

Determine your capacity planning goals - For any capacity planning exercise, it is very important to determine the goals in mind. e.g. for an internet application, managing the ever increasing number of users might be a goal. Similarly, for an intranet application, the number of users might be constant, availability and response time might be the drivers for the capacity planning. Once we understand the nature of the application, and what is the end objective, we can plan the goals. These goals will determine what the measuring points that are required from application. More the measuring points, better will the forecasts.
Collect data and metrics that map to those goals - Once the goals and the measuring points are determined, we need to collect data that maps to the state goals. e.g. if 99.9% availability of the application is the goal, we need to be able to track the application up-time. We need to track and measure all the links in the applications (be it load balancers, application servers, web servers, data base servers, network, storage, routers, switches, power supply). We need to keep track of the various application usage patterns that can affect the application availability.
Plot that data and make forecasts - Once we collect the data, we need to plot that data and create trend curves. We will need to extrapolate and do curve fitting on the time series data. This will give us data in terms of application usage changes and help us in product planning. This is an on-going iterative exercise. Every new data brings in more perspective and helps make the forecast better.
Use those forecasts to enhance the capacity - Based on the forecasts, we need to enhance the capacity to tackle the new growth and achieve our goals.

Any changes in the application (bug fixing or functionality enhancement) will render the old data invalid. We will need to account for these new changes and map into the measuring points. This will help in accounting for the changes in the forecasts.

Remember, capacity planning is something that starts, when you conceive the application and it continues till the application is sun set.

Capacity planning is not an end in itself but a journey.

Enterprise Mobile Web Strategy

noreply@blogger.com (Munish K Gupta) — Wed, 03 Aug 2011 06:11:00 +0000

With the proliferation of devices in various screen sizes and running different OS, having a coherent mobile web strategy has become somewhat of a nightmare for the enterprise. Gone are the days, when the enterprise would just optimize the existing web site for mobile and let the content be served. Today’s consumer with the ever powerful device and increasing bandwidth is looking for experience that is equal to or at times better than the web.

When trying to create an mobile strategy, enterprise need to ponder over this

Should I build a native application for the mobile device
Should I build a mobile optimized web site

To be able to answer these questions, you need to study the consumer usage patterns and ask

What is the value the information provides to the consumer?
What is the frequency of access of information by the consumer?
What is the experience I want to provide to the consumer?

What is my business all about? E.g. If the enterprise is in the business of movie screenings. The consumer is interested in the information with regards to the nearest theatre location, what movies are playing, their timings and how can I buy the tickets. The user will be accessing the application multiple times during the month. In this kind of scenario, having a dedicated application is must.

But there will other consumers who have not yet installed the application. For such user’s your web site should be optimized for mobile access and provide the same information and features. In the end, you can ask him to install the application to make the future search and transaction a lot easier.

Having an application also allows you to tap into the features of the mobile device to provide an enriching experience to the consumer. E.g usage of accelerometer, camera, GPS, compass can greatly enhance the consumer experience.

But, if your enterprise is not direct consumer market, having a mobile optimized web site might will do and you need not create a native mobile application.

So, overall your mobile strategy needs to keep in mind these

You definitely need a mobile web strategy, meaning a mobile optimized web site. Whether you need a dedicated mobile app or not depends upon your answers in the earlier section.
Make sure you optimize the web site across all major mobile platforms (Android, iOS, Windows Phone 7, Blackberry) and across the common device screen sizes
Make sure you try to leverage mobile device features - accelerometer, camera, GPS, compass etc to enhance the experience
Make use of the technologies that allows to build and deploy a common code base with little variations across devices. HTML5, CSS3, webfonts should be used when building your mobile applications
Keep the mobile application design simple. Single column layouts work the best with vertical hierarchy menus. Go easy on the data consumption, today’s devices are faster but consumers do not like application that just hog their bandwidth. Try to keep images to the bare minimum
When building mobile applications, keep the usability aspects of mobile devices in mind. Taps vs clicks – means larger buttons are required, instant feedback (e.g. change the button colors to indicate that the action has been registered), typing is usually difficult – having a tap based options help.

Next step comes to the tools that can be used to build the native mobile applications

Adobe Flash Builder 4.5 offers developers one tool, one development framework and one codebase to build mobile applications on Android, BlackBerry Tablet OS and iOS. It is one integrated development platform that allows you to create mobile application for multiple platforms. You can build very rich UI that make use of device features like accelerometer, camera, GPS, compass etc. You can make use of your enterprise assets - java libraries when building solutions
PhoneGap is an open source development framework for building cross-platform mobile apps using HTML, CSS and JavaScript. The tool enables developers to utilize the core features of the iPhone, Android, Palm, Symbian and Blackberry smartphones, including geolocation, accelerometer, contacts, sound and vibration to build applications
RhoMobile offers Rhodes, an open source, Ruby-based framework that allows for development of native apps for a wide range of smartphone devices and operating systems(iPhone, Android, Windows Mobile, RIM and Symbian)
Appcelerator provides the Titanium Development Platform which allows for the development of native mobile, tablet and desktop applications through typical web dev languages such as JavaScript, PHP, Python, Ruby and HTML

In addition, if you are looking for just the mobile optimization of your web sites and functionality then look for offerings from - netbiscuits, mBlox, iLoop Mobile, Crisp Media, Quattro Wireless, 3Cinteractive.

MapReduce for dummies

noreply@blogger.com (Munish K Gupta) — Mon, 11 Jul 2011 13:28:00 +0000

Continuing the coverage on Hadoop component, we will go through the MapReduce component. MapReduce is a concept that has been programming model of LISP. But before we jump into MapReduce, lets start with an example to understand how MapReduce works.

Given a couple of sentences, write a program that counts the number of words.

Now, the traditional thinking when solving this problem is read a word, check whether the word is one of the stop words, if not , add the word in a HashMap with key as the word and set the value to number of occurrences. If the word is not found in HashMap, then add the word and set the value to 1. If the word is found, then increment the value and word the same in HashMap.

Now, in this scenario, the program is processing the sentence in a serial fashion. Now, imagine if instead of a sentence, we need to count the number of words in encylopedia. Serial processing of this amount of data is time consuming. So, question is is there another algorithm we can use to speed up the processing.

Lets take the same problem and divide the same into 2 steps. In the first step, we take each sentence each and map the number of words in that sentence.

Once, the words have been mapped, lets move to the next step. In this step, we combine (reduce) the maps from two sentences into a single map.

That’s it, we have just seen how, individual sentences can be mapped individually and then once mapped, can be reduced to a single resulting map. Advantage of the MapReduce approach is

The whole process got distributed in small tasks that will help in faster completion of the job
Both the steps can be broken down into tasks. In the first, instance, run multiple map tasks, once the mapping is done, run multiple reduce tasks to combine the results and finally aggregate the results

Now, imagine this MapReduce paradigm working on the HDFS. HDFS has data nodes that splits and store the files in blocks. Now, if map the tasks on each of the data nodes, then we can easily leverage the compute power of those data node machines.

So, each of the data nodes, can run tasks (map or reduce) which are the essence of the MapReduce. As each data nodes stores data for multiple files, multiple tasks might be running at the same time for different data blocks.

To control the MapReduce tasks, there are 2 processes that need to be understood

JobTracker - The JobTracker is the service within Hadoop that farms out MapReduce tasks to specific nodes in the cluster, ideally the nodes that have the data, or at least are in the same rack.
TaskTracker - TaskTracker is a process that starts and tracks MapReduce Tasks in a cluster. It contacts the JobTracker for Task assignments and reporting results.

These Trackers are part of the Hadoop itself and can be tracked easily via

http://<host-name>:50030/ - web UI for MapReduce job tracker(s)
http://<host-name>:50060/ - web UI for task tracker(s)

HDFS for dummies

noreply@blogger.com (Munish K Gupta) — Fri, 08 Jul 2011 04:30:00 +0000

Whenever a newbie wants to start learning the Hadoop, the number of elements in a Hadoop stack are mind bogling and at times difficult to comprehend. I am trying to de-crypt the whole stack and help explain the basic pieces in my own way. Before we start talking about the Hadoop Stack, let us take a step back and try to understand what led to the origins to the Hadoop.

Problem – With the prolification of the internet, the amount of data stored growing up. Lets take an example of a search engine (like Google), that needs to index the large of amount of data that is being generated. The search engine crawls and indexes the data. The index data is stored and retrieval from a single storage device. As the data generated grows, the search index data will keep on increasing.

As the number of queries to access data increase, the current file system I/O becomes inadequate to retrieve large amounts of data simuntaneously. Further, the model of one large single storage starts becoming a bottleneck. To overcome the problem, we move the file system from a single disk storage to a clustered file system. But as the amount of data keeps growing the underlying data that can go one one machine starts to become a bottleneck.

As data reaches TB’s, existing file system based solutions starts faltering. Data access, multiple writers, large file sizes soon become a problem in scaling up the system.

Solution – To overcome the problems, an distributed file system was concieved that provided solution to the above problems. The solution tackled the problem as

When dealing with large files, I/O becomes a big bottleneck. So, we divide the files into small blocks and store in multiple machines. [Block Storage]
When we need to read the file, the client sends a request to multiple machines, each machine sends a block of file which is then combined together to pierce the whole file.
With the advent of block storage, the data access becomes distributed and leads to a faster retrieval/write
As the data blocks are stored on multiple machines, it helps in removing single point of failure by having the same block on multiple machines. Meaning, if one machine goes, the client can request the block from another machine.

Now, any solution that implements file storage as blocks needs to have the following characteristics

Manage the meta data information – Since the file gets broken into multiple blocks, somebody needs to keep track of no of blocks and storage of these blocks on different machines [NameNode]
Manage the stored blocks of data and fulfill the read/write requests [DataNodes]

So, in the context of Hadoop –The NameNode is the arbitrator and repository for all metadata. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also determines the mapping of blocks to DataNodes. DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode. All these component together form the Distributed File System called as HDFS (Hadoop Distributed File System).

Reference - http://hadoop.apache.org/common/docs/current/hdfs_design.html

HDFS has an inbuild redundancy and replication feature that makes sure that any failure of the machine can be dealt without any loss of data. The HDFS balances itself whenever a new data node is added to the cluster or any of the existing datanode fails.

In addition to the distributed file system called HDFS (Hadoop Distributed File System), there are 2 other core components

Hadoop Common – are set of utilities that support the Hadoop subprojects. Hadoop Common includes FileSystem, RPC, and serialization libraries.
Hadoop MapReduce - is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes

So, effectively, when you start working with Hadoop, HDFS and Hadoop MapReduce are the first 2 things you encounter. I will cover MapReduce in subsequent posts.

Building High Performance Applications

noreply@blogger.com (Munish K Gupta) — Sun, 12 Jun 2011 15:24:00 +0000

Performance is one word which is used to describe multiple scenarios’ when talking about the application performance. When someone says, I need a High Performance Application; it might mean any/all of the following

Low Web latency Application ( meaning low Page Loading times)
Application that can serve ever increasing number of users (scalability)
Application that does not go down (either highly available or continuously available)

For each of the above, as an Architect you need to dig deeper to find out what the user is asking for. With the advent of cloud, every CIO is looking to build applications that meet all of the above scenarios. With the advent of elastic compute, one tends to think that by throwing hardware to the application, we may be able to achieve all of the above objectives.

The techniques employed to achieve the above scenarios at times are different and it is important to find the right approach to the solution that meets the above objectives. We will examine some of the common techniques that can help us achieve the objectives

Latency Contributors

Application Tier ing – One of the biggest contributors to the latency is the application tier ing. The hops from WebServer -> Application Server -> Database and back, data serialization/deserialization are some of the biggest contributor to the overall latency. Having Web and Application tier within the same box or even within same JVM can help reduce the network latency factor. One can have logical separation in the application code between Web Tier and Application Tier but need not have physical separation. Using Spring Container that has Web/App tier can help achieve the same. If the application is making use of SOA and making multiple web services or JMS message calls, network latency and serialization of data once again adds to the latency. Solutions like IBM Datapower XML Accelerators can be used to reduce the XML overheads. Similarly, the application can use Solace Message Router’s to speed up the messaging.
Bring Data closer to Application – Data needs to be close to the application so that making all those Database connection calls and getting data from DB can be reduced. Application can cache data to reduce the calls to DB. One can also use cache servers like memcached / ehCache to cache data at the Web/application Tier. Web Tier can cache data like static HTML fragments/images/javascript/CSS files. Application Tier can cache non-transactional data (like lookup maps). OR Mapping tools like Hibernate also support data caching. If it is an Internet Web Application, one can also make use of CDN (content delivery networks)/ Edge Networks (like Akamai) to speed up the delivery of static content.
Disk I/O – Another weak link in the application performance chain is Disk I/O. One way to overcome the limitations with regards to the Disk I/O is too keep data in memory. In Memory databases (like Volt DB or Solid DB or Oracle TimesTen), XTP solutions (like Oracle coherence, IBM eXtreme Scale, GigaSpaces eXtreme Application Platform) can used to speed up the application performance.
Optimized Hardware – The hardware on which application is hosted can also be tuned to reduce latency. Optimization s like 10G/20G network, fiber channels, low latency switches, SSD (Solid State Drives), not using virtualization can make sure the application latency is reduced.
Transport Mechanism – At times, the transport mechanism can also add to the application latency. E.g. secure communication (like https) can add to the latency with the additional overhead of deciphering the data at the receiving end. One way is to offload the SSL at the Load Balancer/Firewall.

Application Scalability – Scalability means ability of an application to handle growing amount of data and concurrency in an efficient manner without impacting performance. Important thing to notice is scalability should not be at the cost of application performance. Some of the techniques that can help scale the application

Stateless Application/Service – The application should store its state in some centralized repository, but the application itself should be stateless. It means no storing of data or state on local file systems. Stateless application allows one to add any number of application instances to accommodate the increasing growth. But soon, the centralized repository starts becoming the bottleneck. With ever increasing data, repositories like (RDBMS) may start buckling down. One approach to this issue is to minimize mutable state in the database. To handle such scenarios, techniques like data sharding need to be applied. Another approach to managing write contention in the database is to look at the possibility of using NoSQL data stores for some or all of the application data.
Load Balancing – As the traffic starts going up, the application can handle the additional load by adding additional server instances to service the requests. The load balancer will make sure none of the servers are working beyond their stated load and new instance should be automatically added as and when the load goes up (auto scaling). One can also add load balance to database with techniques like Master-Master topology or Master-Slave(with partitioning read and write data) to handle the additional load. But if the data is going in Petabytes ranges, data sharding with data replication techniques need to be used. The in-memory data grid architecture can also be utilized to scale the data.
Fault Tolerance / Dynamic Discoverable Elements – When dealing with application that is running in large clusters, it is very important to avoid manual interventions. E.g. when the application load reaches a defined load, the application monitoring should be able to add a new instance and load balancer should be able to recognize the same to utilize it. Similarly, when data gets shard, the applications should be able to recognize and look up the new IP to connect. Similarly, if the application is not able to connect to particular resource, the application should be intelligent enough to recognize the fault and try accessing the alternate resource availability. The application will need to have a central meta data repository for all such fault tolerance scenarios that can be tapped by the application.

Application availability – Availability of an application is very much a function of scalability. Following factors have an impact on the application availability

Redundancy – The application needs to be scalable to be able to compensate for the loss of any instance (whether hardware or software). The redundancy needs to be build at all layers, Software, Hardware, Power and even at data center levels. Even if the data center goes, the user should be able to access the application. Many at times, the level of redundancy and down time is a factor of how money is being thrown at the solution. Remember some problems have no solutions within the context of today’s technology. E.g. real time data mirroring or data sync across data centers that are located geographically apart.
Fault Tolerance – The application needs to be fault tolerant (e.g. retry mechanism) to make sure it can take advantage of dynamically allocated resources to keep functioning.
Monitoring/Testing – Another overlooked factor of application availability is application monitoring. If application is not properly monitored, outages can go undetected leading to application unavailability. Ability to monitor the entire application stack and take corrective actions is very important. This capability is build over a period of time. Once the application has monitoring, auto-scaling features, testing to make sure they work is also important. Something like Chaos Monkey used by Netflix is very helpful.
Configuration Data – Any application that needs to be continuously available needs to be able to run using configuration. E.g. if the application introduces the new service interface, the application should have the ability to either make use of the new interface or keep using the old one. This factor becomes very important when rolling out new features/services and all of them cannot be rolled out at once.

Let me know, if you have used some other techniques to achieve the 3 factors of availability, scalability and low latency.

Maladies of Large scale Software Design

noreply@blogger.com (Munish K Gupta) — Tue, 26 Apr 2011 11:38:00 +0000

When designing and maintaining large scale software architecture, certain rules need to be adhered religiously. In the absence of which the software starts degrading, maintenance becomes difficult, adding features becomes a nightmare and soon everything comes crashing down as a house of cards.

Tiering/Layering your logical architecture – a clear logical separation between different layers lead to simplified code navigation and comprehension. The layers should also follow a strict and clear naming convention for all packages and types. One can make of tools to make sure the code follows the logical architecture and all dependencies are in order.
Cyclical dependency – Adhere to the principle of well defined and cycle free application. Cyclic dependencies can soon lead to bloating of code. Even package should be validated for any cyclical dependencies
NCCD (Normalized cumulative component dependency) is another factor that needs to be adhered. NCCD of compilation units must not be bigger than 7. If this value grows over the threshold, one should isolate layers and subsystem by only letting them have interfaces as entry points. Breaking cyclic dependencies can also shrink this metric considerably.

In addition to the above rules, one can also gauge the health of the software using additional metrics like

Code Coverage– describes the degree to which the source code of a program has been (unit) tested. Popular tools to measure code coverage in Java are Emma.
Code Complexity - is another measure that can used to identify and quantify the code maintainability. Factors like Cylometric Complexity, Halstead's software science metrics and Lines of code metrics are some of the factors. In fact, there is a measure called Maintainability Index that can be calculated by using the factors.
Documentation - is another factor that be used to quantify code maintainability. Documentation can be your normal JavaDocs or the inline source code commenting. You have tools like JavaNCSS where NCSS ( non commenting source statements) that can be applied at class, method/function level
Code Format - is not really a measure but is an indicator of the readability of the code. Tools like CheckStyle can help identify the code formatting problems (from non standard class headers, indentation issues to spacing, variable naming convention and so on). Any issues reported by the tool can be used an indicator of the maintainability of the code

From a programmer’s perspective, the following rules should be followed, that will help in building and maintaining a robust and lively system

Use a consistent formatting and naming scheme
Limit the access to types and methods
Number of types in a package must not exceed 50
Lines of code (compilation unit) must not exceed 700
Number of method parameters must not exceed 7
Cyclomatic Complexity must not exceed 20
Use assertions to ensure pre-conditions and post conditions

Do share your best practices that you followed or came across when build or maintaining large scale software architectures.