Tech Spot

Is Fargate for EKS "Serverless"?

noreply@blogger.com (Aravind Ajad Yarra) — Wed, 11 Dec 2019 04:47:00 +0000

Is Fargate for EKS "Serverless"?

Last week at re:Invent, AWS has announced EKS running on Fargate. Fargate is often described as Serverless offering for containers. If you have used AWS Lambda or equivalent FaaS services like Azure Functions, you might wonder if this is really a Serverless offering? Before answering this question, let's see what Fargate brings on to the table.

What is Fargate?

When Fargate was introduced in 2017, it had brought in a new paradigm of how containers can be run on Cloud. Instead of manually setting up container clusters (using ECS) on top of EC2 instances, Fargate had automated the provisioning of containers as required and eliminated the need for planning and provisioning the compute. Fargate uses lightweight virtualization technology called 'Firecracker' to run the containers within virtual machines, offering strong isolation, security and multi-tenancy, which AWS describes as "MicroVMs". Firecracker is what makes Fargate so unique.
Fargate for EKS

Fargate so far only supported ECS, but had no support for EKS (managed Kubernetes service from AWS). With this announcement EKS can run on Fargate based compute for hosting containers. With EKS on Fargate, setting up an EKS cluster is a matter a single command plus creation of Fargate profile and some IAM tweaking. Once done, EKS cluster can scale based on application scaling requirements as determined by the pod scaling behavior. Most importantly, the scaling is elastic and customers are charged based on the containers provisioned.

Here is an animation from @ArchanaSrikanta that shows how extensible Kubernetes architecture is, and how simple it is to use Fargate.

Curious about the internals of Fargate, Fargate-on-Firecracker, and EKS-on-Fargate? I will be doing a session tomorrow with @onurfz: CON423 AWS Fargate under the hood. Here is a teaser of my EKS Fargate architecture slide! #Fargate #EKS #reInvent2019 #Kubernetes pic.twitter.com/mf7IBXx7ch
— Archana Srikanta (@ArchanaSrikanta) December 5, 2019

Is it Serverless?

Serverless computing is a word that is overloaded to each vendors' conveniences. But if we have to scope in words, here is how I characterize 'Serverless'.

Cost - Strictly usage based pricing
Dev Experience - Source-centric applications
Scale - Truly elastic scaling
Setup - No infrastructure provisioning

More the boxes checked by an offering, better Serverless it is. How does Fargate or Fargate for EKS fare against these characteristics?

Fargate pricing is based on containers provisioned by it to handle the requests, and definitely not based on the actual requests. This box is not checked.
Containers are very close to source, so I would say this box is checked.
Fargate scales containers elastically up and down as needed. This box is checked.
Creating an EKS cluster on Fargate is fairly simple. There is no need to plan and setup the underlying compute, network and storage. This box is checked too.

Fargate for EKS is quite close to be called as Serverless offering, except for the way it is costed.
Fargate for EKS is quite innovative in how Kubernetes is extended to run on a completely different compute infrastructure, albeit with a few limitations.

AWS claims they run 80% of all containers that run on public cloud. This claim show the dominance of AWS. With such dominance, Fargate for EKS is all set to be a leading Container Serverless offering for Kubernetes. Are there any other Container Serverless offerings for Kubernetes that compare to Fargate for EKS? That would probably be the next topic to write about.

This post originally appeared at www.techspot.co.in

Building Digital Capabilities

noreply@blogger.com (Munish Gupta) — Wed, 31 Aug 2016 04:04:00 +0000

Most enterprises are struggling to innovate and are not investing enough into new technologies to counter disruptive measures. As a result, in a few years, their products are becoming irrelevant in the market. Examples of Blackberry, Nokia, Kodak, Blockbuster have been quoted numerous times. Technology has lowered the entry barrier to innovation, and new entrants can easily disrupt the entire business models in a matter of months. Geographical borders do not guarantee any competitive advantages anymore. Any breakthrough or new idea travels around the world in a matter of minutes.

For an enterprise embarking on the digital transformation, following key things need to fall in place for building digital capability within the enterprise:

Business vision – digital transformation initiatives are cross division or cross-functional, as a result, having a common business vision to drive the transformation is very important
Leader led – the digital initiatives need to be top driven and led by a senior leader. The cross-functional team should report to the leader
People skills – cross-functional team needs to be staffed with people that bring varied experience and are adept at defining and refining digital journey focused on the customer experience. In case, the team is relatively new to digital, then hiring for appropriate skills need to be undertaken. Existing team members can undergo training for required skills. People that bring multiple skills to the table are core to the team
Platform approach and Automation - when looking for technology choices in implementing the identified projects, start thinking in terms of platforms instead of systems. Digital transformation is not about one project or program, the continuous iterations and interconnection of functionality, requires platforms that provide the requisite services to build such experience in a rapid manner. The implementation team should not get bogged with scalability and performance issues, things that mar the customer experience. The platform should handle the scalability and performance aspects of the application. Similarly, horizontal capabilities like data analytics, data aggregation, content services, logging services should be viewed and built as platform services. Again, if platform building skills are not available in-house, they need to be hired from market to jumpstart the process. Another key capability is to automate the Development to Operations application lifecycle (DevOps). When you working on scale, manual processes and intervention is error prone and not at all feasible. Investment in automation of all the DevOps process is an important part of the building digital capabilities
Open source software – in the digital products and services space, given that the product(s) can become viral overnight, scalability and performance of the product/service is a key criterion. Use of commercial proprietary products limits the ability of the enterprise to scale the product. Licensing cost is also the key factor for handling user base in excess of 100 thousand. Open source software allows benefiting from the larger community, cutting down on the costly license, use of commodity hardware and ability to dig deeper into code (if required). The platform approach is executed using a combination of open source software.

Enterprises threatened with changing business models have already built or started building digital capabilities. Companies like Walmart are no longer just brick and mortar players but giving tough competition in the online e-commerce space also. Investment into future is required for enterprise to reinvent them for the digital era.

This post originally appeared at www.techspot.co.in

Technology’s Role in Digital Transformation

noreply@blogger.com (Munish Gupta) — Wed, 31 Aug 2016 04:02:00 +0000

For enterprises disrupting the marketplace, technology is the key enabler that is helping them create new business models and processes. Open source software has played an enabler role in developing these new scalable and high-performing systems. From the digital transformation perspective, following four levers play an important role.

Connected – with the advent of IoT everything is getting connected. All this data is being transmitted over mobile and fixed bandwidth networks where the enterprises are making sense out of it leading to real-time decision making. New service models are being created around data. E.g. Rolls Royce aircraft engines capture data around the engine performance which can be given as subscription to airlines for predictive maintenance. This data can also be used to develop models for better engines.
Data – All the data from Things, device location, social interactions etc needs to be captured, processed and analyzed for patterns, behavior discovery to allow better predictions and decisions to be made.
Automation – Given the scale of the systems and data being emitted/generated for analysis, automation is the key enabler. Traditional technologies need to be combined with Artificial intelligence and machine learning models to increasingly create systems that can work autonomously. Lots of supervisor roles can be automated using the AI models. E.g. Uber has eliminated the complete middle management role found in a traditional taxi services company.
Users First – All the data and automation need to be used to build and provide contextualized products and services for the users. Enterprises or intermediaries can get direct access to users/customers via mobile apps. Users can have fully transparent access to their data and enterprises can offer services around that. E.g. Tesla have built a model for direct selling of cars to the end customer bypassing the franchise/dealer network. This allows them to understand the customer behavior and usage of car. They can sell additional services based on the user context.

It is still early days of digital transformation. Lots of enterprises are taking steps with digital as digital marketing which is a very narrow and siloed focus. Digital transformation is about rethinking the business, operational and technology processes (e.g. customer onboarding, customer cross/up sell, customer service, core products/services etc). Technology is the key enabler and drives the entire business model in the digital economy. Every company is being transformed in a technology company where technology is enabling all these digital interactions.

This post originally appeared at www.techspot.co.in

Digital Transformation impact on Value Chain

noreply@blogger.com (Munish Gupta) — Wed, 31 Aug 2016 04:00:00 +0000

New forces are impacting the entire value chain of an enterprise. Availability of large amounts of digital data, automation of the production processes, inter-connectivity of the supply chains, and digitization of the customer experience is re-organizing entire industries leading to the transformation of business models.

Let’s look at the some of the critical pieces of the value chain and see the impact of the new technologies.

Manufacturing – For any enterprise that is manufacturing or sourcing raw materials/parts/products, global sourcing has become the norm. Inter-connectivity of the entire supply chain via telecom networks has synchronized supply chains and shortened the production cycles. The advent of the Internet of Things (IoT) has enabled this inter-connectivity across the value chain
Distribution and Logistics – the distribution model is moving online, market boundaries are blurring, business and people are ordering from all over the world, leading to changing dynamics in the distribution and logistics part of the supply chain. The advent of 3D printing will bring changes in the way logistics are managed. Parts can be printed on location, saving on costly logistics and [asks for differential] warehousing strategy from today. Third party warehouse and fulfillment centers (e.g. Amazon fulfillment) changes the whole cost model. The advent of Geo location has allowed real time tracking of the item enabling more transparency allowing faster decisions.
Customer, Employees – With smart devices everywhere, consumers are starting to ask for information that is contextual, real-time and pushed to them. When the same individuals go to work in their companies, they expect the same quality of customer experience. The entire social revolution focused only on the social aspects of the customer journeys, but what digital millennial need is the digital transformation of the end to end customer journey. Large amounts of customer from social networks combined with knowledge of the customer from internal systems needs to be combed for creating a personalized, contextualized customer experience - a 720 degree of the customer.
Enterprise Systems – Businesses are not investing enough in Change to keep pace with the digital transformation happening all around them. Digital transformation is being enabled by technology and enterprises need to upgrade/enhance their legacy systems and technologies to support the new digital processes/models. Availability of SaaS solutions, open source software, big data, cloud computing is changing the enterprise landscape. The skill sets required to manage and make sense of these new plethora of technologies is driving the transformations.

The impact of the transformation can be seen in pioneering industries such as music (digital products are delivered via digital channels), media (traditional print news media is already on downward spiral, media giant Facebook does not create any content), retail (brick and mortar stores are getting replaced with online eCommerce stores, today largest retailer Alibaba does not own any inventory), taxi services - online ride aggregators are changing the dynamics of traditional taxi services, Uber does not own any taxis and electric cars are changing the dynamics of entire auto industry - imagine no car recalls but OTA software updates to fix the issues. Signs of similar digital transformation can be seen across most of the industry domains.

What signs are you seeing impacting your industry value chains?

This post originally appeared at www.techspot.co.in

A Four-Pronged Approach to Microservices Adoption

noreply@blogger.com (Aravind Ajad Yarra) — Tue, 19 Jul 2016 12:37:00 +0000

Enterprises have been struggling for years to develop applications that are agile and quick to change. Microservices architecture provides a way to address this challenge and has therefore caught the attention of enterprise IT teams. Beyond the initial enthusiasm, proof of concepts and pilot implementations, enterprises are struggling to adopt this architecture. In this blog, I explain why enterprises need to consider a strategic approach to Microservices architecture and suggest a four pronged approach to adoption.

Microservices Enable Evolutionary Architecture

There are quite a few architecture styles that have emerged over the last few years from the internet companies that don’t have the constraints of legacy systems. One of the key characteristics of these architectures is their ability to respond to changing business needs, known as evolutionary architecture.Microservices architecture is one such architecture style that can address variety of concerns that effect the agility of systems. From a technical standpoint, Microservices uses service orientation principles and functional partitioning to “monolithic” applications to make systems function as relatively independent services. From a business benefit standpoint, it makes enterprises digital ready and offers multiple advantages to them. Some of which are:

Allows experimentation & innovation, leveraging cutting edge technologies, but localize the related risk only to a specific service.
Because of independent services, scaling of teams is easier, as Microservices reduces coupling between teams and enables more autonomy and accountability.
Improves resiliency of mission critical systems, by scaling and replicating parts of systems rapidly. In case of failure of a specific service, rest of the system can continue to run.
Enables cloud adoption and cloud native architectures by deployment of services over multiple VMs or containers working together

Thinking about Microservices Adoption & Challenges

Embracing Microservices architecture at an enterprise level poses unique challenges especially if we consider large enterprises that have complex application landscapes. Some of the common questions that both large and mdi-size organizations ask are:

Where do we start the adoption of Microservices?
Coupling within existing systems can be complex. Is there a way to prioritize application portfolio?
What capabilities do we need to start adopting this architecture style at scale?
What practices should we use to ensure successful adoption and get best of Microservices architecture?
There are no proven enterprise adoption patterns, as there is very little enterprise wide adoption outside Internet native companies. How do I do enterprise wide adoption?

Approaching Adoption Strategically

Often enterprises take a tactical approach and use Microservices to douse fires in specific problem areas. Such thinking will do more harm than good. Hence, I recommend a logical and strategic four pronged approach towards adoption. These activities need not happen in the mentioned order though. Necessary capabilities will have to build systematically, which will help realize full benefits of Microservices.

Discover: Articulating the business benefits and creating a case for Microservices adoption. Based on the maturity of enterprise this can be optional. However, one might have to build a case to get the necessary investments and budget allocation.
Plan: Assess the current state capabilities and identify the gaps in terms of capability that is required for adoption at scale. These capabilities include frameworks, technology choices, governance etc. Another important activity is to identify the applications of priority where Microservices architecture can be applied.
Act: Build the base capabilities needed for Microservices. Choose the right technology stacks. Build out platform capability. Setup guidelines and prepare playbooks. Put forward service design guidelines and start implementing services.
Optimize: Take Microservices architecture into production at scale. Ensure services are organized per domains. Automate the deployment processes. Bring in strong management & monitoring capabilities. Keep an eye for potential pitfalls.

This post is cross posted from www.wipro.com to www.techspot.co.in

Architecture concerns for Distributed Systems

noreply@blogger.com (Munish Gupta) — Tue, 15 Dec 2015 09:54:00 +0000

Application architectures are evolving from the era of large monoliths to a more distributed design based model. One of the key initiators of this movement is the advent of cloud computing and the ability it brings in terms of handling ever increasing scale. When an enterprise primarily soaked (people and processes) with the model of building and managing monolithic applications, the journey to build new distributed systems requires re-learning some of the older design techniques and adopting some new patterns. As part of this, I will detail certain architecture concerns that become prominent when moving to a distributed model of application

Scheduler/Orchestration management – From managing 100s of instances to managing 1000s of instances require the ability to orchestrate/schedule service instances/containers across hosts in a seamless manner. To handle increasing scale, workload scheduling/orchestration is a key ingredient of distributed system. Products like Docker Swarm, Kubernetes, Mesos, Marathon etc are some of the leading products in this space
Service Discovery/Registration – As the container based services go up and down, there need to be mechanism to register/unregister the services along with the mechanism to discover the service end points at run time. Products like Consul, Zookeeper, etcd, Confd, Eureka are some of the leading products in this space. Most of these products support load balancing of the incoming traffic across the service instances.
System State Management / Cluster Management – As the cluster grows, there is a need to manage the system state of the cluster. What are the SRV for each of the services, how many instance, on what hosts, what is load etc. To manage this, there is a need for cluster management that keep track of the system state. Products like Docker Swarm Agents, Kubernetes Nodes/Masters, Mesos Slaves, Containership etc are some of the leading products in this space
Data storage – the container storage is ephemeral, which means the any data that needs to be retained beyond the container lifecycle need to be persisted outside. Projects like Docker Volume Plugin, Flocker, Kubernetes Persistent volumes etc are some of the key products
Network – with each of the containers running different processes, there is a need to manage and at time isolate which container services can access which other services. Multiple containers are running on same host sharing the network resources might require security groups to be created for container isolation. Similarly, containers might want to discover services that are hosted across hosts and need simple model to access those. Products like Flannel, Weaveworks, Calico are some of the products in this space.
Monitoring/Auditing/Logging – With 1000s of containers running, monitoring/auditing /logging each of the containers become a tough problem. Data/Logs need to be pulled from each of the container for analysis. Products like Loggly, Fluentd, logentries, datadog, ELK stack are some of the key products in this space.

Besides this, other factors that need to be considered are Container OS and Container Runtime when architecting a distributed application. Other factors like application runtime, deployment management, DNS, Security, SSO/OAuth, API Gateways, Circuit breakers, Performance/Scalability Patterns etc still need to be handled. In your experience, anything else that is a key architecture concern for distributed application, please do share.

This post originally appeared at www.techspot.co.in

Microservices & Containers

noreply@blogger.com (Munish Gupta) — Mon, 01 Jun 2015 05:33:00 +0000

Microservices & Containers - Docker Bangalore Meetup #12 from Aravind Ajad

This post originally appeared at www.techspot.co.in

Microservices Orchestration Design Patterns

noreply@blogger.com (Munish Gupta) — Tue, 14 Apr 2015 11:34:00 +0000

With the advent of micro-services, the application design paradigm has undergone a major shift. The days of developing monolithic applications are over. We are bringing in the principles (read SOA) hereto the preserve of applications or system integration space into the application development world.

General steps for solution architecture are –

Break down/decompose the application into functional areas. These functional area’s provide us the with the bounded context
Within the functional bounded context, design/define your micro services
Functional areas talk to each other over micro services or use some kind of event queue models
Each of the functional areas only expose services to be consumed by the application

Since the micro-services are consumed within the application, the need of ESB is not there. There is no message transformation or mediations required. But service discovery & load balancing of service instance still need to be done, new tools have come up (e.g. Netflix Eureka)

Now from the design perspective, you have the service identified, exposing endpoints.

Qs comes, who will do the orchestration of service invocation and aggregation of data from multiple services?

Where does the intelligence lies ?

Options

Server Side - Create an aggregator service that internally invokes other services (across functional domains) and aggregates the results. This aggregator service is invoked by the application presentation tier
Presentation/Application Side – If you implementing JSP or server side view creation, the presentation tier can perform the orchestration across the services and aggregate the data. One can use Lambda/Futures to implement non blocking call model also
Client Side – In this case, the invocation of services and aggregation of data happens at client side. The UI is composes of multiple widgets, which call up individual services and aggregate the response on the client side

Where should you intelligence lie?

There is no correct answer, depending on the use case, different options can be applied. One can also apply multiple options within the same application depending on the type of the client to be supported or at times, you do not want to expose individual services and an aggregator service might be better.

This post originally appeared at www.techspot.co.in

Architecting Multi-Cloud Applications

noreply@blogger.com (Aravind Ajad Yarra) — Sat, 20 Jul 2013 08:19:00 +0000

Slide deck of my talk at Cloud Connect 2013 in Mumbai. I spoke on the topic of architecting multi-cloud applications.

Architecting Multi-Cloud Applications - Myth or Reality? from aravindajju

This post originally appeared at www.techspot.co.in

Review: “ZeroMQ” by Pieter Hintjens; O'Reilly Media

noreply@blogger.com (Munish Gupta) — Thu, 23 May 2013 12:14:00 +0000

Pieter Hintjens, one of pioneers of the distributed computing and author of ZeroMQ library explains the basic premise behind ZeroMQ design and community. He covers fundamentals of designing large distributed applications using ZeroMQ library.

First and foremost, the book is meant for intermediate to advanced programmers. You need to be aware of the networking concepts. All examples are in C, so you should have good enough grounding in the language. Once you are past these, understanding the concepts in the book is fairly easy.

First part of the book is about Learning to work with ZeroMQ. This section covers all the patterns – Request-Reply, Dealer-Router, Pub-Sub, Proxy, Load Balancing and their combination and usage with each other. The example scenarios detail the scenario’s where these patterns fit in. This section also covers how to build reliability and availability as part of the distributed application.

Second part of the book is about Software Engineering with ZeroMQ. This section covers the basic philosophy behind the ZeroMQ community, advanced architecture using ZeroMQ. The advanced architecture covers things like Contract definition, Protocols, Security, State machines and Data serialization. The author takes a sample application FileMQ – ability to exchange files between two devices la Dropbox and works through the design of the API and Protocol. We also see how to simulate real life scenario’s to test the application.

Overall, ZeroMQ is a must have book on your shelf if you are building or ready to build distributed applications.

This post originally appeared at www.techspot.co.in

Post Agile - State of Software Development

noreply@blogger.com (Aravind Ajad Yarra) — Thu, 21 Feb 2013 08:21:00 +0000

There is a significant change in how software is developed over the last decade. Agile had been the buzzword over the last decade and probably is most significant of changes in software development to date. Now that the hype around Agile has been subsided, it is worth an effort to understand what it is like post agile. Often we see Agile with a narrow perspective, limiting it to a few programming practices and nothing more. We also fail to notice the indirect influence and the larger impact of Agile over the last decade.

State of post Agile Software Development will have to be seen in multiple dimensions in order to really understand its impact:

Software Development & Life cycle tooling landscape has completely changed
Software Project management methods (apart from development method) have changed
Focus on Software craftsmanship
Re-imagined Technology stack

TOOLING

It is very interesting to see the way tooling has been changed over the decade. I must say this is the area where most innovation has happened. Almost all the aspects of software tooling has been changed; from version control systems to functional testing software. Several new categories of tools have born, which we couldn’t have imagined in 90s. Monolithic IDEs and tightly integrated eco systems are replaced with more specialized tools integrated via loose interfaces. Tools like GIT (distributed version control systems), Hudson (Continuous Integration Systems) are few examples of this change.

METHODS
While agile methods initially focused on changes related to software programming practices; over the time they went beyond. New ways of project management, new ways of testing have extended the focus much beyond. According to “State of Agile Development” survey in 2011 (http://www.versionone.com/state_of_agile_development_survey/11/), 80% of the organizations are using agile practices. That number sure tells a little about the adoption of agile methods and its impact.

CRAFTSMANSHIP
Thinking about Software craftsmanship is not very surprising given the agile method’s focus on people. In order to be successful, almost all the agile methods needs skill maturity in people, hence the stress on craftsmanship. Craftsmanship needs a learning organization & people and the focus on quality of work. Mentoring, a standardized toolbox, familiarity with tools is essential for a developer/tester to be a craftsman.

STACK
Last but not least, Technology stack also has been changing to fit the agile practices. Languages like Ruby, Groovy, frameworks like Rails and Grails and NoSQL databases have brought-in fundamental changes to reflect agile practices. Test cases integrated to the programming frameworks is not something one could have thought of a decade ago. Conventions over configuration, method missing and other such features have evolved to keep up with the expectations of agile practices.

PREPARING FOR POST AGILE
Most software development shops look at the impact of Agile in silos. For example, technology groups focus on agile tools and stack, but not methods. Process & Quality groups focus on methods but have neither have clue on tools nor the technology stacks. Craftsmanship is ignored altogether. Looking at impact together in all dimensions (across tools, stack, craftsmanship and methods) is half the work done. Aligning the team to fully benefit from the agile is the other half, which is probably the most difficult part.

Akka Essentials - book published

noreply@blogger.com (Munish Gupta) — Fri, 21 Dec 2012 05:50:00 +0000

My book on Akka titled - Akka Essentials published by Packt was released on 26th Oct 2012. The book is available from PacktPub site as well on amazon.com.

"Akka Essentials" will show you the current challenges with Java Scalability and concurrency model and how Akka’s Actor Model can help you design and build applications that are inherently scalable and fault-tolerant. Whether you are building new applications or want to refactor an existing application, you will learn the right techniques to build and scale up in no time.

This book is aimed at developers, architects who are building large distributed concurrent and scalable applications using Java/Scala. The book assumes knowledge of Java/JEE concepts but no knowledge of Actor model is assumed.

I dedicate this book to my father who passed away on 30th Oct 2012. All earnings from this book will be donated to my father's favorite charity.

This post originally appeared at www.techspot.co.in

Application Scalability: Still elusive for Enterprises Apps

noreply@blogger.com (Munish Gupta) — Tue, 21 Aug 2012 04:46:00 +0000

The advent of the Consumer Business application like Facebook, Twitter has changed the definition of Application Scalability. Decade back 10 million+ was a large user base, Facebook will touch 1Billion+ users by end of this year. There are hordes of applications in 100+ million user range. The techniques and approaches employed by these large Consumer Business Applications are different from the traditional enterprise application design and architecture techniques.

If we study the architectures principles employed by these large Consumer Business Applications, then we can conclude to the following

All the large consumer business applications are build and make use of Cloud Computing
Applications are build using a combination of open source products and platforms
Create solutions where the current set of solution do not meet requirement or scale up (e.g. HipHop, Hadoop, ChaosMonkey etc)
Constant knowledge sharing within the community (FB/Twitter/Google open sources lot of their internal products)

From the Application scalability perspective, the following key architecture patterns have emerged that are being used to scale the application

Stateless Applications - Modern Web applications maintain state, which means that they remember what you were did as part of your last request, and they remember all this data as part of the Session. The Stateful nature of the application means if the server holding that state goes down, the entire user state is lost. Traditional techniques like sync user session data across server nodes or persisting in data store do not scale very well. To overcome this shortcoming, the whole concept of creating stateless applications where the servers do not maintain state. The application uses cookies and makes use of RESTful API’s to craft the user experience. Newer frameworks like Play Framework, Node.js, Vertx.io all promote this stateless application development that scales with the increasing user load very well. Another option used is to offload session state is high throughput memory based key value store, which is common across stateless nodes.
Data Sharding – Another issue encountered with scale is the increasing load on DB servers. DB Servers do employ techniques like Master/Slave or Clustering techniques coupled with large powerful boxes but beyond 100+ million users, these techniques also fail. Further, the costly hardware and software licenses make the traditional DB options very costly. So, companies have used the MySQL as the base and created lots of solutions/topologies around the same. One of those techniques is data sharding. A Data shard is a horizontal partition of the data, meaning rows of the tables are held separately. There are numerous advantages to this partitioning approach. Since the tables are divided and distributed into multiple servers, the total number of rows in each table in each database is reduced. This reduces index size, which generally improves search performance. A database shard can be placed on separate hardware, and multiple shards can be placed on multiple machines. This enables a distribution of the database over a large number of machines, allowing database performance to be spread over multiple machines, greatly improving overall performance.Many new non-relational databases (commonly known as NoSQL databases) such as Cassandra, MongoDb and Hbase have been created with database sharding as a key feature.
Bring Data closer to user (Caching) – Caching has been adopted and used very well by most of the Consumer Business Apps. Open source products like memcache provide caching options across the tiers (web tier/app tier and data tier). Memcache provides a reliable alternative to the traditional caching solutions – coherence, terracotta. Further, the NoSQL solutions employ the caching engine to speed up the database read and write operations. Couchdb uses the memcache to provide an in memory read/write solution.
Service provider/consumer model – Another pattern that has emerged is the Service provider and consumer. It is derivative of the SOA model but leaner and simpler. The business functionality is exposed as a set of services via RESTful API with JSON data format. The presentation layers are the consumers that use these services to build up the user experience. The technologies to build the service and provider can be completely separate. Technologies are chosen based on what is best for the use case. Contracts are enforced by Service version and definition only. Front ends are typically built using PHP or Ruby over rails. Services are built using Scala, Akka, C++, Java etc.

Why Enterprises are looking at these patterns?

Enterprises have started adopting cloud (public/private) by and large. Elastic scaling means responding to the changing load pattern on real time basis. In the absence of application ability to scale, all that investment into cloud is waste
The traditional CPU/Core based licensing models are proving to be costly in the cloud world. Enterprise need to adopt the OSS frameworks/solutions that do not come with licensing baggage
The user expectations have gone up by few notches. Enterprise applications are compared and expected to have same resiliency & performance as any other Internet Scale application
Scale up model(adding more CPUs) for enterprise applications comes with its own headache of additional capital expenditure. Scale out models(add more smaller servers) that rely on commodity servers or can run over a hybrid cloud are becoming more cost effective solutions
Enterprises are going global which requires the systems to be up and available 24X7. Currently tightly coupled application designs do not lend very well to the scalability pattern

This post originally appeared at www.techspot.co.in

Top 5 Picks of Google IO 2012

noreply@blogger.com (Aravind Ajad Yarra) — Mon, 02 Jul 2012 14:39:00 +0000

Google IO 2012 developer conference has just concluded last week amid lot of fanfare. I for one think that Google has lot more influence on Enterprise technology than it seems. Some of what Enterprises see as latest and greatest of technology (such as Map Reduce) has been pioneered in Google a while ago. This is one the main reasons why I followed this event closely.

Over the 3 days Google has several technology and product announcements across the product lines. Here are the top 5 picks at Techspot.

#1 Google Compute Engine

With Compute Engine, Google made it intentions clear as a Cloud Service provider. Google entering Infrastructure as a Service (IaaS) market brings more innovation and cheaper compute resources; enhancing its portfolio beyond PaaS. How exciting!! It is interesting to see the reverse trend of PaaS offerings to IaaS offerings (Microsoft also recently started offering Linux VMs as a service). Does this mean uptake of PaaS is very low? Compute Engine offers Linux VMs running Ubuntu 12.04 and CentOS 6.2. Most of the concepts that Amazon EC2 uses such as zones, ephemeral versus persistent disks are present in Compute Engine tool. That said, this is a reasonable start for Google but a long way to be a threat to Amazon EC2. It is interesting to see Enterprises like Bestbuy listed as their beta customers.

#2 Jelly Bean, Android 4.1

Jelly bean, Android 4.1 is the newest version of the popular mobile operating System announced in IO. Apart from the several performance optimizations and various user level features, Jelly bean packs several interesting technology innovations. Prime among them is “Google Now”. Gartner has been talking about Context Delivery Architecture for few years, Google Now is probably first main stream execution of the same. Google now combines user location, calendar details, past searches, traffic, weather and time in interesting ways and provides useful information before even asking. One of the use case shown is, when you have an appointment at a certain location, it checks traffic and tells how long it'll take to get there. It notifies when you should leave, so that you can reach the destination on time.There are many other interesting features including Project Butter, Systrace tool, peer-to-peer service discovery, cloud messaging, smart app updates.

#3 Packaged Apps & Chrome Everywhere

Google chrome is the new generally available on Android 4.1.With good HTML 5 support, mobile web applications on Chrome in Android are very powerful. Chrome is also available on iOS. Packaged apps are one of the interesting features that are announced for Chrome. Packaged apps allow access to Chrome API, but are written in HTML 5, JavaScript and CSS. Packaged Apps are loaded locally and supports offline mode.In a way these are like Adobe AIR applications, but are written using standard web technologies. This is probably another category defining feature; developing cross platform hybrid applications running locally.

Some interesting stats on chrome are revealed. Chrome now has 310M active users across the world.

#4 Project Glass

Lot has been written and said about Project Glass from Google. Wearable computing is about to start. Project glass probably can make a new way of how we use and interact with computers. Technically not much has been revealed on project glass, other than the fact that it has camera, microphone, gyroscope and wi-fi connectivity. Key-note demo by Sergey Brin that involved blimp, sky divers, project glass and Google+ hang-outs is an epic.

#5 Nexus 7 Tablet

Technically Nexus 7 is just an Android Jelly Bean device. There is no reason to pick this other than for the fact that it is the first official Tablet from Google.At a price of $199 and with Quad Core Tegra 3 architecture and 12 GPUs, this tablet is a real power house. Wi-fi, gyroscope, GPS, accelerometer, Gorilla Glass, front facing camera, this is a sure main stream device and probably puts android tablets into millions of hands, another great revenue opportunity for Mobile developers. Interestingly Nexus 7 ships with Chrome as the default browser instead of Android’s native web-kit based browser. I guess Google is trying to avoid an IE 6 scenario here by separating browser from OS.

There are many more interesting announcements that didn’t make it to the top 5 list but I found them interesting enough to mention here:

Offline maps and offline voice recognition for Android
Enhanced Gesture support in Android
Apps Script for Good Apps Automation
Google Docs Offline

What are your top 5 picks?

This post originally appeared at www.techspot.co.in

Cloud Connect 2012, Bangalore

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 25 May 2012 15:59:00 +0000

Cloud Connect is a global Cloud Computing event that has come to Bangalore, India this year. The event is organized by UBM, a reputed media organization that publishes Information week magazine. Other popular events organized by UBM includes Interop. Bringing Cloud Connect to India, shows the importance of India as a Cloud market. Event happened this week 5/24 to 5/25 at Nimhans Convention Centre.
IBM is a platinum sponsor and the event started with key note by Vamsi from IBM.
IBM's Vamsicharan Mudiam will keynote@cloudconnect_in on May 24 in Bengaluru, #India. View his profile: bit.ly/KnahAi #CloudConnect

IBMCloud

4 days ago

Reply Retweet
IBM's session had several quotes form their recent C-Study report. Some of the interesting findings include the changing CEO priorities. IBM says cloud adoption is increased from 13% to 26% this year.
Ibm c-study: Technology factors and People skills are at the top. Glad to see that. #cloudconnect

Aravind Ajad

2 days ago

Reply Retweet
Like lot of cloud conferences earlier, this conference has lot of focus on Infrastructure Cloud. Most of the sponsors and speakers are focused on the Infrastructure aspects of the Cloud.

Some of the key observations I made were:

* There is quite a bit of talk on private clouds from the attending vendors. There are quite a few sessions covering this. There are sessions from VMware, L&T infotech, Citrix, Cisco, Brocade and so on. This is not surprising, given Infrastructure is the focus for vendors. There are however, few refreshing sessions that took a different take. Director from Citrix cloud.com talked about cloudstack and how it can be used to build amazon like clouds.
Architecting aws style clouds - Ram Chinta. cloud.com #CloudConnect

Aravind Ajad

9 hours ago

Reply Retweet
* There is significant focus on security as well. In fact, one of the conferences rooms almost dedicated to security. And there seems to be good response to these sessions as well. Probably shows cloud security still is a major concern. There is also some pragmatism on how the trade-offs can be made.
Common thread in al presn.@ #cloudconnect on#cloudsecurity -impln.depends largely on orgn.biz goals & risk appetite thereof!

Runa Desai Dalal

5 hours ago

Reply Retweet
1 mre good talk by Akhil Behl of @CiscoSecurity busting myths of #cloudsecurity!Cloud consumer just as responsible as a CSP!#cloudconnect

Runa Desai Dalal

4 hours ago

Reply Retweet
* Another interesting point is the focus on Applications. This is probably a departure from many cloud conferences. There are several talks on this topic.
Krishnan, editor at CloudAvenue and a popular name in Cloud circles spoke on the future of PaaS platforms. He says PaaS is the future of Cloud computing. He backs his arguments by what is happening and the trends he is seeing. He also thinks future of PaaS is really different from what it is now. Future PaaS platforms will be platforms beyond the current capabilities, with lot of Intelligence. They will have several services and capabilities built-into them such as Data Markets, Mobile Support and Social Connectivity. Data plays a big role in the future PaaS. These services makes building intelligent applications on Cloud very easy.
.@krishnan on stage - Intelligent platforms, future of pass. #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
Big data is at the core of next Gen of PaaS@krishnan #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
He gave some examples of which platforms that shows this potential of what future PaaS can be.
Candidates for future of pass: flow.net, brroklyn, pervasive #cloudconnect @krishnan

Aravind Ajad

10 hours ago

Reply Retweet
There is also a session on Cloud platform comparison from Jankiraman. He says Amazon Bean Stalk compared to CloudFoundry, GAE a good compromise between control and flexibility.
Good introduction to PaaS platforms and their current state. Like the PaaS comparison by @janakiramm at#cloudconnect

vivekjuneja

10 hours ago

Reply Retweet
Jason Bloomberg of Zapthink talked about how to used RESTful hypermedia state to make applications Elastic. He says adopting applications to the cloud is not lift and shift, but a change in the architectural thinking. He says RESTful architecture can help build elastic applications that leverage the potential of Cloud. By keeping the state on in the hypermedia, the applications can be more elastic. Jason also shared free poster on Zapthink's vision of future of IT. It is very interesting and full of Cloud, Mobile, Social, Democratization of technology. In a way pointing to demise of Enterprise IT :-).
RESTful Hypermedia state to make applications elastic.@TheEBizWizard #cloudconnect

Aravind Ajad

10 hours ago

Reply Retweet
Free poster 'Vision for Enterprise IT 2020'@TheEbizwizard #cloudconnect

Aravind Ajad

11 hours ago

Reply Retweet
There are also couple of sessions focused on Data. One session from Nokia architect on Big Data adoption. There is a session form CTS architect on NoSQL database constructs.
* Popularity of AWS is evident through sessions. Being a proto cloud provider, there are references to AWS in almost every session. There are quite a few talks on how to deliver highly scalable applications on AWS. One such interesting session by Harish, explained how to achieve a peak tps 80000 requests using elasticity of AWS.
Good session on infrastructure elasticity using AWS.#cloudconnect

Aravind Ajad

a day ago

Reply Retweet
Sessions from me and my Wipro colleagues (not officially sponsored :-) ):

Munish Gupta spoke on Designing Performance of Cloud Applications, while I spoke on Delivering Enterprise Scale Apps on Cloud.
RT @write2munish: Speaking at the CloudConnect Conference in Bangalore on May 24 bit.ly/JFFXkB #cloud #cloudconnect @cloudconnect_in

Cloud Connect India

2 days ago

Reply Retweet
RT @aravindajad: All set for my speaking session at#cloudconnect conference tomorrow. goo.gl/V5Nsf @cloudconnect_in #in

Cloud Connect India

a day ago

Reply Retweet
Srinivas Adhyapak another Wipro colleague spoke about the Impact of cloud on Telcos and how they are in a good position to become public computing providers.
Sessions from my collegue Munish and me are in cloud (Slideshare) for your reference. Thanks to all of those who attended our sessions.
Cloud connect - Delivering Enterprise Scale Applications on Cloud

aravindajju
Thanks to UBM for bringing this event to India. UBM team has put a fantastic show and my appreciations to Srikanth and team.
Cloud Connect Bengaluru draws to a close! Thank you all for making it a success! We will keep in touch herebit.ly/KxNrEy

Cloud C

This post originally appeared at www.techspot.co.in

Blackberry 10 – Putting Developers First

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 04 May 2012 15:37:00 +0000

Blackberry 10 OS is unveiled this week at Blackberry World. In many ways Blackberry 10 is probably the last chance for Blackberry to gain back the market share and be a formidable competition to other Smartphones. If the announcements at the conference are any indication, Blackberry is going all out this time to make their next generation of OS as the best smartphone platform.

Some of the interesting features of BB10 mentioned in the key note:

UI Overhaul – More modern user interface based on Panels (similar to WP7) and the flow interface that switching between applications smooth and seamless. At last, BB user interface is on par with the competition.
Intelligent Keyboard – Finally BB is betting on touch interface. BB is trying to bring its legendary physical keyboard expertise to touch interfaces. Keyboard with gesture support and intelligent prediction keeping the usability a key priority.
Camera with timeline - Camera has a timeline history, so if there is an issue with a photo captured (such as somebody eyes are closed), one can rewind and capture another moment which is more perfect.

The most notable of all, is the focus on developers. There has been significant focus on Developers in the CEO keynote. One of the BB’s problems has been to attract developers to build quality apps to be able to compete with other smartphone platforms. To address this problem, BB has spent significant time and effort and put the developers first in BB10.

Here is how -

For starters, Blackberry 10 is not based on the older platform BB7, but is based on QNX (acquired by BB in 2010). QNX is a RTOS that has been existence for a while and is used in telematics and other mission critical applications used in medical and telecom industries. QNX is also the OS used in Playbook tablet released last year. QNX uses micro-kernel architecture and has great tooling support, making it a solid platform, attractive enough for developers.

Keeping developers first, BB has taken quite a different approach for applications (on BB 10) compared to the competition. All of the Blackberry competition has a single deeply integrated application development framework and uses a single programming language. Apple uses Objective C, Google with Java and Microsoft with C#. BB however, took a multi-language multi-framework approach, to keep a low entry barrier for all kinds of application developers. Following graphic illustrates how much BB10 is doing to include all kinds of developers, leaving nobody behind.

Blackberry WebWorks SDK:

Webworks SDK supports BB application development using HTML 5 and CSS. This is really the lowest barrier for application developers. Millions of web developers can potentially become blackberry developers with very little training. Unlike other smartphone platforms where HTML 5 applications are secondary citizens, BB10 has embraced the web development paradigm.

Ripple mobile environment emulator for testing the applications for different form factors
Local storage of HTML 5 and CSS applications, does not server access
Javascript extensions providing access to Blackberry messenger, PIM and hardware
bbUI.js for native look and feel

Blackberry Android Runtime:

Android has pioneered the use of virtual machine as the first class development platform on smartphones. Dalvik on android provides a bytecode compiler and a runtime allowing applications developed in Java language, to run on Android. Given there are thousands of android applications already running Dalvik, BB has made a smart move in transitioning these applications onto BB 10, tapping into large existing Android developer base. While there are quite a few API compatibility limitations, nevertheless BB Android runtime provides an easy option for porting existing Android applications.

Supports Android 2.3.3 (Gingerbread) applications
Online repackaging tool to convert APK files (android application package format) to BAR files
Support for multiple windows and in app payment

Blackberry Adobe AIR SDK:

Adobe AIR is a popular UI application development platform based on Action Script and Flash runtime. Action script is a variant of Java Script hence majority of the web application developers uses AIR to develop cross platform UI applications. AIR already supports Android, iOS and even desktop OSes and thousands of applications already written. Given AIR is support, all the AIR applications can be ported to BB10 easily. What is interesting to note is that, AIR applications are developed using Flex, which is a quite popular with Enterprise to develop in-house rich UI applications.

Blackberry Native SDK:

BlackBerry 10 Native SDK is targeted at building high-performance native applications. Supporting C++ as the language, it is not really meant for developing business applications, but meant for applications that need good response time, such as games.

Scoreloop social gaming SDK provides support for user profiles and global leaderboards, awards
Push Management APIs allow developers to create Push-based apps
Visual Studio IDE plug-in will now make it easier for VS developers
Cascades SDK – New UI framework for developing native UI applications supporting C++ and Declarative Qt Markup Language (QML)

As it happened with HP webOS and to an extent with Microsoft WP7, applications can make or break the smartphone platform. With BB 10, Blackberry is taking all the right steps to get the developer support, which is crucial to their success. BB is also pulling all the right strings to make it easy for developers to develop their applications on BB OS. That said, not supporting legacy Java applications (developed using BB 7 java SDK) is a significant issue.

It will be interesting to see how successful BB 10 will be and whether it can bring back the lost glory for the Blackberry.

Further reading:

Blackberry 10 announcement - http://www.blackberryworld.com/event-info/about
Blackberry developer blog - http://devblog.blackberry.com

This post originally appeared at www.techspot.co.in

Developing Scalable Apps for deploying on IaaS cloud

noreply@blogger.com (Munish Gupta) — Tue, 01 May 2012 03:03:00 +0000

My presentation on Developing Scalable Apps for deploying on IaaS cloud, which i presented at Cloud Developer Conference 2012
Developing Scalable Apps for deploying on IaaS Cloud CDC April 2012

This post originally appeared at www.techspot.co.in

Plumbing the Web

noreply@blogger.com (Aravind Ajad Yarra) — Mon, 30 Apr 2012 14:22:00 +0000

Cloud based services are so ubiquitous that it is almost impossible to imagine our daily life without these. These services range from web e-mail services like Gmail to productivity apps like Evernote to Social Networks like Twitter to Information services like Weather.com. I can count at least twenty different services that I use on a daily basis and probably hard to live without. I get tremendous value from these services, both personally and professionally. However, there is an ugly side to these services. Each of these services live in their own island and it is hard to go on to each service to keep up with it.

Isn’t it great if I can push my favorite Google Reader article to Instapaper for a later reading?
Isn’t it great if I can get an SMS alert, if the weather.com forecasts rain in Bangalore today evening?
Isn’t it great if I can move my personal tweets to my Gmail account for later reference?
Isn’t it great if I can get an email alert if my twitter follower mentions me?

What we need is a way to plumb all these discrete services together to create more valued experience.

Ifttt.com
Ifttt.com is a web plumbing service that connects multiple web services together to create services that are much more useful than those individually are. Ifttt.com stands for If This Then That. It essentially creates a macro environment for web (à la macros in MS Office), enabling knowledge web workers to automate tasks across web. Given the workforce in Enterprises are increasingly populated with Gen Yers, such web plumbing services have an important place in the future of Enterprise IT.

It provides basic constructs to monitor and evaluate an event and then take an action on the same. To show how this service works, please see the screen grab below. This graphic shows a Task which sends an SMS alert if there is Rain predicted tomorrow on Weather.com.

As can be seen in the graphic there are two simple constructs Trigger and Action. There are predefined channels which can be used to define the triggers. In this case Weather.com is used as the channel and the trigger is “Tomorrow forecast calls for Rain”. Action is defined as an SMS message, which another pre-defined channel with my phone number attached to it.

Ifttt.com runs this task every 15 minutes and checks if there rain is forecasted tomorrow and then sends an SMS alert. That is how simple using this service. The user interface is barebones and is very simple and easy to use. One doesn’t need to have any Programming background to use. But don’t let the simplicity undermine its power. As Douglas Adams quotes in ‘The Salmon of Doubt: Hitchhiking the Galaxy One Last Time’, “The future of computer power is pure simplicity”.

Under the hood of Ifttt, REST APIs and RSS Feeds are hard at work. These technologies enable easy programmable access to various web services. Another important, but often overlooked technology that forms the technological foundation of Ifttt is OAuth. OAuth enables token based end user authentication and authorization between multiple web services.

Ifttt.com is not the first service to attempt web plumbing. There is a great service called Yahoo! Pipes, which probably started the idea of web plumbing. Yahoo! Pipes is very visual with a graphical interface reminiscent of popular WISYWIG image editors. Yahoo has done a great job in abstracting the complexity of REST APIs, RSS feeds as visual widgets that can be dragged and dropped. Unfortunately the service couldn’t get the traction with masses, however is quite popular among web geeks and programmers.

Extending the Idea to Enterprise:
It doesn’t take much imagination to see how useful web plumbing (in specific ifttt.com) can be in Enterprises. Enterprises have been trying hard for a long time to seamless integration of internal IT services. Years ago Enterprises have embarked the journey of Service Orientation Architecture, but are yet to reach the distant dream of seamless integration. Adoption of cloud SaaS services have only made reaching the dream harder.

Given how ifttt.com has brought web plumbing to masses, there is lot of speculation that ifttt.com is pursuing Enterprise Interests. Recently ifttt got a funding of $1.5 million, not bad for a two person company. Can’t wait for an Enterprise edition of ifttt.com.
Cloud Integration Platforms are already rage in Enterprises. To name a few, IBM’s Cast Iron and Dell’s Boomi are already heavy lifting the integration between on-premise services and SaaS service hosted in the cloud. SnapLogic (www.snaplogic.com) is another interesting Cloud Integration Platform. It has pioneered the use of App Store for its cloud integration platform. However, all these platforms are targeted towards the IT developers and not the Business Users. One day, I hope these services learn something from ifttt.com.

This post originally appeared at www.techspot.co.in

Social Networks Going Visual

noreply@blogger.com (Aravind Ajad Yarra) — Fri, 13 Apr 2012 16:13:00 +0000

Social Networks are traditionally focused on sharing of status messages and links. The focus of these networks was about sharing of all kinds, but not as much as about the Visual aspects. This focus has been shifting recently to Visuals. This can be seen in the recent growth of the Visual Social Networks such as Instagram and Pinterest. These networks put the Visual aspects in front and center of the sharing.

Some of the facts that support this observation:

This week Facebook announced to buy Instagram for $1bn. (Instagram has just 13 employees but has about 50 million plus users. Instagram works by providing several photo filters that can make ordinary photos / pictures visually appealing. After applying filters these pictures can be share.)

Pinterest has 12 million users and has grown 145% from January to March. (Pinterest operates like an online pin board. Pinterest allows to bookmark (called pinning) any pictures that we see online and share them on a board. Anybody who likes your picture can also pin it. Popularity in terms of Pins most likely to get noticed and drive traffic)

The reasons for such focus on Visual aspects could be attributed to:

High quality Cameras on Mobile Devices:

With Smartphones, People now have access to high quality cameras all the time, leading to sharing pictures more and more. Some of the smartphones such as iPhone 4S have high quality cameras that can give many dedicated Point and Shoot cameras a run for their money.

Signal to Noise ratio on Traditional Social networks:

Traditional social networks such as Facebook, Orkut, MySpace, Twitter and Google+ can do so many things; so much so that it is hard to align with People’s interest. People refer this as Information Overload. I refer to this phenomenon as Social Pollution and Social Fatigue. This exact phenomenon is what is leading to Interest Based Social Networks. Visual Social Networks are a subset of Interest based Social Networks. Other examples of Interest based Social networks are – Quora.com (Q&A based Social Network based on Topics interesting to a User), Foodspotting (Recipe based Interest Networks) and Endomondo and Fitocracy (Fitness based Interest networks)

Visual Appeal:

Humans are inherently visual. Visually appealing object stimulates our brains and makes us enjoy and share them :-)

While the reasons can be many, the impact of Visual social networks especially is astounding. Especially Pinterest has an astounding effect on online commerce. On-line businesses are flocking to have their presence felt on Pinterest. Here are some interest facts on how Pinterest is contributing to Social Commerce.

Impact on On-line / Social Commerce:

Impact on On-line Commerce can be measured by the amount of referral traffic a Network brings. As can be seen in the graphic below Pinterest referral traffic has increased significantly within a month. Amount of referral traffic Pinterest brings is more than Twitter and Google plus. Given the size of Pinterest this is really significant. This can explain why on-line commerce world is eager to have their presence felt on Pinterest.

In order to capitalize on this trend, on-line businesses are now focusing on having high quality and artistic product images on their sites. This is one of the many ways how On-line businesses are leveraging the Social networks generate more leads and better sales.

References & Further Reading:

Everything you need to know about Pinterest http://techcrunch.com/2012/03/14/this-is-everything-you-need-to-know-about-pinterest-infographic/
Pinterest drives more referral traffic than LinkedIn http://blog.shareaholic.com/2012/03/pinterest-referral-traffic-2/
Why Instagram is worth #1bn http://mashable.com/2012/04/12/instagram-worth-1-billion/

This post originally appeared at www.techspot.co.in

Consumerization of IT – what it means for the Architect

noreply@blogger.com (Munish Gupta) — Tue, 24 Jan 2012 07:30:00 +0000

Consumerization is described as the trend for IT to first emerge in the Consumer space and subsequently makes its way into the enterprise world. But what exactly in the consumer world, that is making the users, demand the similar things from the enterprise IT. To understand the underlying need, we need to first examine the basic requirements of the user.

Kathy Sierra, co-creator of the Head First series of books and founder of javaranch.com, describes the hierarchy of needs from the user(s) perspective. The needs are stacked in the order of increasing engagement from the user. Starting with the basic needs of a defined functionality and its correctness, moving on to the learnability, efficiency & usability and finally culminating in intuitiveness and enchantment. Merely provision of correctly working functionality is not guarantee of the success of the application(s). The idea is to hook the user; the application needs to do something extra.

Using the hierarchy of needs as the background, we will see how the applications in the consumer and enterprise world stack up.
Now, in the consumer world, the advent of Consumerization started with the proliferation of the mobile devices. In the connected world, the device (smartphone or tablet) imposed certain constraints

Screen size is small and limited
Processing power of the device is limited
Interface is touch enabled
Connectivity is not constant (read patchy) and speed/data is limited (no unlimited data plans)
User audience demography not known

Now, the Apps needed to be designed keeping in mind the constraints, which meant that the Apps are

Focused on providing only one functionality
Have simple and intuitive messages/steps in the absence of any help/guides
Providing simple interfaces (cater to worldwide audience)
Having uncluttered screens as real estate is limited
Using UI Controls that are big and usable (touch interface)
Having intuitive workflows
Engaging to stand apart among the millions of other Apps

This meant that the functionality does not require the user to be a geek. The average individual user can understand and use the App functionality without any external help. Majority of the Consumer Apps are heavily focused on the Efficiency and Usability part. Few of the Apps have been able to enchant and engage the user (e.g. Angry Birds).

In the enterprise world, the device is most likely to be powerful desktop/laptop with a large screen. The connectivity is via LAN or broadband. As a result the Enterprise Applications are more industrial with high focus on providing long list of functionality and ensuring the correctness of the same. Consumers facing business applications usually focused on usability, Specialized enterprise applications (e.g. Call Center Applications) focus on the overall workflow efficiency, but majority of the applications are resource intensive, lack efficiency and have high learnability curves.

The enterprise user experiences the simple, intuitive consumer apps on their mobile devices. With the advent of BYOD (Bring Your Own Device), enterprise user’s started bringing their smart devices into the enterprise. Soon the users are comparing the experience of consumer apps with the enterprise business apps; they wonder why the enterprise apps can’t provide a similar experience. Why are the enterprise apps so low on learnability, efficiency, intuitiveness factors?

The whole idea behind the Consumerization of IT is not just measures like BYOD but to bring the missing intuitiveness, usability and efficiency into the enterprise applications.

So, what does Consumerization means for the Architect?

When designing and architecting enterprise business applications, following considerations need to be kept in mind

Connected users means the applications need to be available 24 X 7 ( Always On, Always Available), 24 X 7 availability requires application solution(s) to be elastic – expand or shrink based on the load
Intuitiveness and Usability have to be high on the agenda when designing the interface and workflows
Solution needs offer APIs for additional applications to be build
Integration with systems and other applications (including SaaS) need to be simple, straight forward and well documented
User experience is the key to successful business application (it was always the key) but need to be (re)designed for the connected & mobile devices
Product evaluation(s) will now include SaaS applications that can provide the functionality
Saas adoption will increase leading to newer challenges in integration, data security and privacy measures
Applications need to be designed/tested keeping in mind various devices/OS combinations (days of designing apps that worked on IE6 only are over)

The impact of consumerization on the Enterprise IT will be felt in the years to comes, BYOD is just the harbinger of things to come.

This post originally appeared at www.techspot.co.in

Bigdata Trends in 2012

noreply@blogger.com (Munish Gupta) — Tue, 17 Jan 2012 07:30:00 +0000

Bigdata has caught the fancy of each and everyone. Consumer and enterprise world is looking to adopt the Bigdata. Analysts are talking about it, proclaiming it to the next big thing!

This is my take on how the Bigdata market space will evolve in 2012 and years to come.

Data Analytics Service – We will start seeing SaaS vendors that will start offering data analytics as a service (e.g. 1010data). These vendors will be able to pull in your unstructured data and provide you standardized reports and data services. This is very similar to the Web analytics (like Omniture) but access lot more data to provide patterns. 70-80% of the data analysis pattern will be similar across industries. These vendors will be able to provide these reports as off the shelf capabilities. For the other 20%, the web interface will allow the user to create patterns to analyze the data

Data Visualization Service – Analysis is only one part of the data, the other big thing will be able to visualize the data (more like infographics) that help make sense of the scale and data points. Business will need help to make sense of all this analysis and again 3rd party vendors (via products or Saas) will be fulfilling this space

Crowdsourcing Model – enterprise will start using the crowdsourcing model to help analyze and find patterns in the data. Goldcorp was one the original pioneers that used the crowdsourcing model to analyze the geological data. This model will be seen more when dealing large geospatial data in the areas of mining, drilling, oil ad gas

RDBMS vendors will make a comeback – Bigdata has been scoring high on account of their ability to analyze and process unstructured data. The NoSQL vendors and products are all claiming to be the holy grail of data storage, fast processing and analytics. I believe, the Big Vendors –Oracle, IBM and to some extent Microsoft will release their version of relational databases that can store (using automated data sharding) and process large amount of data (with a SQL MapReduce layer). The vendors will incorporate techniques from the NoSQL world coupled with in-memory storage (think Coherence, eXtreme Scale) to provide sub second read write response times. The market is too big for the existing vendors to not claim supremacy

NoSQL market will shake up – The current crop of NoSQL solutions has emerged as need to fulfill certain use cases. Their base features and functionality have been decided by the patronizing user client/project co-sponsors (e.g. Cassandra – Facebook, Hadoop-Yahoo, membase-Zynga). These NoSQL solutions work very well for the specific use case. Attempts to broad base the use case, mean a lot of effort needs to put in. Along the way, some products will merge, some will lose community support and other will get acquired by product vendor companies

Data Analysis Libraries - To help the developers analyze data, the common patterns and designs will be available as off the shelf libraries. This means, analyzing data for most common patterns will be easier and will become no-brainer. The projected need for Data scientists will not materialize to the same extent with the proliferation of these libraries. E.g. MS Cloud Numerics

Bigdata will follow the Consumerization model, where the core infrastructure either comes as a service or an appliance. The data analytics, data visualization will be provided as a standard set of services on top of the raw data with an ability to create your own model.

Do let me know, how do you see the Bigdata market moving forward and do you agree or disagree with my predictions?

Application Performance and Antipatterns

noreply@blogger.com (Munish Gupta) — Wed, 04 Jan 2012 13:35:00 +0000

Any application you pick up, there are some issues – big or small. There will be copy-paste code, mistakes, algorithms which could have better thought through. But what distinguishes an antipattern from these normal errors is that like patterns these antipatterns are recurring throughout the code base. In my recent experience in dealing with performance issues, I had observed certain recurrent themes that are undermining the overall application performance. Most of these antipatterns are well documented but it seems we do not learn from others mistakes. We need to make our own mistakes. I am recounting some of the common patterns that I observed in the recent months.

Excessive Layering - Most of the underlying performance starts with the excessive layering antipattern. The application design has grown over the usage of controllers, commands and facades. In order to decouple each layer, the designers are adding facades at each of the tiers. Now, for every request at the web tier, the request call goes through multiple layers just to fetch the results. Imagine doing this for thousands of requests coming in and the load the JVM need to handle to process these requests. The number of objects that get created and destroyed when making these calls add to the memory overhead. This further limits the amount of requests that can be handled by each server node.

Based on the size of the application, deployment model, the number of user’s, appropriate decision need to be taken to reduce the number of layers. E.g. if the entire application gets deployed in the same container, there is no need to create multiple layers of process beans, service beans(business beans), data access objects etc. Similarly, when developing an internet scale application, large number of layers start adding overheads to the request processing.

Remember, large number of layers means large number of classes which effectively start impacting the overall application maintainability.
Round Tripping- With the advent of ORM mappings, Session/DAO objects, the programmer starts making calls to beans for every data. This leading to excessive calls between the layers. Another side issue is the number of method calls each layer start having to support this model. Worse case is, when the beans are web service based. Client tier making multiple web service calls within a single user request have a direct impact on the application performance.

To reduce the round tripping, the application needs to handle or combine multiple requests at the business tier.
Overstuffed Session- Session object is a feature provided by the JEE container to track user session during the web site visit. The application start with the promise of putting very minimal information in the session but over a period of time, the session object keeps on growing. Too much of data or wrong kind of data is stuffed into the session object. Large data objects will mean that the objects placed in the session will linger on till the session object is destroyed. This impacts the number of user’s that can be served by the application server node. Further, I have seen, application using session clustering to support availability requirements but adding significant overheads to the network traffic and ability of application to handle higher number of users.

To unstuff the session object, take an inventory of what all goes there, see what is necessary, what objects can be defaulted to request scope. For others, remove the objects from session when their usage is over.
Golden Hammer (Everything is a Service) – With the advent of SOA, there is tendency to expose the business services, which can be orchestrated into process services. In the older applications, one can observe similar pattern being implemented with EJBs. This pattern coupled with the bottom up design approach at times, means exposing each and every data entity as a business service. This kind of design might be working correctly functionally, but from the performance and maintenance point of view, it soon becomes a night mare. Every web service call adds overhead in terms of data serialization and deserialization. At times, the data(XML) being passed with web service calls is also huge leading to performance issues.

The usage of services or ejb’s should to be evaluated from application usage perspective. Attention needs to be paid on the contract design.
Chatty Services – Another pattern observed is the way the service is implemented via multiple web service calls each of which is communicating a small piece of data. This results in explosion of web services and which leads to degradation of performance and unmaintainable code. Also, from the deployment perspective, the application starts running into problems. I have come across projects which have hundred plus services all getting crammed into a single deployment unit. When the application comes up, the base heap requirement is already in 2Gb range leaving not much space for application to run.

If the application is having too many fine grained services, then it an indication towards the application of this antipattern.

The above mentioned antipatterns are frequent causes of application performance issues. The teams usually start with the right intentions but over a period of time, things will start slipping. Some of the common reasons

Lack of design standards and reviews processes – even if these exists, the delivery pressure is leading to skipping these processes
Team members inexperience or narrow view leads to every programmer only looking at their module and nobody is looking at the overall application performance
Continuous Integration(CI) tools not integrated with compliance check tools like PMD, Checkstyle, FindBugs etc
No focus on profiling of the application on regular basis during the code construction phase
Not evaluating the results from the Load tests to decipher and fix the underlying issue (blaming the poor infrastructure setup)

What are the other antipatterns you have observed that have contributed to the degradation in the application performance. Do share!

Deciphering an Outage

noreply@blogger.com (Munish Gupta) — Mon, 19 Dec 2011 13:40:00 +0000

Recently, there was a major fire outage reported. The building maintenance team informed the fire brigade department. The fire brigade came in, helped douse the fire and tried to salvage whatever they could. The owner of the building demanded an independent assessment of the reasons behind the fire, what triggered it and how to make sure this does not happen again. The building maintenance team called in an independent assessment team and asked them to comb the carnage and find out what might have caused this!

The independent assessment team went through the motions of accumulating the evidence, looking for telltale signs of what might have gone wrong and then eventually did an event reconstruction on how events would have unfolded on that day. Subsequently, the team came out with the recommendations that need to be implemented to make sure that such an outage does not happen again.

Typical reasons for most of fire outages are –not conducting regular fire safety drills, violation of the building codes, not designing the fire exit paths, not keeping common area’s clean, storing hazards materials in the basement and building unauthorized extensions. This was the same case here and the recommendations were mostly in same line. Do regular fire safety drills, design fire exit paths, adhere to the building codes, demolish unauthorized extensions, stop storing hazards materials in the basement.

Can you guess the reasons for?

why the building was built in violation of so many norms
how did all the inspectors certified the building
why did the client not got an independent assessment of the building done before possession
why did the maintenance team accepted the building without looking into the cost of maintenance in the absence of basic safety
why should the people using the building not lose confidence in the safety of the building

The above scenario is analogous to what I encountered recently. Only difference is that it was a customer facing web application that faced the outage.

Managing Enterprise Group Mail Content

noreply@blogger.com (Munish Gupta) — Mon, 07 Nov 2011 05:56:00 +0000

How many times, we have seen that one of the easiest way of cleaning up your mail box is to delete all the enterprise announcements and group e-mails? Are we getting bombarded with too many policy change mails, group emails and announcements? After a few days of the email bombardment, are our brain gets conditioned to just ignore them ?

All this leads to a point where we start becoming clueless about

Happenings in the Enterprise
Policy/Process Changes that might be affecting us

which means we might be working harder and not smarter.

Imagine, the enterprise announces some template for making presentations. The same is announced with enterprise wide email. We ignore the email contents and continuing using the older templates. One fine day, the boss looks at the presentation and says why are you using the older template and there is no body else to blame except self.

Imagine, you did saw the email and know about the new template. But you deleted the email. When a fort night later, there was a requirement to make a presentation, you are searching high and dry for the link from where you can download the template. There is no direct link available on your intranet that points to the template. There is no way to retrieve the email also.

Why do enterprise not make available all those emails available on the intranet? Is there an easier way to manage this flood of important information? How can the individual employee choose to subscribe/unsubscribe to such emails? Is there a way the employee can refer to these emails later on - where the content is available tagged and ready to be searched?

The answer to these question is simple tool - Blogs.

Enterprise can start publishing all the announcements as blog entries. This simple tool alleviates all the pains and provide a fantastic way of organizing all the content that is properly tagged and available to be searched, subscribed and even allow debates on some of those emails. How many times we wanted to comment on some policy change but could not do so because it is a group email and there is no way to provide feedback.

Having a blog does not mean, the enterprise stops sending out emails but the blog can become the one stop content repository for all such announcements. Today, publishing an entry is as easy as sending an email.

Modern Web Social Collaboration tools provide blog as a feature out of box. Enterprises can make use of the Blog as a means of storing enterprise group email content.

Most of the internet based business already make use of the blogs to connect and disseminate information about their products and services. It is just a matter of picking up the same best practice within the enterprise.

Building Scalable Enterprise Systems

noreply@blogger.com (Munish Gupta) — Mon, 17 Oct 2011 11:04:00 +0000

Successful enterprise are all about business agility and able to introduce new products and services in the market. All this business agility coupled with reduced IT overheads means, the enterprise need to find better ways to improve and transform their enterprise systems.

The advent of Cloud, Social, Mobile and Consumerization of IT means enterprises applications need to be adapt to the changing environment. Today, every enterprise is looking to fulfill the following demands

Adopt cloud for their enterprise applications (whether private / public / hybrid is a matter of enterprise priorities)
Replace or retire in-house enterprise applications ( where applicable) in favor of the equivalent SaaS applications
Expose enterprise data for third party consumption
Make the enterprise functionality available over variety of channels (web, mobile) - Consumerization
Make the systems available 24X7 to meet the ever growing business demands

The Enterprise application patterns for creating applications – Portals, CMS, SOA, Centralized DB’s are not functionally capable or geared to meet the new business demands. The enterprise needs to adopt the newer application patterns that are coming out of the consumer web world. Some of the patterns emerging out of new generation consumer web applications that can be applied for enterprise applications

Partition by Function – Divide the functionality by usage patterns or resource consumption. The functionality can be exposed using RESTful or Web services. This functional grouping will help in achieving optimization at the resource level. This helps to scale individual functional groups based on their resource characteristics (High CPU, More RAM, High I/O). This pattern organizes the enterprise platform around services which in turn provides agility. Things can be done in parallel because the output is a service. This allows fast time to market. Helps create an infrastructure that allows services to be built very fast.
Segregate the providers and Consumers – Instead of building large monolithic applications, segregate the functionality providers versus consumers. The enterprise should have separate teams – one that deal with the functionality provider and other that consume the functionality. The functionality is exposed via simple API’s having standard versioned interfaces
Public/private cloud integration – Enterprises need to have standardized integration strategy as the need on board multiple applications from multiple vendors start becoming common.
- Enterprise need to start integrating with SaaS vendors that can provide the required apps for the consumption of the enterprise users. At times, the integration might need to be handled at the API levels specifically for authentication/authorization.
- Enterprise applications specially that are consumer facing can also be hosted in public clouds.
Enterprise need to integrate services that handle data coming in or going out from public cloud / SaaS vendors to their data centers.

Tech Spot

Is Fargate for EKS "Serverless"​?

Is Fargate for EKS "Serverless"​?

What is Fargate?

Is it Serverless?

Building Digital Capabilities

Technology’s Role in Digital Transformation

Digital Transformation impact on Value Chain

A Four-Pronged Approach to Microservices Adoption

Architecture concerns for Distributed Systems

Microservices & Containers

Microservices Orchestration Design Patterns

Architecting Multi-Cloud Applications

Review: “ZeroMQ” by Pieter Hintjens; O'Reilly Media

Post Agile - State of Software Development

Akka Essentials - book published

Application Scalability: Still elusive for Enterprises Apps

Top 5 Picks of Google IO 2012

Cloud Connect 2012, Bangalore

Blackberry 10 – Putting Developers First

Developing Scalable Apps for deploying on IaaS cloud

Plumbing the Web

Social Networks Going Visual

Consumerization of IT – what it means for the Architect

Bigdata Trends in 2012

Application Performance and Antipatterns

Deciphering an Outage

Managing Enterprise Group Mail Content

Building Scalable Enterprise Systems

Is Fargate for EKS "Serverless"?

Is Fargate for EKS "Serverless"?