Cloud Computing and Hadoop

What is Big Data and three V's of Big Data

2012-08-23T10:02:00.000-07:00

SUMMARY

This section of the blog will explain the basics of Big Data and the three V’s that contributes to the Big Data.

This blog is targeted at architects, IT pros, and other leaders in enterprise who are interested to move towards Big Data.

INTRODUCTION

What is really Big Data?

Every day, organization collects so much of data from several terabytes to petabytes and in one case it’s in yottabytes (10²⁴ bytes). These data comes from various systems internally as well as externally and these data adds so much value to the organization. Traditionally organization used the historical data and analyzed them to understand the value but today we need the information in real-time as the competitive organization can take over our business if we don’t act on the data in real-time. Big Data is not just about the size, it’s the opportunity to get the value out of the data collected in real-time.

What contributes to Big Data?

Big Data has emerged in recent time to describe the challenges in data management. There are several challenges with the amount of data that are flowing in and out of systems at a speed that is unimaginable. There are several factors that drives the big data and three major factors/dimensions that contributes to Big Data are three V's - Volume, Velocity and Variety

Volume:

The first challenge every organization facing today deals with Volume, the amount of data that are flowing in and out of the system, as the volume increases several other factors needs to be considered like scaling out, scaling up, replication, failover, etc...

Velocity:

Velocity is another challenge, it’s the speed at which the new data is flowing as well as the modification to existing data and in many cases these new and updated data must be available to systems immediately in real-time, there are several tasks has to happen in real time, like indexing, searching, analytics, etc… on the new and updated data.

Variety:

This is one the most important challenge as we are moving more towards mobility and social. Organization today collects enormous amount of data, these data fall into one of three different forms: structured, semi-structured and unstructured data.

Conclusion

The above post is to quickly give an overview on the Big Data and the three dimensions of Big Data.

For more information:

Check out several white papers on the three V’s from EMC, IBM and other market leader’s site.

Was this helpful? Please give me your feedback.

HADOOP Overview

2012-03-30T12:04:00.001-07:00

SUMMARY

This section of the blog will explain the basics of Big Data and Hadoop and the need to move towards the direction.

As well as describing the technologies in detail, it also discusses at high-level about the various benefits like scalability, fault-tolerance, ROI, and how to evaluate between developing in-house vs. utilizing appliances.

This blog is targeted at architects, IT pros, and other leaders in enterprise who are interested to move towards Big Data. It assumes the reader is familiar with database, big data and BI in general.

INTRODUCTION

Today many organization collects data in various forms, email, logs, blogs, rss feeds, even sometime the transactional data , etc… and not use those data to the economical potential . These data are key to the growth of every organization, it can brings new business, it identifies the capability of different products , some might be risky to the organization to continue, some might be yielding high profits but may need more attention, customer satisfaction, etc…

Many organizations profile their data to measure and manage its quality characteristics, but surprisingly few organization made an attempt to evaluate the economic potential of their information assets to amplify the realized value from their information portfolios.

The data we are talking here are in several tera-bytes and are growing exponential. Most of these data are in unstructured format and do we have a technology that can scale the data processing needs? without spending millions of dollar and time?

we need a new way to address these issues, luckily we have solution to these challenges from technology leaders like Google, yahoo, Facebook, Twitter, Linked in etc…

Google in fact was facing the same challenge way before any one else as the data was growing tremendously and the current technology was not helping them to process, Google decided to design a proprietary tool that will solve their needs and they came up with a new processing model called Map-Reduce. Later they shared their findings and solution to the community via White paper. Dough Cutting saw the opportunity and with the help of other open source community (Mainly Yahoo and some Google engineers) come up with a new open source version of Map-Reduce system called Hadoop.

What is Hadoop?

Hadoop is a large scale distributed data processing using commodity hardware, it is the new way to store, processes and analyze high performance large-scale data, the Project is an Open Source administered by Apache Software foundation.

Hadoop provides two key services: 1)Reliable data storage using HDFS and 2) high-performance parallel data processing using Map-Reduce framework.

Lets dive into the technology challenges and the solution and how the solution works in detail….

Challenge with current Large-Scale Computation

Traditionally, computation has been processor-bound and for decades, the primary push was to increase the computing power of a single machine which requires faster processor, more RAM and more $$$

Programming for traditional distributed systems is complex and data exchange requires synchronization

It is difficult to deal with partial failures of the system

Typically, data for a distributed system is stored on a SAN, which is very expensive and at compute time, data is copied to the compute nodes.

How Hadoop solves the current challenge?

Hadoop is designed to run on cheap commodity hardware with automated data replication and node failure capabilities.

With the commodity hardware it brings a huge cost saving to the organization

It scales linear to handle large datasets by adding nodes to the cluster, by increasing the nodes supports load capacity and the failure of any node on the cluster will not stop the process, the other nodes in the cluster takes the responsibility in completing the task. The failure node can join the cluster once it recovers from its failure without restarting the system (Fault-tolerance).

Its is a simple architecture to manage the process of data in parallel as well as to write efficient parallel programming.

Hadoop is built on the major three components along with several other toolset (Eco-System).

Hadoop Common: Set of Hadoop libraries written in Java to support the core file system, serialization, etc…

HDFS: Primary storage system used by Hadoop applications, Data are split into blocks of 64 or 128 mb (Configurable) and replicated into multiple nodes on the cluster to enable reliable, extremely rapid computations.

Map/Reduce: the core framework for distributed processing of large data sets. It uses some common design pattern in processing data (this is designed based on the Google White paper on Map-Reduce).

Eco-System:

Hive	is a Hadoop-based data warehouse developed by Facebook. It allows users to write queries in SQL, which are then converted to MapReduce. This allows SQL programmers with no MapReduce experience to use the warehouse and makes it easier to integrate with business intelligence and visualization tools such as Micro strategy, Tableau, Revolutions Analytics, etc.
Pig	Pig Latin is a Hadoop-based language developed by Yahoo. It is relatively easy to learn and is adept at very deep, very long data pipelines
HBase	is a non-relational database that allows for low-latency, quick lookups in Hadoop. It adds transactional capabilities to Hadoop, allowing users to conduct updates, inserts and deletes. EBay and Facebook use HBase heavily.
Flume	is a framework for populating Hadoop with data. Agents are populated throughout ones IT infrastructure – inside web servers, application servers and mobile devices, for example – to collect data and integrate it into Hadoop.
Oozie	is a workflow processing system that lets users define a series of jobs written in multiple languages – such as Map Reduce, Pig and Hive -- then intelligently link them to one another. Oozie allows users to specify, for example, that a particular query is only to be initiated after specified previous jobs on which it relies for data are completed.
Whirr	is a set of libraries that allows users to easily spin-up Hadoop clusters on top of Amazon EC2, Rackspace or any virtual infrastructure. It supports all major virtualized infrastructure vendors on the market.
Avro	is a data serialization system that allows for encoding the schema of Hadoop files. It is adept at parsing data and performing removed procedure calls.
Mahout	is a data mining library. It takes the most popular data mining algorithms for performing clustering, regression testing and statistical modeling and implements them using the Map Reduce model.
Sqoop	is a connectivity tool for moving data from non-Hadoop data stores – such as relational databases and data warehouses – into Hadoop. It allows users to specify the target location inside of Hadoop and instruct Sqoop to move data from Oracle, Teradata or other relational databases to the target.
BigTop	is an effort to create a more formal process or framework for packaging and interoperability testing of Hadoop' s sub-projects and related components with the goal improving the Hadoop platform as a whole.

Fundamental Concept of Hadoop :

Applications are written in high-level programming model with no dependency on network programming.

Nodes on the cluster talk to each other as little as possible and the communication between the nodes are managed by pre-defined daemons that runs on the NameNode (Master) and the DataNode (Slave). Data is spread among machines in advance and computation happens where the data is stored

How it all works?

Building blocks of Hadoop:

Hadoop is comprises of five main daemons and these daemons have specific roles some exist in on Main node and some exist across multiple nodes

NameNode (Master): the main daemon that does the vital part of the Hadoop framework. It holds the metadata for HDFS, it is the master that directs the data node daemons to perform the I/O tasks. It tracks how the files are broken down to blocks and where these blocks reside on the nodes, etc… also, performs overall health of the DFS. Master Node do not store any data or perform any map/reduce computation. He acts as a supervisor making sure everything is working as expected.

Secondary NameNode: Performs housekeeping functions for the NameNode and it Is not a backup for the

NameNode! It takes the metadata snapshot from the Master NameNode at different intervals. This is useful when the main NameNode fails, the engineers can configure the system to treat the Secondary NameNode to main NameNode by changing few configuration settings. It helps to minimize the downtime and preventing from the loss of data and its doesn’t work as a recovery for main NameNode. NameNode is a single point of failure for Hadoop cluster.

DataNode(s) (Slave): Stores the actual HDFS data blocks. Data node performs the map/reduce computation as it holds the actual data blocks. It also manages the replication of its data block to other nodes for redundancy. The daemon in data node communicates with NameNode Daemon constantly and updates the status of the process.

JobTracker: This daemon runs on the main NameNode and acts as a liaison between the application and Hadoop. Manages MapReduce jobs, distributes individual tasks to machines running in the cluster. If the node or the job fails, it takes the responsibility of assigning the work to other nodes with the same data.

There is one and only Job Tracker Daemon in the entire Hadoop cluster.

TaskTracker: Instantiates and monitors individual Map and Reduce tasks

The data are loaded into the file system and the Hadoop command is used to move the data to Hadoop DFS. After the data is loaded into the HDFS, it is split into many ‘blocks’ of 64MB (Default size and configurable)

Map tasks (the first part of the Map Reduce system) work on relatively small portions of data

A master program allocates work to nodes such that a Map task will work on a block of data stored locally on that node whenever possible. Many nodes work in parallel, each on their own part of the overall Dataset

Who Provides the service/training/support/etc…?

There are several distribution of Hadoop as well as appliance that has Hadoop built in are available in the market and now the choice comes to us (the enterprise) to decide which direction to go and as an architect myself list down few pros and cons.

Vendor distribution vs. In-House build:

Hadoop implementations consist of more independent "moving parts" and version numbers than most. Enterprises can construct a Hadoop solution themselves if they have the time, resources and expertise to do so, as well as a commitment to ongoing internal support, which involves evaluating new releases of each project before adding it to the infrastructure. Preferably, they will choose a commercial Apache Hadoop distribution whose component projects are pre-integrated and may be backed by the vendor's support.

Vendors:

Cloudera:

The creator of Hadoop Mr. Doug Cuttings is now part of Cloudera

Was first to market as a commercialized and evangelist.

Includes Apache Hadoop (CDH) and most other sub-projects

Provides a broad range of features for storage, computation, integration and end-user access with its own deployment and management tools.

It boasts a rich and expanding global partner network and numerous enterprise clients.

Oracle's Big Data Appliance

NetApp's Open Solution for Hadoop,

Dell/ Cloudera Hadoop Solution and

SGI's Optimized Hadoop Solution all use Cloudera' s distribution

Distributes and sells CDH in Asia/Pacific.

Hortonworks:

Led the design and development effort for the next-generation MapReduce and HDFS architectures for Hadoop 0.23 (presumably soon to be Apache Hadoop 2.0).

It has partnered with Microsoft to bring Apache Hadoop to Windows and Azure.

IBM's InfoSphere BigInsights:

It adds text analytics, Big Sheets for data exploration

Development tools

IBM's "extended" Adaptive MapReduce and a flexible scheduler

General Parallel File System (GPFS), as an alternative to HDFS

Enterprise-grade security,

A Web-based management console and high-speed data connectors

Includes Lucene.

EMC Greenplum:

Greenplum MR software based on MapR's M5 distribution

Greenplum Data Computing Appliance, which can include Greenplum Database and/or Greenplum Data

Integration Accelerator modules along with Greenplum HD in a single rack.

EMC Isilon storage would add native support for HDFS

Isilon storage will ease enterprise installations with high availability and automated data import/export.

Licensing models:

Cloudera, Hortonworks, MapR and DataStax use the Apache license for Apache project components and add commercial licenses for their own added offerings.

IBM and Oracle makes its distribution available under commercial license

Various Appliance from leading database companies:

Oracle Selects Cloudera to Provide Apache Hadoop Distribution and Tools for Oracle Big Data Appliance …more

Teradata Corporation (NYSE: TDC), the world’s largest company solely focused on data warehousing and enterprise analytics, today announced a partnership with Cloudera, the leading provider of Hadoop-based data management software and services …more

Microsoft and Horton works have formed a strategic relationship to accelerate and extend the delivery of Apache Hadoop-based distributions for Windows Server and Windows Azure.

EMC launched a Greenplum data warehousing appliance designed to take on the likes of IBM’s Netezza, Oracle’s Exadata and Teradata ..more

Conclusion

The above post is to quickly give an overview on Hadoop, the eco-system and the technology in general. Hope this helps.

My next blog will have a detail overview of the internals as well as the basics of Map-Reduce and its architecture with some sample.

For more information:

Apache Hadoop	http://hadoop.apache.org/
Cloudera	http://www.cloudera.com/
Greenplum	http://www.greenplum.com/
Hadoop Wiki Introduction	http://hadoop.apache.org/core/
Getting Started	http://wiki.apache.org/hadoop/GettingStartedWithHadoop
Map/Reduce Overview	http://wiki.apache.org/hadoop/HadoopMapReduce
DFS	http://hadoop.apache.org/core/docs/current/hdfs_design.html
Java doc	http://hadoop.apache.org/core/docs/current/api/index.html

Training: I took Hadoop programming class from Cloudera and I highly recommend the training.

Was this helpful? Please give me your feedback.

Who can use the cloud

2012-01-26T13:59:00.001-08:00

In my opinion from a small start-up company to a big enterprise can use Cloud. There are few areas of concerns especially when it comes to security and key confidential data residing outside the network, in these situation we can still use Cloud, probably in a hybrid mode or private cloud or even buying cloud appliances (some Cloud service providers offer appliances that can be installed on locally)

For startup companies, this is the best solution as they don’t need to invest heavily on infrastructure and the resources that supports them. They can spend valuable amount on their product design, development tools and resources.

For companies that are more centric to data processing , analytical, batch processing, heavy algorithmic/scientific calculation processing etc… Cloud solution will be great candidate

For enterprise still public cloud is ideal solution but sometime with the compliance, legal and other regulatory issues, Hybrid model will work fine.

For organization who do business across the globe need to be careful when dealing with the cloud solution, especially those who work with countries that have strict data protection and governing policies. There are few Cloud solution providers that supports the above said polices by having infrastructure specific to those place. In these situation Hybrid model might help.

Cloud = Big Savings $$$

2012-01-26T13:15:00.001-08:00

Overview:

Cloud based solution brings a great benefit to building application especially for organization that spends millions of dollar in deploying, hardware, maintenance , monitoring the application, etc... as well as the other overhead costs like electricity, space, other utilities to manage the hardware, etc…

I listed down few key list of savings from Cloud solution:

No-Infrastructure Cost:

This one topic is enough for moving towards cloud as this has quadruple effort on cost saving

If we are developing application, we don’t need to think about procuring an infrastructure, that includes servers, Racks/Server closets, Air-conditioner to keep them cool, electricity to run these appliances, etc… All we need is a simple desktop/Laptop to develop application and when its ready for testing, deploy them on the cloud and pay for the service that you use. This is a huge cost saving.

Caution: If you are an infrastructure person, do not read the paragraph below:

One big savings from this approach is, we do not need to hire resources to manage these day-to-day tasks on infrastructure as well as other resource whose primary job is to procure the hardware. If we hire a resource to manage our infrastructure, we still need to pay for the resource at least 8 hr. a day wages + other benefits + HR overheads cost (Pay stub, tax calculation, Leave, bonus etc…), even if the resource is not fully occupied.

Just-in-time Infrastructure with no upfront cost:

In my experience working with several organization, procuring an infrastructure and building them to our need requires hell out of time. Just listed down few thing we all do with new infrastructure

Hardware capacity planning (Memory, Disk space, clustering, etc…)
Adding to the inventory system/Procurement/financial systems/etc…
Installing required software's (OS, Database, etc…)
Setting up monitoring
Setting up backup/recovery plan
Network planning
Security settings
Etc...

and when our needs are increasing (volume/traffic/users/etc…) we need to repeat the process again and need to wait till we get all these procured.

With cloud, all this can be achieved in no time, I mean really in no time. We can achieve the above said tasks in less than an hour which normally takes months and sometime 6 months (if you work for big organization).

Just-In time scale up and scale down:

When we want to scale up and down, Cloud solution helps to meet the same in no time. All these can be done using few clicks on the providers site. Consider we are a tax company, we may need 24/7 running application with high volume users flowing in to our system during tax season and during that time we can scale up and during the off-season we can scale down, this saves tons of money.

Pay for what you consume:

Cloud payment works just like our electricity bill and telephone bill, pay for what you use. If our application/process do not need 24/7, then why to pay for 24/7. you can turn it off when you do not want to consume., you pay for the usage and there are several options available within this service providers.

There are much more benefits to list….

Summary

Cloud Computing is a new revolution to the Information Technology. It makes great impact on Cost savings and time to market (Speed).

In the next post, we'll look at who should use Cloud Computing.

Hope this helps,

Your comments are important to me. Please use the comment section form below to provide your valuable feedback.

Cloud Computing Overview

2012-01-25T11:13:00.001-08:00

Overview:

Cloud computing is simply a set of services that are provided by some third party companies or within an enterprise as a centralized service. There are lots of advantages and I listed some of them below:

1. Separation of Hardware from the development team - Developers job is to develop the application based on the business requirement and not worrying about the hardware availability, space, licenses, software’s/tools, etc... Provider of this service will be responsible for all these services. In short no licensing or server expenses, development team simply register their services with the provider.

2. Maintenance: The provider of these services handles the traffic, security, network, etc... and it reduces the maintenance cost as well as the resource cost.

3. The cost incurred for these services are truly based on the usage, rather than paying for the complete hardware, network, etc...

The services that are provided by the industry are classified as three different services

1. Software as a service (SaaS)

othis service provides you an application package or a piece of software that allows you to do some tasks that you normally do at work

oe.g.: MS-Office 360, Google Docs, Drop box, Sales Force, Skype, Workday, Custom application, etc...

2. Platform as a service (PaaS)

othis service provides a platform with the tools that allows to develop and deploy application on the cloud

oe.g.: MS-Visual Studio, Ruby on Rails, Java, LAMP, Python, etc...

3. Infrastructure as a service (IaaS)

othis service provides you a physical/virtual hardware like servers, storage and networking related hardware that you can leverage for infrastructure needs

Cloud service providers (I listed few that I'm aware of):

Microsoft introduced Cloud computing called 'AZURE' in two platforms, one as a Development platform and other one as a Delivery platform. They provide all the above three services (SaaS, PaaS, IaaS). We can completely outsource the application service on Microsoft Azure platform or we can buy the complete package that includes the hardware, Azure service, SQL Azure Service, etc... as a bundle that can be housed in the enterprise.

Amazon introduce Cloud Computing called 'Amazon EC2/AWS' and they provide all the above three services (SaaS, PaaS, IaaS) and support wide range of development platform. It supports wide range of database from SQL Server, Oracle, MySQL to some open source database. Amazon also supports the Hadoop technology on their platform.

Other Cloud providers like VMWare, Rackspace, Red Hat, they generally provide IaaS and PaaS service.

The Future of Cloud Computing: though cloud computing seems like a brand new concept, this has been in the market for quite few years and some of these providers already worked on the private cloud model for some time. Some of the companies already started consuming these services, especially the new ventures are all targeting towards the cloud technology.

Risks: though the cloud services providers offers a great level of security and SLA, still there are some risk that needs to be considered before choosing the service/s. Sometime the external factor also plays a role not to choose the cloud option like Compliance, legal and other regulatory needs, also when the organization heavily rely on the vendor applications as well as application that are heavily dependent on frequent configuration changes.

Characteristics of Cloud: there are several characteristics cloud computing provides but I would like to pick the key important characteristics that I believe cloud computing offers

Self-Servicing

Consumers can provision or de-provision on the fly, you can increase the storage capacity, bandwidth capacity, load balancing, etc...

Elasticity and Scalability

As the name implies, cloud is all about elasticity, you can stretch and shrink based on your need at given point of time.
You can scale your application by increasing or decreasing the capacity

Standardization

It provides a standard interface, once you are familiar with couple of application on the cloud network; it follows the same mechanism for the rest (in most cases).
You can define the standardized security mechanism for your business.

Pay as per usage

Pay for what you use. Cloud service is more like our electricity meter, pay for the service you consumed and you can turn on or off when needed.

Resource management:

A cloud service provides a great deal of cost saving especially resource management, since most of critical tasks are managed by the cloud providers.

Deployment Models: Cloud services can be deployed and consumed in different ways

Public Cloud - in this model the entire service (IaaS, SaaS, PaaS) are hosted externally

Private Cloud- in this model the entire services are setup internally or sometime by utilizing the appliances provided by the Cloud Service providers.

Hybrid Cloud - in this model the services are in mix mode, some part of the cloud sits in public loud and some are internally hosted e.g.: Database can be managed internally and the application that consumes the database sits on the cloud.

Summary

Cloud Computing is a new revolution to the Information Technology. It makes great impact on Cost savings and time to market (Speed).

In the next post, we'll look at what are differnt ways we can save money from Cloud Computing.

Hope this helps,

Your comments are important to me. Please use the comment section form below to provide your valuable feedback.