The Next Computing Era

Introduction to Hadoop

noreply@blogger.com (Praveen Prakasan) — Sat, 07 Jul 2012 17:26:00 +0000

Apache Hadoop is a Java software framework that allows for the distributed processing of large data sets across clusters of computers spread across the world using a simple programming model.

Multiple autonomous systems appear as one, interacting via a message passing interface, no single point of failure. Hadoop has components which take care of all complexities for us and by using a simple map reduce framework we are able to harness the power of distributed computing without having to worry about complexities like fault tolerance, data loss. It has replication mechanism for data recovery and job scheduling and blacklisting of faulty nodes by a configurable blacklisting policy.

Following are major components.
1. Map-reduce (Job Tracker and task tracker)
2. Namenode and Secondary namenode (A HDFS NameNode stores Edit logs and File system Image).
3. Datanode (Runs on slaves)
4. JobTracker (Runs on server)
5. TaskTracker (Runs on slaves)

Hadoop can be configured in the following three modes:
1. Local Standalone (Non-distributed)
    All hadoop deamons run as a single java process on a single system
    Useful for debugging

2. Pseudo Distributed
    Daemons run a single-node
    Each Hadoop Daemons runs in a separate java process

3. Fully Distributed
    Master-slave Architecture
    One machine is designed as the NameNode and other as JobTracker (Can be with same

machine as well)
Reset of the machines in the cluster act as both DataNode and TaskTracker

MapReduce:
The map reduce framework is introduced by google. A simple and powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large clusters of commodity PCs. (definition by Google paper on mapred)

This broadly consists of two mandatory functions to implement: “Map and reduce”. A map is a function which is executed on each key-value pair from an input split, does some processing and emits again a key and value pair. After map and before reduce can begin there is a phase called shuffle which copies and sorts the output on key and aggregates values.

These key and “aggregated value” pairs are captured by reduce and outputs a reduced key value pair . This process is also called aggregation as you get values aggregated for a particular key as input to reduce method. Again in reduce you may play around as you want with key-values and what you emit know is also key value pairs which are dumped directly to a file.

Now simply by expressing a problem in terms of map-reduce we can execute a task in parallel and distribute it across a broad cluster and be relieved of taking care of all complexities of distributed computing. Indeed “Life made easy”, had you tried doing the same thing with MPI libraries you can understand the complexity there scaling to thousands or even hundreds of nodes.

There is a lot more going in map-reduce than just map-reduce. But the beauty of the hadoop is that it takes care of most of those things and a user may not dig into details for simply running a job, though it is good if one has knowledge of those features and can help in tuning parameters and thus improve efficiency and fault tolerance.

HDFS:
Hadoop has its own implementation of distributed file system called hadoop distributed filesystem, which is coherent and provides all facilities of a file system. It implements ACLs and provides a subset of usual UNIX commands for accessing or querying the filesystem and if one mounts it as a fuse dfs then it is possible to access it as any other linux filesystem with standard unix commands.

Namenode:
A single point of failure for an HDFS installation. It contains information regarding a block’s location as well as the information of entire directory structure and files. By saying It is a single point of failure - I mean, if namenode goes down - whole filesystem is offline. Hadoop also has a secondary namenode which contains edit log, which in case of a failure of namenode can be used to replay all the actions of the filesystem and thus restore the state of the filesystem. A secondary namenode regularly contacts namenode and takes checkpointed snapshot images. At any time of failure these checkpointed images can be used to restore the namenode. Current efforts are going on to have high availability for Namenode.

Datanode:
Datanode stores actual blocks of data and stores and retrieves blocks when asked. They periodically report back to Namenode with list of blocks they are storing.

Jobtracker and tasktracker:
There is one JobTracker(is also a single point of failure) running on a master node and several tasktracker running on slave nodes. Each tasktracker has multiple task-instances running and every task tracker reports to jobtracker in the form of heart beat at regular intervals which also carries message of the progress of the current job it is executing and idle if it has finished executing. Jobtracker schedules jobs and takes care of failed ones by re-executing them on some other nodes. In Mrv2 efforts are made to have high availability for Jobtracker, which would definitely change the way it has been.

Hadoop and Vertica

noreply@blogger.com (Praveen Prakasan) — Fri, 22 Jun 2012 14:17:00 +0000

Hadoop Summit 2012 - Hadoop and Vertica: The Data Analytics Platform at Twitter

View more presentations from Bill Graham

Top ten tips tricks for hadoop success r9

noreply@blogger.com (Praveen Prakasan) — Wed, 25 Apr 2012 15:42:00 +0000

Top ten tips tricks for hadoop success r9

View more presentations from Cloudera, Inc.

cloud computing

noreply@blogger.com (Praveen Prakasan) — Tue, 06 Mar 2012 12:35:00 +0000

cloud computing

More PowerPoint presentations from neha dubey

Questions & Answers

noreply@blogger.com (Praveen Prakasan) — Sat, 03 Mar 2012 17:51:00 +0000

Salesforce

Proposed Q&A site for serious Salesforce administrators, implementation experts, developers and anybody in-between! Covering declarative (clicks not code) as well as code-based development as well as system administration.

Certification Questions and Answers 1

noreply@blogger.com (Praveen Prakasan) — Sat, 03 Mar 2012 17:16:00 +0000

Certification Q&A 1

What, When and Where Salesforce,com Now!

noreply@blogger.com (Praveen Prakasan) — Tue, 28 Feb 2012 19:32:00 +0000

What, When and Where Salesforce,com Now!: The New Evolution In Cloud Computing

Marc Benioff Is Thinking Bigger

noreply@blogger.com (Praveen Prakasan) — Mon, 27 Feb 2012 13:55:00 +0000

The New Evolution In Cloud Computing

How to Install Force.com IDE

noreply@blogger.com (Praveen Prakasan) — Sun, 26 Feb 2012 18:01:00 +0000

How to Install Force.com IDE: How to Install Force.com IDE Software Requirements Eclipse 3.3: http://www.eclipse.org/downloads/packages/eclipse- i...

Salesforce Interview Q&A PART 2

noreply@blogger.com (Praveen Prakasan) — Sun, 26 Feb 2012 02:11:00 +0000

Salesforce Interview Q&A PART 2: Part 2 21.What are Validation rules ? Ans:- Validation rules verify that the data a user enters in the application meets ...

Salesforce Interview Q&A PART 1

noreply@blogger.com (Praveen Prakasan) — Fri, 24 Feb 2012 13:57:00 +0000

Salesforce Interview Q&A PART 1

Hadoop Interview Questions

noreply@blogger.com (Praveen Prakasan) — Fri, 24 Feb 2012 06:56:00 +0000

Hadoop Interview Questions

Salesforce Interview Q&A PART 1

noreply@blogger.com (Praveen Prakasan) — Fri, 24 Feb 2012 06:02:00 +0000

Salesforce Interview Q&A PART 1

Salesforce.com Agile Transformation - Agile 2007 Conference

noreply@blogger.com (Praveen Prakasan) — Tue, 21 Feb 2012 10:54:00 +0000

Salesforce.com Agile Transformation - Agile 2007 Conference

View more PowerPoint from Steve Greene

Scrum Gathering 2008 Stockholm - Salesforce.com

noreply@blogger.com (Praveen Prakasan) — Tue, 21 Feb 2012 06:30:00 +0000

Scrum Gathering 2008 Stockholm - Salesforce.com

View more PowerPoint from Steve Greene