MY TECH LEARNERS

Eclipse with Hadoop on Ubuntu

2013-11-08T16:57:00.003+05:30

hi friends this tutorial is for you

To configure Eclipse IDE in Hadoop 0.20.2 with Ubuntu 12.04 LTS

After successfully configured hadoop setup on ubuntu follow below steps for to setup eclipse

(see step by step documentation in below link)

http://mytechlearners.com

Step 1: Download Eclipse Europa from below link

for Ubuntu 64 bit

for Ubuntu 32-bit

Step 2: copy eclipse file to usr/local from downloads

sudo cp $HOME/Downloads/<eclispe file nane> /usr/local/

Step 3: extract eclipse file in /usr/local directory

first change direcoty to /usr/local

$ cd /usr/local (enter)

$ sudo tar -xzvf <eclipse-file-name>

$ ls (you will get ecllispe named directory)

and now open usr/local/eclipse directory from file system (not from Terminal) then open eclipse (diamond symbol file)

Step 4: copy eclipse jar from hadoop to eclipse

$ sudo cp /usr/local/hadoop/contrib/eclipse-plugin/<jar-file-name> /usr/local/eclipse/plugins

now restart (close and open) eclipse .. then goto windows-->open perspective--> other --> select Map/Reduce and click on ok.

Step 5: Right click in the Map/Reduce Locations view and select 'New Hadoop Location'

In the window that appears, type in 'HDFS Cluster' for the Location name. Under Map/Reduce Master, type in localhost for the Host, and 9101 for the Port. For DFS Master, make sure the 'Use M/R Master host' checkbox is selected, and type in 9100 for the Port. For the User name, type in 'User'. Click 'Finish'

Step 6: In the Project Explorer window on the left, you should now be able to expand the DFS Locations tree and see your new location. Continue to expand it and you should see something like the following file structure:

DFS Locations

-> HDFS Cluster(1)

-> (2)

-> tmp(1)

-> huser(0)

At this point you can create directories and files and upload them to the HDFS from Eclipse, or you can create them through the cygwin window as you did in step 7 in the previous section.

Hadoop Important URL's Information

2013-10-09T21:06:00.003+05:30

Hadoop Important URL's Information

#   Running Hadoop On Ubuntu Linux (Single-Node Cluster)

#   Running Hadoop On Ubuntu Linux (Multi-Node Cluster)

#   Big Data University Hadoop Information

#   Hadoop Tutorial

#   Hadoop in Practice

#   Hadoop Presentation

#   Hadoop Talks

#   Big Data and NoSQL database

#   Hadoop fundamentals

#   Introduction to Hadoop

#   Apache Hadoop Topics for Certification

#   Hadoop for Transcriptomics

#   What is Hadoop

#   Power Exchange for Hadoop

#   What is Apache Hadoop

#   Overview Of Hadoop

#   Big Data analytics ,Hadoop

#   An Overview Of Hadoop (How it works)

#   Hadoop architecture

#   Understanding Hadoop Clusters and the Network

#   How To Build Optimal Hadoop Cluster

#   Big Ideas: Demystifying Hadoop

#   Hadoop Cluster

#   Understanding storage in the Hadoop cluster

#   First Hadoop Platform Solution for Parsing Big Data

#   Map Reduce Clusters

#   Next Hadoop Configuration

#   Cloudera Releases Next-Generation Hadoop Platform

#   Enabling Big Data Analytics Inside of Hadoop

#   What is it, Hadoop

#   Apache Hadoop

#   Hadoop Tutorial

#   Hadoop Apps

#   Setup Hadoop on Ubuntu

#   Practical Problem Solving with Apache Hadoop and Pig

# Download Good eBooks for free

# Big Data Circus

Hadoop Information

2013-10-09T21:04:00.001+05:30

Who Can Learn?

It is appropriate for developers, system administrators, managers, architects, or anyone who wants to jump start their adventure with Hadoop. Prior Hadoop knowledge is not required.

Introduction

Apache Hadoop is an open-source software framework that supports data-intensive distributed applications, licensed under the Apache v2 license. It enables applications to work with thousands of computational independent computers and petabytes of data. Hadoop was derived from Google's MapReduce and Google File System (GFS) papers.

The entire Apache Hadoop “platform” is now commonly considered to consist of the Hadoop kernel, MapReduce and HDFS, as well as a number of related projects – including Apache Hive, Apache Hbase, and others.

Hadoop is a top-level Apache project being built and used by a global community of contributors, written in the Java programming language. The Apache Hadoop project and its related projects (Hive, HBase, Zookeeper, and so on) have many contributors from across the ecosystem.

History

Hadoop was created by Doug Cutting and Michael J. Cafarella. Doug, who was working since 2009 as chief architect at Cloudera,named it after his son's toy elephant. It was originally developed to support distribution for the Nutch search engine project.

Architecture

Hadoop consists of the Hadoop Common which provides access to the filesystems supported by Hadoop. The Hadoop Common package contains the necessary JAR files and scripts needed to start Hadoop. The package also provides source code, documentation, and a contribution section which includes projects from the Hadoop Community.

For effective scheduling of work, every Hadoop-compatible filesystem should provide location awareness: the name of the rack (more precisely, of the network switch) where a worker node is. Hadoop applications can use this information to run work on the node where the data is, and, failing that, on the same rack/switch, reducing backbone traffic. The Hadoop Distributed File System (HDFS) uses this when replicating data, to try to keep different copies of the data on different racks. The goal is to reduce the impact of a rack power outage or switch failure so that even if these events occur, the data may still be readable.

A multi-node Hadoop cluster

A small Hadoop cluster will include a single master and multiple worker nodes. The master node consists of a JobTracker, TaskTracker, NameNode, and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only worker nodes, and compute-only worker nodes; these are normally only used in non-standard applications. Hadoop requires JRE 1.6 or higher. The standard startup and shutdown scripts require ssh to be set up between nodes in the cluster.

In a larger cluster, the HDFS is managed through a dedicated NameNode server to host the filesystem index, and a secondary NameNode that can generate snapshots of the namenode's memory structures, thus preventing filesystem corruption and reducing loss of data. Similarly, a standalone JobTracker server can manage job scheduling. In clusters where the Hadoop MapReduce engine is deployed against an alternate filesystem, the NameNode, secondary NameNode and DataNode architecture of HDFS is replaced by the filesystem-specific equivalent.

Hadoop Singlenode Cluster on CentOS

2013-08-26T19:38:00.003+05:30

Configuring Hadoop Singlenode Cluster on CentOS 6.3 64bit

Step:1

Download ,Install and Configure Java 1.6

Step:2

Download and Configure Hadoop 0.20.2

Step:3

Download and Configure Eclipse

Step:1

I. In CentOS , it's very easy to install JDK 1.6 follow

below steps

i. Goto (at top panel)System-->Administration--> Add/Remove Software-->(sometimes it ask authentication --click anyway)--> on Search Box Type -- JDK then click on find

ii. Search for OpenJDK Developement Environment (1.6 vesrion)and select it then click apply

iii. Once Installation Done, then open terminal and type java -version(if you got version no.. then you installed java successfully)

II. Setting Environment Variables in Terminal

export JAVA_HOME= /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64

and

and in terminal type ---> $ vi ~/.bash_profile (enter)

you will see one visual editor--> press i and type

export JAVA_HOME= /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64

Step 2:

I. Create a system user account to use for hadoop installation.

# useradd huser

# passwd huser

Changing password for user hadoop.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

II. Configuring Key Based Login

Its required to setup haddop user to ssh ifself without password. Using following method it will enable key based login for hadoop user.

# su - huser

$ ssh-keygen -t rsa

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ chmod 0600 ~/.ssh/authorized_keys

$ exit

III. Download the stable version 0.20.2 of Hadoop

i. go to below link and download

http://www.mediafire.com/download/dtv7k2bhvkfoq64/hadoop-0.20.2.tar.gz

ii. Extract Hadoop tar file

# mkdir /opt/huser

# cp /home/huser/Downloads/hadoop-0.20.2.tar.gz hadoop

# cd /opt/huser/

# tar -xzf hadoop-0.20.2.tar.gz

# mv hadoop-0.20.2 hadoop

# chown -R huser /opt/huser

# cd /opt/huser/hadoop /

iii. Configure Hadoop

Make the following additions to the corresponding files: in Hadoop/conf

* core-site.xml (inside the configuration tags)

<name>fs.default.name</name>

<value>hdfs://localhost:8020</value>

</property>

* mapred-site.xml (inside the configuration tags)

<name>mapred.job.tracker</name>

<value>localhost:8021</value>

</property>

* hdfs-site.xml (inside the configuration tags)

<name>dfs.replication</name>

</property>

* hadoop-env.sh

uncomment the JAVA_HOME export command, and set the path to your Java home

export JAVA_HOME= /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64

Set JAVA_HOME path as per your system configuration for java.

iv. Format Name Node

# su - huser

$ cd /opt/huser/hadoop

$ bin/hadoop namenode -format (you will get )

29/06/13 07:24:20 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = srv1.tecadmin.net/192.168.1.90 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1479473; compiled by 'hortonfo' on Mon May 6 06:59:37 UTC 2013 STARTUP_MSG: java = 1.7.0_17 ************************************************************/ 29/06/13 07:24:20 INFO util.GSet: Computing capacity for map BlocksMap 29/06/13 07:24:20 INFO util.GSet: VM type = 32-bit 29/06/13 07:24:20 INFO util.GSet: 2.0% max memory = 1013645312 13/06/02 22:53:48 INFO util.GSet: capacity = 2^22 = 4194304 entries 29/06/13 07:24:20 INFO util.GSet: recommended=4194304, actual=4194304 29/06/13 07:24:20 INFO namenode.FSNamesystem: fsOwner=hadoop 13/06/02 22:53:49 INFO namenode.FSNamesystem: supergroup=supergroup 29/06/13 07:24:20 INFO namenode.FSNamesystem: isPermissionEnabled=true 29/06/13 07:24:20 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 29/06/13 07:24:20 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 29/06/13 07:24:20 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 29/06/13 07:24:20 INFO namenode.NameNode: Caching file names occuring more than 10 times 29/06/13 07:24:20 INFO common.Storage: Image file of size 112 saved in 0 seconds. 29/06/13 07:24:20 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits 29/06/13 07:24:20 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/opt/hadoop/hadoop/dfs/name/current/edits 29/06/13 07:24:20 INFO common.Storage: Storage directory /opt/hadoop/hadoop/dfs/name has been successfully formatted. 29/06/13 07:24:20 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at srv1.tecadmin.net/192.168.1.90 ************************************************************

v. Start Hadoop Services

$ bin/start-all.sh

vi. Test and Access Hadoop Services

$ jps <you will get like this>

26049 SecondaryNameNode

25929 DataNode

26399 Jps

26129 Jobtracker

26249 Tasktracker

25807 Namenode

Web Access URLs for Services

http://localhost:50030/ for the Jobtracker

http://localhost:50070/ for the Namenode

http://localhost:50060/ for the Tasktracker

Tip: set hadoop home path environment variable

export PATH=$PATH:/opt/huser/hadoop/bin

that's it, now you can access Hadoop from anywhere from terminal

Congratulations , your successfully configured Hadoop on CentOS

Hadoop Cluster on UBUNTU 12.04 LTS

2013-08-26T19:37:00.002+05:30

Configuring Hadoop Singlenode Cluster on Ubuntu 12.04 64bit

Step:1

Download ,Install and Configure Java 1.6

Step:2

Download and Configure Hadoop 0.20.2

Step:1

I. In Ubuntu , it's very easy to install JDK 1.6 follow

below steps

i. Go to Terminal and type

$ sudo apt-get install openjdk-6-jdk (enter)

ii. Setting Environment Variables in Terminal

$ export JAVA_HOME= /usr/lib/jvm/java-6-openjdk-amd64

(usual path. check once in your system)

and

in terminal type --->

$ sudo vim ~/.bashrc (enter)

you will see one visual editor--> press i and type

export JAVA_HOME= /usr/lib/jvm/java-6-openjdk-amd64

Step:2

I. Create a system user account to use for hadoop installation.

type following in terminal

$ sudo addgroup hgroup (enter)

$ sudo adduser --ingroup hgroup huser (enter)

Adding user `huser' ...

Adding new user `huser' (1002) with group `hgroup' ...

Creating home directory `/home/huser' ...

Copying files from `/etc/skel' ...

Enter new UNIX password:

Retype new UNIX password:

passwd: password updated successfully

Changing the user information for huser

Enter the new value, or press ENTER for the default

Full Name []:

Room Number []:

Work Phone []:

Home Phone []:

Other []:

Is the information correct? [Y/n] Y (enter)

II. Configuring SSH Key Based Login

Its required to setup haddop user to ssh itself without password. Using following method it will enable key based login for hadoop user.

$ su - huser

"huser@localhost:~$" you will get like this

$ ssh-keygen -t rsa -P ""

$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

$ ssh localhost

(if success you will get welcome message)

(if fail you will get

ssh connect to host localhost port 22 connection refused)

then run following command on super user... only if fails

$ sudo apt-get install openssh-server (if fails then run)

then run again

$ ssh localhost

III. Disabling IPV6

i. you need to run this command as super user

$ sudo vim /etc/sysctl.conf

(press i.. copy below lines end of the file)

# disable ipv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

ii. test ipv6 disabled or not.. run below command

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

You have to reboot your machine in order to make the changes take effect

A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want)

IV. Download the stable version 0.20.2 of Hadoop

i. go to below link and download

http://www.mediafire.com/download/dtv7k2bhvkfoq64/hadoop-0.20.2.tar.gz

ii. Extract Hadoop tar file (as super user)

$ sudo cp /home/huser/Downloads/hadoop-0.20.2.tar.gz /usr/local

$ cd /usr/local/

$ sudo tar -xzf hadoop-0.20.2.tar.gz

$ sudo mv hadoop-0.20.2 hadoop

$ sudo chown -R huser:hgroup hadoop

$ su - huser

s cd /usr/local/hadoop/ (now we are in hadoop home directory)

iii. Configure Hadoop

Make the following additions to the corresponding files: in hadoop/conf

run from huser--->

$ vim conf/core-site.xml

* core-site.xml (inside the configuration tags)

<name>fs.default.name</name>

<value>hdfs://localhost:8020</value>

</property>

$ vim conf/mapred-site.xml

* mapred-site.xml (inside the configuration tags)

<name>mapred.job.tracker</name>

<value>localhost:8021</value>

</property>

$ vim conf/hdfs-site.xml

* hdfs-site.xml (inside the configuration tags)

<name>dfs.replication</name>

</property>

$ vim conf/hadoop-env.sh

* hadoop-env.sh

uncomment the JAVA_HOME export command, and set the path to your Java home

export JAVA_HOME=/usr/lib/jvm/java java-6-openjdk-amd64

Set JAVA_HOME path as per your system configuration for java.

iv. Format Name Node

(from huser only)

$ cd /usr/local/hadoop

$ bin/hadoop namenode -format (you will get )

v. Start Hadoop Services

$ bin/start-all.sh

the output will be look like

huser@ubuntu:/usr/local/hadoop$ bin/start-all.sh

starting namenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hduser-namenode-ubuntu.out

localhost: starting datanode, logging to /usr/local/hadoop/bin/../logs/hadoop-hduser-datanode-ubuntu.out

localhost: starting secondarynamenode, logging to /usr/local/hadoop/bin/../logs/hadoop-hduser-secondarynamenode-ubuntu.out

starting jobtracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hduser-jobtracker-ubuntu.out

localhost: starting tasktracker, logging to /usr/local/hadoop/bin/../logs/hadoop-hduser-tasktracker-ubuntu.out

huser@ubuntu:/usr/local/hadoop$

vi. Test and Access Hadoop Services

$ jps <you will get like this>

26049 SecondaryNameNode

25929 DataNode

26399 Jps

26129 Jobtracker

26249 Tasktracker

25807 Namenode

Web Access URLs for Services

http://localhost:50030/ for the Jobtracker

http://localhost:50070/ for the Namenode

http://localhost:50060/ for the Tasktracker

Creating HOME Paths for Hadoop and JAVA in bashrc file

run from huser

$ vim $HOME/.bashrc

and press i, add following line at end of the file

# Set Hadoop-related environment variables

export HADOOP_HOME=/usr/local/hadoop

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)

export JAVA_HOME=/usr/lib/jvm/java java-6-openjdk-amd64

# Some convenient aliases and functions for running Hadoop-related commands

unalias fs &> /dev/null

alias fs="hadoop fs"

unalias hls &> /dev/null

alias hls="fs -ls"

# If you have LZO compression enabled in your Hadoop cluster and

# compress job outputs with LZOP (not covered in this tutorial):

# Conveniently inspect an LZOP compressed file from the command

# line; run via:

# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo

# Requires installed 'lzop' command.

lzohead () {

hadoop fs -cat $1 | lzop -dc | head -1000 | less

}

that's it, now you can access Hadoop from anywhere from terminal

Congratulations , your successfully configured Hadoop on Ubuntu

Hadoop Pseudo Mode in Windows OS

2013-08-26T19:25:00.002+05:30

SETTING UP HADOOP DAEMONS ON WINDOWS OS

Step:1

Install and Configure Cygwin 2.x

Step:2

Install and Configure Java 1.6

Step:3

Install and Configure Hadoop 0.20.2

Step:4

Install and Configure Eclipse Europa

Step:1

1. Download 'setup.exe' from Cygwin website

2. Right-click on 'setup.exe' And Click on Run as Administrator

3. Leave settings as they are, click through until you come to the plugin selection window

3.1 - Make sure that the installation directory is 'C:\cygwin'

4. In the plugin selection window, select below packages and install

i) openssh

ii) openssl

iii) diffutils

iv) tcp_wrappers

5. Click 'Next', then go do something productive while installation runs its course.

6. Once installed, go to Start -> All Programs -> Cygwin, right-click on the subsequent shortcut and select the option to 'Run as Administrator'

This is needed in order to allow the sshd account to operate without a passphrase, which is required for Hadoop to work.

8. Once the prompt window opens up, type 'ssh-host-config' and hit Enter

9. Should privilege separation be used? NO

10. Do you want to install sshd as a service? YES

11. Enter the value of CYGWIN for the daemon: <LEAVE BLANK, JUST HIT ENTER>

12. Do you want to use a different name? (default is 'cyg_server'): NO

13. Please enter the password for user 'cyg_server': <LEAVE BLANK, JUST HIT ENTER>

14. Reenter: <LEAVE BLANK, JUST HIT ENTER>

At this point the ssh service should be installed, to run under the 'cyg_server' account. Don't worry, this will all be handled under the hood.

To start the ssh service, in services --> CYGWINsshd (right-click)--> start and ok. When you log in next time, this will automatically run.

To test, type in 'ssh localhost' in your cygwin window. You should not be prompted for anything.

15. Set Cygwin Path in MyComputer-->Properties-->Advanced System Settings-->Environment variables-->create NEW System variables like below

Step2:

1. Download Java 1.6 from Oracle Website

If Can't find Check this link

http://www.oracle.com/technetwork/java/javase/downloads/index.html

2. Right-click on Java(downloaded file) and install like Run as Administrator

Note: When installing Change your path for both JDK and JRE files like c:/Java/<jdk or jre>

3. Leave settings as they are, click through until you come to the Finish prompt

4. Set Java Home Path in MyComputer-->Properties-->Advanced System Settings-->Environment variables-->create NEW user variables like below

Step3:

This is assuming the installation of version 0.20.2 of Hadoop. Newer versions do not get along with Windows OS (mainly, the tasktracker daemon which requires permissions to be set that are inherently not allowed by Windows OS, but are required by more recent versions of Hadoop e.g. 0.20.20x.x)

1. Download the stable version 0.20.2 of Hadoop

2. Using 7-Zip (you should download this if you have not already, and it should be your default archive browser), open up the archive file. Copy the top level directory from the archive file and paste it into your home directory in C:/cygwin. This is usually something like C:\cygwin\usr\local

3. Once copied into your cygwin home directory, navigate to {hadoop-home}/conf. Open the following files for editing in your favorite editor (I strongly suggest Notepad++ ... why would you use anything else):

* core-site.xml

* hdfs-site.xml

* mapred-site.xml

* hadoop-env.sh

4. Make the following additions to the corresponding files:

* core-site.xml (inside the configuration tags)

<name>fs.default.name</name>

<value>localhost:9100</value>

</property>

* mapred-site.xml (inside the configuration tags)

<name>mapred.job.tracker</name>

<value>localhost:9101</value>

</property>

* hdfs-site.xml (inside the configuration tags)

<name>dfs.replication</name>

</property>

* hadoop-env.sh

* uncomment the JAVA_HOME export command, and set the path to your Java home (typically /cygdrive/c/Java/jdk1.6.0/}

5. In a cygwin window, inside your top-level hadoop directory, it's time to format your Hadoop file system. Type in 'bin/hadoop namenode -format' and hit enter. This will create and format the HDFS.

6. Now it is time to start all of the hadoop daemons that will simulate a distributed system, type in: 'bin/start-all.sh' and hit enter.

You should not receive any errors (there may be some messages about not being able to change to the home directory, but this is ok).

Double check that your HDFS and JobTracker is up and running properly by visiting http://localhost:50070 and http://localhost:50030, respectively.

To make sure everything is up and running properly, let's try a regex example.

7. From the top level hadoop directory, type in the following set of commands:

$ bin/hadoop dfs -mkdir input

$ bin/hadoop dfs -put conf input

$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'

$ bin/hadoop dfs -cat output/*

This should display the output of the job (finding any word that matched the regex pattern above).

8. Assuming no errors, you are all set to set up your Eclipse environment.

FYI, you can stop all your daemons by typing in 'bin/stop-all.sh', but keep it running for now as we move on to the next step.

Step4:

1. Download Eclipse Europa

2. Copy the hadoop plugin jar located at:

Normally you could use the plugin provided in the 0.20.2 contrib folder that comes with Hadoop.

3. Copy the jar and paste it into your Eclipse plugins directory (e.g. C:/eclipse/plugins)

4. In a regular command prompt, navigate to your eclipse folder (e.g. 'cd C:/eclipse')

5. Once Eclipse is open, open a new perspective (top right corner) and select 'Other'. From the list, select 'MapReduce'.

6. Go to Window -> Show View, and select Map/Reduce. This will open a view window for Map/Reduce Locations

7. Now you are ready to tie in Eclipse with your existing HDFS that you formatted and configured earlier. Right click in the Map/Reduce Locations view and select 'New Hadoop Location'

8. In the window that appears, type in 'localhost' for the Location name. Under Map/Reduce Master, type in localhost for the Host, and 9101 for the Port. For DFS Master, make sure the 'Use M/R Master host' checkbox is selected, and type in 9100 for the Port. For the User name, type in 'User'. Click 'Finish'

9. In the Project Explorer window on the left, you should now be able to expand the DFS Locations tree and see your new location. Continue to expand it and you should see something like the following file structure:

DFS Locations

-> (1)

-> tmp (1)

-> hadoop-{username} (1)

-> mapred (1)

-> system(1)

-> jobtracker.info

At this point you can create directories and files and upload them to the HDFS from Eclipse, or you can create them through the cygwin window as you did in step 7 in the previous section.

Practice on Ubuntu with JAVA

2012-09-26T10:12:00.000+05:30

Uninstall OpenJDK (if installed)

First we’ll check which OpenJDK packages are installed.

sudo dpkg --list | grep -i jdk

Next, let’s uninstall OpenJDK related packages. Edit the list of packages below if you have additional or different OpenJDK packages installed.

sudo apt-get purge icedtea-* openjdk-*

Let’s check that all OpenJDK packages have been removed.

sudo dpkg --list | grep -i jdk

HDFS - A Short Story

2012-09-24T10:12:00.002+05:30

Part - 1

Part - 2

Part - 3

Part - 4

Part - 5

Interview Questions & Answers for Hadoop MapReduce developers

2012-09-22T09:32:00.004+05:30

What is a JobTracker in Hadoop? How many instances of JobTracker run on a Hadoop Cluster?

JobTracker is the daemon service for submitting and tracking MapReduce jobs in Hadoop. There is only One Job Tracker process run on any hadoop cluster. Job Tracker runs on its own JVM process. In a typical production cluster its run on a separate machine. Each slave node is configured with job tracker node location. The JobTracker is single point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. JobTracker in Hadoop performs following actions(from Hadoop Wiki:)

Client applications submit jobs to the Job tracker.
The JobTracker talks to the NameNode to determine the location of the data
The JobTracker locates TaskTracker nodes with available slots at or near the data
The JobTracker submits the work to the chosen TaskTracker nodes.
The TaskTracker nodes are monitored. If they do not submit heartbeat signals often enough, they are deemed to have failed and the work is scheduled on a different TaskTracker.
A TaskTracker will notify the JobTracker when a task fails. The JobTracker decides what to do then: it may resubmit the job elsewhere, it may mark that specific record as something to avoid, and it may may even blacklist the TaskTracker as unreliable.
When the work is completed, the JobTracker updates its status.

Client applications can poll the JobTracker for information.

How JobTracker schedules a task?

The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated. When the JobTracker tries to find somewhere to schedule a task within the MapReduce operations, it first looks for an empty slot on the same server that hosts the DataNode containing the data, and if not, it looks for an empty slot on a machine in the same rack.

What is a Task Tracker in Hadoop? How many instances of TaskTracker run on a Hadoop Cluster

A TaskTracker is a slave node daemon in the cluster that accepts tasks (Map, Reduce and Shuffle operations) from a JobTracker. There is only One Task Tracker process run on any hadoop slave node. Task Tracker runs on its own JVM process. Every TaskTracker is configured with a set of slots, these indicate the number of tasks that it can accept. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. The TaskTracker monitors these task instances, capturing the output and exit codes. When the Task instances finish, successfully or not, the task tracker notifies the JobTracker. The TaskTrackers also send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated.

What is a Task instance in Hadoop? Where does it run?

Task instances are the actual MapReduce jobs which are run on each slave node. The TaskTracker starts a separate JVM processes to do the actual work (called as Task Instance) this is to ensure that process failure does not take down the task tracker. Each Task Instance runs on its own JVM process. There can be multiple processes of task instance running on a slave node. This is based on the number of slots configured on task tracker. By default a new task instance JVM process is spawned for a task.

How many Daemon processes run on a Hadoop system?

Hadoop is comprised of five separate daemons. Each of these daemon run in its own JVM.Following 3 Daemons run on Master nodes NameNode - This daemon stores and maintains the metadata for HDFS. Secondary NameNode - Performs housekeeping functions for the NameNode. JobTracker - Manages MapReduce jobs, distributes individual tasks to machines running the Task Tracker. Following 2 Daemons run on each Slave nodes DataNode – Stores actual HDFS data blocks. TaskTracker - Responsible for instantiating and monitoring individual Map and Reduce tasks.

What is configuration of a typical slave node on Hadoop cluster? How many JVMs run on a slave node?

Single instance of a Task Tracker is run on each Slave node. Task tracker is run as a separate JVM process.
Single instance of a DataNode daemon is run on each Slave node. DataNode daemon is run as a separate JVM process.
One or Multiple instances of Task Instance is run on each slave node. Each task instance is run as a separate JVM process. The number of Task instances can be controlled by configuration. Typically a high end machine is configured to run more task instances.

How NameNode Handles data node failures?

NameNode periodically receives a Heartbeat and a Blockreport from each of the DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is functioning properly. A Blockreport contains a list of all blocks on a DataNode. When NameNode notices that it has not recieved a hearbeat message from a data node after a certain amount of time, the data node is marked as dead. Since blocks will be under replicated the system begins replicating the blocks that were stored on the dead datanode. The NameNode Orchestrates the replication of data blocks from one datanode to another. The replication data transfer happens directly between datanodes and the data never passes through the namenode.

Does MapReduce programming model provide a way for reducers to communicate with each other? In a MapReduce job can a reducer communicate with another reducer?

Nope, MapReduce programming model does not allow reducers to communicate with each other. Reducers run in isolation.

Can I set the number of reducers to zero?

Yes, Setting the number of reducers to zero is a valid configuration in Hadoop. When you set the reducers to zero no reducers will be executed, and the output of each mapper will be stored to a separate file on HDFS. [This is different from the condition when reducers are set to a number greater than zero and the Mappers output (intermediate data) is written to the Local file system(NOT HDFS) of each mappter slave node.]

Where is the Mapper Output (intermediate key-value data) stored ?

The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.

What are combiners? When should I use a combiner in my MapReduce Job?

Combiners are used to increase the efficiency of a MapReduce program. They are used to aggregate intermediate map output locally on individual mapper outputs. Combiners can help you reduce the amount of data that needs to be transferred across to the reducers. You can use your reducer code as a combiner if the operation performed is commutative and associative. The execution of combiner is not guaranteed, Hadoop may or may not execute a combiner. Also, if required it may execute it more then 1 times. Therefore your MapReduce jobs should not depend on the combiners execution.

What is Writable & WritableComparable interface?

org.apache.hadoop.io.Writable is a Java interface. Any key or value type in the Hadoop Map-Reduce framework implements this interface. Implementations typically implement a static read(DataInput) method which constructs a new instance, calls readFields(DataInput) and returns the instance.
org.apache.hadoop.io.WritableComparable is a Java interface. Any type which is to be used as a key in the Hadoop Map-Reduce framework should implement this interface. WritableComparable objects can be compared to each other using Comparators.

What is the Hadoop MapReduce API contract for a key and value Class?

The Key must implement the org.apache.hadoop.io.WritableComparable interface.
The value must implement the org.apache.hadoop.io.Writable interface.

What is a IdentityMapper and IdentityReducer in MapReduce ?

org.apache.hadoop.mapred.lib.IdentityMapper Implements the identity function, mapping inputs directly to outputs. If MapReduce programmer do not set the Mapper Class using JobConf.setMapperClass then IdentityMapper.class is used as a default value.
org.apache.hadoop.mapred.lib.IdentityReducer Performs no reduction, writing all input values directly to the output. If MapReduce programmer do not set the Reducer Class using JobConf.setReducerClass then IdentityReducer.class is used as a default value.

What is the meaning of speculative execution in Hadoop? Why is it important?

Speculative execution is a way of coping with individual Machine performance. In large clusters where hundreds or thousands of machines are involved there may be machines which are not performing as fast as others. This may result in delays in a full job due to only one machine not performaing well. To avoid this, speculative execution in hadoop can run multiple copies of same map or reduce task on different slave nodes. The results from first node to finish are used.

When the reducers are started in a MapReduce job?

In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.

If reducers do not start before all mappers finish then why does the progress on MapReduce job shows something like Map(50%) Reduce(10%)? Why reducers progress percentage is displayed when mapper is not finished yet?

Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The progress calculation also takes in account the processing of data transfer which is done by reduce process, therefore the reduce progress starts showing up as soon as any intermediate key-value pair for a mapper is available to be transferred to reducer. Though the reducer progress is updated still the programmer defined reduce method is called only after all the mappers have finished.

What is HDFS ? How it is different from traditional file systems?

HDFS, the Hadoop Distributed File System, is responsible for storing huge data on the cluster. This is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant.

HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
HDFS provides high throughput access to application data and is suitable for applications that have large data sets.
HDFS is designed to support very large files. Applications that are compatible with HDFS are those that deal with large data sets. These applications write their data only once but they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS supports write-once-read-many semantics on files.

What is HDFS Block size? How is it different from traditional file system block size?

In HDFS data is split into blocks and distributed across multiple nodes in the cluster. Each block is typically 64Mb or 128Mb in size. Each block is replicated multiple times. Default is to replicate each block three times. Replicas are stored on different nodes. HDFS utilizes the local file system to store each HDFS block as a separate file. HDFS Block size can not be compared with the traditional file system block size.

What is a NameNode? How many instances of NameNode run on a Hadoop Cluster?

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself. There is only One NameNode process run on any hadoop cluster. NameNode runs on its own JVM process. In a typical production cluster its run on a separate machine. The NameNode is a Single Point of Failure for the HDFS Cluster. When the NameNode goes down, the file system goes offline. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives.

What is a DataNode? How many instances of DataNode run on a Hadoop Cluster?

A DataNode stores data in the Hadoop File System HDFS. There is only One DataNode process run on any hadoop slave node. DataNode runs on its own JVM process. On startup, a DataNode connects to the NameNode. DataNode instances can talk to each other, this is mostly during replicating data.

How the Client communicates with HDFS?

The Client communication to HDFS happens using Hadoop HDFS API. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file on HDFS. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data.

How the HDFS Blocks are replicated?

HDFS is designed to reliably store very large files across machines in a large cluster. It stores each file as a sequence of blocks; all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later. Files in HDFS are write-once and have strictly one writer at any time. The NameNode makes all decisions regarding replication of blocks. HDFS uses rack-aware replica placement policy. In default configuration there are total 3 copies of a datablock on HDFS, 2 copies are stored on datanodes on same rack and 3rd copy on a different rack.

What is the purpose of the secondary name-node?

The term "secondary name-node" is somewhat misleading. It is not a name-node in the sense that data-nodes cannot connect to the secondary name-node, and in no event it can replace the primary name-node in case of its failure.

The only purpose of the secondary name-node is to perform periodic checkpoints. The secondary name-node periodically downloads current name-node image and edits log files, joins them into new image and uploads the new image back to the (primary and the only) name-node.

So if the name-node fails and you can restart it on the same physical node then there is no need to shutdown data-nodes, just the name-node need to be restarted. If you cannot use the old node anymore you will need to copy the latest image somewhere else. The latest image can be found either on the node that used to be the primary before failure if available; or on the secondary name-node. The latter will be the latest checkpoint without subsequent edits logs, that is the most recent name space modifications may be missing there. You will also need to restart the whole cluster in this case.

Hadoop Installation

2012-09-21T15:55:00.000+05:30

Steps are following:

1. The steps are tested on Linux Ubuntu 10.

2. Download Hadoop from http://hadoop.apache.org/common/releases.html . This document is done with 23 May, 2012: Release 2.0.0-alpha available

3. Unzip Hadoop in a directory. This will create a direcotry "hadoop-2.0.0-alpha"

5. Set the environment variable for Java and Hadoop and also put Hadoop in the path. For bashrc shell it lools like:

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
export HADOOP_INSTALL=/home/enlighted/hadoop-2.0.0-alpha
export PATH=$PATH:/home/enlighted/programs/apache-maven-3.0.4/bin:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin

6. Check if hadoop is installed properly by using hadoop command

hadoop version

It will print something similar to

Hadoop 2.0.0-alpha

Subversionhttp://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.0.0

alph... –r1338348

7. Hadoop can be configured using XML configuration files which are at etc/hadoop directory inside hadoop folder. The improtant onces are:

core-site.xml -> core porperties

hdfs-site.xml -> hdfs properties

yarn-site.xml -> yarn properties which is the next generation of Map Reduce.

8. By default hadoop is configured to run in standalone mode. The other two modes are pseudo distributed and clustered mode

Hadoop Introduction

2012-09-21T15:36:00.000+05:30

Persistence is the ability for data to live beyond the lifecycle of a process. Software as part of interaction generate a lot of data which needs to live beyond them. In the beginning, the data storage systems were file oriented. However files did little to provide a good structure to the stored data. The other main issue with files was there proprietary format, which made it impossible to read them in the absence of certain programs. Also processing of data in file across versions of software was another challenge to deal with, as run time structures to hold the data do change across releases. Also in terms of scalability, files were not very good in terms of searching and doing local updates on data. To solve this, industry came up with RDBMS type of system.

RDBMS is a very structured way of defining the data. The rigorous structuring and ability to constrain provided the capability to keep the integrity of data also popularly known as ACID properties. For the amount of data the world was dealing with. RDBMS systems were sufficient and provided the required scalability to an extent. This was the time when to get indexed in a search engine one has to submit the web link to search engines. People managed higher scalability with concepts like sharding however at the cost of increased complexity. (Developers love to make things complex).

Google came and changed the world. The amount of data they start dealing with just got larger by any sane means of measurement. They inverted the concept of web indexing. Rather then people submitting links, they started crawling web, downloading pages locally and indexing them. However with this approach came the problem of large data popularly termed as "Big Data" now. There was a parallel development happening. Doug cutting was working on a search engine infrastructure project called Nutch however he ran into issues of scaling it further. A paper from Google around that time on how Google handles Big Data using GFS or Google File System. Based on the paper, Doug started working on Hadoop on the line of GFS and used the concept of Map Reduce. Around those time, Doug was employed by Yahoo and the project got he required funding and resources. Today Hadoop is a clear winner in terms of handling Big Data problems and is used in my enterprises who deal with Big Data. The core of Hadoop is the ability to push the processing logic to the data itself without moving a lot of data around. This is different from technologies like Grid computing which move both the data and processing logic. Compared to Hadoop, RDBMS systems do the opposite of pulling the data to the processing logic. Having said that it does not means that we are going to throw RDBMS and adopt Hadoop everywhere. Both tend to complement each other and have their own unique strength and weaknesses.

A high level comparison of Hadoop is as follows:

Data integrity low	Data integrity high (ACID compliant)
Non structured data	Structured Data
Linear/Horizontal scaling	Veritcal/non linear scaling
good for initial insert and then multiple reads	Good for multiple updates

Apache Hadoop – Background and an Overview

2012-09-09T14:02:00.000+05:30

MapReduce – The Paradigm

Essentially, the MapReduce model consists of a first, embarrassingly parallel, map phase where input data is split into discreet chunks to be processed. It is followed by the second and final reduce phase where the output of the map phase is aggregated to produce the desired result. The simple, and fairly restricted, nature of the programming model lends itself to very efficient and extremely large-scale implementations across thousands of cheap, commodity nodes.

Apache Hadoop MapReduce is the most popular open-source implementation of the MapReduce model.

In particular, when MapReduce is paired with a distributed file-system such as Apache Hadoop HDFS, which can provide very high aggregate I/O bandwidth across a large cluster, the economics of the system are extremely compelling – a key factor in the popularity of Hadoop.

One of the keys to this is the lack of data motion i.e. move compute to data and do not move data to the compute node via the network. Specifically, the MapReduce tasks can be scheduled on the same physical nodes on which data is resident in HDFS, which exposes the underlying storage layout across the cluster. This significantly reduces the network I/O patterns and keeps most of the I/O on the local disk or within the same rack – a core advantage.

Apache Hadoop MapReduce, circa 2011 – A Recap

Apache Hadoop MapReduce is an open-source, Apache Software Foundation project, which is an implementation of the MapReduce programming paradigm described above. Now, as someone who has spent over six years working full-time on Apache Hadoop, I normally like to point out that the Apache Hadoop MapReduce project itself can be broken down into the following major facets:

The end-user MapReduce API for programming the desired MapReduce application.
The MapReduce framework, which is the runtime implementation of various phases such as the map phase, the sort/shuffle/merge aggregation and the reduce phase.
The MapReduce system, which is the backend infrastructure required to run the user’s MapReduce application, manage cluster resources, schedule thousands of concurrent jobs etc.

This separation of concerns has significant benefits, particularly for the end-users – they can completely focus on the application via the API and allow the combination of the MapReduce Framework and the MapReduce System to deal with the ugly details such as resource management, fault-tolerance, scheduling etc.

The current Apache Hadoop MapReduce System is composed of the JobTracker, which is the master, and the per-node slaves called TaskTrackers.

The JobTracker is responsible for resource management (managing the worker nodes i.e. TaskTrackers), tracking resource consumption/availability and also job life-cycle management (scheduling individual tasks of the job, tracking progress, providing fault-tolerance for tasks etc).

The TaskTracker has simple responsibilities – launch/teardown tasks on orders from the JobTracker and provide task-status information to the JobTracker periodically.

For a while, we have understood that the Apache Hadoop MapReduce framework needed an overhaul. In particular, with regards to the JobTracker, we needed to address several aspects regarding scalability, cluster utilization, ability for customers to control upgrades to the stack i.e. customer agility and equally importantly, supporting workloads other than MapReduce itself.

We’ve done running repairs over time, including recent support for JobTracker availability and resiliency to HDFS issues (both of which are available in Hortonworks Data Platform v1 i.e. HDP1) but lately they’ve come at an ever-increasing maintenance cost and yet, did not address core issues such as support for non-MapReduce and customer agility.

Hadoop with HDFS and Map Reduce

2012-09-08T19:21:00.002+05:30

Hadoop is an open source project for processing large datasets in parallel with the use of low level commodity machines.

Hadoop is build on two main parts. An special file system called Hadoop Distributed File System (HDFS) and the Map Reduce Framework.

The HDFS File System is an optimized file system for distributed processing of very large datasets on commodity hardware.

The map reduce framework works in two main phases to process the data. Which are the Map phase and the Reduce phase.