Hadoop Tutorial

Multi-Node Cluster Setup

2011-12-27T12:28:00.000-08:00

Administration Lab 6: Multi-node Cluster Setup

Create Additional VM Image

Repeat steps outlined in Lab 1 to create a new Virtual Machine and install JDK and Hadoop

Configure Bridged Adapter

Power off both Virtual Machines (Machine => ACPI Shutdown)

Highlight each VM and click on Settings => Network

In the “Attached To” field, select “Bridged Adapter

In the “Name” field, select the correct network card adapter (wired/wireless) that you use on your primary Operating System to connect to the Internet

Uncheck all devices except “Hard Disk”

Find out Master & Slave IPs

Run ifconfig on master and slave

Write down IPs for both hosts

Execute Installation from Lab1

Except the following steps:

sudo apt-get install hadoop-0.20-namenode
sudo apt-get install hadoop-0.20-jobtracker

They are not needed on slave node

Configuration For Both Nodes

/etc/hadoop-0.20/conf/mapred-site.xml

Provisioning IPs

/etc/hosts

Add entires for master and slave node to ensure proper network communication

Reformat namenode and delete data from data directory

sudosuhdfs

Execute hadoop-0.20 namenode format

Delete data from data directories on both machine

/var/lib/hadoop-0.20/cache/hdfs/dfs/data

Start Distributed Cluster

Start master node normally

Start datanode and tasktracker on slave node only

sudo /etc/init.d/hadoop-0.20-tasktracker start

sudo /etc/init.d/hadoop-0.20-datanode start

Verification

Check Web interface to verify that second node is connected

Run sleep example to make sure that mapreduce is working properly

Format Conversion

2011-12-27T12:06:00.000-08:00

Programming Lab 3: Format Conversion

This lab demonstrates how to use different map-reduce formats for input and output
Also shows how to convert from one format to another

Problem to Solve

We will work with one set of data OrgChart and will represent that data in a different format:

Create Project and Link with Libraries

Copy provided libraries and java code from USB drives
Create project in NetBeans or Eclipse
Link with libraries
Create new class and copy provided code
Modify input and output directory
Run code and examine result
For detailed instructions on project creating refer to Lab1.

Task1. Parse existing Key Value Text Input

We need to parse existing Key Value Text input and output it in the Key Value Text.
Data representation is not going to change, but data will be sorted by its key
Set proper input format

Set proper output format

Don’t forget to change input path to the correct path on your workstation
Expected output

Task2. Generate Sequence File From Key Value Text Input

We need to parse existing Key Value Text input and output it in the Sequence File
Data representation is not going to change, but data item will be sorted by its key and it will be in a compressed form
Set proper input format

Set proper output format

Enable compression

Expected output will not be readable since it is compressed

Task3. Generate Map File From Key Value Text Input

We need to parse existing Key Value Text input and output it in the Map File
Data representation is not going to change, but data will be sorted by its key and it will be in compressed form. Also, Map file will create index for faster access of the data.
Set proper input format

Set proper output format

Compression remains enabled

Expected output will not be readable since it is compressed and map file output will generate index file

Task4. Generate Key Value Text Format From Map File

We need to parse existing Map File and output it in the Key Value Text input
Data representation is not going to change, but data will be sorted by its key and has to match the input data
Keep in mind that if you keep compression enabled the data will be compressed
Set proper input format

Set proper output format

We can choose to keep compression enabled

Expected output will not be readable since it is compressed and map file output will generate index file

Summary

This lab has demonstrated how to work with different data format
You should be comfortable in using any of them

Map Side Join

2011-12-27T09:22:00.000-08:00

Programming Lab 4: Map Side Join

This lab demonstrates Map-Side join technique used to quickly and efficiently join data in multiple directories
In a lot of cases this is the fastest way to process data.

Problem to Solve

We will work with two sets of data OrgChart and Salary and will will need to join them together using Map Reduce join technique

Create Project and Link with Libraries

Copy provided libraries and java code from USB drives
Create project in NetBeans or Eclipse
Link with libraries
Create new class and copy provided code
Modify input and output directory to SalaryMap and OrgChartMap folders
For detailed instructions on project creating refer to Lab1.

Task1. Create Map Files Using Previous Lab

Using lab 3 code point input directory to lab4/input/Salary directory and point output directory to lab4/input/SalaryMap directory
Run the code and it will generate map file (sequence file with index)
Repeat the same procedure with OrgChart

Task2. Point MapSide Directories to Newly Generated Map Files

Change directory to point to newly generated map files
Run code and see the outcome

Please, note that unlike regular MR job, map-side joint quarantines order of the data.

Walk Through

Map-Side join takes array of the path to join

Set specific Map Side join input format

Walk Through Mapper

Mapper provides access to values via TupleWritable value

value.has(0) checks if you have left value

value.get(0) gives you the left value

value.has(1) checks if you have right value

value.get(1) gives you the right value

Task3. Adding Vacation Time

Generate Map File for vacation time
Extend Map-Reduce job to add another folder in join
Modify Mapper code to add vacation time into output value
Hint: to add two text object:

Reduce Side Join

2011-12-26T23:38:00.000-08:00

Programming Lab 5: Reduce Side Join

This lab demonstrates reduce-side join with different reducers

Problem to Solve

We will work with two sets of data OrgChart and Salary, and we will need to join them together using Reduce Side join technique
However, in some cases position description can consist of two different words and we only will compare the first one

Create Project and Link with Libraries

Copy provided libraries and java code from USB drives
Create project in NetBeans or Eclipse
Link with libraries
Create new classes for each file with the same name as file in the directory or you can copy java files to your source directory
For detailed instructions on creating a project please refer to Programming Lab1

Walk Through

Please, make sure to adjust input path to your local machine

We must assign different mappers to different input folders

Engineer salary should be applicable to Engineer and Engineer 2
Position will be represented by Position class and we will compare and match only the first word
The position class will implement a comparator method

Expected Result

The expected result is the following:

Input Format

2011-12-26T03:18:00.001-08:00

Programming Lab 2: Input Format

We will learn how to join different sets of data on the same key using regular Map-Reduce approach.
Also, this lab will show limitations of the regular Map-Reduce approach and what developer need to do in order to overcome those limitations

Problem to Solve

We have two sets of data : position information and salary information
We need to join them together!!!

Produce:

Create Project and Link with libraries

Copy provided libraries and java code from USB drives
Create project in NetBeans or Eclipse
Link with libraries
Create new class and copy provided code
Modify input and output directory
Run code and examine result
For detailed instructions on project creating refer to Lab1.
Our key is position in the company (CEO, Engineer, etc.)
Our data is separated by “=”. We will use KeyValueTextInputFormat.
We also need to set special separator since its different from default tab

Expected output

Please, note that order is not quarantine. We need to do special processing to guarantee order.

Task

Additional input folder contains data for vacation time
Additional input folder can be specified by calling FileInputFormat.addInputPath
We need to add that data for every record

Word Count

2011-12-26T03:09:00.000-08:00

Programming Lab 1: Word Count

We will learn how to setup development environment for Hadoop projects
Run “Word Count ” applications
Create new application to count letters in text documents

Prerequisites

Java 1.6
Hadoop and log4j libraries
NetBeans or Eclipse

Create Project and Link with Libraries

Copy provided libraries and java code from USB drives
Create project in NetBeans or Eclipse (specific instructions on the next page)
Link with libraries
Create new class and copy provided code
Modify input and output directory
Run code and examine result

Create Project with NetBeans

Click on File -> New Project
Select Java Application type
Set main class to WordCount
Name your project HadoopLab1WordCount
Follow instructions on the screen
See the next page for a screenshot

Open a file from the provided material in ProgLabs/lab1/original
Copy everything except the package name
Insert code into newly created class right after the package name

Let’s link with appropriate libraries

Right click on project name and select properties
Pick Libraries option and click on the Compile tab
Click on Add JAR/Folder button
Add everything from ProgLabs/lib folder
NetBeans will re-evaluate dependencies and you should not see any errors at this point
Adjust input/output values in run method to ProgLabs/lab1/<<input|output>> accordingly

Link the project with appropriate libraries

Right click on project name and select properties
Pick Libraries option and click on Compile tab
Click on Add JAR/Folder button
Add everything from ProgLab/lib folder
NetBeans will reevaluate dependencies and you should not see any errors at this point

Adjust the input/output values in run method to ProgLabs/lab1/<<input|output>> accordingly
Right-click and run!

Create Project with Eclipse

Create New Project: Click on File -> New Project
Name project HadoopLab1WordCount
Select Java 1.6
Click Next and add libraries from ProgLabs/lib

Create a new class – WordCount
Right-click on the project -> select New -> Class
Copy code from Labs/ProgLabs/lab4/original into new class
Right click and run it!

Lets examine output directory

You will see two files: one indicating result of the Map Reduce job and the second one containing result of the job

Result

Adam 1

Brandon 1

Graig 1

Kim 1

Marty 1

Mike 1

Nancy 1

Nick 1

Nishani 1

Steve 2

Tracy 2

Vidur 1

Exercise: Let’s Count Letters!

Modify word count application to count letters in the document
Create another class that implements Reducer and switch application to use it in run method
Hint:

String ch = String.valueOf(line.charAt(i));

MapReduce Filesystem Recovery

2011-12-21T08:09:00.000-08:00

Administration Lab 5: MapReduce Filesystem Recovery

Restore the Last State of VM

Open Virtual Box application

Start the last VM state

Stop Hadoop cluster

for x in /etc/init.d/hadoop-* ; do sudo $x stop; done

Procedure to Recover Namenode

Change user to hdfs

sudo su hdfs

Lets try to import image

hadoop-0.20 namenode –importCheckpoint

This will fail!!! (Why???)

Lets move data to home directory

mkdir ~/name

mv /var/lib/hadoop-0.20/cache/hadoop/dfs/name/* ~/name

ls /var/lib/hadoop-0.20/cache/hadoop/dfs/name

Lets try import again

hadoop-0.20 namenode –importCheckpoint

This will work!

Ctrl-C to terminate node and exit from shell

Start the cluster

for x in /etc/init.d/hadoop-* ; do sudo $x start; done

Checking Filesystem

hadoop-0.20 fsck <file name> -files -blocks -racks

MapReduce Performance Tuning

2011-12-21T07:58:00.000-08:00

Administration Lab 4: MapReduce Performance Tuning

Restore the Last State of VM

Open Virtual Box application

Start the last VM state

Bounce Hadoopcluster

for x in /etc/init.d/hadoop-* ; do sudo $x stop; done

for x in /etc/init.d/hadoop-* ; do sudo $x start; done

2 mappers, 2 reducers

1 CPU 2 Cores

In Windows: All Programs => Accessories => System Tools => System Information

Execute Waiting Job

hadoop-0.20 jar
/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-*examples.jar sleep -m 4 –r 4 -mt 10000 -rt 10000

-m number of mappers

-r number of reducers

-mt milliseconds to sleep at map step

-rt milliseconds to sleep at reduce step

Map Scheduling

Only two mappers will initialized in the same time

Go to port 50030 and click on running job and pick map step

Reduce Scheduling

Only two reducers will initialized in the same time

Go to port 50030 and click on running job and pick reduce step

Overall Job Summary

Take a note of your job running time

How can we improve that???

Reduce number of mappers and reducers

hadoop-0.20 jar
/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-*examples.jar sleep –m 2 –r 2 –mt 20000 –rt 20000

Increase number of mappers/reducers

Go to /etc/hadoop-0.20/conf (please, use tab for auto completion )

Open mapred-site.xml with sudo permissions

Increase number of reducers and mappers to 4

Config File Example

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>

</property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>

</property>

FAQ: Where are Defaults ?

Defaults are located inside hadoop-core.jar

Locate hadoop-core.jar

Default: /usr/lib/hadoop-0.20/hadoop-core.jar

Copy jar to home directory: cp hadoop-core.jar ~/

Check content: jar tfhadoop-core.jar | grep default

Extract content: jar xfv hadoop-core.jar

Default files are a good source of information

<property>
<name>hadoop.job.history.location</name>
<value></value>
<description> If job tracker is static the history files are storedin this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history.
</description>
</property>

<property>
<name>hadoop.job.history.user.location</name>
<value></value>
<description> User can specify a location to store the history files ofa particular job. If nothing is specified, the logs are stored inoutput directory. The files are stored in "_logs/history/" in the directory.User can stop logging by giving the value "none".
</description>
</property>

Bounce cluster and wait for safemode exit

for x in /etc/init.d/hadoop-0.20-*; do sudo $x stop; done;

Shortcut: history | grep stop
!<number of the command>

The same command with start

hadoop-0.20 dfsadmin –safemode wait

Note: You might see an error while stopping cluster. This is related to a current open bug that should be fixed in next release of hadoop. It is safe to ignore it and can proceed.

Config File Example

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>

</property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>

</property>

FAQ: Where are Defaults

Defaults are located inside hadoop-core.jar

Locate hadoop-core.jar

Default: /usr/lib/hadoop-0.20/hadoop-core.jar

Copy jar to home directory:

cp hadoop-core.jar ~/

Check content:

jar tfhadoop-core.jar | grep default

Extract content:

jar xfv hadoop-core.jar

Default files are a good source of information

Bounce cluster and wait for safemode exit

for x in /etc/init.d/hadoop-0.20-*; do sudo $x stop; done;

Shortcut: history | grep stop
!<number of the command>

The same command with start

hadoop-0.20 dfsadmin –safemode wait

Note: You might see an error while stopping cluster. This is related to a current open bug that should be fixed in next release of hadoop. It is safe to ignore it and can proceed.

New Capacity

You should see 4 mappers and 4 reducers

Let’s execute the same job

hadoop-0.20 jar
/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-*examples.jar sleep -m 4 -r4 -mt 10000 -rt 10000

All four tasks are initializing in same time

Mappers

Reducers

Total Summary

All Summary

	Request Map	Request Reduce	Avail Map	Avail Reduce	Average Map	Average Reduce	Total Time
Case 1	4	4	2	2	33 sec	33 sec	100 sec
Case 2	2	2	2	2	26 sec	34 sec	69 sec
Case 3	4	4	4	4	123 sec	47 sec	172 sec
Case 4	1	1	4	4	42 sec	47 sec	94 sec

FAQ: How to kill a job?

Retrive job id:

hadoop-0.20 job -list

Kill the job:

hadoop-0.20 job –kill <job-id>

Note: this is a hard kill, some additional clean up might be required

Working with MapReduce

2011-12-21T07:26:00.001-08:00

Administration Lab 3: Working with MapReduce

Restore the Last State of VM

Open Virtual Box application

Start the last VM state

Bounce Hadoopcluster

for x in /etc/init.d/hadoop-* ; do sudo $x stop; done

for x in /etc/init.d/hadoop-* ; do sudo $x start; done

Calcualte PI

Execute the following command:

hadoop-0.20 jar /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-*examples.jar pi 10 10000

Result: job finished in 103.293 seconds

Estimated value of PI is 3.1412000000000

Execute Word Count

Create some input: vi test.txt

Move input to HDFS:

hadoop-0.20 dfs -put ./test.txt /test.txt

Execute word count job

hadoop-0.20 dfs -ls/

hadoop-0.20 jar /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u1-*examples.jar wordcount /test.txt /output

Check what happen with output directory

hadoop-0.20 dfs -ls /output

View output

hadoop-0.20 dfs -cat /output/part

Working with HDFS

2011-12-21T07:15:00.001-08:00

Administration Lab 2: Working with HDFS

Directory Creation and Copying Files

Open VM
Open new terminal windows
Make sure you are in your home directory
Create new file using command vi test.txt
Press ‘a’ for editing, enter some text and press ‘ESC’ ,’wq’ to save and exit
Check that file was created by running ‘ls’ command

Create a Directory

Create new directory in HDFS

hadoop-0.20 dfs –mkdir /test

Check that directory exist

hadoop-0.20 dfs –ls /

Save file in HDFS

hadoop-0.20 dfs –put ./test.txt /test/test.txt

View file in HDFS

hadoop-0.20 dfs –cat /test/test.txt

Examine File and Space

Examine available space

hadoop-0.20 dfs -df /test/test.txt

Make another copy of the file

hadoop-0.20 dfs –cp /test/test.txt /test/test2.txt

Increase replication factor

hadoop-0.20 dfs –setrep –R 9 /test/test.txt

View HDFS Files via Web Interface

Browse HDFS via Web Interface

Parallel Copying

Lets try to copy a file with distcp application: hadoop-0.20 distcp /test/test.txt /test/test2.txt

This will launch map-reduce job for copying and create new file

This procedure will be slower then a regular copy, but you will see significant improvement when copying large directories

Hadoop Install

2011-12-21T07:07:00.001-08:00

Hadoop Installation

Prerequisites
Installation of Virtual Box to run Virtual Machine Images (VMs)

Installation of Ubuntu 10 Image to run Hadoop

Installing Virtual Box

Download the corresponding installation package from: http://www.virtualbox.org/wiki/Downloads (Windows or Mac OS X recommended)

Close all applications and run the installation package following the on screen instructions Note: The current tested version is 4.0.8 (05/26/2011)

Installing Ubuntu 10 Image

Download Ubuntu OS version 10 from http://www.ubuntu.com/download/ubuntu/download Note: Make sure to download version 10. The latest version 11 is not supported by Hadoop distribution (as of 05/26/2011)

Start Virtual Box from application selection menu.

Click on “New” button to create new virtual machine and click continue
Provide a name for your VM and select Linux and Ubuntu in OS options

Keep the rest of the settings as defaults and continue with instructions
Start the VM after it was created by selecting VM and clicking “Start”

Select installation media as the downloaded Ubuntu installation package

Proceed with default settings during installation. Please, note the user “hadoop” is reserved and don’t select that as your user.
Restart OS after installation is completed you should see the following screen on your monitor

Installing JDK 1.6 and Hadoop

Install Java JDK 1.6 and Hadoop on Ubuntu

Open new terminal by going to Applications -> Accessories -> Terminal;
Check the release version of the Ubuntu by running the following command: lsb_release –c

The expected output should be lucid .

Verifying Installation

You should see the following screens in your browser: