Asp.net Codes

Hadoop SDK, HIVE, Mapreduce and .NET Developers

2015-07-13T21:34:00.001+05:30

Hadoop SDK and Tutorials for Microsoft .NET Developers

http://hortonworks.com/blog/hadoop-sdk-and-tutorials-for-microsoft-net-developers/

How to Communicate to Hadoop via Hive using .NET/C#

http://www.codeproject.com/Tips/738141/How-to-Communicate-to-Hadoop-via-Hive-using-NET-Cs

Creating .NET-based Mappers and Reducersfor Hadoop with JNBridgePro

How to Create MapReduce Jobs for Hadoop Using C#

http://www.codeguru.com/columns/experts/how-to-create-mapreduce-jobs-for-hadoop-using-c.htm

BIG DATA for .NET Devs: HDInsight, Writing Hadoop Map Reduce Jobs In C# And Querying Results Back Using LINQ

http://www.amazedsaint.com/2013/03/taming-big-data-with-c-using-hadoop-on.html

What type of MongoDB deployment would you like to build?

2015-06-25T21:29:00.001+05:30

Three ways we can deploy mongodb

Standalone Instance

A standalone MongoDB provides a single copy of your data and is appropriate for projects where data loss is not a concern.

Replica Set

A replica set is a group of MongoDB instances that host the same data set. A replica set provides both redundancy and high availability.

Sharded Cluster

Sharding is a method for storing data across multiple machines. MongoDB uses sharding to support deployments with very large data sets and high throughput operations.

HIVE 0.14 Update , delete, insert is possible

2015-06-20T17:15:00.002+05:30

Attempt to do update or delete using transaction manager that does not support these operations in hive-0.14.0

CRUD operations are supported in Hive from 0.14 onwards.
See Wiki

Hive supports data warehouse software facility,which facilitates querying and managing large datasets residing in distributed storage. In data warehouse there are situation where we need to update, delete etc transactions.In hive later versions UPDATE was not supported,but there were workarounds to do update a transaction

1. Update Statement In Hive For Small Tables
2. Update Statement In Hive For Large Tables using INSERT

Lets see how to do INSERT,UPDATE,DELETE in newer version of hive.

Create a table "test"

CREATE EXTERNAL TABLE 
    test (EmployeeID Int,FirstName String,Designation  
        String,Salary Int,Department String) 
    ROW FORMAT DELIMITED FIELDS TERMINATED BY  "," 
    LOCATION '/user/hdfs/Hive';

We will try to update the salary of employee id 19 from 45,000 to 50,000.

 hive> UPDATE test 
           SET salary = 50000 
           WHERE employeeid = 19;

 FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction m anager that does not support these operations.

While applying above query it shows a semantic Exception.In order to allow update and delete we need to add additional settings in hive-site.xml and create table with ACID output format support.

To achieve the same follow below steps:

1. New Configuration Parameters for Transactions

 hive.support.concurrency – true
 hive.enforce.bucketing – true
 hive.exec.dynamic.partition.mode – nonstrict
 hive.txn.manager –org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
 hive.compactor.initiator.on – true
 hive.compactor.worker.threads – 1

You can set these configuration in hive-site.xml (after setting restart Hive ) for ever or via terminal.
Dont Forget to restart Hive once the above settings are applied, else you will get the same error again.
2. Below query creates HiveTest table with ACID support
(To do Update,delete or Insert we need to create a table that support ACID properties)

 create table HiveTest 
   (EmployeeID Int,FirstName String,Designation String,
     Salary Int,Department String) 
   clustered by (department) into 3 buckets 
   stored as orc TBLPROPERTIES ('transactional'='true') ;

3. Load data into HiveTest from a staging table,which contains the original data.

 from stagingtbl 
   insert into table HiveTest 
   select employeeid,firstname,designation,salary,department;

4. UPDATE,DELETE and INSERT operations

1.UPDATE

 update HiveTest 
    set salary = 50000 
    where employeeid = 19;

SYNOPSIS

The referenced column must be a column of the table being updated.
The value assigned must be an expression that Hive supports in the select clause. Thus arithmetic operators, UDFs, casts, literals, etc. are supported. Subqueries are not supported.
Only rows that match the WHERE clause will be updated.
Partitioning columns cannot be updated.
Bucketing columns cannot be updated.
In Hive 0.14, upon successful completion of this operation the changes will be auto-committed.

2. INSERT

 insert into table HiveTest 
     values(21,'Hive','Hive',0,'B');

SYNOPSIS

Each row listed in the VALUES clause is inserted into table tablename.
Values must be provided for every column in the table. The standard SQL syntax that allows the user to insert values into only some columns is not yet supported. To mimic the standard SQL, nulls can be provided for columns the user does not wish to assign a value to.
Dynamic partitioning is supported in the same way as for INSERT...SELECT.
If the table being inserted into supports ACID and a transaction manager that supports ACID is in use, this operation will be auto-committed upon successful completion.

3. DELETE

 delete from HiveTest
     where employeeid=19;

SYNOPSIS

Only rows that match the WHERE clause will be deleted.
In Hive 0.14, upon successful completion of this operation the changes will be auto-committed.

MySQL: Multiple Inserts for a single column

2015-06-09T16:02:00.000+05:30

INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);

once the bracket close it will take the values and look for another value in next bracket.

Incorrect table definition; there can be only one auto colum

2015-06-09T15:51:00.001+05:30

Create table Tablename (id int not null AUTO_incremment ..........
.....................
.............................
);

IF you have not defined the primary key in above syntax u will face this kind of error.

Data introduction

2015-06-08T00:14:00.000+05:30

Data

New York Stock exchange - 4-5 terabyte data per day
Facebook - 240 billion photos, 7 petabyte data per month
Internet archive stores 18.5 data.
Ancestory.com stores 10 petabyte data.

Good news is that big data is here and problem

How to store
how to anylyze this much of huge data.

Apache Oozie : Element In workflow and callback mechnism

2015-05-26T17:29:00.001+05:30

Apache OOZIE : Workflow

2015-05-26T17:25:00.001+05:30

Oozie

2015-05-26T17:23:00.000+05:30

HIVE Vs RDBMS

2015-05-21T18:52:00.001+05:30

HIVE

RDBMS

· Hive is a data warehouse which provides SQL like queries for data residing in HDFS. A RDBMS wrapper for files available in HDFS.

· Hive is strictly used for select. NO UPDATE/DELETE is provided. It makes it a data warehouse. The SQL code written in Hive eventually turns into MapReduce code and is implemented on database.

· And also RDBMS uses SQL, PL/SQL and relational calculus for execution on data set

· RDMBSs are schema-on write

· RDBMS is structured data which store data's in the form of tables and we use SQL to retrive data.

· RDBMS as said my Vigneshwara is structured database used in traditional databases for the OLTP and OLAP purpose.

· HIVE only SQL is used

· Hive is schema-on read

SOLVED : Java - filepath - Invalid escape sequence

2015-05-21T15:56:00.001+05:30

It's a compile-time error, so it can't be to do with permissions etc.

The problem is that you're not escaping the backslashes. You need:

String filePath = request.getRealPath("\\\\10.0.1.18\\downloads\\upload)";

Then the contents of the string will be just

\\10.0.1.18\downloads\upload

This is exactly the same as in the first line you showed, where this:

String filePath = "D:\\location";

... will actually create a string with contents of:

D:\location

HBase Architecture : Write-Ahead Log

2015-05-20T17:34:00.002+05:30

WAL is the lifeline that is needed when disaster strikes, it records all changes to the data.So if the server crashes, it can effectively replay that log to get everything up to where the server should have been just before the crash. It also means that if writing the record to the WAL fails, the whole operation must be considered a failure

HLOG

The class that implements the WAL is called HLog.

Versioning in hbase

2015-05-20T16:31:00.003+05:30

Create table

create 'htab48','colfam1'

htab48 = table name
colfam1 = column name

put 'htab48', 'row1', 'colfam1:mapreduce','mapreduce is ment for parallel processing'
put 'htab48', 'row1', 'colfam1:hdfs','hdfs is ment for storage'

The beauty of hbase is it is versioned datastore

put 'htab48', 'row1', 'colfam1:hdfs','hdfs1 is ment for storage'

but it is not not to work first disable table then give version to column

disable 'htab48'

alter 'htab48', NAME=>'colfam1',VERSIONS=>3

put 'htab48', 'row1', 'colfam1:hdfs1','hdfs1 is ment for storage'
put 'htab48', 'row1', 'colfam1:hdfs2','hdfs1 is ment for storage'

get 'htab48' ,'row1',{COLUMN=>'colfam1:hdfs',VERSIONS=>3}

COLUMN CELL
colfam1:hdfs timestamp=1234567876543 , value=hdfs is ment for storage
colfam1:hdfs timestamp=1234467876543 , value=hdfs1 is ment for storage

Chapter 03 : HBASE ARCHITECHTURE

2015-05-19T11:38:00.001+05:30

Chapter2 :Column Oriented database

2015-05-19T11:32:00.002+05:30

Column Oriented database

RDBMS

C1,C2,C3 = Columns

R1,R2,R3 = Rows

C1 C2 C3

A	100	HYD
B	200	BANG
C	300	Delhi
D	400	PUNE

R1 R2 R3

A	B	C	D
100	200	300	400
HYD	BANG	Delhi	PUNE

Here rows converted into column , column converted into rows......just to understand.

Timestamp will be provided by framework for each value.The reason to store values on column basis instead is based on the assumption is for specific queries not all the values needed. This is by it is analytical database and good for storing schema.

Directly fetching values no need of sequential accessing like RDBMS


	A1000

Chapter : 1 Hbase introduction

2015-05-19T11:08:00.002+05:30

Hbase Introduction

Hbase is one of the component of hadoop built on top of HDFS and is ment for Real time random read/write. In against to the sequential file accessing of HDFS.

Hbase is opensource , distributed, scalable , fault tolerance , non -relational, multi-dimensional versioned, column oriented database which is built on top of BIG TABLES (BUILT ON THE ORIGIN OF GFS).

Google big tables ment for structured data only.

All the database will be organized by the means of tables only and whenever we inserting values in Hbase table for each and every values there will be a unique timestamp which will be generated by HBASE TABLES automatically.

HBASE is not a [part of default installatipon of hadoop and hence therefore we need to install it on top of hadoop only.

IN SVN when we insert data and from other location B modified it it creates version.

SVN

Input.log

Input_1.log

A has saved it =Input.log

Some modification done = SVN Framework generated another copy Input_1.log

Timestamp :- its a row and column orientesaction point where data is stored, multiple data can be stored on same timestamp.

APACHE PIG BUILT IN TRANSFORMATION

2015-05-13T16:30:00.006+05:30

COGROUP

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = COGROUP A by a1,B by a4;
Dump C;

Group BY

A = load 'data3' using PigStorage(' ') as (a1:chararray,a2:int,a3:int);
B = Group A by a2;
C = foreach B generate group,COUNT(A);
Dump C;

CROSS

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = CROSS A,B;
Dump C;

DISTINCT

A = load 'data7' using PigStorage(' ') as (a1:int,a2:int,a3:int);
B = DISTINCT A;
Dump B;

Filter

A = load 'data7' using PigStorage(' ') as (a1:int,a2:int,a3:int);
B = Filter A by a2==5;
Dump B;

COGROUP

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = COGROUP A by a1,B by a4;
D = Foreach C generate group,FLATTEN (A);
Dump D;

A = load 'data3' using PigStorage(' ') as (a1:chararray,a2:int,a3:int);
B = Group A by a2;
Describe B;
Dump B;
C = foreach B generate group,Count(A);
Dump C;

JOIN

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = JOIN A by a1,B by a4;
Dump C;

LIMIT

A = load 'data6' using PigStorage(' ') as (a1:int,a2:int,a3:int,a4:int,a5:int);
B = LIMIT A 3;
Dump B;

Order by

A = load 'data6' using PigStorage(' ') as (a1:int,a2:int,a3:int,a4:int,a5:int);
B = Order A by $1;
Dump B;

SPLIT

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
SPLIT A into B if a1==2, C if a1==3;
Dump B;
Dump C;

UNION

A = load 'data4' using PigStorage(' ') as (a1:int,a2:int);
B = load 'data5' using PigStorage(' ') as (a3:int,a4:int);
C = UNION A,B;
Dump C;

Managed Vs ExternalTables

2015-05-13T15:44:00.002+05:30

-A Hive table that's not external is called a managed table.

-One of the main differences between an external and a managed table in Hive is that when an external table is dropped,

the data associated with it doesn't get deleted, only the metadata (number of columns, type of columns, terminators,

etc.) gets dropped from the Hive metastore. When a managed table gets dropped, both the metadata and data get dropped.

I have so far always preferred making tables external because if the schema of my Hive table changes,

I can just drop the external table and re-create another external table over the same HDFS data with the new schema.

However, most (if not all) of the changes to schema can now be made through ALTER TABLE or similar commands so my

recommendation/preference to use external tables over managed ones might be more of a legacy concern than a contemporary

one.

PARTITIONING vs BUCKTING-IN-HIVE

2015-05-13T15:42:00.004+05:30

Basically both Partitioning and Bucketing slice the data for executing the query much more efficiently than on the non-sliced data. The major difference is that the number of slices will keep on changing in the case of partitioning as data is modified, but with bucketing the number of slices are fixed which are specified while creating the table.

Bucketing happen by using a Hash algorithm and then a modulo on the number of buckets. So, a row might get inserted into any of the bucket. Bucketing can be used for sampling of data, as well also for joining two data sets much more effectively and much more.

Hive Performance tuning

2015-05-13T15:34:00.001+05:30

1) Where Time = int

When the time field within the where clause is specified query parser will automatically detect partition to be processed.

Please not it work with integer type not with float.

Syntax

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020 + 3600

[GOOD]: SELECT field1, field2, field3 FROM tbl WHERE time > 1349393020 – 3600

2) Don not Use Distinct with Count in hive basically it do process this kind of query Only one reducer is used.

Syntax : select count(Distinct Field1) from tablename;

Go for this

Select Count(1)

From ( Select Distinct Field1 from tablename ) t;

distribute by in hive

2015-05-13T15:14:00.002+05:30

DISTRIBUTE BY
=============
1. DISTRIBUTE BY controls how map output will be divided among Reducers.
2. In the below example DISTRIBUTE BY ensures that records of ename will go to the same reducer and then to SORT the same
   data the way we want (ascending order of empid).
3. DISTRIBUTE BY works similar to GROUP BY in the sence that how it controls the reducers to receives the rows for processing.

NOTE: Hive requires DISTRIBUTE BY clause comes BEFORE the SORT BY if we are using both in a query.
-----------------------------------
hive> select * from disttab;
OK
108   Sandeep     22000
109   Sandeep     22500
110   Sandeep     23500
111   Sandeep     24340
112   Karan 45000
113   Karan 45600
101   Ravi 46000
102   Prakash     34000
103   Murali      23000
104   Madan 45555
105   Kanth 56000
106   Varma 33333
Time taken: 0.409 seconds
hive>
-------------------------------------
hive> select empid , ename , esal from disttab DISTRIBUTE BY ename SORT BY ename asc , esal asc;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201304160610_0008, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201304160610_0008
Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201304160610_0008
2013-04-16 07:05:36,346 Stage-1 map = 0%, reduce = 0%
2013-04-16 07:05:40,480 Stage-1 map = 100%, reduce = 0%
2013-04-16 07:05:49,639 Stage-1 map = 100%, reduce = 33%
2013-04-16 07:05:50,765 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201304160610_0008
OK
108   Sandeep     22000
109   Sandeep     22500
110   Sandeep     23500
111   Sandeep     24340
105   Kanth 56000
112   Karan 45000
113   Karan 45600
104   Madan 45555
103   Murali      23000
102   Prakash     34000
101   Ravi 46000
106   Varma 33333
Time taken: 24.721 seconds
hive>

hadoop hive cluster by

2015-05-13T15:12:00.000+05:30

CLUSTER BY
==========
1.CLUSTER BY clause is a short-hand way of expressing the DISTRIBUTE BY with SORT BY

PFB example how we have changed the way of DISTRIBUTE BY with SORT BY
---------------------------------------------------------------------

hive> select empid , ename , esal from disttab CLUSTER BY ename;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201304160610_0009, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201304160610_0009
Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201304160610_0009
2013-04-16 07:07:57,918 Stage-1 map = 0%, reduce = 0%
2013-04-16 07:08:05,362 Stage-1 map = 100%, reduce = 0%
2013-04-16 07:08:14,526 Stage-1 map = 100%, reduce = 33%
2013-04-16 07:08:15,535 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201304160610_0009
OK
NULL NULL NULL
108   Gopal 22000
109   Gopal 22500
110   Gopal 23500
111   Gopal 24340
105   Kanth 56000
113   Karan 45600
112   Karan 45000
104   Madan 45555
103   Murali      23000
102   Prakash     34000
101   Ravi 46000
106   Varma 33333
Time taken: 32.426 seconds
hive>

Hive VS Impala

2015-05-13T15:07:00.001+05:30

What is Impala?

Cloudera Impala is an open source Massively Parallel Processing (MPP) query engine that runs natively on Apache Hadoop. The Apache-licensed Impala project brings scalable parallel database technology to Hadoop, enabling users to issue low-latency SQL queries to data stored in HDFS and Apache HBase without requiring data movement or transformation

With Impala, analysts and data scientists now have the ability to perform real-time, “speed of thought” analytics on data stored in Hadoop via SQL or through Business Intelligence (BI) tools. The result is that large-scale data processing (via MapReduce) and interactive queries can be done on the same system using the same data and metadata – removing the need to migrate data sets into specialized systems and/or proprietary formats simply to perform analysis

How does impala provide faster query response compared to hive for the same data on hdfs ?

You should see Impala as "SQL on HDFS", while Hive is more "SQL on Hadoop".

In other words, Impala doesn't even use Hadoop at all. It simply has daemons running on all your nodes which cache some of the data that is in HDFS, so that these daemons can return data quickly without having to go through a whole Map/Reduce job.

The reason for this is that there is a certain overhead involved in running a Map/Reduce job, so by short-circuiting Map/Reduce altogether you can get some pretty big gain in runtime.

That being said, Impala does not replace Hive, it is good for very different use cases. Impala doesn't provide fault-tolerance compared to Hive, so if there is a problem during your query then it's gone. Definitely for ETL type of jobs where failure of one job would be costly I would recommend Hive, but Impala can be awesome for small ad-hoc queries, for example for data scientists or business analysts who just want to take a look and analyze some data without building robust jobs. Also from my personal experience, Impala is still not very mature, and I've seen some crashes sometimes when the amount of data is larger than available memory.

Impala Use Cases

What are good use cases for Impala as opposed to Hive or MapReduce?

Impala is well-suited to executing SQL queries for interactive exploratory analytics on large data sets. Hive and MapReduce are appropriate for very long running, batch-oriented tasks such as ETL.

Is MapReduce required for Impala? Will Impala continue to work as expected if MapReduce is stopped?

Impala does not use MapReduce at all.

Can Impala be used for complex event processing?

For example, in an industrial environment, many agents may generate large amounts of data. Can Impala be

used to analyze this data, checking for notable changes in the environment?

Complex Event Processing (CEP) is usually performed by dedicated stream-processing systems. Impala is not

a stream-processing system, as it most closely resembles a relational database.

Is Impala intended to handle real time queries in low-latency applications or is it for ad hoc queries for the purpose of data exploration?

Ad-hoc queries are the primary use case for Impala. We anticipate it being used in many other situations where

low-latency is required. Whether Impala is appropriate for any particular use-case depends on the workload,

data size and query volume.

Apache Hive : ORDER BY VS SORT BY

2015-05-13T15:00:00.001+05:30

ORDER BY VS SORT BY

===================

ORDER BY

--------

1. The ORDER BY clause is familar from other SQL dialects.

2. It performs the 'total ordering of the query result set' i.e. ' all the data will be passed to a single reducer' which

will take longer time if we have larger datasets.

3. Hive requires the LIMIT clause for ORDER BY as it will take longer time (if hives property

hive.mapred.mode is set to 'strict' which is by default 'nonstrict'

NOTE: In /usr/lib/hive/conf/hive-default.xml--> below property

<name>hive.mapred.mode</name>

<value>nonstrict</value>

<description>The mode in which the hive operations are being performed. In strict mode, some risky queries are not allowed to run</description>

</property>

SORT BY

-------

3. Where as SORT BY (Hive's alternative to ORDER BY) , will orders the data only within each reducer and performs the 'local ordering of the data' where each REDUCER'S OUTPUT will be SORTTED.

4. Better perfomance is traded for total ordering.

hive> select * from testudf;

100 Karan 24000

101 Gopal 25000

102 RaviKanth 26000

103 Ravi Prakash 270000

104 Gopi Prakash 28000

105 Ravi Prakash 29000

106 Vishnu Prakash 31000

107 Kasi Prakash 34000

Time taken: 0.419 seconds

hive> desc testudf;

empid int

ename string

esal int

Time taken: 0.197 seconds

hive> select empid , ename , esal from testudf order by empid asc , esal desc;

Total MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks determined at compile time: 1

In order to change the average load for a reducer (in bytes):

set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

set mapred.reduce.tasks=<number>

Starting Job = job_201304160610_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201304160610_0001

Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201304160610_0001

2013-04-16 06:15:57,723 Stage-1 map = 0%, reduce = 0%

2013-04-16 06:16:08,764 Stage-1 map = 100%, reduce = 0%

2013-04-16 06:16:18,977 Stage-1 map = 100%, reduce = 100%

Ended Job = job_201304160610_0001

100 Karan 24000

101 Gopal 25000

102 RaviKanth 26000

103 Ravi Prakash 270000

104 Gopi Prakash 28000

105 Ravi Prakash 29000

106 Vishnu Prakash 31000

107 Kasi Prakash 34000

Time taken: 36.859 seconds

hive> select empid , ename , esal from testudf sort by empid asc , esal desc;

Total MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks not specified. Estimated from input data size: 1

In order to change the average load for a reducer (in bytes):

set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

set mapred.reduce.tasks=<number>

Starting Job = job_201304160610_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201304160610_0002

Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201304160610_0002

2013-04-16 06:18:23,766 Stage-1 map = 0%, reduce = 0%

2013-04-16 06:18:31,316 Stage-1 map = 100%, reduce = 0%

2013-04-16 06:18:40,440 Stage-1 map = 100%, reduce = 33%

2013-04-16 06:18:41,451 Stage-1 map = 100%, reduce = 100%

Ended Job = job_201304160610_0002

100 Karan 24000

101 Gopal 25000

102 RaviKanth 26000

103 Ravi Prakash 270000

104 Gopi Prakash 28000

105 Ravi Prakash 29000

106 Vishnu Prakash 31000

107 Kasi Prakash 34000

Time taken: 28.06 seconds

hive>

failed with exception null hive

2015-05-11T18:26:00.003+05:30

LFS

When We are loading data from LFS to table then it will load on second try to load same data now data exist so it will create a copy new newfile suffixed by *_copy_1.

/user/hive/warehouse/databasename/tablename

HDFS

When We are loading data from HDFS to table then it will try -mv on second try to load same data now data exist so it will not perform -mv DUE to Write Once read many HDFS Feature and rename the source file.