BigDataNoob

2015-09-12T23:17:00.000-07:00

Brief Overview of HBase, its architecture, components, and data model

Introduction To HBase from Anil Gupta

Connect Phoenix to Secure HBase Cluster

2013-09-14T13:32:00.000-07:00

Blogpost Update(Dec 21, 2015) :

In Phoenix3.0+ we added the feature to natively connect to secure HBase cluster. Following is the JDBC connection URL format:
jdbc:phoenix:<quorom>:<port>:[zk_rootNode]:<principal>:<keytab>

So, users just need to use above JDBC connection string while connecting to secure cluster via sqlline, java code, or your favorite sql browser.

Still need more info?
Have a look at https://issues.apache.org/jira/browse/PHOENIX-19 for implementation details.

-------------------------------------------------------------------------------------------------------------------------

Steps for connecting Phoenix to a secure cluster prior to Phoenix3.0 release

By default, Phoenix distribution is unable to connect to secure cluster. But, Phoenix can connect to a secure HBase cluster by modifying sqlline.sh script and setting up some more java parameters. Great thing about this solution is that no java code modification is required.
So, it hardly takes 5 minutes. :)

Following are the steps required to do the set-up:
1. Create jaas.conf file with following content:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
};
2. Modify sqlline.sh script to add -Djava.security.auth.login.config=<path to jaas.conf file> property.
3. Modify sqlline.sh script to add to following folder/jars in classpath of Phoenix(add them in order of this list):

Folder where hadoop configuration files are present
Folder where hbase configuration files are present
hbase jar
hadoop common jar
zookeeper jar
hadoop auth jar
phoenix-x.x.x.jar

Your resultant java command in sqlline.sh script should look somewhat like this:
java -Djava.security.auth.login.config=/abcd/conf/jaas.conf -cp ".:/abcd/hadoop/conf:/abcd/hbase/conf:/cloudera/parcels/CDH/lib/hbase/hbase-0.94.6-cdh4.3.0-security.jar:/cloudera/parcels/CDH/lib/hadoop/hadoop-common-2.0.0-cdh4.3.0.jar:/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/zookeeper/zookeeper-3.4.5-cdh4.3.0.jar:/cloudera/parcels/CDH/lib/hadoop/hadoop-auth-2.0.0-cdh4.3.0.jar:/abcd/phoenix-2.0.0.jar:$phoenix_client_jar" -Dlog4j.configuration=file:$current_dir/log4j.properties sqlline.SqlLine -d com.salesforce.phoenix.jdbc.PhoenixDriver -u jdbc:phoenix:$1 -n none -p none --color=true --fastConnect=false --silent=true --isolation=TRANSACTION_READ_COMMITTED $sqlfile

How to run:
1. Run kinit command on shell to get the kerberos ticket.
2. From the same shell invoke sqlline.sh <zookeeper_quorum>

Please note that i have tested this functionality with cdh4.3(HBase0.94.6) and Phoenix2.0.0. This should work with other HBase distros like Apache, Hortonworks, etc.

Have fun with HBase sql interface on secure cluster!

My Random findings in Hadoop/HBase

2013-01-31T21:53:00.002-08:00

In this post i list my random findings on Hadoop. These facts or behavior are usually not highlighted/documented in Hadoop so it requires debugging or code walk through.

My Random Findings in Hadoop/HBase:
1. MapReduce job ignores the input files that starts with a underscore("_") character. So, please avoid "_" as prefix in input file names for MapReduce job.
2. Default separator between <key,value> in TextOutputFormat is "\t". Separator can be changed by setting the property "mapreduce.output.textoutputformat.separator".

HBase vs Cassandra

2012-11-30T12:19:00.003-08:00

Apache HBase vs Apache Cassandra

This comparative study was done by me and Larry Thomas in May, 2012. Cassandra stuff was prepared by Larry Thomas.
This information is NOT intended to be a tutorial for either Apache Cassandra or Apache HBase. We tried our level best to provide the most accurate information. Please comment or email me if you find any corrections. I would be happy to maintain this list with the most accurate and updated information.

Point

HBase

Cassandra

Foundations

HBase is based on BigTable (Google)

Cassandra is based on DynamoDB (Amazon). Initially developed at Facebook by former Amazon engineers. This is one reason why Cassandra supports multi data center. Rackspace is a big contributor to Cassandra due to multi data center support.

Infrastructure

HBase uses the Hadoop Infrastructure (Zookeeper, NameNode, HDFS). Organizations that will deploy Hadoop anyway may be comfortable with leveraging Hadoop knowledge by using HBase

Cassandra started and evolved separate from Hadoop and its infrastructure and Operational knowledge requirements are different than Hadoop. However, for analytics, many Cassandra deployments use Cassandra + Storm (which uses Zookeeper), and/or Cassandra + Hadoop.

Infrastructure Simplicity and SPOF

The HBase-Hadoop Infrastructure has several "moving parts" consisting of Zookeeper, Name Node, Hbase Master, and Data Nodes, Zookeeper is clustered and naturally fault tolerant. Name Node needs to be clustered to be fault tolerant.

Cassandra uses a a single Node-type. All nodes are equal and perform all functions. Any Node can act as a coordinator, ensuring no SPOF. Adding Storm or Hadoop, of course, adds complexity to the infrastructure.

Read Intensive Use Cases

HBase is optimized for reads, supported by single-write master, and resulting strict consistency model, as well as use of Ordered Partitioning which supports row-scans. HBase is well suited for doing Range based scans.

Cassandra has excellent single-row read performance as long as eventual consistency semantics are sufficient for the use-case. Cassandra quorum reads, which are required for strict consistency will naturally be slower than Hbase reads. Cassandra does not support Range based row-scans which may be limiting in certain use-cases. Cassandra is well suited for supporting single-row queries, or selecting multiple rows based on a Column-Value index.

Multi-Data Center Support and Disaster Recovery

HBase provides for asynchronous replication of an HBase Cluster across a WAN. HBase clusters cannot be set up to achieve zero RPO, but in steady-state HBase should be roughly failover-equivalent to any other DBMS that relies on asynchronous replication over a WAN. Fall-back processes and procedures (e.g. after failover) are TBD.

Cassandra Random Partitioning provides for row-replication of a single row across a WAN, either asynchronous (write.ONE, write.LOCAL_QUORUM), or synchronous (write.QUORUM, write.ALL). Cassandra clusters can therefore be set up to achieve zero RPO, but each write will require at least one wan-ACK back to the coordinator to achieve this capability.

Write.ONE Durability

Writes are replicated in a pipeline fashion: the first-data-node for the region persists the write, and then sends the write to the next Natural Endpoint, and so-on in a pipeline fashion. HBase’s commit log "acks" a write only after *all* of the nodes in the pipeline have written the data to their OS buffers. The first Region Server in the pipeline must also have persisted the write to its WAL.

Cassandra's coordinators will send parallel write-requests to all Natural Endpoints, The coordinator will "ack" the write after exactly one Natural Endpoint has "acked" the write, which means that node has also persisted the write to its WAL. The writes may or may not have committed to any other Natural Endpoint.

Ordered Partitioning

HBase only supports Ordered Partitoning. This means that Rows for a CF are stored in RowKey order in HFiles, where each Hfile contains a "block" or "shard" of all the rows in a CF. HFiles are distributed across all data-nodes in the Cluster

Cassandra officially supports Ordered Partitioning, but no production user of Cassandra uses Ordered Partitioning due to the "hot spots" it creates and the operational difficulties such hot-spots cause. Random Partitioning is the only recommended Cassandra partitioning scheme, and rows are distributed across all nodes in the cluster.

RowKey Range Scans

Because of ordered partitioning, HBase queries can be formulated with partial start and end row-keys, and can locate rows inclusive-of, or exclusive of these partial-rowkeys. The start and end row-keys in a range-scan need not even exist in Hbase.

Because of random partitioning, partial rowkeys cannot be used with Cassandra. RowKeys must be known exactly. Counting rows in a CF is complicated. It is highly recommended that for these types of use-cases, data should be stored in columns in Cassandra, not in rows.

Linear Scalability for large tables and range scans

Due to Ordered Partitioning, HBase will easily scale horizontally while still supporting rowkey range scans.

If data is stored in columns in Cassandra to support range scans, the practical limitation of a row size in Cassandra is 10's of Megabytes. Rows larger than that causes problems with compaction overhead and time.

Atomic Compare and Set

HBase supports Atomic Compare and Set. HBase supports supports transaction within a Row.

Cassandra does not support Atomic Compare and Set. Counters require dedicated counter column-families which because of eventual-consistency requires that all replicas in all natural end-points be read and updated with ACK. However, hinted-handoff mechanisms can make even these built-in counters suspect for accuracy. FIFO queues are difficult (if not impossible) to implement with Cassandra.

Read Load Balancing - single Row

Hbase does not support Read Load Balancing against a single row. A single row is served by exactly one region server at a time. Other replicas are used ony in case of a node failure. Scalability is primarily supported by Partitioning which statistically distributes reads of different rows across multiple data nodes.

Cassandra will support Read Load Balancing against a single row. However, this is primarily supported by Read.ONE, and eventual consistency must be taken into consideration. Scalability is primarily supported by Partitioning which distributes reads of different rows across multiple data nodes.

Bloom Filters

Bloom Filters can be used in HBase as another form of Indexing. They work on the basis of RowKey or RowKey+ColumnName to reduce the number of data-blocks that HBase has to read to satisfy a query. (Bloom Filters may exhibit false-positives (reading too much data), but never false negatives (reading not enough data).

Cassandra uses bloom filters for key lookup.

Triggers

Triggers are supported by the CoProcessor capability in HBase. They allow HBase to observe the get/put/delete events on a table (CF), and then execute the trigger-logic. Triggers are coded as java classes.

Cassandra does not support co-processor-like functionality (as far as we know)

Secondary Indexes

Hbase does not natively support secondary indexes, but one use-case of Triggers is that a trigger on a "put" can automatically keep a secondary index up-to-date, and therefore not put the burden on the application (client).

Cassandra supports secondary indexes on column families where the column name is known. (Not on dynamic columns).

Simple Aggregation

Hbase CoProcessors support out-of-the-box simple aggregations in HBase. SUM, MIN, MAX, AVG, STD. Other aggregations can be built by defining java-classes to perform the aggregation

Aggregations in Cassandra are not supported by the Cassandra nodes - client must provide aggregations. When the aggregation requirement spans multiple rows, Random Partitioning makes aggregations very difficult for the client. Recommendation is to use Storm or Hadoop for aggregations.

HIVE Integration

HIVE can access HBase tables directly (uses de-serialization under the hood that is aware of the HBase file format).

Work in Progress (https://issues.apache.org/jira/browse/CASSANDRA-4131)

PIG Integration

PIG has native support for writing into/reading from HBase.

Cassandra 0.7.4+

Point

HBase

Cassandra

CAP Theorem Focus	Consistency, Availability	Availability, Partition-Tolerance
Consistency	Strong	Eventual (Strong is Optional)
Single Write Master	Yes	No (R+W+1 to get Strong Consistency)
Optimized For	Reads	Writes
Main Data Structure	CF, RowKey, Name Value Pair Set	CF, RowKey, Name Value Pair Set
Dynamic Columns	Yes	Yes
Column Names as Data	Yes	Yes
Static Columns	No	Yes
RowKey Slices	Yes	No
Static Column Value Indexes	No	Yes
Sorted Column Names	Yes	Yes
Cell Versioning Support	Yes	No
Snapshots	Yes	Yes
Bloom Filters	Yes	Yes(only on Key)
CoProcessors	Yes	No
Triggers	Yes(Part of Coprocessor)	No
Push Down Predicates	Yes(Part of Coprocessor)	No
Atomic Compare and Set	Yes	No
Explicit Row Locks	Yes	No
Row Key Caching	Yes	Yes
Partitioning Strategy	Ordered Partitioning	Random Partitioning recommended
Rebalancing	Automatic	Not Needed with Random Partitioning
Availability	N-Replicas across Nodes	N-Replicas across Nodes
Data Node Failure	Graceful Degredation	Graceful Degredation
Data Node Failure - Replication	N-Replicas Preserved	(N-1) Replicas Preserved + Hinted Handoff
Data Node Restoration	Same as Node Addition	Requires Node Repair Admin-action
Data Node Addition	Rebalancing Automatic	Rebalancing Requires Token-Assignment Adjustment
Data Node Management	Simple (Roll In, Role Out)	Human Admin Action Required
Cluster Admin Nodes	Zookeeper, NameNode, HMaster	All Nodes are Equal
SPOF	Now, all the Admin Nodes are Fault Tolerant	All Nodes are Equal
Write.ANY	No, but Replicas are Node Agnostic	Yes (Writes Never Fail if this option is used)
Write.ONE	Standard, HA, Strong Consistency	Yes (often used), HA, Weak Consistency
Write.QUORUM	No (not required)	Yes (often used with Read.QUORUM for Strong Consistency
Write.ALL	Yes (performance penalty)	Yes (performance penalty, not HA)
Asynchronous WAN Replication	Yes, but it needs testing on corner cases.	Yes (Replica's can span data centers)
Synchronous WAN Replication	No	Yes with Write.QUORUM or Write.EACH-QUORUM
Compression Support	Yes	Yes

Store data in HFileV1 format in HBase 0.92 and further releases

2012-10-06T20:05:00.001-07:00

Till HBase0.90, data in HBase was stored in HFileV1 format. If you happen to upgrade the cluster to HBase 0.92 and further releases then by default HBase will store the data in HFileV2 instead of HFileV1.

If you would like to store the data in HFileV1 in HBase 0.92 and releases after that, then set the following property in hbase-site.xml file:

<property>
<name>hfile.format.version</name>
<value>1</value>
</property>

Once you change this property then you need disable all the table that are stored in HFileV2 format, if there are any. The only caveat is that at a given point of time HBase can use only use one file format to read the tables.

If you would like to revert back to using HFileV2 then:
1. Change the value of "hfile.format.version" to "2"
2. Disable all the tables that are using HFileV1 format and enable all the tables using HFileV2 format.
3. Restart the cluster.

Bulk Load a CSV file into HBase

2012-03-18T14:55:00.003-07:00

By default the bulk loader class ImpotTsv in HBase imports a tab separated files. Files with other separator like ',' ,'|' ,etc can be imported by using "-Dimporttsv.separator" option at the command line.At present, only single character separators are supported by Bulk Loader.

One of the things missing from the Hbase Bulk Load documentation is that all the option starting from "-D" are mapreduce job configuration parameters and all of them should be specified after "importtsv" and before Tablename. If this convention is not followed then job configuration wont be set as per user expectation because the genericoptionsparser method in HBase BulkLoading module is called in such a way that it stops parsing the command line parameters once it encounters the first non "-D" option.

For bulk loading a CSV file into HBase use the following command:
hadoop jar <path to hbase jar> importtsv -Dimporttsv.columns=a,b,c '-Dimporttsv.separator=,' <tablename> <inputdir>

Start Cloudera Hadoop/Hbase cluster

2012-02-15T11:52:00.000-08:00

Once you are done with installing and configuring the Cloudera Hadoop, Hbase and Zookeeper. Here are the following steps required to start the Hadoop/Hbase cluster.

This post might not hold good for Hadoop Yarn i.e.Hadoop1.0.0 and later versions.

Bringing-up the Cloudera Hadoop/Hbase Cluster
NOTE: Step 1-7 are enough to start a Hadoop cluster. Step 8-11 are only required when the cluster needs to run Hbase also.

Format the namenode by running: sudo -u hdfs hadoop namenode -format
Start the namenode service.
Start the datanode service on all the slaves.
If user intends to run map-reduce job on cluster then create and configure the mapred.system.dir directory in HDFS.
The mapred.system.dir parameter present in mapred-site.xml is represented by the following /mapred/system path example:

$ sudo -u hdfs hadoop fs -mkdir /mapred/system
$ sudo -u hdfs hadoop fs -chown mapred:hadoop /mapred/system
Set the correct owner and permissions of the mapred.system.dir directory in HDFS:

Directory Owner Permissions

mapred.system.dir mapred:hadoop drwx------

/ (root directory) hdfs:hadoop drwxr-xr-x
If step 3 is performed then start the Jobtracker.
If step 3 is performed then start the tasktracker on nodes which are supposed to partcipate in Hadoop(map-reduce) jobs.
Create and set correct permission on the hbase.rootdir directory in HDFS. hbase.rootdir is set as hdfs://namenode_hostname/hbase for this example:

$ sudo -u hdfs hadoop fs -mkdir /hbase
$ sudo -u hdfs hadoop fs -chown hbase /hbase
Start the servers present in zookeeper quorum.
Start the Hbase master service.
Start hbase-regionserver service on all the hosts which are supposed to store Hbase data.

Directory	Owner	Permissions
mapred.system.dir	mapred:hadoop	drwx------
/ (root directory)	hdfs:hadoop	drwxr-xr-x

upload/copy a file/folder on VMware ESXi server from local filesystem

2012-02-09T11:03:00.000-08:00

Upload/copy a file/folder on VMware ESXi 5.0 server from local filesystem.
Following steps need to be followed:

Login to ESXi server using VMware vSphere client.
Select ESXi server from the list on the left panel.
Go to Summary tab.
Right click and select “Browse Datastore” on any of the Storage device listed under Storage in Resources panel.
“Datastore Browser” window will pop-up. Upload the file on the Datastore by pressing the Upload button( 4^thbutton from left) on the toolbar.

HBase ERROR zookeeper.ZKConfig: no valid quorum servers found in zoo.cfg

2012-02-07T13:27:00.001-08:00

While running HBase if you get the following error message:
ERROR zookeeper.ZKConfig: no valid quorum servers found in zoo.cfg

Cause: The zoo.cfg file which contains the zookeeper configuration does not contains any valid entries for zookeeper server or the hostname provided for the servers are incorrect.

Resolution: Add the valid server.x values in the zoo.cfg and restart the Hbase cluster.

Here is a valid example:
server.0=zookeeper-host:2888:3888

Note: '0' in the "server.0" denotes the id of the zookeeper server present in the quorum. It should match with "myid" of the zookeeper server.

Run hdfs master( viz. namenode) with debug log level in hadoop

2012-01-05T11:49:00.000-08:00

Modify the log4j.properties which is present in your classpath. Usually this log4j.properties file is present in your conf folder of hadoop.

1.Add the following statement in log4j.properties file:
log4j.logger.org.apache.hadoop.hdfs=DEBUG

2. Restart or start the namenode.

Jobtracker dead but pid file exists message while checking the status of jobtracker service

2011-12-28T18:02:00.000-08:00

I am using the cloudera-hadoop 3 update 2 version(cdh3u2) on centos6.0. If you get the message"jobtracker dead but pid file exists" then the jobtracker service had an ungraceful exit and some clean up of files is required and you probably need to look at the log of jobtracker to investigate the problem.

Following is the command to check the status of jobtracker:
1. [host@jobtracker ~]$ sudo service hadoop-0.20-jobtracker status
jobtracker dead but pid file exists

2. Remove the file /var/run/hadoop-0.20/hadoop-hadoop-jobtracker.pid

3. After removing the file check the status of jobtracker service again:
[host@jobtracker init.d]$ sudo service hadoop-0.20-jobtracker status
jobtracker is stopped

If you get the above message then you are good to start the service again.

How to do contains only query in hive on a String column

2011-07-29T16:47:00.000-07:00

Suppose you have a flat file repository and you would like to project on the rows which contains only certain set of values in a multivalued column. Then the example below would come handy...

For the schema of student_classes table please refer to this previous post....
Objective: List the student who are only taking classes of Computer Science(cs) and Electrical Engineering(ee) departments.

Idea: Replace the dept code and course id with a blank using regexp_replace and after replacement if the length of resulting string is equal to zero then the student is only taking classes of the cs and ee dept.

Query: hive -e 'select * from student_classes where length(regexp_replace(classes,"cs:\\d{1,9},?|ee:\\d{1,9},?",""))= 0'

The regular expression above in the query is searching for more than one pattern in the column and every match is replaced by a blank.

Useful One Liners in Scripting Languages and Vi Editor

2011-07-19T12:14:00.000-07:00

1. Delete all blank lines {even lines which contains only white-space chars}
cat file_with_blank_line.txt | sed '/^[\t]*$/d' > file_without_blank_lines.txt

2. For each line, print all the non-empty field in tab separated file along with field number.
awk -F'\t' '{ buf=""; for(i=1;i<=NF;i++)if($i!="") buf=buf" ~||~ "i"=>"$i; print buf "\n"}' input_file.tsv

3. Count the number of rows in a file and group the rows on the basis of number of columns.
awk -F'\t' '{cnt[NF]++} END {for (n in cnt) print n ":" cnt[n]}' input.tsv

4. 0,$s/,//ng -- Vi command to count number of ',' in the entire file.

5. 0,$s/,//n -- Vi command to count number of lines with at least once ',' in file.

6. g/<regular expression>/d -- Vi command to search and delete the lines matching the regular expression.

7. g!/<regular expression>/d -- Vi command to search and delete the lines not matching the regular expression.

More to be added as i find more and more stuff related.... :)

Copy and uncompress file to HDFS without unziping the file on local filesystem

2011-07-15T15:38:00.000-07:00

If your file is in GB's then this command would certainly help to avoid out of space errors as there is no need to unzip the file on local filesystem.

put command in hadoop supports reading input from stdin. For reading the input from stdin use '-' as source file.

Compressed filename: compressed.tar.gz

gunzip -c compressed.tar.gz | hadoop fs -put - /user/files/uncompressed_data

Only Disadvantage: The only drawback of this approach is that in HDFS the data will be merged into a single file even though the local compressed file contains more than one file.

Use regexp_replace function of Hive to count number of values in a multivalued column

2011-07-14T21:04:00.000-07:00

Example: student_classes table has student_id(string), classes(string)
classes is a multivalued column consisting only of alpha-numeric and ':' character where ':' is the separator between department code and class_id in the corresponding department.
Sample row of Student_Classes: 234 CSE:1001,EE:2005,ME:1256

Objective: Find out the student id who are enrolled in more than 2 classes
Solution: Count the number of comma in the classes column. For this, replace every character other than ',' by blank and check if the length of remaining string is greater than 1.
Here is the Hive query:
hive -e 'select * from student_classes where length(regexp_replace(classes,"\\w*:?","")) > 1'

This method can be pretty handy to leverage the power of NoSql paradigm and do processing on multivalued column in a flat file repository.

replace newline or tab with comma in vi editor

2011-06-30T16:53:00.000-07:00

1. Open the file in Vi editor
2. Switch to command mode.
3. To replace all newline characters with comma use ":1,$s/\n/,/g"
If you want to replace all the tab characters in file with comma then instead of the above command use this => :1,$s/(Press CTRL+v) then (Press TAB)/,/g
4. Press Enter

Using find_in_set to emulate functionality of IN Clause of SQL in Hive

2011-06-28T17:11:00.000-07:00

Like me, if you are still forced to use Hive0.5 due to business decisions then here is a cool trick to simulate the functionality of "IN clause" of SQL for matching a set of values which are known beforehand.

int find_in_set(string str, string strList) : Returns the first occurance of str in strList where strList is a comma-delimited string. Returns null if either argument is null. Returns 0 if the first argument contains any commas. e.g. find_in_set('ab', 'abc,b,ab,c,def') returns 3 {PS: As of Hive 0.7}

Students table has id(string), name(string) and age(int) columns

Query in Hive to find specific student with their ID : hive -e 'select * from students where find_in_set(id, "2323, 4345,5674,6753,54654,234234")>0'

The above query will emulate the following in SQL:
select * from students where id in ('2323', '4345','5674','6753','54654','234234')

Stdout or cat a bz or any compressed file from HDFS

2011-06-24T17:13:00.000-07:00

hadoop fs -cat /usr/data/list.bz | bzcat

Similarly, for other compressed file : use the corresponding cat command.
For example use if it's gz file then use zcat.

Grepping a tab character in file

2011-06-24T17:04:00.000-07:00

For grepping a tab character press Ctrl+v and then press Tab.

example: grep ' ' test.txt

Ctrl+v needs to be pressed after the opening single quote char.