The Adventure of "R"

S3 / Linux

2011-12-14T14:29:00.002+05:30

Here are the instructions to use s3cmd tool to access and upload data to your bucket on Amazon S3:

· Download s3cmd from: http://sourceforge.net/projects/s3tools/files/s3cmd/1.0.1/s3cmd-1.0.1.tar.gz/download

· Uncompress the file using the cmd: tar zxf s3cmd-1.0.1.tar.gz

· Go inside the directory: cd s3cmd-1.0.1

· First time configure it: ./s3cmd –configure

o Supply the values: access key, secret key
s3cmd mb s3://my-new-bucket-name (To Create Bucket)

· Run the command to list the contents of your bucket: ./s3cmd ls s3://yourbucket

· Put (write/deposit) a file as; ./s3cmd put <localfilename> s3://yourbucket/<remote-folder-optional>/<remote-file-name>

· Make sure there are no additional ‘/’ – because the S3 does not optimize them to single one.

DATAPIPE Combo -- 1

2011-12-09T12:00:00.000+05:30

Another one of the various datapipe combo available out there

AVRO

HADOOP--PIG--VOLDEMORT--SINATRA

JAVA,PIG,RUBY/JRUBY

http://datasyndrome.com/post/13707537045/booting-the-analytics-application-events-ruby

OOZIE HADOOP And STAGING DIR

2011-12-07T15:46:00.002+05:30

For Oozie to work properly /user/<user>/.staging needs 700 as permission.

If not the silent exceptions (like local exception in oozie log and FS permission exception in name node) will possibly waist a lot of your time

SVN Revert

2011-11-18T10:03:00.002+05:30

svn merge --dry-run -r 73:68
svn merge -r 73:68
svn commit -m "Reverted to revision 68."

Perl List INC Paths

2011-11-15T12:59:00.002+05:30

perl -e 'print join "\n", @INC'

List all Perl Modules

2011-11-14T12:12:00.002+05:30

instmodsh

Google Doodle : Freddie Mercury

2011-09-05T23:15:00.000+05:30

In spite of being such a heavy user of google I hardly had a chance to say thank you but the doodle today makes me to do so. So thank You Google.

http://www.youtube.com/watch?v=Xe0gIFxYhrk

PIG and Skewed JOIN Issue with Empty Files

2011-02-08T11:41:00.001+05:30

Infra

HADOOP 0.20.2

PIG 0.6

ISSUE

SKEWED JOIN on EMPTY FILES

RESULT

JOB FAILS

CONCLUSION

DON'T DO SKEWED JOIN IF INPUT DATA IS EMPTY if You Still want JOB Not to come in Failed JOB List

Recursively Delete A Directory

2011-02-08T11:36:00.003+05:30

Sweet and Powerful. Isn't It

rm -rf `find . -type d -name <dir>`

Input Params for Pig Script

2010-11-22T14:37:00.008+05:30

A way to list all the input parameters of a pig script

open FILE, "<", "$pigfile" or die $!;
#GOING through the file to list out Parameters#
while (my $line = <file>) {
      #Only taking non commented line and content inside single quotes#
      if($line !~ /--/ && $line =~ /(')\$(.+?)(\1)/){
           print("\n$2");
}
close(FILE);

GIft YourSelf A Road Mishap

2010-11-17T09:30:00.004+05:30

Here is the ALGORITHM for getting yourself a perfect GIFT.

1) Pick out a beautiful Day for a ride. When the weather is perfect with a little rain before

2) It helps if you pick out a day after the night when you had a fight with your affable ones. (You can blame them or get enlightened of your mistake)

3) Pick out the day which is officially a holiday

4) In spite of the day being a holiday choose your destination as your office.

5) Enjoy the ride towards your office and try to filter away your negativity.

6) That's it, let the COSMOS do the rest.

:)

Not possible

Here is what will Happen

7) For a change you will enjoy the weather.

8) For a change you shall cruise

9) For a change you shall not find any traffic

10) Then you will have to halt for some bugger in front of you finds the need to do so.

11) Rest leave it to the people on your behind to not find the need to do the same

12) Now enjoy the beautiful feeling of kissing the tar.

13) Enjoy even more the feeling of nakedness of being in a tee and jeans realising the one who desired to ride over you is well dressed with jacket and gloves .

14) Enjoy the fact the day is cooler than the rest, hence intensity is more than usual.

SO ENJOY and GIFT Yourself Something Different today

Diff of two DIR

2010-11-15T11:33:00.003+05:30

Love this. Amazingly powerful

diff --brief --recursive <dir1> <dir2>

Hadoop 0.21, Pig 0.7, Hive 0.4.1, Sqoop 1.10, Hadoop-LZO, Elephant-Bird

2010-11-03T20:57:00.005+05:30

Strangely but truly a lot of time has gone by when the last official hadoop release was made.

Unfortunately Hadoop 0.21 never became an official release but a release candidate was made by Tom White and co.

Felt happy to hack my way through along with a friend to make various software work with Hadoop 0.21

1) Hive 0.4.1
https://github.com/hirohanin/hive0.4.1hadoop21

2) Sqoop 1.1.0
https://github.com/hirohanin/sqoop-1.1.0hadoop21

3) Hadoop-LZO
https://github.com/hirohanin/hadoop-lzo-hadoop21

4) Elephant Bird
https://github.com/hirohanin/elephant-birdpig7hadoop21

5) Pig 0.7

https://github.com/hirohanin/pig7hadoop21

Disclaimer: The software are unofficial and unsupported, unless some one gives a better reason to do so.

Apache Tika

2010-10-16T11:06:00.004+05:30

I was amazed with the kind of extraction facility has been made available in open source world. Apache Tika has has gone a long way to help developers come with better crawlers. I was happy to use it in a project

HADOOP+PIG+HIVE

2010-10-16T10:36:00.003+05:30

Its been fun working on humongous data with an abstraction created by the realm of different projects under hadoop. PIG, HIVE being the query language which provides even better idiots layers on top it. Aspects of keeping a cluster running, with managing data sizes with various compression initially driven by LZO (elephant-bird) drifting towards block compression. Making ones life easier by using scheduler like Azkaban. Humongous data and heavy loading, thats the way to go

Murder + Capistrano

2010-10-04T11:34:00.002+05:30

Deployments are one painful task in any production environment and it shows its ugly face in a resource heavy environment. PUPPET for OS centric deb/config could be useful. Recently got a chance to use a combo of capistrano and murder. Murder is a beautiful concept of torrent inspired production deploy and capistrano brings in the feature of other abstracted functionalities.

In the end I wonder why so many deployment tools is in RUBY or support a tool written in ruby by providing a gem of itself

Ant , Maven, Ivy

2010-10-02T12:10:00.007+05:30

Moving from the trustworthy ant to abstracted maven. The only thing I realised, unless applied company wide maven becomes hard although its usp is well established. Time to dibble with ivy for it promises a lot with transitive dependencies.

One interesting realisation in maven that one can fake the repo if you have a local set of jars in your machine (for the jars not present in repo)

<dependency>
      <groupid>...</groupid>
      <artifactid>....</artifactid>
      <version>...</version>
      <scope>system</scope>
      <systempath>/absolute path to lib/XXX.jar</systempath>
</dependency>

And I am inclining more and more towards a combo of ant + ivy. In my view and use case and even generally it seems most promising.

JAVACC

2010-10-02T02:39:00.000+05:30

http://en.wikipedia.org/wiki/JavaCC

As it turns out, I finally get to use it in a system. Not bad at all

Elephant Bird with PIG 0.7

2010-09-29T09:34:00.002+05:30

A Fork on twitter's elephant-bird which is an untested version compatible with PIG 0.7 for LZO compression

http://github.com/hirohanin/elephant-bird

DROP ALL TABLES : MYSQL

2010-09-29T09:27:00.004+05:30

One of the ways

mysqldump -u<user> -p<pwd> --add-drop-table --no-data <db> | grep ^DROP | mysql -u<user> -p<pwd> <db>

The Game : Neil Strauss

2010-08-02T02:02:00.004+05:30

I have no clue what to make out of this book. Except it formed an easy read, probably very engrossing for it dwells in common insecurities and desires.

BOAT

2010-07-30T00:57:00.000+05:30

Some say heart is just like a wheel..
If It is so..
Then at the moment its spinning aimlessly..
Dangerously rocking the boat of my life..
Get hold of it my gravest and trustworthy captain..
Before it sinks into the oblivion of the deep blue sea..

A Moment

2010-07-30T00:55:00.000+05:30

Strange are the ways of human thoughts..
Stranger are the illusion it creates..
The unanswered calls which puts me down..
Sometimes uplifts me with strangest hope..
Only for a moment..
But its a moment when the clock stops ticking..
The seconds between two consecutive rings..
My heart stops beating..
Where my mind tells me you picked up the call..
Ecstatic is what I am..
Happiness takes over me..
Only for a moment, If for a moment.

Only if Known was a Known

2010-07-28T14:15:00.001+05:30

Waiting for months for a response..
Its hard to remain in a stoic pose..
Empty thoughts and empty response..
Leads one to misleading unknowns..
Never knew my wrong were so wrong..
Only if I knew the doer that doeth..
Only if the knowledge was profound...
The blindness which overwhelms..
Darkness that over cometh..
Makes the default at every moment..
Make'th the heart default with every action..
Only if the known was a KNOWN..

Where ??

2010-07-27T23:29:00.000+05:30

My eyes seek you..
My Mind seeks you..
My soul seeks you..
I am so much over the edge..
Where are you..
Why do you run..
Why don't you communicate..