Kaniyarasu's Weblog

Yusuf Pathan Obstructing the field

2013-05-15T07:16:00.000-07:00

Today in KKR Vs PWI, Yusuf pathan obstructs the field by kicking the balls with his legs.
PWI asked for out and wow umpire gave it which gave the victory exactly after a month.

Apache JMeter - First Step

2013-05-15T04:16:00.000-07:00

Load testing is aimed at putting demand on a system or device and measuring its response. It is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation.

The Apache JMeter desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance.

Setup
As simple as this - Download the zip and play with that
JMeter needs JVM 1.5 or Higher to be installed in the system
Download the Zip from Apache JMeter site - jmeter.apache.org/‎
Extract the zip file
To run the GUI goto JMeter_Home/bin
Execute the jmeter file

- For windows machines, Double click on the jmeter.bat file
- For Linux/mac machines, In the Bash Shell go to Jmeter_home/bin location, Just type jmeter and execute the command
Sometimes during the test run you may get into a familiar exception called as "Out of Memory Exception" for that you may need to run the jmeter with the jar and the possible actions as
$ java -Xms128m -Xmx1028m -jar apacheJMeter.jar
Xms - Specifies the Min heap size
Xmx - Specifies the Max heap size
Integrating JMeter Plugins
JMeter plugins are very useful in several ways, Provides options for making a request with some aditional options, It improves better representation of results and many more

Download the Jmeter Plugins from http://code.google.com/p/jmeter-plugins/wiki/JMeterPluginsCMD
Extract the Zip file
Copy JMeterPlugins.jar and paste it into JMeter_Home/lib/ext/ Directory
Copy CMDRunner.jar and paste it into JMeter_Home/lib/ext/ Directory
This Plugin is to have additional graphs - result representations
Re-Launch the Jmeter

OAuth Plugin
For some of the APIs we need to pass the secret keys to get the response, for that we need to add OAuth plugin to make the request with the secret keys
Download this oauth-plugin for jmeter - http://code.google.com/p/jmeter-oauth/
Add the jar file to JMeter_Home/lib/ext Directory
To add a OAuth Request just Choose Add -> Sampler -> Oauth Request

L&P Testing - Making it effective and simpler
Here are the some additional and structured ways to make the performance test simpler, easier in effective manner
1. Integrating with Ant
2. Adding the build jobs in hudson/jenkins or any other opensource build tools

Integrating with Ant
Setup
Download apache ant from ant.apache.org/‎
Extract the Zip file

There is an excellent article on integrating ant with jmeter http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php

Download the ant-jmeter.jar mentioned in above article
Add the jar to ANT_HOME/lib to include ant-jmeter.jar in classpath
Add the jar also into the JMeter_Home/lib/ext Directory

Once done with the ant setup go for integrating the build process in any one of the opensource build management tools

Server side monitoring

Though JMeter has in-built server monitoring tool perfMon, we can go for other better ways to monitor the servers performance

There are numerous opensource tools available for monitoring the server like dstat, Zenoss, nagios, ganglia
dstat is the one which is easier & simpler to setup, It gives the accurate server metrics
dstat is a real-time server monitoring tool
There are options to capture the real-time resutls in a csv file. We can also upload the csv file in michenux.net to see the results in colorful graphs

Installing and configuring Dropbox on Linux server

2013-05-15T00:31:00.000-07:00

Installation

The Dropbox daemon works fine on all 32-bit and 64-bit Linux servers. To install, run the following command in your Linux terminal.

32-bit:

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86" | tar xzf -

64-bit:

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

Next, run the Dropbox daemon from the newly created .dropbox-dist folder.

~/.dropbox-dist/dropboxd

If you're running Dropbox on your server for the first time, you'll be asked to copy and paste a link in a working browser to create a new account or add your server to an existing account. Once you do, your Dropbox folder will be created in your home directory.

Problem I Faced

I've got a Linux server running in Amazon AWS and OS is on a 10GB drive and I use a Amazon EBS drive attached for storage.

Now I wanted to set up Dropbox on this machine. But I haven't synched yet because there won't be enough room in my home dir. My /home is on my 10GB drive since I never use it and all my data is on the 2TB drive mounted in /media/.

What is the best way to set this up? Dropbox doesn't support moving the folder in Linux yet. I found a script to move the folder but it seems outdated. Perhaps I can use symlinks in some clever way? Or maybe move my /home folder to my SATA drive?

Solution

You gave the answer yourself, Use symlinks. Move your Dropbox folder to your HDD, then create a symlink in your home folder For example: ln -s /media/MyHDD/Dropbox /home/yourname/

Thanks for visting our blog.

Raja Rani - Official Teaser(Starring Arya and Nayanthara)

2013-05-14T07:08:00.000-07:00

Raja Rani is a upcoming tamil movie starring arya and nayanthara in the lead roles which is directed by a debutant Atlee kumar who works as assistant in Enthiran with Shankar.
This is the first official teaser released by the team. Hope it will be a entertaining love story.
Watch and Enjoy!

Want to shop responsibly? Log on to TheResPage

2013-05-13T02:27:00.000-07:00

The flourishing eCommerce industry in India has something in it for all concerned parties. Competitive prices, multiple choices and convenience for the shoppers, a great market for the retailers, booming investment opportunities for the funds and a good buck for associated service providers. TheResPage, which has recently stepped into this space, has gone out-of-the-box by including a social angle to online shopping.

Please Visit: therespage.com
Source: http://yourstory.in/2013/05/want-to-shop-responsibly-log-on-to-therespage/

Amazing Shots - 360 degree

2013-04-30T06:48:00.000-07:00

1. Lake Hillier, is a lake on Middle Island, the largest of the islands and islets that make up the Recherche Archipelago, Western Australia. The most notable feature of this lake is its pink color.

2. Sagano bamboo forest (located near Kyoto, Japan) is one of the most beautiful bamboo forests in the world.

3. This is Hala Fruit from Hawaii.

4. If Saturn were as close to Earth as the Moon is, this is how it would look like.

5. There's a butterfly (Glasswing Butterfly) with transparent wings.

6. The Tsunami Cloud !!! This incredible photograph was taken by Gary Brink at the Holland State Park in Michigan, USA. The incredible cloud formation formed on Lake Michigan just off the beach.

7. Amazing shot of lenticular clouds over Lake Crowley, California.

8. Mount Roraima one of the oldest geological formations on Earth!

9. Cotton Grass, Iceland.

10. "Blue Emperor Scorpion". One of the largest scorpion in the world.

Interesting but weird - Omar Borkan Al Gala : one of the three irresistibly handsome men who were deported

2013-04-29T06:11:00.000-07:00

If you thought that there’s no flip side to being impossibly handsome, think again

Three men from United Arab Emirates were attending the Jenadrivah Heritage & Culture Festival in Riyadh, the capital of Saudi Arabia. They were forcibly evicted from the festival by Saudi police officers. The grounds for this harsh move? The visitors were too handsome.

“A festival official said the three Emiratis were taken out on the grounds they are too handsome and that the Commission [for the Promotion of Virtue and Prevention of Vices] members feared female visitors could fall for them,” a Saudi Arabian newspaper Elaph reported.

After evicting them, the police immediately deported the trio back to Abu Dhabi, lest any Saudi women see them and find them too irresistible to not fall in love.

Saudi Arabia is extremely conservative and offers little liberty to its women citizens. The female population of the country is not permitted to talk to any men to whom they are not related.

asaintown.net recently posted some pictures of Omar Borkan Al Gala, saying that he was one of the three irresistibly handsome men who were deported.

Awesome editor for any Scripting Language - Sub lime Text Editor

2013-04-25T04:42:00.000-07:00

In Linux and Mac, most people use Vi editor and text editor for editing any scripting languages like shell, Python..etc,

This editor will be a bit complex and will not be handy to use.

One of my friend, A Linux enthusiast has introduced me a editor called "Sub Lime".

This is an awesome text editor. I found it very useful and handy. So thought of sharing it with you all.

Advantages of Sublime Text editor:

1. It gives auto suggestions for keywords and variables.

2. You can easily manage files and folders.

Steps to install :

sudo add-apt-repository ppa:webupd8team/sublime-text-2
sudo apt-get update
sudo apt-get install sublime-text

After installing you can type the following command to open it.

sudo sublime-text

You can also find this in the applications after installing.

Do easy programming....

Reference Link : http://ellislab.com/forums/viewthread/232759/

Here is the preview ....

Python Programming Tutorial 2 - Strings

2013-04-24T08:01:00.000-07:00

Python has a built-in string class named 'str'(Java - String) with lot of features. Python strings are immutable which means they can't be changed after creation(Java strings are also immutable). String can be enclosed by either single or double quotes.

Characters in a string can be accessible through [] syntax and python is also zero-based index as Java and C++. Built-in function len(str) returns the length of string(number of characters in input string)

Note: Don't use "len" as a variable name, it will block the len(str) functionality.

[code]>>> s='hi'
>>> s[1]
'i'
>>> len(s)
2[/code]

Below given video, demonstrates the some of the python String literals.

Note: Content was Scrapped from youtube.

In Java the '+' automatically converts the any types to String while appending with String literal but in Python this is not a case. The function str(data) will convert the input data to String.

[code]>>> pi=3.14

>>> str(pi)
'3.14'
>>> 'rrr'+pi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot concatenate 'str' and 'float' objects
>>> 'rrr'+str(pi)
'rrr3.14'
>>> num=18
>>> print "ff" + repr(num)
ff18
>>> print "ff" + `num`
ff18[/code]

Below given video, demonstrates the some of the python String representation.

Note: Content was Scrapped from youtube.

If the value to String literal is prefixed with 'r' and interpreter will pass all the chars without special treatment of backslash '\'.

[code]>>> raw="test\nraw\t.."
>>> print raw
test
raw ..
>>> raw=r"test\nraw\t.."
>>> print raw
test\nraw\t..[/code]

A 'u' prefix allows you to write a unicode string literal (Python has lots of other unicode support features)

[code]>>> data=u'Hello World !'
>>> data
u'Hello World !'
>>> data=u'Hello\u0020World !'
>>> data
u'Hello World !'[/code]

The escape sequence \u0020 indicates to insert the Unicode character with the ordinal value 0x0020 (the space character) at the given position.

Python Programming Tutorial 1 - Numbers and Math

2013-04-24T04:55:00.000-07:00

This tutorials gives overview of the Python numbers and Math functions.

Using the Python as Calculator

Let's try some python command line options. The python Interpreter act as an Calculator, we can do simple mathematical function in the command line.

Below given video, demonstrates the some of the python number and mathematical functions.

Note: Content was Scrapped from youtube.

A values can be assigned to several variable simultaneously.

[code]>>> x = y = z = 0 # Zero x, y and z
>>> x
0
>>> y
0
>>> z
0[/code]

Complex numbers are also supported; imaginary numbers are written with a suffix of j or J. Complex numbers with a nonzero real component are written as (real+imagj), or can be created with the complex(real, imag) function.

[code]>>> 1j * 1J
(-1+0j)
>>> 1j * complex(0,1)
(-1+0j)
>>> 3+1j*3
(3+3j)
>>> (3+1j)*3
(9+3j)
>>> a=(1+2j)/(1+1j)
>>> a
(1.5+0.5j)
>>> a.real
1.5
>>>a.imag
0.5
>>> abs(a) # sqrt(a.real**2 + a.imag**2)
5.0[/code]

In interactive mode, the last printed expression is assigned to the variable _. This means that when you are using Python as a desk calculator, it is somewhat easier to continue calculations, for example:

[code]>>> tax = 12.5 / 100
>>> price = 100.50
>>> price * tax
12.5625
>>> price + _
113.0625
>>> round(_, 2)
113.06[/code]

Python Programming Tutorial 2 - Strings

Subscribe for next update. Thanks you.

Technology Driven Startups

2013-04-24T03:18:00.000-07:00

Struts Video Tutorial

2013-04-24T02:33:00.000-07:00

http://www.youtube.com/watch?v=f46WEeM8HTA&list=PLB7BB551126EDD5E0

Spring Video Tutorials

2013-04-24T02:29:00.000-07:00

Ways to Contribute to Open Source

2013-04-23T02:50:00.000-07:00

Plenty of people want to get involved in the open source and they do not know where to start. Below source will give plenty of tips and ways of contribution.

Open source software has changed computing and the world, and many of you would love to contribute. Unfortunately, many people are daunted by what they imagine is a high barrier to entry into a project. I commonly hear people say that they'd love to contribute but can't because of three reasons:

"I'm not a very good programmer."

"I don't have much time to put into it."

"I don't know what project to work on.”

There are three core principles to remember as you look for opportunities to contribute:

Projects need contributions from everyone of all skills and levels of expertise.

The smallest of contributions is still more than none.

The best project to start working on is one that you use already.

Continue Reading.

Significance of Host File in Windows

2013-04-16T07:31:00.000-07:00

What is host file?

Almost every operating system that communicates via TCP/IP, the standard of communication on the Internet, has a file called the HOSTS file. This file allows you to create mappings between domain names and IP addresses.

You can command your machine to contact the only a particular machine for replies.

How to set your local machine to contact only a particular machine for the replies with the host file on Windows?

1.Click on Win+R or Start -> run and type drivers as below:

2. Goto the “etc” directory and open the hosts file

3. Add the entries in the following format and save it

{IP}         {Domain Name}

Example:

127.0.0.1              localhost

10.90.112.101     domainname

Note:

You must have the administrative rights to modify/add the entries in hosts file.

4. After saved the above host entries, whenever you gave the request the particular domain configured in the hosts file then response will come directly from the ip which we configured(example: 10.90.112.101)

5. If you need to host entries, just delete the line which you added or insert “#” in front of the line as below

#10.90.112.101 domainname

iOS UI Automation

2013-04-11T04:30:00.000-07:00

Introduction / Overview

iPhone apps are developed using native or native-hybrid bridge.

native-hybrid : The datas are rendered in HTML and the native actions are performed using js callbacks

Automation testing framework for these applications is mainly to,

Reduce the manual testing effort

Reduce the testing cycles involved for any changes to the application

Provide maximum test coverage

Why UIAutomation ?

Selenium Automation

Using selenium the basic level functionalities like tapping, swiping could not be done in the iPhone app. so,selenium could not be an option to automate the iPhone app

Jasmine framework

Jasmine is a BDD (Behavior Driven Development) framework.

Jasmine is mainly used for Unit testing

More or less jasmine is same as tuneup framework

Jasmine provides good reporting structure

Reports can be in xml or html format

UI Automation + tuneup.js

It is the Apple's inbuilt Automation testing framework.

It is highly stable and also reliable testing framework.

It uses javascript to automate the testcases.

Automation can also be performed in the specified devices

It can also be accessible from command line so that we can also integrate this with Hudson

Considering its salient features,simplicity UIAutomation is preferred for automating the test cases over other testing frameworks

Tools used

Xcode IDE (Version 4.X and above)

Instruments

Instruments Tool

Instruments is a tool for UIAutomation provided by Xcode. Instruments provides us the record and playback feature which will be very useful in doing automation

It can be used to collect data about the performance and behavior of one or more processes on the system and track that data over time.

Test Case Log

The test case when executed on the iOS application target, using Instruments,

The Instruments console will show the details of,

Log Message - What Test case is being executed

Log Type - Test case is Pass or Fail.

If a test case fails, it would display the

Element tree in the failed page meaning, what are the UI elements present in the current page.

Also the screen shot of the page in which error occured.

Automation Framework

Tune-up is the framework used for automation

Tune-up is a collection of JavaScript utilities that builds upon and improves the UIAutomation library provided by Apple for testing iOS applications via Instruments.
You can find more details about the Tune-up framework from here:https://github.com/alexvollmer/tuneup_js/blob/master/README.md
an also in http://alexvollmer.com/posts/2010/07/03/working-with-uiautomation/

Test Structure in Tune-up

To create a test case, use the test() function, which takes the name of the test case as a string, and a function. The function will be given a UIATarget instance and a UIAApplication instance.

For example:

test("Sign In to the app", function(target, app) {

// The target and app arguments are root elements

/* (i.e) The element structure of every element in the app goes in this way
UIATarget.localTarget().frontMostApp().(position of the particular element in the view)
for reusing UIATarget.localTarget().frontMostApp() we store it in target,app */

// The UIATarget is the primary portal into the application running on the device or simulator

// Steps for Sign-in

});

test("Sign out from the app", function(target, app) {

// Steps for Sign-out

});

Integrating with the Project

MakeFile configuration:

Configure the MakeFile which is in the project home , with the command lines which is used to build the project and Launch the instrument.

Add the following configuration,

1.TRACE_PATH : Location to store the .trace file, which will create after the execution of test script using instruments.

2.TEMPLATE_PATH : Location where the test case TemplateTrace files are stored, which is created from the instruments by saving the testSuite script as Template.

3.uiautomation :

Shell command which

Builds the project ,

Executes the test case script by launching the Instruments tool.

And this executes the test Suite which has the entire flow.

Ex.

/* Specify the Trace path, Template path and App path */

TRACE_PATH = TraceDocument/UIAutomationTrace

TEMPLATE_PATH = Test/Automation/Resources/TestCaseTemplates

APP_PATH = /Users/${projectpath}//Products/Debug-iphonesimulator/sample.app

/* Shell commands to Run the Test cases */

uiautomation:

xcodebuild -workspace sample.xcworkspace -scheme Release -sdk iphonesimulator5.0

instruments -t ${TEMPLATE_PATH}/testSuiteTemplate.tracetemplate -D ${TRACE_PATH} ${APP_PATH}

Running the Automation Script

Command to run automation from Terminal using the MakeFile ,

In Terminal ,Move to the directory where the ‘.xcodeproj‘ file resides

And execute the following Command,

make uiautomation

Additional Info about Shell Commands

Write a Shell Script to execute the following commands

1) Building the Project

At first the Project should be build through ‘xcodebuild’ command

xcodebuild:

This command is used to build the xcode project.

Xcodebuild Usage

Xcodebuild -- workspace [PROJECT-WORKSPACE] --scheme [SCHEME-NAME] --sdk [DEVICE/SIMULATOR NAME]

Option	Description
-- workspace	PROJECT-WORKSPACE Name of the Project file(ex sample.xcworkspace)
--scheme	SCHEME-NAME Name of the Sheme
--sdk	Specify the device/simulator name(ex: simulator name – iPhoneSimulator5.0)

Move to the directory where the ‘.xcodeproj‘ file resides and then the command Should be executed

Ex:

Move to the directory where the project resides

Execute the xcodebuild command

$bash: xcodebuild -workspace sample.xcworkspace -scheme Release -sdk iphonesimulator5.0

This command will build the project with configuration as release and sdk as iphone simulator5.0

2) Launching the Instruments tool

Execute a command to launch the Instrument with the latest build

This command executes the saved ‘.tracetemplate’ file and saves the result in the path specified

Instruments Command Usage

instruments [-t template] [-d document] [-w device] [-p pid] | [application [-e variable value ]]

Options	Description
-t	templateFile name of type ‘ .tracetemplate ‘ with the path of the file location
-s	Show list of known templates and exit
-d	documentThe path to save the trace document data to (This may already exist, in which case a new Run will be added)
-p	pidThe ID of the process to attach to
application	The path to the application or command to launch
-w	hardware deviceThe identifier of the hardware to target
-e	variable valueAn environment variable setting (You may specify more than one)

Ex :

instruments -t $TEMPLATE_PATH -D $TRACE_PATH $APP_PATH

Option	Description	Example
TEMPLATE_PATH	Template path is the path of the saved template file	$PROJECT_HOME/Test/Automation/Resources/TestCaseTemplates/testSuite.tracetemplate
TRACE_PATH	It is the path where the .trace will be saved after executing the automation	$PROJECT_HOME/TraceDocument/UIAutomationTrace
APP_PATH	It is the path of the latest build of the APP	~/Library/Developer/Xcode/DerivedData/sample-gxfepasnjxtmhjcevjkpyunezjca/Build/Products/Debug-iphonesimulator/sample.app

Note: After the execution of the xcodebuild command the app will be build in a directory; APP_PATH should have that directory path

The Build path can be customized. To change the app build path, In the Xcode Choose the Xcode option int the Menu bar choose Preferences – Locations and then give the custom location in the derived data field

On executing this command, the script loaded in the .tracetemplate file will get executed on the specified simulator/device and stores the .trace file in the specified path

Light smart by going green

2013-04-05T06:59:00.000-07:00

I was in dilemma, when I started to write this blog. Because the content that am going to narrate here is a well known fact. But still, a recent incident has urged me post this blog.

How many of us have cursed the Tamil Nadu or Indian government for inconsistent power supply. We blame political parties, politicians and even God at times for lack of rain which has a direct influence in power production. But we fail to think that there is only few that can be done by the government and we as a citizen of this country need to take responsibility and do some possible deed.

There is a simple step that you can contribute to nature in two ways, one by consuming less energy and another by polluting the environment less.

Its nothing but replacing incandescent bulbs with CFL bulbs. How this will help us ???.

A CFL consumes 25% less energy when compared to an incandescent bulb. This saves electricity by 70%. An incandescent bulb glowing for 1 hr consumes 60 watts, whereas a CFL glowing for 1 hour consumes 10 watts only. You will get the same glow from both these bulbs. We save 50 Watts of energy per hour, if we do a simple change. This is how we consume less energy.

"We never know the worth of water till the well is dry". ~Thomas Fuller,Gnomologia. Same is the case for power we waste most of the time without knowing its value.

On the other side of the story, If we light an incandescent lamp for one hour it produces 5 kg of co2, which is an endanger for the environment. This is equal to riding a bike for 8 km. So the first immediate step we need to do is to change all the incandescent bulb with CFL wherever we could. So that you can contribute something valuable to the environment and to the nation by saving a decent amount of electricity.

You may have lot of questions that, who is buying incandescent these days and where will be these kind of bulbs found now a days. Yes, we have many people around who are unaware of it. The reason why am writing this here is, one of my well educated friend has recently bought a incandescent bulb to our home. That's the time I have decided to post this blog and create awareness about the same. You can find these kind of bulbs in rural areas mostly than the urban. It is not uncommon to find these kind of bulb in rural areas.

If we change every such Bulb in our state(Tamil Nadu), we can get an extra 15 mins of power for each day in our state. We can do this instead blaming our government or politicians. If we could make it happen and if we do the same in all states of our country, we could save a lot of power and that can be used for many other industrialization purposes.

I thought of this blog a long time back and drafted this, But I need to be a "Doer before a Sayer". It took a months time for me to change all incandescent lamps wherever I could (In my home, My Grandma's home, Our current rented home in Chennai). Totally 5 is what I have changed so far. Even its a little am happy that I have contributed something to the environment and did a good deed as a citizen to my country.

Little progress together makes a brighter and greener future. I want my friends to do and Share the same.

Lets create a better future together.....

GenericOptionsParser, Tool, and ToolRunner for running Hadoop Job

2013-03-26T21:34:00.000-07:00

Hadoop comes with a few helper classes for making it easier to run jobs from the command line. GenericOptionsParser is a class that interprets common Hadoop command-line options and sets them on a Configuration object for your application to use as desired. You don’t usually use GenericOptionsParser directly, as it’s more convenient to implement the Tool interface and run your application with the ToolRunner, which uses GenericOptionsParser internally:

public interface Tool extends Configurable {
        int run(String [] args) throws Exception;
    }

Below example shows a very simple implementation of Tool, for running the Hadoop Map Reduce Job.

public class WordCountConfigured extends Configured implements Tool {
        @Override
        public int run(String[] args) throws Exception {
        Configuration conf = getConf();

        return 0;
        }        
    }
    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new WordCountConfigured(), args);
        System.exit(exitCode);
    }

We make WordCountConfigured a subclass of Configured, which is an implementation of the Configurable interface. All implementations of Tool need to implement Configurable (since Tool extends it), and subclassing Configured is often the easiest way to achieve this. The run() method obtains the Configuration using Configurable’s getConf() method, and then iterates over it, printing each property to standard output.

WordCountConfigured’s main() method does not invoke its own run() method directly. Instead, we call ToolRunner’s static run() method, which takes care of creating a Configuration object for the Tool, before calling its run() method. ToolRunner also uses a GenericOptionsParser to pick up any standard options specified on the command line, and set them on the Configuration instance. We can see the effect of picking up the properties specified in conf/hadoop-localhost.xml by running the following command:

hadoop WordCountConfigured -conf conf/hadoop-localhost.xml -D mapred.job.tracker=localhost:10011 -D mapred.reduce.tasks=n

Options specified with -D take priority over properties from the configuration files. This is very useful: you can put defaults into configuration files, and then override them with the -D option as needed. A common example of this is setting the number of reducers for a MapReduce job via -D mapred.reduce.tasks=n. This will override the number of reducers set on the cluster, or if set in any client-side configuration files. The other options that GenericOptionsParser and ToolRunner support are listed in Table.

GenericOptionsParser and ToolRunner option Description

Property>	Description
-D property=value	Sets the given Hadoop configuration property to the given value. Overrides any default or site properties in the configuration, and any properties set via the -conf option.
-conf filename ...	Adds the given files to the list of resources in the configuration. This is a convenient way to set site properties, or to set a number of properties at once.
-fs uri	Sets the default filesystem to the given URI. Shortcut for -D fs.default.name=uri
-jt host:port	Sets the jobtracker to the given host and port. Shortcut for -D mapred.job.tracker=host:port
-files file1,file2,...	Copies the specified files from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS) and makes them available to MapReduce programs in the task’s working directory.
-archives archive1,archive2,...	Copies the specified archives from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS), unarchives them, and makes them available to MapReduce programs in the task’s working directory.
-libjars jar1,jar2,...	Copies the specified JAR files from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS), and adds them to the MapReduce task’s classpath. This option is a useful way of shipping JAR files that a job is dependent on

Apache Ant - Tutorial

2013-03-25T03:20:00.000-07:00

1. Build Tools
Build tools are used to automate the repetitive task like compiling source code, generating documentation, running tests, uilding the jar and so on. Some of the well known build tools are Apache Ant, Maven.

2. Overview of Ant

Ant(Another Neat Tool) is the java library and mostly used to building and deploying the java application. Ant provides the built-in tools to compile, build, test and packing the java application. Ant builds are based on three blocks.
Tasks: Task is the unit of work. For example compile, packing.
Targets: Targets can be invoked via Ant.
Extension Points: Extension points are same as targets and it won't any operation and just coordinate the targets.

3. Building the Java Application: Using Apache Ant

Create the build.xml(not as same) in your project root directory. Given below is the sample build xml(self explanatory - comments inline)

<?xml version="1.0" ?>
<project name="nRelate Analytics" >
    <!-- Sets variables which can later be used. -->
    <!-- The value of a property is accessed via ${} -->
    <property name="src.dir" location="src" />
    <property name="build.dir" location="build" />
    <property name="dist.dir" location="dist" />
    <property name="lib.dir" location="lib" />

    <!--
      Create a classpath container which can be later used in the ant task
    -->
    <path id="build.classpath">
        <fileset dir="${lib.dir}">
            <include name="**/*.jar" />
        </fileset>
    </path>

    <!-- Deletes the existing build, dist directory-->
    <target name="clean">
        <delete dir="${build.dir}" />
        <delete dir="${dist.dir}" />
    </target>

    <!-- Creates the  build, dist directory-->
    <target name="makedir">
        <mkdir dir="${build.dir}" />
        <mkdir dir="${dist.dir}" />
    </target>

    <!-- Compiles the java code (including the usage of library -->
    <target name="compile" depends="clean, makedir">
        <javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="build.classpath">
        </javac>
    </target>

    <!--Creates the deployable jar file  -->
    <target name="createjar" depends="compile">
        <jar destfile="${dist.dir}LogNormalizer.jar" basedir="${build.dir}">
        <!-- Adding to Manifest file, from which class to start exceution -->
        <manifest>
            <attribute name="Main-Class" value="test.MainClass" />
        </manifest>
        </jar>
    </target>
</project>

4) Run your Ant build from the command line

Open a command line and switch to your project directory. Type in the following commands.

# Run the build
ant -f build.xml
# build.xml is default you can also use
ant
Specify the target as below.
#Run the build.xml by specifying the target.
ant -f build.xml createjar

The build should finish successfully and generate the build artifacts under the dist directory.

Subscribe to get updates on this article.

My Personal collections

2013-03-22T07:06:00.000-07:00

This is my personal collection of books that I wish to read and also books that I have accomplished so far. I firmly believe in Thomas Jefferson's quote - "A life without a book is meaningless". I want my life to be meaningful and so here comes this blog. I will keep on updating this list and also welcome people who suggest me some good books and give me some feedback.

Drowned ...

Two States - Chetan bhagat

This is beautiful love story between an IIM guy from Delhi and a girl from Tamil nadu. This is how they fall in love and carrying it till their marriage. Over coming their problems of being from different caste, culture, language and family back ground. This is the author's own story. A good one to read.

One night at the call center - Chetan bhagat

Another drama from Chetan, where the entire book is about the story that happens over a night. This explains the life at call center and along with a love drama in between. A kind of interesting, though it doesn't sound as two states.

Who will cry when you die - Robin Sharma

Author Robin speaks about life's simple things which will make life happier. This will be a collection of different lessons where he touch base all areas from being kind to others, mental and physical fitness, going green, being positive and lot more.

The Greatness guide - Robin Sharma

Robin explains how one should lead his life both as a leader and a follower. He has embedded great sayings of Legends and proverbs that will sting to your mind. He simply states that , "the first point of how a world sees you, should begin from you.

Five point someone - Chetan bhagat

This another real life incident happened to author in IIT. A mix of fun and love in his college days. Well written in simple language as like his other books. Once you start, you can never close without finishing it.

What young India Wants - Chetan Bhagat

Its the collection of essays and articles written by the author in various newspapers in the north. He has also some good letters written by him to Political leaders. Most of the incidents in this book speaks about things that cling around northern part of India.

The Alchemist - Paul Coelho

This is a story where the author speaks about treasure, desert, travel and love. Its about a young boy who risks his life in fulfilling his destiny. He insists that "“If you really desire to achieve something, the whole world conspires in helping you to achieve it”.

Revolution 2020 - Chetan Bhagat

A triangle love story between three. One guy who failed to IIT but becoming a director of an Engineering college. Another guy who cracked the exams to get in to the best college and at last ended up in running his own news paper publication who is willing to correct the corrupted nation with his thoughts and broad mind. The final touch of whom the girl will be married to is an interesting ending.

Kane and Abel - Jeffrey Archer

This is one of the best books that I have ever read. It explains life of 2 persons from their birth to death. The journey they went through, difficulties in bringing themselves up their life. I highly recommend this book for anyone. The author has kept his nerve through out the story. I have decided to read the few other books of this author. He deserves the title to be one of the best authors in the world.

To be drowned....

A Thousand Splendid Suns - Khaled Hosseini

Google Launches Realtime API for Google Drive

2013-03-20T22:13:00.000-07:00

Google has launched a Realtime API for Google Drive, enabling developers to create apps that can tap into Drive's real-time editing capabilities.

The API provides developers with collaborative versions of data objects such as maps, lists, strings, and JSON values, which are automatically synchronized, and all modifications to them are stored. Developers can create apps that read from and write to these objects like any local object.

If the basic set of objects isn't enough, developers can also create custom objects and references, which includes trees and arbitrary graph structures.

interested developers can check out the Google Drive Realtime API technical documentation here.

Source: Mashable.com

Excellence is a drive from inside

2013-02-19T01:54:00.000-08:00

Shared in the Cognizant Internal Mail Thread.

Excellence is a drive from inside

A tourist once visited a temple under construction where he saw a sculptor making an idol of God. Suddenly he noticed a similar idol lying nearby.

Surprised, he asked the sculptor, "Do you need two statues of the same idol?"

"No," said the sculptor without looking up, "We need only one, but the first one got damaged at the last stage."

The gentleman examined the idol and found no apparent damage.

"Where is the damage?" he asked.

"There is a scratch on the nose of the idol." said the sculptor, still busy with his work.

"Where are you going to install the idol?"

The sculptor replied that it would be installed on a pillar twenty feet high.

"If the idol is that far, who is going to know that there is a scratch on the nose?" the gentleman asked...

The sculptor stopped his work, looked up at the gentleman, smiled and said, "I will know it."

Moral :

Excellence is not for someone else to notice but for your own satisfaction and efficiency.

The desire to excel is exclusive of the fact whether someone else appreciates.

Writing MapReduce Program on Hadoop - Context

2013-02-14T05:46:00.000-08:00

Map and Reduce

Map Reduce works by breaking the processing into two phases: Map phase and Reduce phase. Each phase has the key-value pair as input and type of key and value can be chosen by the programmer.

Data flow in the Map and Reduce:

Input ==> Map ==> Mapper Output ==> Sort and shuffle ==> Reduce ==> Final Output

Steps to Write the Hadoop Map Reduce in Java

Map Reduce program need three things: Map, Reduce and Some code to run job(Here we will call it as Invoker)

1). Create the Map(Any Name) class and map function was represented by org.apache.hadoop.mapreduce.Mapper.class which declares an abstract map() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

public class Map extends Mapper<;LongWritable, Text, Text,IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException {
word.set(value.toString());
context.write(word, one);
}
}

[/code]
Explanation:
The Mapper class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key is LongWritable(Long representation by hadoop), input value is the Text(String representation by hadoop), output key is text(keyword) and output value is Intwritable(int representation by hadoop). All above hadoop datatypes are same as java datatype expect that are optimized for network serialization.

2). Create the Reducer(Any Name) class and reduce function was represented by org.apache.hadoop.mapreduce.Reducer.class which declares an abstract reduce() method.
[code lang="java"]import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;
import java.util.Iterator;

public class Reduce extends Reducer<Text, IntWritable, Text,IntWritable> {
@Override
protected void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
int sum = 0;
for(IntWritable intWritable : values){
sum += intWritable.get();
}
context.write(key, new IntWritable(sum));
}
}
[/code]
Explanation:
The Reducer class is the generic class with four formal parameters(input key, input value, output key and output value). Here input key and input value type must match with Mapper output, output key is text(keyword) and output value is Intwritable(number of occurence).

3) We are ready with Map and Reduce implementation, then we need to have the invoker for confguring the Hadoop job and invoke the Map Reduce program.
[code lang="java"]import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount{
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
configuration.set("fs.default.name", "hdfs://localhost:10011");
configuration.set("mapred.job.tracker","localhost:10012");

Job job = new Job(configuration, "Word Count");

job.setJarByClass(WordCount.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//Submit the job to the cluster and wait for it to finish.
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
[/code]
4). Compile code by using following command

mkdir WordCount
javac -classpath ${HADOOP_HOME}/hadoop-0.20.2+228-core.jar -d WordCount path/*.java

5). Create the jar by using command

jar -cvf ~/WordCount.jar -C WordCount/ .

6). Create the input file in the local file system

Eg : mkdir /home/user1/wordcount/input

cd /wordcount/input
gedit file01
gedit file02

and so on..

7). Copy the input file in local file system to HDFS

$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file01 /home/user1/dfs/input/file01
$HADOOP_HOME/bin/hadoop fs -cp ~/wordcount/input/file02 /home/user1/dfs/input/file02

8). Execute the jar as follows:

$HADOOP_HOME/bin/hadoop jar WordCount.jar WordCount /home/user1/dfs/input /home/user1/dfs/output

9). After execution get completed, below set of commands is used to view the reduce file that are generated

$HADOOP_HOME/bin/hadoop fs -ls /home/user1/dfs/output/

10). To view the output, use this below given command

$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00000
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00001
$HADOOP_HOME/bin/hadoop fs -cat hdfs:///home/user1/dfs/output/part-00002

and so on...

Upcoming Post: Using Distributed Cache in Java Hadoop MapReduce.

Recovering corrupted files in Linux

2013-01-24T23:05:00.000-08:00

There are many compression technique that you use and compress a file in Linux.

Few techniques are gzip, tar and lot more.

At times when you decompress/uncompress a file after a long time you may get few errors like.

"Unexpected EOF." "Files not in gzip format" or "Files not in gz format" something like this.

This is because the files may be corrupted.

There is a recovery tool kit and it is mainly for gzip but this can also be used for tar.gz too.

STEPS FOR RECOVERY :

Go to the link http://www.urbanophile.com/arenn/coding/gzrt/

And click on the gzrt-0.6.tar.gz link, then gzrt-0.6.tar.gz file will be downloaded to your machine.

Go to the specified location and do the following.

$ tar -xzvf gzrt-0.6.tar.gz
$ cd gzrt-0.6
$ make

Then copy the corrupted file to this folder and execute the following command.

$./gzrecover ABC.gz

Then the recovered file will be in the name of ABC.recovered

Hence there is no problem in dealing with corrupted files in Linux any more. :)

Downloading videos in Linux

2013-01-22T02:27:00.000-08:00

There is a simple software (clip grab) that allows you to download online videos in Linux.

Execute the following commands in your terminal.

############################

sudo add-apt-repository ppa:clipgrab-team/ppa

sudo apt-get update

sudo apt-get install clipgrab

##########################

The software is now installed , to run the software execute the following command

#############

clipgrab

################

Steps to download :

1. Go to the "SETTINGS" tab in the GUI of Clip grab and mention the destination path where you need to save the video.

2. Then under downloads tab, Paste the web-link from which u want to download the video.

3. Click the "grab this clip" button.

The video will start getting downloaded in the specified destination.