Senthilkumar Rajendran's Blog

Joining ORACLE

2013-06-13T22:13:00.001-07:00

Well I am joining ORACLE again after 6 years of service at Wipro Technologies. At Wipro I have been supporting Continental-Germany all the six years in supporting their end to end Business Intelligent Team.
At Oracle I am joining the Implementation Service Team at The Cloud Computing Group.

BIG DATA HADOOP Testing with MapReduce Examples Part 3

2013-03-14T08:57:00.000-07:00

In BIG DATA HADOOP Testing with MapReduce Examples Part 1 and BIG DATA HADOOP Testing with MapReduce Examples Part 2 I have resolved some of the issues in getting the HADOOP running but still I have some issues left over and this time it is "Invalid shuffle port number -1 returned" when the mapreduce jobs are submitted.

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output

13/03/13 14:59:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.

13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.

13/03/13 14:59:40 INFO input.FileInputFormat: Total input paths to process : 1

13/03/13 14:59:41 INFO mapreduce.JobSubmitter: number of splits:1

13/03/13 14:59:41 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar

13/03/13 14:59:41 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class

13/03/13 14:59:41 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class

13/03/13 14:59:41 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

13/03/13 14:59:41 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name

13/03/13 14:59:41 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

13/03/13 14:59:41 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir

13/03/13 14:59:41 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

13/03/13 14:59:41 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

13/03/13 14:59:41 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class

13/03/13 14:59:41 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

13/03/13 14:59:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363184126427_0001

13/03/13 14:59:41 INFO client.YarnClientImpl: Submitted application application_1363184126427_0001 to ResourceManager at /0.0.0.0:8032

13/03/13 14:59:42 INFO mapreduce.Job: The url to track the job: http://bigdataserver1:8088/proxy/application_1363184126427_0001/

13/03/13 14:59:42 INFO mapreduce.Job: Running job: job_1363184126427_0001

13/03/13 14:59:53 INFO mapreduce.Job: Job job_1363184126427_0001 running in uber mode : false

13/03/13 14:59:53 INFO mapreduce.Job: map 0% reduce 0%

13/03/13 14:59:54 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_0, Status : FAILED

Container launch failed for container_1363184126427_0001_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_0

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:55 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_1, Status : FAILED

Container launch failed for container_1363184126427_0001_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_1

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:57 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_2, Status : FAILED

Container launch failed for container_1363184126427_0001_01_000004 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_2

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:59 INFO mapreduce.Job: map 100% reduce 0%

13/03/13 14:59:59 INFO mapreduce.Job: Job job_1363184126427_0001 failed with state FAILED due to: Task failed task_1363184126427_0001_m_000000

Job failed as tasks failed. failedMaps:1 failedReduces:0

13/03/13 15:00:00 INFO mapreduce.Job: Counters: 4

Job Counters

Other local map tasks=3

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=0

Total time spent by all reduces in occupied slots (ms)=0

hadoop@bigdataserver1:~/hadoop>

Solution is to update yarn-site.xml with the values below and then restart the HADOOP cluster.

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output4

13/03/13 15:23:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

13/03/13 15:23:15 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.

13/03/13 15:23:15 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.

13/03/13 15:23:16 INFO input.FileInputFormat: Total input paths to process : 1

13/03/13 15:23:16 INFO mapreduce.JobSubmitter: number of splits:1

13/03/13 15:23:16 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar

13/03/13 15:23:16 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class

13/03/13 15:23:16 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class

13/03/13 15:23:16 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

13/03/13 15:23:16 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name

13/03/13 15:23:16 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

13/03/13 15:23:16 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir

13/03/13 15:23:16 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

13/03/13 15:23:16 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

13/03/13 15:23:16 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class

13/03/13 15:23:16 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

13/03/13 15:23:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363188167312_0001

13/03/13 15:23:17 INFO client.YarnClientImpl: Submitted application application_1363188167312_0001 to ResourceManager at /0.0.0.0:8032

13/03/13 15:23:17 INFO mapreduce.Job: The url to track the job: http://bigdataserver1:8088/proxy/application_1363188167312_0001/

13/03/13 15:23:17 INFO mapreduce.Job: Running job: job_1363188167312_0001

13/03/13 15:23:27 INFO mapreduce.Job: Job job_1363188167312_0001 running in uber mode : false

13/03/13 15:23:27 INFO mapreduce.Job: map 0% reduce 0%

13/03/13 15:23:32 INFO mapreduce.Job: map 100% reduce 0%

13/03/13 15:23:38 INFO mapreduce.Job: map 100% reduce 100%

13/03/13 15:23:38 INFO mapreduce.Job: Job job_1363188167312_0001 completed successfully

13/03/13 15:23:38 INFO mapreduce.Job: Counters: 43

File System Counters

FILE: Number of bytes read=2369

FILE: Number of bytes written=140677

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=1474

HDFS: Number of bytes written=1631

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=3851

Total time spent by all reduces in occupied slots (ms)=4999

Map-Reduce Framework

Map input records=200

Map output records=199

Map output bytes=2165

Map output materialized bytes=2369

Input split bytes=104

Combine input records=199

Combine output records=183

Reduce input groups=183

Reduce shuffle bytes=2369

Reduce input records=183

Reduce output records=183

Spilled Records=366

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms)=57

CPU time spent (ms)=2730

Physical memory (bytes) snapshot=358436864

Virtual memory (bytes) snapshot=926806016

Total committed heap usage (bytes)=303431680

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=1370

File Output Format Counters

Bytes Written=1631

hadoop@bigdataserver1:~/hadoop>

Finally I have verified my HADOOP cluster.

BIG DATA HADOOP Testing with MapReduce Examples Part 2

2013-03-14T08:51:00.002-07:00

In BIG DATA HADOOP Testing with MapReduce Examples Part 1 the map reduce example was not complete. I ran the wordcount with the proper lower case to see what happens and again it error-ed.

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:58:06 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "localhost:9001"
13/03/13 14:58:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:76)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1188)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1184)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1183)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
hadoop@bigdataserver1:~/hadoop>

Solution is to update configuration file mapred-site.xml with the correct property values.

Running again results in another round of errors

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:59:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/03/13 14:59:39 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/03/13 14:59:40 INFO input.FileInputFormat: Total input paths to process : 1
13/03/13 14:59:41 INFO mapreduce.JobSubmitter: number of splits:1
13/03/13 14:59:41 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/03/13 14:59:41 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/03/13 14:59:41 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/03/13 14:59:41 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/03/13 14:59:41 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/03/13 14:59:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1363184126427_0001
13/03/13 14:59:41 INFO client.YarnClientImpl: Submitted application application_1363184126427_0001 to ResourceManager at /0.0.0.0:8032
13/03/13 14:59:42 INFO mapreduce.Job: The url to track the job: http://bigdataserver1.f1:8088/proxy/application_1363184126427_0001/
13/03/13 14:59:42 INFO mapreduce.Job: Running job: job_1363184126427_0001
13/03/13 14:59:53 INFO mapreduce.Job: Job job_1363184126427_0001 running in uber mode : false
13/03/13 14:59:53 INFO mapreduce.Job: map 0% reduce 0%
13/03/13 14:59:54 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_0, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_0
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:55 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_1, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_1
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:57 INFO mapreduce.Job: Task Id : attempt_1363184126427_0001_m_000000_2, Status : FAILED
Container launch failed for container_1363184126427_0001_01_000004 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1363184126427_0001_m_000000_2
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:399)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

13/03/13 14:59:59 INFO mapreduce.Job: map 100% reduce 0%
13/03/13 14:59:59 INFO mapreduce.Job: Job job_1363184126427_0001 failed with state FAILED due to: Task failed task_1363184126427_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

13/03/13 15:00:00 INFO mapreduce.Job: Counters: 4
Job Counters
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
hadoop@bigdataserver1:~/hadoop>

Hunting for fix

BIG DATA HADOOP Testing with MapReduce Examples Part 1

2013-03-14T08:45:00.000-07:00

hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar - jar file for testing hadoop

wordcount example reads text files and counts how often words occur and here I am passing the name.txt which was copied to the HDFS

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
13/03/13 14:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/03/13 14:58:06 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "localhost:9001"
13/03/13 14:58:06 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:76)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1188)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1184)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1183)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
hadoop@bigdataserver1:~/hadoop>

Solution to resolve the above error was to source the HADOOP_MAPRED_HOME in the hadoop-env.sh file.

Ran again and it resulted in another error

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar wordcount /bigdata1/name.txt /bigdata1/output
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/partition/InputSampler$Sampler
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getMethod0(Class.java:2670)
at java.lang.Class.getMethod(Class.java:1603)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.(ProgramDriver.java:60)
at org.apache.hadoop.util.ProgramDriver.addClass(ProgramDriver.java:103)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.partition.InputSampler$Sampler
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 12 more
hadoop@bigdataserver1:~/hadoop>

Solution is to source the mapreduce classpath in the hadoop-env.sh file.

# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=/home/hadoop/hadoop/share/hadoop/mapreduce/*:$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done

hadoop@bigdataserver1:~/hadoop> hadoop classpath

/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/*:/home/hadoop/hadoop/share/hadoop/common/*:/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop/share/hadoop/hdfs/*:/home/hadoop/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/*:/home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*

hadoop@bigdataserver1:~/hadoop> ls /home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*

/bin/ls: /home/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*: No such file or directory

hadoop@bigdataserver1:~/hadoop> pwd

/home/hadoop/hadoop

hadoop@bigdataserver1:~/hadoop> echo $CLASSPATH

hadoop@bigdataserver1:~/hadoop> vi etc/hadoop/hadoop-env.sh

hadoop@bigdataserver1:~/hadoop> echo $HADOOP_HOME

hadoop@bigdataserver1:~/hadoop> export HADOOP_HOME=/home/hadoop/hadoop

hadoop@bigdataserver1:~/hadoop> $HADOOP_HOME/contrib/capacity-scheduler/*.jar

hadoop@bigdataserver1:~/hadoop> ls $HADOOP_HOME/contrib/capacity-scheduler/*.jar

/bin/ls: /home/hadoop/hadoop/contrib/capacity-scheduler/*.jar: No such file or directory

hadoop@bigdataserver1:~/hadoop> echo $HADOOP_CLASSPATH

hadoop@bigdataserver1:~/hadoop> ls /home/hadoop/hadoop/share/hadoop/mapreduce

hadoop-mapreduce-client-app-2.0.0-cdh4.2.0.jar hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0.jar lib

hadoop-mapreduce-client-common-2.0.0-cdh4.2.0.jar hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0-tests.jar lib-examples

hadoop-mapreduce-client-core-2.0.0-cdh4.2.0.jar hadoop-mapreduce-client-shuffle-2.0.0-cdh4.2.0.jar

hadoop-mapreduce-client-hs-2.0.0-cdh4.2.0.jar hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar

hadoop@bigdataserver1:~/hadoop>

hadoop@bigdataserver1:~/hadoop> vi etc/hadoop/hadoop-env.sh

update class path

hadoop@bigdataserver1:~/hadoop> hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar WordCount /bigdata1/name.txt /bigdata1/output

Unknown program 'WordCount' chosen.

Valid program names are:

aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.

aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.

bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.

dbcount: An example job that count the pageview counts from a database.

distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.

grep: A map/reduce program that counts the matches of a regex in the input.

join: A job that effects a join over sorted, equally partitioned datasets

multifilewc: A job that counts words from several files.

pentomino: A map/reduce tile laying program to find solutions to pentomino problems.

pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.

randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.

randomwriter: A map/reduce program that writes 10GB of random data per node.

secondarysort: An example defining a secondary sort to the reduce.

sort: A map/reduce program that sorts the data written by the random writer.

sudoku: A sudoku solver.

teragen: Generate data for the terasort

terasort: Run the terasort

teravalidate: Checking results of terasort

wordcount: A map/reduce program that counts the words in the input files.

hadoop@bigdataserver1:~/hadoop>

Looks positive that the mapreduce is working but with a wrong syntax.

BIG DATA Moving a file to HDFS

2013-03-14T07:59:00.002-07:00

Here is a simple step to copy a file into HDFS Hadoop File System.

First Create a Directory

hadoop@bigdataserver1:> hadoop fs -mkdir /bigdata1
hadoop@bigdataserver1:>

Copy the a sample file

hadoop@bigdataserver1:> hadoop fs -put /home/hadoop/hadoop/bigdata/name.txt /bigdata1
hadoop@bigdataserver1:>

Let us check if we can see the directory

hadoop@bigdataserver1:> hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2013-03-13 13:14 /bigdata1
hadoop@bigdataserver1:>

Let us check if we can see the file

hadoop@bigdataserver1:> hadoop fs -ls /bigdata1
Found 1 items
-rw-r--r-- 1 hadoop supergroup 1370 2013-03-13 13:14 /bigdata1/name.txt
hadoop@bigdataserver1:>

BIG DATA HADOOP Services Startup and Shutdown

2013-03-14T07:53:00.000-07:00

When using start-all.sh , I got the message "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"

So it is clear that start-all.sh should not be user and the same hold true with stop-all.sh

Using start-dfs.sh it gives me JAVA_HOME is not set

hadoop@bigdataserver1:~/hadoop/sbin> sh start-dfs.sh
which: no start-dfs.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin)
13/03/13 11:44:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Error: JAVA_HOME is not set and could not be found.
13/03/13 11:45:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop@bigdataserver1:~/hadoop/sbin> echo $JAVA_HOME
/home/hadoop/jdk1.6.0_18
hadoop@bigdataserver1:~/hadoop/sbin>

But you can see that the environment is sources properly.

Solution is to update hadoop-env.sh with the JAVA_HOME

hadoop@fravm097023:~/hadoop/etc/hadoop> grep -i JAVA_HOME hadoop-env.sh
# The only required environment variable is JAVA_HOME. All others are
# set JAVA_HOME in this file, so that it is correctly defined on
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/home/hadoop/jdk1.6.0_18
hadoop@fravm097023:~/hadoop/etc/hadoop>

After that DFS and YARN services will come up

hadoop@bigdataserver1:~/hadoop/sbin> sh start-dfs.sh

which: no start-dfs.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin)

13/03/13 11:51:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-namenode-bigdataserver1.out

localhost: starting datanode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-bigdataserver1.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-bigdataserver1.out

13/03/13 11:51:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

hadoop@bigdataserver1:~/hadoop/sbin>

hadoop@bigdataserver1:~/hadoop/sbin> sh start-yarn.sh

starting yarn daemons

which: no start-yarn.sh in (/home/hadoop/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/home/hadoop/hadoop/bin)

starting resourcemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-bigdataserver1.out

localhost: starting nodemanager, logging to /home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-bigdataserver1.out

hadoop@bigdataserver1:~/hadoop/sbin>

BIG DATA CDH Single Node Setup

2013-03-14T07:31:00.000-07:00

To get started with a single node setup here are some simple steps.

- Get a machine which supports CDH , in my case I had a SUSE Linux
- Create a hadoop user
- Download hadoop tar ball hadoop-2.0.0-cdh4.2.0.tar.gz
- Download and Install JDK 1.6 or 1.7
- Unpack the tar ball hadoop-2.0.0-cdh4.2.0.tar.gz
- Source the JAVA_HOME variable
- Modify the hadoop configuration files
- Set password less ssh to the localhost
- Format hadoop namenode
- Start the services
- Validate the setup

I am jumping directly to the configuration files configuration and then to formatting namenode

Formatting the hadoop nodename

hadoop@bigdataserver1:~> hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

13/03/13 11:15:08 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = bigdataserver1/10.216.9.25
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.0.0-cdh4.2.0
STARTUP_MSG: classpath = /home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jline-0.9.94.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.6.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kfs-0.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-sources.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0-test-sources.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.0.0-cdh4.2.0.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/zookeeper-3.4.5-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jline-0.9.94.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-sources.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-test-sources.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.2.4.Final.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.2.4.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.4.0a.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.2.0-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.0.0-cdh4.2.0.jar
STARTUP_MSG: build = file:///var/lib/jenkins/workspace/CDH4.2.0-Packaging-Hadoop/build/cdh4/hadoop/2.0.0-cdh4.2.0/source/hadoop-common-project/hadoop-common -r 8bce4bd28a464e0a92950c50ba01a9deb1d85686; compiled by 'jenkins' on Fri Feb 15 10:42:32 PST 2013
STARTUP_MSG: java = 1.6.0_18
************************************************************/
13/03/13 11:15:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-5d53a7be-005c-4d8b-9f93-088c795cbb35
13/03/13 11:15:10 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list
13/03/13 11:15:10 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
13/03/13 11:15:10 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
13/03/13 11:15:10 INFO blockmanagement.BlockManager: defaultReplication = 1
13/03/13 11:15:10 INFO blockmanagement.BlockManager: maxReplication = 512
13/03/13 11:15:10 INFO blockmanagement.BlockManager: minReplication = 1
13/03/13 11:15:10 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
13/03/13 11:15:10 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
13/03/13 11:15:10 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
13/03/13 11:15:10 INFO blockmanagement.BlockManager: encryptDataTransfer = false
13/03/13 11:15:10 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
13/03/13 11:15:10 INFO namenode.FSNamesystem: supergroup = supergroup
13/03/13 11:15:10 INFO namenode.FSNamesystem: isPermissionEnabled = false
13/03/13 11:15:10 INFO namenode.FSNamesystem: HA Enabled: false
13/03/13 11:15:10 INFO namenode.FSNamesystem: Append Enabled: true
13/03/13 11:15:11 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
13/03/13 11:15:11 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
13/03/13 11:15:11 INFO namenode.NNStorage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
13/03/13 11:15:11 INFO namenode.FSImage: Saving image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
13/03/13 11:15:11 INFO namenode.FSImage: Image file of size 121 saved in 0 seconds.
13/03/13 11:15:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
13/03/13 11:15:11 INFO util.ExitUtil: Exiting with status 0
13/03/13 11:15:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at bigdataserver1/10.216.9.25
************************************************************/
hadoop@bigdataserver1:~>

Start the service using start-all.sh

Health Check : you can verify the service using the below url
http://bigdataserver1.bigdata.com:50070/dfshealth.jsp

BIG DATA Cloudera and Oracle

2013-03-14T07:14:00.004-07:00

Cloudera and Oracle partnered to bring a reliable mainstream enterprise solution of Big Data through the Oracle Big Data Appliance. It is designed and optimized for big work load and built using standard hardware.

Oracle Big Data Appliance is a supported big data infrastructure which is affordable , salable and integrated with the key components of big data platform.

CDH (Cloudera Distribution Including Apache Hadoop) is for enterprise deployment which is included with the Oracle Big Data Appliance. It has a free version and it supports up to 50 node but it can scale up with the paid version.

BIG DATA Getting Started with HADOOP

2013-03-14T05:34:00.001-07:00

Hadoop was founded by Douglass Read Cutting and picked the name from one of his son's toy elephant.

It is a open source project from Apache that evolved rapidly into a major technology movement. It is capable to handle large data set which are structured and unstructured. It had ability to run on low cost clusters and can scale up rapidly.

Hadoop architecture helps applications run on nodes which has thousands of terabytes of data. It has a distributed file system called HDFS (Hadoop Distributed File System) with fast data transfer rates between the clustered nodes and also support failures. It does not require RAID Storage as it achieves reliability by replication technology on multiple hosts.

Hadoop is a collection of components like MapReduce , Hive , Pig , NoSQL Database , ZooKeeper , Ambari , HCatalog , Oozie, Hue and more.

MapReduce is one key component. It is a framework for writing applications that processes large amount of data which are structured and unstructured in nature. MapReduce applications can be designed for a single node Hadoop and then deployed on a 100 node Hadoop.

MapReduce engine had one JobTracker to which jobs are submitted. The JobTracker then pushes work to the TaskTracker nodes in the cluster. JobTracker and TaskTracker help to complete the allocated job.

I will share my experience in getting a single node hadoop running and then running a mapreduce sample application in details.

BIG DATA is getting Bigger and Bigger

2013-03-14T04:34:00.001-07:00

BIG Data is getting Bigger and Bigger with Oracle , SAP , IBM , EMC , HP , Software AG and with more other players.

With the cost of storage & CPU going down and with the big vendors pitching in with reliable solutions it is now time for companies to look for crunching all those data's which are large and complex.

So how Oracle is helping Companies to eat the Big Mac ?

Oracle Big Data Appliance with clusters of servers.

Cloudera distribution including Apache Hadoop to acquire and organize data.

Oracle NoSQL Database Community Edition to acquire data.

Additional supporting system software like Oracle Linux , Oracle JVM VM and R distribution.

So what is that had to do with Administrator like me ?

No more terabytes , I have to think big like petabyte , exabyte , zettabyte , yottabyte and more

Hadoop , future of the big data

No more SQL , I have to know NoSQL (Not only SQL)

Big data connectors to RDBMS integrating with Oracle Database and Exadata

For analyze and visualizing with Oracle Exalytics.

Cloud computing and Big Data

So what are the likely responsibilities for Big Data Administrators ?

Managing fast growing big data clusters , NoSQL database and linux systems.

Meeting operational performance and availability

Setup monitoring and health check of the clusters and all its associated components.

Define and enforce operational procedures , processes and best practices.

Provide leadership and take ownership in collaboration with teams/vendors/3rd parties to resolving issues.

Effectively influence , partner and deliver results in a team environment.

With all the above said I will dedicate some time to snack the Big Mac.

Relevant Posts

BIG DATA Getting Started with HADOOP
BIG DATA Cloudera and Oracle
BIG DATA CDH Single Node Setup
BIG DATA HADOOP Testing with MapReduce Examples Part 1
BIG DATA HADOOP Testing with MapReduce Examples Part 2
BIG DATA HADOOP Testing with MapReduce Examples Part 3

CeBIT CeBIT CeBIT CeBIT CeBIT

2013-03-05T13:27:00.001-08:00

CeBIT is the world's leading high-tech event showcasing digital IT and telecommunications solutions, from 5 to 9 March, 2013 in Hanover, Germany.

This year I had the opportunity to be there.

It was a long day and lots and lots of walk.

Here are some of the important information I found in CeBIT

German Chancellor Angela Merkel officially opened the world’s biggest annual high-tech trade fair.
SAP's presence was very good with all of its partners.
Could see SAP HANA's presence more and had a hand's on Smart Energy demo of a 512GB HANA Appliance.
Every body looks like in-memory computing
Could see Chinese dominance over phones, tablets and other electronics.
Android is every where
Saw the 3D printer and it is really interesting (spend 2 hours to see a model getting printed)
Lots of space for Gaming

Planned to visit CeBIT another day in this week and will share some interesting stuffs.

Enterprise Manager Grid Control 12c Upgrade

2013-03-04T02:58:00.002-08:00

I am in the process of upgrading to 12c Cloud Control from version 11.1.0.1.0 using the one system upgrade.

All looks good with the prerequisite checks but having the problem when authenticating with the database.
The installer gives me ORA-00942 : table or view does not exist but I have no idea.

I am seeing the below message in the installer log files

java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:851)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1477)
at oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:392)
at oracle.sysman.install.oneclick.queries.DBConnect.executeQuery(DBConnect.java:118)
at oracle.sysman.install.oneclick.queries.executeSql.performQuery(executeSql.java:56)
at oracle.sysman.install.oneclick.EMGCInputValidation.validate(EMGCInputValidation.java:2370)
at oracle.sysman.install.oneclick.EMGCDatabaseConnectionDlgonNext.actionsOnClickofNext(EMGCDatabaseConnectionDlgonNext.java:395)
at oracle.sysman.install.oneclick.EMGCNoSeedInstallDetailsDlg_new$PageValidationListener.wizardValidatePage(EMGCNoSeedInstallDetailsDlg_new.java:667)
at oracle.bali.ewt.wizard.WizardPage.processWizardValidateEvent(WizardPage.java:710)
at oracle.bali.ewt.wizard.WizardPage.validatePage(WizardPage.java:680)
at oracle.bali.ewt.wizard.BaseWizard.validateSelectedPage(BaseWizard.java:2367)
at oracle.bali.ewt.wizard.BaseWizard._validatePage(BaseWizard.java:3072)
at oracle.bali.ewt.wizard.BaseWizard.doNext(BaseWizard.java:2152)
at oracle.bali.ewt.wizard.dWizard.DWizard.doNext(DWizard.java:405)
at oracle.bali.ewt.wizard.BaseWizard$Action$1.run(BaseWizard.java:3944)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:642)
at java.awt.EventQueue.access$000(EventQueue.java:85)
at java.awt.EventQueue$1.run(EventQueue.java:603)
at java.awt.EventQueue$1.run(EventQueue.java:601)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:612)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

So far no clue on the errors. Will keep updated if this can help someone.

Instant InnoDB

2013-02-19T00:57:00.003-08:00

InnoDB is ACID complaint storage engine for MySQL that has Oracle like commit , rollback and crash recovery to secure data. To quickly start with the InnoDB configuration and setup Packt Publishing has come up with the Instant InnoDB reference book which helps to quickly get on track with the setup.

Preconfiguring Oracle Unified Directory (OUD)

2013-02-13T05:51:00.003-08:00

When Preconfiguring Oracle Unified Directory (OUD) for OIM and OAM 11gR2 Integration the first step say to add OUD Containers using ldapadd command.

http://docs.oracle.com/cd/E27559_01/install.1112/e27301/preconfigoud.htm#CHDCEJKD

ldapadd -h testserver -p 1389 -D "cn=Directory Manager" -w "password" -c -f OUDContainers.ldif

But ldapadd command is not available in OUD 11gR2 , so the same can be loaded using ldapmodify using a special flag. (--defaultAdd)

ldapmodify -h testserver -p 1389 -D "cn=Directory Manager" -w "password" -c --defaultAdd -f OUDContainers.ldif

ldapadd command is available in OID not in OUD. So this is a documentation bug.

Oracle Cloud Trial is Active

2013-02-04T03:36:00.000-08:00

Well my Oracle Cloud trial versions are active now and I have something to play around for a month.
The activation took a long time for me and there are lot of people out there who already had their hands on.

So what are the tools that are basically used on the Oracle Cloud offering

Oracle Database

version provided is 11g release 2 enterprise edition
managed and secured on exadata
data access and manupulation with sql and plsql standards
apex - my favorite
data movement using sql developer
it has a lot of productive applications
it is mobile enabled

Oracle Java

it is all java out there
enterprise manager for managing
it runs on exalogic
its associated database runs on exadata
it runs on weblogic
it uses identity managment for user/group management
it uses access manager for single signon

Good to here from Oracle and I am going to explore it in the next up coming days.

SAP BO and DS with Oracle Database Part 6 Installing DS Client

2013-01-29T08:04:00.003-08:00

SAP BO and DS with Oracle Database Part 6 Installing DS Client

I downloaded the "SBOP DATA SERVICES DESIGNER 4.0 WITH SP03 WINDOWS (64B)" and used the generic trial licenses for the installation.

Using the installed tool access the designer and it would prompt to enter the administrator password.

now we have the console ready for deploying jobs. Wait the jobserver is still not connected to the repository and it had a red cross over the icon which is at the lower right corner.

The fix took may be a week for me to figure out in the SAP world. It is unlike Oracle world where most of the bug fixes are open and available on the internet. SAP world is still not open based on my 10 years of experience but this is only my view. But still I solved it and I will blog about it later.

SAP BO and DS with Oracle Database Part 5 Verifying CMC and DS

2013-01-29T07:54:00.001-08:00

SAP BO and DS with Oracle Database Part 5 Verifying CMC and DS

Now that we are done with the installation , we have to verify CMC and DS information.

Access CMC and make sure you are able to login via the administrative user

URL : http://bodsserver001:8080/BOE/CMC

Access Data Services option to see if the repository is active

Edit the repository and confirm if you can connect using the Test Connection button.

Access the Data Services console to see if you can see the repository and the rest of the components.

URL : http://bodsserver001:8080/DataServices

SAP BO and DS with Oracle Database Part 4 Installing DS on SUSE Linux

2013-01-29T07:01:00.001-08:00

SAP BO and DS with Oracle Database Part 4 Installing DS on SUSE Linux

Now that we are doing with installing BO , I wanted to create the DS environment so that we have a complete setup before we go into the rest of the experiments.

I downloaded "SBOP DATA SERVICES 4.0 WITH SP03 LINUX (64B)" and used the generic trial licenses for implementation.

After downloading unzip the files in Windows and then ftp into the linux server.

Set environment variables relevant for the installation
$ export LANG=en_US.utf8
$ export LC_ALL=en_US.utf8
$ export ORACLE_HOME=..../client/
$ PATH=$PATH:$ORACLE_HOME/bin

Now create a directory to store the installation files.

$ mkdir -p /oraapp/bo/bo4
$ sh setup.sh InstallDir=/oraapp/bo/bo4

Now Choose the Installation language

Make sure that you select the BO installation folder. Initially I created another directory /bo/ds4 and installed the complete package and nothing happened. So the recommendation is to install DS into BO folder.

Check the prerequisite , which is fine

accept the trial license

input the trial license key , this will expire in 30 days

Choose option to install data services with cms server , so that it can be controlled via the cms console.

Update the administrative password

select the product which has to be installed , you can custom install after expanding and selecting components which is required.

if there is already running data services configuration than that can be reused here but I have skipped it because of the new install

I am selecting the option to use the existing installed database. remember I created the user in the BO installation procedure.

select Oracle , pay attention if you have Oracle RAC (soon we will see HANA here )

give the details of the oracle environment

create the repository

now configure a new job server to process your jobs , this can be done later with the client tool.

I choose to configure the job server and now I gave new name and a port for the job server to listen

Configure access server here

ports and other inputs has to mentioned

start the installation

it starts

it started

it runs

it finished soon ( no i did not know how long it took , I started the installation in the evening and the next day I saw it got finished )

Now we have BO and DS running. Lets play a bit more with the other options.

SAP BO and DS with Oracle Database Part 3 Installing BO on SUSE Linux

2013-01-29T05:25:00.000-08:00

SAP BO and DS with Oracle Database Part 3 Installing BO on SUSE Linux

I downloaded "SBOP BI PLATFORM 4.0 SP04 (incl. FP03) SERVER LNX" and used the generic trial licenses for the implementation.

After downloading unzip the files in Windows and then ftp into the linux server.

Set environment variables relevant for the installation
$ export LANG=en_US.utf8
$ export LC_ALL=en_US.utf8
$ export ORACLE_HOME=..../client/
$ PATH=$PATH:$ORACLE_HOME/bin

Now create a directory to store the installation files.

$ mkdir -p /oraapp/bo/bo4
$ sh setup.sh InstallDir=/oraapp/bo/bo4

Now Choose the Installation language

Select the destination folder

Now the prerequisite check will be done , seems all is fine.

Accept the license agreement

Input the trial key (in my case)

Choose the application language , pay attention if you are deploying BO in the European world as you may have to select mostly all the languages.

Choose the install option

then the install type (here custom installation can be done with option 2) , I choose Full (:-

Choosing the database , here we already have running Oracle Database so option 2

Select Oracle for the CMS repository

Select Oracle for the Audit repository

I used Tomcat application server and there is a possibility to use a wide range of application server. One such is Weblogic which I will try out in a separate installation.

I opt to use the default version management system.

Configure the Server Intelligence Agent which is very helpful when it comes to managing BO components.

Configure the CMS port for the administrative console

Supply the password for the administrator accounts.

Here we have to give the Oracle TNS Name (this should be the name service name that we have mentioned in the tnsnames.ora of the Oracle Client Installation , to create the file source Oracle Home and run netca )

Login into the repository Oracle database and create the below two users.

SQL > grant connect,resource to boecms identified by boecms; (CMS Database User Name)
SQL > grant connect,resource to boeaud identified by boeaud; (Audit Database User Name)

Supply the port number for Tomcat application server

Configure the HTTP listening port

Configure the version management port and administrative password

I do not want to configure SDM agent at this moment so I skip it

I do not want to configure IEM at this moment so I skip it.

Give the confirmation for the installation to start

here we go

go it went to 8% and I can have coffee

now I am going to have lunch will be back and check the status

lunch break is over but still ??? my server is less powerful *may be*

common ....

tea break

good

very good

Now SAP BO 4.0 SP4 is installed and there is something for me to play this evening.

Next

SAP BO and DS with Oracle Database Part 4 Installing DS on SUSE Linux

SAP BO and DS with Oracle Database Part 2 Installing Oracle Client

2013-01-29T03:16:00.002-08:00

SAP BO and DS with Oracle Database Part 2 Installing Oracle Client

BO - Business Objects

DS - Data Services

Before starting to install BODS we have install the Oracle Client Software on the machine where BO and DS are going to be installed. Since my implementation was on SUSE Linux I downloaded Oracle Client for Linux and installed it.
The below screenshots will give a overview of the installation process.

With this we are done with the Oracle Client Installation that is required for BO DS Installation.

Next

SAP BO and DS with Oracle Database Part 3 Installing BO on SUSE Linux

SAP BO and DS with Oracle Database Part 1 Installing Oracle Database

2013-01-29T03:11:00.000-08:00

SAP BO and DS with Oracle Database Part 1 Installing Oracle Database

BO - Business Objects
DS - Data Services

Every body should be aware of the need to store data on to a database. But when it comes to choosing the database it is always Oracle who wins at Corporate Companies. I have made a BO and DS implementation with Oracle Database running at the back end.

To start with the BO and DS implementation first a database has to be installed. I choose HP Itanium server for the database and SUSE 11 for the BO Server. There are very less implementation of SAP BO and DS on linux environment so I decided to work with linux platforms instead of Windows.

Installing Oracle Database

This is the fun part and I am not going into detailed level as there are many references where the installation are covered. After installing the Oracle home invoke DBCA and install a health database for the repository. Additional databases has to be installed for all the necessary supporting objects , example a data warehouse database.

Next

SAP BO and DS with Oracle Database Part 2 Installing Oracle Client

New Year Greetings

2012-12-31T01:04:00.003-08:00

Wish every one a very happy new year.

Happy New Year 2013 with G Style

2012-12-26T08:58:00.001-08:00

Watch " Psy - Gangnam Style Official Music Video [HD] (강남스타일) M/V " on YouTube

12/12/12 12:12

2012-12-12T03:35:00.001-08:00

Just a blog post for the remarkable time

12/12/12 12:12

Oracle APEX Best Practices from Packt Publications

2012-11-26T03:00:00.002-08:00

Many of my IT Colleagues and Managers come to me asking for a simple and easy solution to deploy an application quickly and the first tool that hints me is Oracle Apex. I started to use when it was called HTMLDB and it was a fun to learn it.

I would like to recommend a book for all who are going to be using or already using Oracle Apex. This book is published by Packtpub and it is written by Oracle Apex Experts. The interesting part I liked in this book is using advanced SQL and PLSQL in Apex for developing real-world applications.

So here is a snapshot of what you are going to learn from this book

Lesser known features of SQL and PL/SQL
Incorporate printing capabilities
Create secure applications
Troubleshooting and Debugging
Setting up your environment
Best practices for building real life applications

You can view the sample chapters and prefaces of this title on PacktLib or download sample chapters in PDF format.