Data, Data, Data

We are living in the clouds era. Internet provide us with such a great resource to help our lives. In the progress, we created a lot of data. Consider a search engine like Google or Bing. They indexed all of sites across the network. If we are talking about sites these days, that’s a big number we are talking about. Netcraft reported that there are more than 200 Millions sites in the world. It means the search engine must process and analysis a lot of data.We all know that the development in hardware, especially CPU, is fast. We can easily find a multi-core processor nowadays. But still, processing a large scale of data in a single computer is time consuming. This could happens because there is speed gap between CPU and storage devices. Storage devices read and write process is much slower than CPU processing speed. So, distributing the process into some computers working together might be faster than on a single computer for large data. There comes the distributed computing.

Distributed Systems

Distributed computing is the process of using multiple autonomous computers that working together and communicating using computer network. Each computers process some parts of the tasks given to them concurrently on their own memory. This method is good at handling large scale of data, but there are some challenges that must be considered. One of the common problems in distributed environments is network failures. It can be caused by broken link or router error. This can affects data transmission, disrupting the entire process.

The most challenging problem in distributed computing is synchronization between multiple machines. This problem covers how to prevent from deadlock and race condition in the transmission process. Another thing to considered is how to handle nodes failure. If one node among hundreds of node failed, then how to move the jobs in the failure node delivered to the other running nodes or restart the computation. This is not trivial task. Then MapReduce comes in handy.

to be continued in part 2

Credits: MEGWARE Compute-Cluster SlashFive picture by TopSlash. licensed GNU Free Documentation License, Version 1.2 or any later version

Related posts:

1. Research Plan
2. Programming Hadoop in Netbeans
3. Why Generate Database Keys?

Meanwhile, Netbeans is an integrated development environment (or IDE) for programming in Java and many other programming languages. Netbeans (like any other IDE) helps programmer to develop applications easier and as painless as possible with its features. For this case, it helps us to develop Hadoop MapReduce jobs.

In this post, I’ll tell you step-by-step how to use Netbeans to develop a Hadoop MapReduce job. I’m using Netbeans 6.8 in Ubuntu Karmic Koala distribution. The MapReduce program we are going to create here is a simple program called wordcount. This program reads text in some files and lists all the words and how many those words present in all files. The source code of this program is available on the MapReduce tutorials packed with the Apache Hadoop distribution.

We divided this tutorial into three steps. First, we will install Karmasphere Studio for Hadoop, a Netbeans extension. Then, we will type some codes. And finally, we will run the MapReduce job in the Netbeans. Okay, fasten your seat belt.. Here we go..

Install Karmasphere Studio for Hadoop

In order to do this, you must already installed JDK 1.6 and Netbeans (of course). There is a nifty tutorial with pictures about how to install the Karmasphere Studio for Hadoop on their site, but I’ll write it again here.

1. Open your Netbeans, go to Update Center using Tools > Plugins.
2. In the Update Center, go to Settings tab and click the Add button. Enter the following Name and URL in the Update Center Customizer window:
   Name: Karmasphere Studio for Hadoop
   URL: http://hadoopstudio.org/updates/updates.xml
3. Now, select the Available Plugins tab. Find the “Karmasphere Studio for Hadoop” in the list and check it. Then click the Install button.
4. Click Next and accept the license agreement. Click Install for list of will be installed plugins. Then, click Continue to download and install the plugins. The plugins size is about 20-something MB (I forgot). Wait for it and when it’s finished, restart your IDE.
5. Done, we are good to go.

Typing some codes

Now, we are going to type some codes for wordcount program. To do this you must restart your IDE after the plugins installation. If you haven’t do it, then do it now, I’ll wait. Done it? Okay, let’s continue.

1. We need to create a new Java application. To do that, go to File > New Project. Pick Java Application project and click Next.
2. In the next window, give WordCount as the name of the project. Then type WordCount as the Main Class. When you’re done, click Finish.
3. Okay, the editor for WordCount.java is now open. But first, we must added the Hadoop library to the project. To do this right-click on the Libraries on the WordCount project folder at the left side of the IDE, then pick Add Library.
   
4. In the Add Library window, select Hadoop 0.20.0 as the version of Hadoop that we are going to use. Then click the Add Library button.
5. The appropriate library now has been added to the project. Next we are going to the WordCount.java editor. Edit this file with this code below:

   import java.io.IOException;
   import java.util.*;
   
   import org.apache.hadoop.fs.Path;
   import org.apache.hadoop.io.*;
   import org.apache.hadoop.mapred.*;
   
   public class WordCount{
   
   	public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>{
   
   		private final static IntWritable one =  new IntWritable(1);
   		private Text word = new Text();
   
   		public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException{
   				String line = value.toString();
   				StringTokenizer tokenizer = new StringTokenizer(line);
   
   				while(tokenizer.hasMoreTokens()){
   					word.set(tokenizer.nextToken());
   					output.collect(word, one);
   				}
   		}
   	}
   
   	public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable>{
   
   		public void reduce(Text key, Iterator<IntWritable> values,
   			OutputCollector<Text, IntWritable> output, Reporter reporter)
   			throws IOException{
   
   				int sum = 0;
   				while (values.hasNext()){
   
   					sum += values.next().get();
   				}
   
   				output.collect(key, new IntWritable(sum));
   		}
   	}
   
   	public static void main(String[]args) throws IOException{
   
   		JobConf conf = new JobConf(WordCount.class);
   		conf.setJobName("wordcount");
   		conf.setOutputKeyClass(Text.class);
                   conf.setOutputValueClass(IntWritable.class);
   		conf.setMapperClass(Map.class);
   		conf.setReducerClass(Reduce.class);
   		conf.setInputFormat(TextInputFormat.class);
   
   		conf.setOutputFormat(TextOutputFormat.class);
   
   		FileInputFormat.setInputPaths(conf, new Path(args[0]));
   		FileOutputFormat.setOutputPath(conf, new Path(args[1]));
   
                   try{
                       JobClient.runJob(conf);
                   }catch(IOException e){
                       System.err.println(e.getMessage());
                   }
   	}
   }

   This program will take two arguments, the directory path of the input and the output. In this post, I’ll not explain the details about the code above. Please refer to the Apache Hadoop MapReduce tutorial if you wanna know about it

6. After we sure that there is no error or typo, let’s build the program. To do this, right-click the WordCount project in the left side and pick Build. This step will create the JAR file of the program.
7. Next, we will prepare the input for this program. We will create a folder and two text files inside the folder.
   For example, if you are creating input folder at your home directory, then the path will be /home/username/input. Inside it create two text files, let’s name it file01 and file02.
   On the first file type the sentence (without the quotes): “Hello world Bye world”
   And in the second sentence type (without the quotes): “Hello Hadoop Bye Hadoop”
   Actually, you can type anything you want. The two sentences are just examples. Save the files when you’re done
8. We are done in this step. Let’s go to the final step.

Running the MapReduce job

Okay. Now we are going to run the MapReduce job locally in Netbeans. This is how it’s done.

1. On the left side of the IDE, click the Services tab. Right-click on the Hadoop Jobs and pick New Job.
   
2. Give WordCount as Job Name and select the Hadoop Job from pre-existing JAR file type. Click Next when you’re done.
3. Then, browse the JAR file we already created in the previous step. Click browse and go to your Netbeans WordCount Project folder. The JAR file is located in the dist folder. If you’re using Netbeans default settings, then the JAR file will be located in /home/username/NetbeansProjects/WordCount/dist. Click Next when you’re done.
4. In the step Set Job Defaults (Step 5 of 5), choose In-Process Thread (0.20.0) as the default cluster. Then, in the Default Arguments type the arguments needed by the program. In this case, the input and output directory path. Type the input folder that we created earlier and the output folder:
   /home/username/input /home/username/output
   For your information, we don’t need to create the output folder first. The program will create the folder for you. Click Finish when you’re done.
5. Now, we will finally run the MapReduce job. To do this right-click the WordCount under the Hadoop Jobs list and pick Run Job…
6. In the Execute Hadoop Job window, give WordCount as the Job Name and click Run.
7. If your job executes successfully, there will be an output directory and inside it you’ll find a file. Inside the file you’ll find something like this:

   Bye	2
   Hadoop	2
   Hello	2
   World	2

Now we’re done. If you have a question, feel free to ask me. But for your information, I’m still learning about this too. Let’s study about it together. Have a nice try and see you on the next post.

Related posts:

1. Creating Scrollable JDesktopPane on Netbeans
2. CodeIgniter: Removing index.php
3. Conecting to DSL from Ubuntu 8.10

• The bachelor graduate worker lacking practical skills. They can not answer a fundamental question that every IT or computer science graduate should know.
• The bachelor graduate worker also lacking soft skills, like how to speak with the higher-ups and communicate with another workers.

As a result, the companies prefer to hire a vocational IT graduate. Why?

• A vocational graduate sometimes have the practical skills that a bachelor graduate didn’t have. Computer science or IT is a wide spread knowledge. It means you didn’t have to go to the college just the learn how to program. It’s all over the clouds. So the learning materials are reachable to everyone.
• Vocational graduates are easier to manage. Some of them have more respect to the higher-ups than the bachelor graduates.
• The standard salary for the vocational graduates is less expensive than the bachelor graduates. Combine this factor with better skills and higher respect means that bachelor graduates’s job are in a grave danger.

 What to do then? First, college students must realized that all the materials given in the class is not enough. They must learning all the practical skills by themselves using the courses as the foundations. Doing some experiments, make a study group, looking for a project. There are a lot of Open Source project that  they can contribute. As long as it didn’t bother the class, then do it. Another things, take the internship program in a great place, like a software house or a big company. The bigger the company, the higher the amounts of experience that could be taken.

Joel Spolsky gave advice about what a computer science college students should do before graduate:

Without further ado, then, here are Joel’s Seven Pieces of Free Advice for Computer Science College Students (worth what you paid for them):

1. Learn how to write before graduating.
2. Learn C before graduating.
3. Learn microeconomics before graduating.
4. Don’t blow off non-CS classes just because they’re boring.
5. Take programming-intensive courses.
6. Stop worrying about all the jobs going to India.
7. No matter what you do, get a good summer internship.

Now for the explanations, unless you’re gullible enough to do all that stuff just because I tell you to, in which case add: 8. Seek professional help for that self-esteem thing.

That’s a great advice. You should check his article of you have time.

If you are an IT or computer science college students, I suggest you to improve your practical skills so that you can compete with the vocational graduates. Don’t be a smart ass and feeling high and mighty, because there are a lot of skillful people out there. Remember, computer science or information technology is not an exclusive knowledge. 

Stay hungry, stay foolish (Steve Jobs)

Credit:

Graduation at SIAS by Quimbero, licensed as Public Domain

Related posts:

1. Lessons from Apple
2. Short Break
3. Research Plan

When I was in my college, I tried to implement Web Map Service (WMS) and Web Feature Service (WFS) as a foundation for a distributed Geographical Information Systems (or better known as GIS). My academic advisor at that time told me that this idea is not entirely new, but there are still a lot of people didn’t know about it yet. So with this topic as my thesis, he wished that one day people will know about this technology.

The implementation that I made was quite simple actually. But let me tell you the complete story. At first, I was thinking about develop a geographical operation that can be operated via web service in the clouds. After some weeks of analyzing and gathering informations, I found out that this work could be really hard and time consuming. I didn’t have background in geography–I’m a computer science student–and I didn’t have much time before the next graduation. Finally, I just created a spatial data repository and make it accessible across the network using GeoServer,an Open Source implementation of WMS and WFS. I, then, created a simple web application to pull the spatial data and display it to the browser. I also provided a simple data update feature, utilizing one of the feature of WFS. I used OpenLayers to create the application. It’s really simple actually.

In my graduate study, right now, I want to try something entirely different. I want to explore MapReduce, a programming model for processing a large scale of data in a distributed environment. I heard about this model from some mailing lists and websites, surprised that the paper [pdf], the lecture notes and videos are easy to get. So, for the time being, I decided to do some experiments in order to learn something about it.

It’s still a plan in my head actually. I never talked about it to my thesis advisor (because I have none yet). But I can predict some problems that I will be dealing with if I do this research plan. They are:

• The case. I don’t have any idea about the case that I should solve with this research. My college’s advisor told me about doing something in bioinformatics like genome assembling. I think I will cosider it. But I’m open for an idea.
• The machine and its network. The lab are always busy with the other graduate student. Fortunately, one friend of mine told me that there is another place that I can use in the campus to do experiments. But I should create a permission letter first. Okay, I’ll do it.

In the mean time, I’ll focus myself to learn about MapReduce. Maybe I’ll post something about it in this blog. If you have a suggestion about what should I do with this programming model, let me know. I’d be really glad to hear it.

Credits:

EDSAC pictures, copyrighted Computer Laboratory, University of Cambridge, licensed under the Creative Commons Attribution 2.0 Generic license.

Related posts:

1. Another New Theme..
2. Not on the same page
3. The last few months

It has been a long time since my last post in this blog. I just wrote some posts in my old blog when suddenly it’s already 2010. Wow, time really waits for no one. So in this post I wanna sum up what has been happening in the last year. Well, I don’t actually do anything big actually, but I made some progress in live, I guess.

Here I come. I spent my first months in 2009 in my homeland. I was recovering from my illness and successfully gained some weights. I did some theme for this blog, exploring with domain and paid-hosting, maintaining my virtual social live in Facebook and Twitter. Anything that can be done in the house, I’ve done it. Oh yeah, I also created a simple blog aggregator or usually called planet to lists all of my class’s blog posting.

I also have an opportunity to continue my study. My college advisor gave me an opportunity to go abroad. Sadly, because of my health concerns, I couldn’t accept his offer this time (although I really want to). So I registered to a national university, and for the administration necessity I went to my old university. I met some of my old friends and my advisors. When it’s done, I went back to complete my registration.

Then I was accepted in the university. I’m moving to a new city. Finding a new boarding room. Wandering around and found some cool books in the library. Leeching on the university’s software repositories. I like this one. Because the repositories hosted by the university, I could access them locally just by connecting to the university’s network. When I tried to download an Linux image, I couldn’t believe the download speed. I could download an image about 700 MB size in just 15 minutes. Wow!!

Okay, so it’s 2010 now. The new year is celebrated with an examination for the first semester. That’s tough. While everyone else went home to celebrate a new year with their families and beloved ones, I must went to the city to prepare my exam. Oh yeah, I also found a cool stuff. It’s Posterous. It’s just the same blog with different concept. You can post writing to the blog by writing an email to them. I know, Wordpress have post by email feature too. But in Wordpress, that’s not the main posting method. In Posterous, post by email is the main method. You can also Autopost your writing to another (mainly social networking) sites, like Twitter, Facebook, Plurk, and many more. So I decided to create a blog there and will have some fun with it in the future.

I can’t predict the future. That’s why I can wake up in the morning and get excited about what should I do today. Like Steve Jobs said, “let’s invent tomorrow!”

Credits:

Fireworks in Perth picture by Chris Johnson, licensed Creative Commons Attribution 2.0

Related posts:

1. Dang!
2. 500 Roundup: the Statistics
3. Not on the same page

There was once a man who went on a journey to correct his regrets. He tried his best to change the past. Unfortunately, the door to miracles never opened for him. He realized at the end of his journey, no matter how hard he tries to change his past, he cannot change himself. That is why he tought, instead of regretting about the past, what’s most important is to change the present for the future.

(taken from one of my favorite movie series)

Related posts:

1. Ads or No Ads
2. Alternative Softwares
3. Stop It

Okay, I just got this first assignment. Me (and all the other students) must submit a paper about some topics provided by the teacher. Because this is my first time, I just do what I can. I wrote it like I used to write in blog. By the end of the day, I finally finished about seven pages of them. I printed it out, booked it, and submitted it to the academic. Then, I sent the pdf version to the teacher by email.

On the next day, I met my classmate. He asked me about the paper. I told him that I have sent the pdf version of the paper to the teacher email and have submitted the “real” paper at the academic. He told me that he’s using chm format for the paper.

I said, “I think that’s okay, as long as you submitted the paper on the academic.”

He asked, “What do you submit to the academic? I mean in what form?”

I replied, “I printed it out.”

“You printed it out? I thought it was burnt on a CD”, he said it again.

“Huh? If you submitted a CD, then why we should submitted to the email?”, I asked him.

“Yeah, but you see. I’ve created a program to brute-force attack on Word’s encryption. Just to prove that it could be attacked”, he told me.

I was stunned for a moment. “So you created a program?”, I asked him just to make sure.

“I did”, he said. “If I printed all the source code, that could cost a lot of paper”, he added.

“And the code worked?”, I asked him again, disbelieved.

“Yeah, it worked”, he replied as if it’s nothing special.

“Well, I think you could printed your chm version. And then submitted the source code in CD”, I advised him.

“Yeah, that should work”, he said again.

Until now, I still surprised about what he was doing. I mean we’re not on the same page. Even, we’re not in the same book. It’s kinda like I’m still on the first page of the first episode of long going trilogy novels, while he was already at the beginning of the last episode.

I must be better than this. Seriously.

Related posts:

1. Blogroll on a Page
2. Just a Little Bit..
3. Breadcrumbs and Permalink

Today is the last week before the mid term exam. Life is fast, right? I still remember when I got attached by this WSD, and now it’s already a year after that. Six months after that, I finally be able to recover. And got into this graduate study.

It’s a new city, you know? This city is totally different with the one from the old days. I can still found cultural politeness such as say hi and smile to older people. The great thing is, they all do it sincerely. The city itself is exotic. There were so many tourist attractions here, make me feel no boring.

The grad study itself is far different from before. You know, most of the people who took this degree are employed. They are older than mine, and they are busier. Thinking about myself doing nothing just made me feel bad about the time I’ve been wasting. But learning from an older guy sometimes can be really entertaining and inspiring. Adapting to new crowds with different ages is difficult, I know that. But I’m, doing my best. And yeah, I also practicing this book authored by Dale Carnegie about how to make friends.

Yeah, I’m reading a book about how to make friends. I know that I’ve been a little anti-social lately. I just can’t handle in the middle of crowds, especially when I became the point of attention. It just scared the hell out of me. I know I’m not good at making friends. Worse, I’m also not good at keeping friends. Feeling high and mighty, sometimes I could treat my friends really really bad. Although that’s not really my intention, but with this I also say sorry for everyone. I’m really sorry if I made mistakes toward you. I’m sorry if I made you mad. I’m really sorry.

Back to the main line. I also found some inspiring teachers here. Like this teacher. Up until this mid term, he just showed up four times. And I just came to his course twice. Hey, it’s not my fault. The first one because I didn’t know about the course. And the second was because his schedule is overlapping with another schedule. But, with just two class, he inspires me a lot. He reads the book I read. One day, he talked about a research by a PhD student. The research itself wasn’t so special because the products is already used in the real world created by foreign countries. But he stood with the student. He said that if we don’t wanna start innovate by ourselves, we’ll keep depend on others product. So he encourages us to do research, not necessarily a magical things that can change the world, but at least to make us can stand on our own feet.

The second teacher is still young, just got back from his PhD. At first, it looked like he’s kinda textbooks, you know like someone talks like the book says. But after some class, I just realized that his knowledge is huge. I bet he’s reading a lot of books, that explains how he knows a lot. His vast knowledge inspires me to work harder, because there’re so many things in the world that I don’t know yet.

Well, I think that’s enough now. I got something to do too. Wish me luck for the mid term exam. Thanks for reading..

Related posts:

1. Not on the same page
2. It’s a New Year!!
3. Write Free

It happened in the third year of my college. We were having parallel programming course. In the middle of the course, our teacher gave us some cases that we had to solve using parallel principles. The class was divided into some groups with their own cases to solve. Every group must presents the solution twice. The first presentation tells the parallel principle used to solve the case. While in the second, the group must shows the implementation of the solution they presented earlier. There were some weeks range between the first and the second presentation.

Dijkstra shortest-path algorithm ilustration

I was considered lucky as my group got Dijkstra’s shortest-path algorithm. The aim of this algorithm is to find the shortest path between two places using weighted graph. The hardest part is we must paralellized it, so that it can be implemented in more than one processor. So we begin our search at Google. Luckily, some researchers were already implemented it. Well, it took some times for us to understand it. But we’re ready for the first presentation.

So, we presented it. The main idea of the parallelization is to divide the graph into some grids. Each grid consists of some nodes. Then, each processor handle the distance measurement for each grids. I don’t really remember about the details, but I hope my explanation is enough.

At the end of the presentations, one friend of mine asked a question. This man is one of the top-geek persons in our class. He said that our idea will be hard to implement. He said that we couldn’t ensure that every nodes is evenly distributed on every processors. And here it comes my coolest moment.

I didn’t know what I’m thinking at that time. But I must say a word to defend our group. And the words unconsciously came from my mouth.

“Well, we don’t know unless we’re trying. Besides, we’re still students that in the learning progress. It doesn’t matter if it will be failed, but at least we try to do it.”

It looked like every single person in that room astonished by my talk. To be frank, me too. I never thought that I could say something like that. I’m still smiling if I remember that moment. This might be subjective, but I think that was the coolest moment in my life. At least up until now.

So, what is your coolest moment? I’ll be happy to read it.

Related posts:

1. Scheduling
2. Alternative Softwares
3. Wireless at Home

I still remember the old days when I started to write blog. Like a child who found a new toy, I showed it off to all of the people. Not just showed off, but also tempted my friends to start their own blog. And I still remembered it clearly as day, one of my pals said these things. He said that what are blog used for? It’s just a narcissistic writing by some dudes, telling about their lives and bla bla bla. But you know that I don’t buy it, because you’re reading mine, right? And I don’t think I am a narcissist (0kay, maybe a little).

I’ll give you my story. It started some years ago. I was thinking it was cool to have a place where I can write some and access it from everywhere. Yes, I’m talking about writing on the clouds. At that time, I wasn’t thinking about what should I wrote. I just wanna write, that’s all.

Then I stumbled in this tech-paper. It has special page on them, featured some great blogs to visit. At the end of the page, they wrote an article about Wordpress, one of blog engines on the clouds. After read it, I did some research about what this blog things. What is it? How this thing works? And as soon as I finished my preliminary research, I was ready to start a blog.

Back to the main question, why am I blogging? The reason might be weird. I love to read. And as I read, these words suddenly appear in my mind. Every now and then, all the books and materials I’ve read transformed into words stream that almost take over my head. One thing that I know, I should pour these words into something to keep me sane.

At my early blogging days, I arranged these words into something like association maps. I associated one word to another. I tried to find the ideas that can collect these words into a single post. Almost anytime, everywhere, I was always thinking, what I should write next. And here’s the funny part. Sometimes I got a good idea when I was taking a bath or (sorry) defecate.

So here comes the social networks. The microblogging (or status updates) has changed the blogosphere. Now, you do not need to write a number of paragraph to tell other people. At one side it’s good because you can read the main idea quickly. The bad thing, because of its limitation, sometimes you missed the writer’s points.

So, to sum it up:
Why am I blogging? To keep me sane.
Why I wrote this post? I wanna get these words out of my head.
Why are you blogging? Write it down at the comment section. I’ll be glad to hear it.

Related posts:

1. The Copy Paste Tale
2. The Catcher in The Rye
3. Short Break