Ad hoc Share

Local DNS cachin Linux

2013-03-15T00:24:00.002-07:00

Linux Local DNS caching using dnsmasq

Source: http://www.webupd8.org/2009/12/faster-browsing-in-linux-with-local-dns.html

FASTER BROWSING IN LINUX WITH LOCAL DNS CACHE

A local DNS cache can help for faster browsing since you’re caching the DNS request instead of attempting that request multiple times. The internet speed will not get any faster, but the browsing speed will improve, because on each website there are usually quite a few DNS requests for which the local DNS cache will be used, bringing the query time to almost 0. You can find more info about DNS, on Wikipedia.

To see how fast your current domain name servers (DNS) are, open a terminal and paste this:

dig yahoo.com

(Or dig google.com or whatever domain)

You should see something like this:

; <<>> DiG 9.6.1-P1 <<>> yahoo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42045
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;yahoo.com.   IN A

;; ANSWER SECTION:
yahoo.com.  20142 IN A 69.147.114.224
yahoo.com.  20142 IN A 209.131.36.159
yahoo.com.  20142 IN A 209.191.93.53

;; Query time: 50 msec
;; SERVER: 208.67.220.220#53(208.67.220.220)
;; WHEN: Wed Dec  9 13:21:48 2009
;; MSG SIZE  rcvd: 75

Notice the "Query time" in bold. It's usually somewhere near 50 msec. (it depends on your domain name servers).

Run this one more time. If the query time decreases to less than 5 msec, it means your internet service provider DNS already uses some caching method and you do not need to follow this how-to. If the response time is almost the same and you are using a cable (broadband) internet connection, you can use this guide to cache the DNS for faster internet browsing.

Firstly, I would like to thank embraceubuntu for this how-to, I've just made it more newbie-friendly. So the credits go to him.

Before we get started, please note that there is an easier method on doing this (by only installing (well, actually you need to edit /etc/bind/named.conf.options with your ISP DNS) resolvconf and bind9: sudo apt-get install resolvconf bind9) but in my tests, using resolvconf and bind9, the first DNS query time was 200-300 msec (maybe it needs some tweaking but I couldn't figure out why there is such a big query time the first time) and then since it was cached, it went to 0. But the method I am going to explain will get an initial query time equal to your default DNS (~50 msec for me, as opposed to 200-300 msec which I got by using resolvconf and bind9).

Let's get started!

Manually configuring the local DNS cache

1. Install DNSMasq:

sudo apt-get install dnsmasq

2. Configure dnsmasq.conf

Press Alt + F2 and type:

gksu gedit /etc/dnsmasq.conf

Now search for "listen-address" (it's on line 90 on my Ubuntu Karmic installation), remove the "#" character in front of "listen-address" and add "127.0.0.1" after the "=" (all without the quotes). Basically, this is how the "listen-address" line should look like after editing it:

listen-address=127.0.0.1

(Optional) You can also edit the cache size if you want. Search for this in the same file: "#cache-size=150" (it's on line 432 on my Ubuntu Karmic installation), remove the "#" character in front of the line (this uncomments it) and change "150" with the size you want for you DNS cache. This is how the line should look after editing it:

cache-size=500

Obviously, "500" can be any number you want.

Don't forget to save the changes!

3. Edit dhclient.conf

Press Alt + F2 and type:

gksu gedit /etc/dhcp3/dhclient.conf

For newer Ubuntu versions (tested on Oneiric), dhclient.conf has moved, so use the following command instead:

gksu gedit /etc/dhcp/dhclient.conf

And modify the "prepend domain-name-servers" (it's on line 20 on my computer) to look like this:

prepend domain-name-servers 127.0.0.1;

4. Edit resolv.conf

Press Alt + F2 and paste this:

gksu gedit /etc/resolv.conf

Initially, this is how the resolv.conf file looks like:

nameserver ISP_DNS1
nameserver ISP_DNS2

Where ISP_DNS1 and ISP_DNS2 are your ISP domain name servers (or 8.8.4.4, etc if you are using Google DNS and so on).

Put this as the first line in your resolv.conf file:

nameserver 127.0.0.1

Which means this is how your resolv.conf file will look like:

nameserver 127.0.0.1
nameserver ISP_DNS1
nameserver ISP_DNS2

Again, ISP_DNS1 and ISP_DNS2 are your ISP domain name servers.

As an example, this is how my resolv.conf file looks like (using local DNS cache, a Google DNS and an OpenDNS DNS):

nameserver 127.0.0.1
nameserver 8.8.4.4
nameserver 208.67.220.220

4.1 If you are using a DSL connection, you need to make sure the ppp client will not overwrite your /etc/resolv.conf file. To do this, press Alt + F2, and paste this:

gksu gedit /etc/ppp/peers/provider

Search for "usepeerdns" and replace it with "#usepeerdns" (we used "#" to comment that line so it's ignored).

5. Restart your networking and dnsmasq:

-Networking:

sudo /etc/init.d/networking restart

-DNS:

sudo /etc/init.d/dnsmasq restart

Please note that you can use this last command at any time you want to restart your DNS cache (flush DNS, clear the cache - call it whatever you want) without restarting the computer.

6. Testing

To see the peformance improvement, open a terminal and type:

dig yahoo.com

The first time, it should be the same like in the beginning of the post (~50 msec. for me). Now type it again! You should see something like this:

dig yahoo.com

; <<>> DiG 9.6.1-P2 <<>> yahoo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57501
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;yahoo.com.   IN A

;; ANSWER SECTION:
yahoo.com.  20982 IN A 209.131.36.159
yahoo.com.  20982 IN A 69.147.114.224
yahoo.com.  20982 IN A 209.191.93.53

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Dec  9 14:43:41 2009
;; MSG SIZE  rcvd: 75

0 msec. query time, because the domains are now cached.

Note: Using the method above, the DNS cache will be cleared once you reboot your computer. For persistent DNS caching (on the hard disk), see this excellent how-to on Ubuntu Forums.

Smartphones full HD 1080p resolution 2013

2013-03-09T01:52:00.000-08:00

Sony Xperia Z

5.0 inches TFT capacitive touchscreen
Full HD support Resolution of 1080 x 1920 pixels
Protection layer of Shatter proof and scratch-resistant glass and possess Sony Mobile BRAVIA Engine 2

Sony Xperia ZL

5.0 inch TFT capacitive touchscreen
Resolution of 1080 x 1920 pixels
Protection layer of Shatter proof and scratch-resistant glass and possess Sony Mobile BRAVIA Engine 2

ZTE Grand S

5 inch full-HD screen
Resolution of 1920 x 1080 pixels

Lenovo K900

5.0 inch IPS LCD touchscreen
Resolution of 1920 x 1080 pixels

Huawei Ascend D2

5.0 inch IPS LCD capacitive touchscreen
Resolution of 1080 x 1920 pixels
Protection layer of Corning Gorilla Glass

HTC Droid DNA

5.0 inch Super LCD3 capacitive touchscreen
Resolution of 1080 x 1920 pixels
Protection layer of Corning Gorilla Glass 2

Samsung Galaxy S4

5.0 inch Super AMOLED capacitive touchscreen
Resolution of 1080 x 1920 pixels
Protection layer of Corning Gorilla Glass 2

HTC Butterfly

5.0 inch Super LCD3 capacitive touchscreen
Resolution of 1080 x 1920 pixels
Protection layer of Corning Gorilla Glass 2

International shipping to India

2013-03-03T00:43:00.002-08:00

Services

1. iShopInternational.com

Description source: http://forum.xda-developers.com/showpost.php?p=36983040&postcount=2

Try ishopinternational.com
You have to provide them with amazon link and they will give you quotation price which includes product cost from amazon, shipping to india and customs (<10%). You have to pay them in their Indian account and they will purchase from amazon and deliver to you at your doorstep. You can use coupon code amazon200 or amazon250 to get 250 rs discount.

2. www.ppobox.com

3. www.hopshopgo.com

4. www.shopandship.com

5. www.borderlinx.com

5. www.comgateway.com

Android Development resources

2013-03-01T02:51:00.004-08:00

References -

http://mobileorchard.com/android-app-developmentthreading-part-1-handlers/
http://mobileorchard.com/android-app-developmentthreading-part-2-async-tasks/

My AVD commandline

"-scale 0.6 -qemu -m 512 -enable-kvm"

Android emulator taking too much of screen size?

2013-02-27T23:13:00.001-08:00

Source: http://stackoverflow.com/questions/2359895/android-emulator-screen-too-tall

(Little modified)

Using AVD Manager

Open AVD Manager

If using eclipse then Go to Window -> Android SDK and AVD Manager -> Virtual Devices

Select the AVD you want to launch and click Start
Check the "Scale display to real size" button
Enter how big you want it to appear in inches and press Launch. For this to work, you'll have to also enter a reasonable approximation of your mac's screen resolution. I'm using 7 inches and 113 dpi for my 13" Macbook Pro, but you may be able to get away with 8 or 9 inches.

While debugging (add this to command line)

Source: http://stackoverflow.com/questions/2359895/android-emulator-screen-too-tall/4963984#4963984

This is actually possible from your project as well, no need to start the emulator through the AVD manager:

1) go to Run > Run Configurations... > (Select your application on the left hand side) > (Click the "Target" tab on the right hand side). 2) At the bottom there, you'll see 'Emulator launch parameters'. In the 'additional emulator command line options', add '-scale 0.75' (to make the screen 75% of full size)

Next time you start the emulator it will have scaled properly, hooray!

Change when the Emulator is running

Source: http://stackoverflow.com/a/6049246

There is also a way to re size the emulator through a windows command prompt.

From command prompt run: telnet localhost 5554
window scale 0.75
quit

Assuming there is one emulator running with on port 5554.

How to Do a Clean Install of Windows 8 with an Upgrade Disc

2013-02-17T23:25:00.001-08:00

How to Do a Clean Install of Windows 8 with an Upgrade Disc:

Sometimes, you just need to do a clean install. Unfortunately, the Windows 8 Upgrade doesn't always allow for that, throwing you an error when you try to activate after a clean install. Reader uncommoner shows us a workaround for this issue.

If you do a clean install using the Windows 8 Upgrade Assistant, you should be fine—but if you've already formatted your drive or you're moving to a new drive, you can't do a "clean install" without installing an old version of Windows first. It'll let you install Windows 8 cleanly, but when you go to activate, you get an error 0x8007007B, saying your product key can only be used for upgrading.

If you get that error, here's how to fix it:

Press the Windows key and type regedit. Press enter to open the Registry Editor.
Navigate toHKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE/and double-click on the MediabootInstall key in the right pane.
Change the key's value from 1 to 0.
Exit the Registry Editor, press the Windows key again, and type cmd. Right-click on the Command Prompt icon and run it as an administrator.
Type slmgr /rearm and press Enter.
Reboot Windows.

When you get back into Windows, you should be able to run the Activation utility and activate Windows as normal, without getting an error. Obviously, you could use this trick for evil, but it has its legitimate place too—if, say, you're upgrading your hard drive and want to do a fresh install on it, or if you formatted your drive before upgrading.

We haven't had a chance to test it ourselves, but it's been well documented around the net, so we're confident it should work for you if you're getting this particular error. If you give it a shot, let us know how it works for you in the discussions below! Thanks for the tip, uncommoner!

How to Do a Clean Install of Windows 8 with an Upgrade License

Cannot delete files - You'll need to provide administrator permission to delete this folder

2013-01-26T22:23:00.001-08:00

Source: http://forum.thewindowsclub.com/windows-tips-tutorials-articles/18379-how-take-ownership-full-control-permissions-files-folders-windows.html

How to Take Ownership and Full Control Permissions of Files & Folders in Windows

A lot of files and folders in Windows 7 & Vista does not actually belongs to users. Rather, most system files have “Trusted Installer” as owner, the assign or grant read+write, traverse or full control permissions to SYSTEM or CREATOR OWNER user account only. So users must take ownership and grant full access control permissions and rights to themselves if they want to modify, rename or delete these files or folders. Sometimes, users may need to take ownership and grant full rights to themselves on another drive or partition, especially on disk newly installed or inserted if they cannot browse the contents from the drive.

To take ownership and grant full control (or read write) permissions of files or folders in Windows Vista, do these steps.

1. In Windows Explorer window, locate the files or folders that you want to take ownership and grant or change full control or other access permissions.
2. Right click on the file or directory, and then select Properties on the right click menu.

3. Click on Security tab.
4. Click on Advanced button at the bottom.

5. In “Advanced Security Settings” dialog window, click on Owner tab.
6. Here you will be able to see current owner (i.e. TrustedInstaller). To take ownership of the object, click on the Edit button. If UAC prompts for administrator’s password or permission to continue, enter the correct password or press Continue button.

7. Additional “Advanced Security Settings” dialog will appear. Highlight the user name (for example, Administrators) in the Change owner to box that you want to assign as the owner for the object. Click OK to make the change.

8. Back in original parent level “Advanced Security Settings” window, you will see the existing owner of the file or folder has changed to the user you just selected.
9. Click OK button to exit this window.
10. Click OK again to exit completely from the Properties window.
11. The ownership is now belonged to user or user account that been selected. To assign necessary permissions to the user too, repeat step 1 to 3 to open the object’s Properties window again.
12. In object’s Properties window, click on Edit button to change permissions. If UAC prompts for administrator’s password or permission to continue, enter the correct password or press Continue button.

13. Highlight the Administrators or the user who wants the permissions on the object be changed in the “Group or user names” box.

If the user ID or group that you want to manage the permissions for the object doesn’t exist, click on Add button, and type in the user name or group name desired into the Enter object names to select box, and finish off by clicking on OK.
14. In the Permissions for Administrators box below (or any other user name or group name you chose), click on “Full Control” under the “Allow” column to assign full access rights control permissions to Administrators group.

15. Click “OK” twice when done.

Users can now do whatever you like to the files or directories processed as above. If you feel that above process is a little too long, and prefer to use command line to perform above process, then open an elevated command prompt as administrator, and issues the following commands:

For Files:

takeown /f file_name /d y
icacls file_name /grant administrators:F

For Folders or Directories (will perform action recursively):

takeown /f directory_name /r /d y
icacls directory_name /grant administrators:F /t

Replace file_name or directory_name with actual file name or folder name, with path when applicable. The first command will take ownership of the file or folder specified, and the second command will grant full control permissions to administrators user group. Note that when using command for folders, to command will run recursively. To prevent the task been perform recursively, remove the “/r” and “/t” switch.

Above two commands have been scripted in Windows command shell batch script that easily perform the task of taking ownership and grant full control permissions to Administrators user group by typing simple command. Alternatively, users can add “Take Control Of” option to right clickmenu so that the next time you need to take control of a file with full control permissions, it’s just a one click task.

HDInsight mapreduce - Hadoop API for .NET

2013-01-21T04:34:00.002-08:00

Taken from: http://hadoopsdk.codeplex.com/wikipage?title=Getting%20Started%20With%20Map%20Reduce&referringTitle=Map%2fReduce

Hadoop API for .NET
=====================

Introduction

----------
Hadoop Streaming is a facility for writing map-reduce jobs in the language of you choice. Hadoop API for .NET is a wrapper to Streaming that provides a convenient experience for .NET developers. An understanding of the concepts and general functionality provided by Hadoop Streaming is necessary for successful use of this API: see http://hadoop.apache.org/common/docs/r0.20.0/streaming.html for this background information.

The main facilities provided by this API are:

1. Abstraction of job execution to avoid manual construction of streaming command-line.

2. Mapper, Reducer, Combiner base classes and runtime wrappers that provide helpful abstractions. For example, the ReducerCombinerBase class provides input through (string key, IEnumerable<string> value) groups.

3. Detection of .NET dependencies and automatic inclusion in streaming job.

4. Local unit-testing support for map/combine/reduce classes via StreamingUnit class

5. Support for JSON I/O and strongly typed mapper/combiner/reducer via Json* classes. The pattern used by the JSON classes can be used to create other serialization wrappers.

When jobs are submitted via the API a Hadoop Streaming command is generated and executed. The command is displayed on the console and can be used for direct invocation if required.

Input & Output formats

--------------------

The input/output format supported is line-oriented tab-separated records, staged in a Hadoop-supported file system such as HDFS or Azure Blob Storage. The input may comprise many files but each should have a consistent format: records delimited by \n\r, columns delimited by \t.

When a job comprises both a mapper and reducer, the key values emitted by the mapper must be plain text that can be sorted successfully with an ordinal-text-comparer such as provided by .NETs `StringComparison.Ordinal`.

In all other cases the record fields may comprise formatted text such as Json or other text representation of structured data. The API includes support for Json fields via the classes in the `Microsoft.Hadoop.MapReduce.Json` namespace.

If data is in a binary format or document-oriented format (such as a folder full of .docx files), the input to a map-reduce job will typically be files that list the path to each real file, one path per line. The mapper can then look up the files using whatever API is appropriate.

Example Map-Reduce program

------------------------

A .NET map-reduce 'program' comprises a number of parts:

1. A job definition. This declares the `MapperType`, `ReducerType`, `CombinerType` and configuration settings

2. Mapper, Reducer and Combiner classes

4. Input data. Typically staged to HDFS or Azure Storage prior to job execution. The most common approach is via the Hadoop file-system utility. For example,

> hadoop fs -copyFromLocal localFile input/folder/file

5. A job-executor. This can either be the `MRRunner.exe` that is part of the API distribution, or by creating a Main() function in your .NET application that invokes `HadoopJobExecutor`.

To create a mapper-only job:

1. Create a new C# project and reference `Microsoft.Hadoop.MapReduce.DLL`.

2. Create a class the implements `HadoopJob<FirstMapper>`.

3. Create a class called `FirstMapper` that implements `MapperBase`.

For example, the following is a complete map-reduce 'program' that consumes files containing integers and produces output that includes `sqrt(x)` for each input value.

    public class FirstJob : HadoopJob<SqrtMapper>
    {
        public override HadoopJobConfiguration Configure(ExecutorContext context)
        {
            HadoopJobConfiguration config = new HadoopJobConfiguration();
            config.InputPath = "input/SqrtJob";
            config.OutputFolder = "output/SqrtJob";
            return config;
        }
    }

    public class SqrtMapper : MapperBase
    {
        public override void Map(string inputLine, MapperContext context)
        {
            int inputValue = int.Parse(inputLine);

            // Perform the work.
            double sqrt = Math.Sqrt((double)inputValue);

            // Write output data.
            context.EmitKeyValue(inputValue.ToString(), sqrt.ToString());
        }
    }

To run this program, stage some data in HDFS:

1. create a text file called input.txt that has one integer per line.

2. import that text file to HDFS via

> hadoop fs -copyFromLocal input.txt input/SqrtJob/input.txt

3. compile your .NET code to a DLL called FirstJob.dll and run it via

> MRRunner -dll FirstJob.dll

When this runs, the console will display the complete Hadoop Streaming command issued and then the normal console output from the hadoop streaming command itself.

To see detailed information about the execution of current and past jobs, use the Hadoop streaming web front-end, typically accessible at http://localhost:50030.

To explore the HDFS filesystem, use the HDFS web front-end that is typically accessible at http://localhost:50080.

When the job completes, output will be available in HDFS at `/user/user/output/SqrtJob`.

HadoopJob class

-------------

The `Job<>` class defines the users components that are included in a map-reduce program. Generic parameters are used to declare the Mapper, Combiner and Reducer classes that should be used by the job. Configuration parameters are supplied by overriding the method

public abstract HadoopJobConfiguration Configure(ExecutorContext context);

To implement this method, instantiate a `HadoopJobConfiguration` object then set its members and return.

HadoopJobConfiguration class

--------------------------

The configuration for a job is a strongly typed bag of settings that are largely passed directly to the hadoop streaming command-line. Some settings result in non-trivial settings but most are straight forward. Only a subset of Hadoop Streaming settings are directly exposed through the Configuration object. All settings are usable through catch-all factilities: `config.AdditionalStreamingArguments` and `config.AdditionalGenericArguments`.

HadoopJobExectuor class

---------------------

HadoopJobExecutor handles the creation and execution of a complete Hadoop Streaming command-line. It can be called in various ways. The first is to use the MRRunner.exe utility which will invoked HadoopJobExecutor on your behalf

> MRRunner -dll MyMRProgram.dll {-class jobClass} {-- job-class options}

The second is to invoke the executor directly and request it execute a HadoopJob

HadoopJobExecutor.Execute<JobType>(arguments)

A third approach is to avoid the JobType and just invoke directly mentioning MapperType etc and a Configuration object

HadoopJobExecutor.Execute<TMapper,..>(configuration)

MRRunner

------
MRRunner is a command-line utility used to execute a map-reduce program written against the Hadoop for .NET API. To get started, you should have an assembly (a .net DLL or EXE) that defines at least one implementation of HadoopJob<>.

If MyDll contains only one implementation of Hadoop<>, you can run the job with

> MRRunner -dll MyDll

If MyDll contains multiple implementations of HadoopJob<>, indicate the one you wish to run

> MRRunner -dll MyDll -class MyClass

To supply options to your job, pass them as trailing arguments on the command-line, after a double-hyphen

> MRRunner -dll MyDll -class MyClass -- extraArg1 extraArg2

These additional arguments are provided your your job via a context object that is available to all methods on HadoopJob<>

MapperBase

--------

A MapperBase implementation describes how to perform the Map function. The input to Map will be a subset of the rows of the input. On each call to the Mapper.Map(string input, MapperContext context) method, a single line will be provided as input. The Map method can make use of the context object to lookup relevant setting, emit output lines and emit log messages and counter updates.

For example:

    public class MyMapper : MapperBase {
        public override void Map(string inputLine, MapperContext context)
        {
            context.Log("mapper called. input=" + inputLine);
            context.IncrementCounter("mapInputs");
            context.EmitKeyValue(key, value);
        }
    }

The MapperBase class also provides overridable methods to be run at the start/end of each batch. These methods can be used to perform set up and teardown such as initialising a component.

ReducerCombinerBase

-----------------

A ReducerCombinerBase implementation describes how to perform a reduce and/or combine operation. In each case the operation takes a group and emits key/value pairs that typically represent an aggregated representation of the group. For example, the input to a reducer may be: key = 'a', values = 1,2,3 and the output might be {'a',6}. To implement `ReducerCombinerBase`, override the `Reduce(key, values, context)` method and use the context object to emit key/value pairs as necessary.

A common requirement for a map-reduce program is to reuse one reduce function as both the reducer and the combiner. This is achieved by referencing the same reducer class when declaring a HadoopJob class.

    public class MyHadoopJob<MyMapper, MyReducer, MyReducer> {
        ...
    }

Json support

----------

The primary data format for Hadoop Streaming is line-oriented text and so the normal currency of Map and Reduce implementations is System.String. It can often be convenient to transform the strings to/from .NET objects and this requires a serialization mechanism. A set of classes that use Json.NET as the serialization engine are provided in the Microsoft.Hadoop.MapReduce.Json namespace. As an example of their use, consider input data that has JSON format values:

    {ID=2, Name="Alan"}
    {ID=3, Name="Bob"}

Further, assume that a class definition that can represent the values is

    public Employee {
        public int ID {get;set;}
        public string Name {get;set;}
    }

The Json Mapper classes can help perform the deserialization and transformation to Employee instances that is required for convenient processing. Let's assume the output of the Mapper will be simple strings; in this case the appropriate Mapper type to use is JsonInMapperBase<>. For example:

    public class MyMapper : JsonInMapperBase<Employee> {
        public override void Map(Employee value, MapperContext context){
        
        }
    }

JsonInMapperBase performs the deserialization of the input lines and the instantiation of Employee objects. The Map function that must be implemented can deal with Employee inputs rather than strings.

Other classes in `Microsoft.Hadoop.MapReduce.Json` support transferring object-representations between mapper and reducer and as the output of the reducer.

Last edited Oct 25, 2012 at 8:09 PM by mikelid, version 2

Create sample input using windows batch file

2013-01-15T05:17:00.002-08:00

Script that did my job

-------------------------------------------

@echo off

setlocal EnableDelayedExpansion

for /L %%F IN (1,1,100) DO (
set /a year = !RANDOM! %% 12+2000
echo !year! !RANDOM!
)

endlocal
-------------------------------------------

Generates random numbers. First column contains year between 2000 to 2012 second column contains a random number. You would have to tweak for your requirement.

Did your day just got saved? ! :)

Oh and by the way you can direct your output to a text file as
<filename>.cmd > mysampleinput.txt

:p

References

http://ss64.com/nt/set.html
http://www.bish.co.uk/forum/index.php?topic=58.0
http://stackoverflow.com/questions/3439626/windows-batch-file-calculating-filename-from-current-date

Bus Vs Broker

2013-01-13T21:36:00.002-08:00

Bus and Broker Pub/Sub Differences

Source: http://www.udidahan.com/2011/03/24/bus-and-broker-pubsub-differences/

One of the things which often confuses people using NServiceBus for the first time is that it only allows an endpoint to subscribe to a given event from a single other publishing endpoint. The rule that there can only be a single publisher for a given event type is one of the things that differentiates buses from brokers, though both obviously allow you to have multiple subscribers.

Brokers

Message brokers, more broadly known and used on the Java platform, don’t come with this constraint. For example, when using ActiveMQ, you can have any number of endpoints come to the broker and publish a message under a given topic.

So where’s the problem?

It’s all about accountability.

Let’s say you’ve subscribed to a given topic, and have received two events – one telling you that the price of bananas next week will be $1/kg and another telling you that it’ll be $2/kg.

Which one is right?

Especially given that those events may have been published by any other endpoint via the broker.

Is it first one wins? Last one wins? How about first one sent vs. first one received? Ditto for last. As a subscriber, can you really be held accountable for having the logic to choose the right one? Shouldn’t this responsibility have fallen to the publishing side?

This is one of the big drawbacks of the broker, hub and spoke architecture. No responsibility. No single source of truth – unless everybody’s going to some central database, in which case – what’s the point of all this messaging anyway?

Buses

The Bus Architectural Style is all about accountability. If you are going to publish an event, you are accountable for the correctness of the data in that event – there is no central database that a subscriber can go to “just in case”. And the only way that you can be held accountable, is if you have full responsibility – ergo, you’re the only one who can publish that type of event.

If you say bananas are going to cost $1/kg next week, that’s that. Subscribers will not hear from anybody else on that topic.

Now, this is not to say that you can’t have more than one physical publishing endpoint.

You see, buses differentiate between the logical and the physical. Brokers tend to assume that the physical hub-and-spoke topology is also the logical.

In a bus, while there can only be one logical endpoint publishing a given type of event, that endpoint can be physically scaled out across multiple machines. It is the responsibility of the bus to provide infrastructure facilities to allow for that to happen in such a way that to subscribers, it still appears as if there is really only one publishing endpoint.

The same is true about the subscriber – one logical subscribing endpoint may be scaled out across multiple machines.

Product Mix-ups

Unfortunately, there are many broker-style technologies out there that are being marketed under the banner of the Enterprise Service Bus. While some products have the ability to be deployed in both a centralized and distributed fashion (sometimes called “federated” or “embedded” mode), many do not enforce the “single publishing endpoint per event-type” rule.

Without this constraint, it is just too easy to make mistakes.

NServiceBus

By enforcing this constraint, we see the same kind of question appear on the discussion group time and time again:

“I have an Audit event that I’d like all of my machines to publish, and have one machine subscribe to them all, but NServiceBus won’t let me. How do I make NServiceBus support this scenario?”

And the answer is the same every time:

“You should have all the machines Send the Audit message (configured to go to the single machine handling that message), and not Publish. It is not an event until its been handled by the endpoint responsible for it.”

The semantics of the message matter a lot.

When looking at Service-Oriented Architecture, these messages are the contract and, as any lawyer will tell you, contracts need to be explicit and the intentions really need to be spelled out – otherwise the contract is practically worthless.

In closing

Friction is sometimes a good thing – it prevents us from making mistakes. It keeps cars on the road. And because that’s not enough friction, we introduce curbs as well.

If you’re looking for a service bus technology for your next project, check that it’ll give you the friction that you need to keep everybody safe. Really check what it is that the vendors are offering you – more often than not, it’s some ESB lipstick on a broker pig.

To learn more about how NServiceBus supports this kind of publish/subscribe, click here.

Excel - generate random sample data file

2013-01-09T04:47:00.001-08:00

Some formulas that will help you build sample data files

1. Random number generation

=RANDBETWEEN(1,1500)

Generates random number between 1 and 1500

2. Random text generation

=CHAR(INT(RAND()*26+65))&CHAR(INT(RAND()*26+65))&CHAR(INT(RAND()*26+65))

Generates random text of 3 characters

HDInsight on Windows: Building PowerPivot Report from Hive in a Few Easy Steps

2013-01-07T02:33:00.002-08:00

Misc:

32 bit ODBC on 64 bit windows - c:\windows\syswow64\odbcad32.exe

Source:

http://blogs.msdn.com/b/karang/archive/2013/01/04/hdinsight_2d00_on_2d00_windows_2d00_building_2d00_powerpivot_2d00_report_2d00_from_2d00_hive_2d00_in_2d00_a_2d00_few_2d00_easy_2d00_steps.aspx

Karan Gulati (SSAS Maestro)

4 Jan 2013 3:20 PM

What are we Covering:

Hive
- Create table
- Load Data
- Query Data
Configure ODBC
Create PowerPivot
- With DataSource as HDInsight
- Import Data from HDInsight
Create Pivot Chat using imported data

(1)Open Hadoop Command Line

After installing HDInsight for Windows you will get Hadoop Command Line Shortcut on desktop
Click on Hadoop Command Line Shortcut and it will open Cmd Line for Hadoop

(2)Type Hive and it will change to Hive Command

(3)Create Tables

CREATE TABLE emp(id int,name string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
CREATE TABLE sales(id int,sales int)ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';

(4)Load Data using CSV Files

LOAD DATA LOCAL INPATH 'c:/data/emp.csv' OVERWRITE INTO TABLE emp;

LOAD DATA LOCAL INPATH 'c:/data/sales.csv' OVERWRITE INTO TABLE sales;

(5) Open HDInsight Dashboard (You will find shortcut on desktop) and click on Download

(6)Click on the appropriate link for your Excel.

(7)Configure ODBC
For configuring ODBC, Click start and type in ODBC. This will bring up the Data Sources dialog. Go to the System DSN tab and click Add. Configure it with a Data Source Name, Your Host Information, and a Username. In Single Node HDInsight Installation you don’t need to provide Username but if you are using HDInsight on Azure you can provide Username .

(8) Provider Data Source Name and Default Port is 10000

(9)Open Excel Sheet, select PowerPivot in ribbon and click on Manage

(10)Click on Get External Data

(11)Select – Get External Data and Data Import window and Select Others(OLEDB/ODBC)

(12)Click Build

(13)Select OLE DB Provider for ODBC

(14)Select DSN which we have created earlier in step number 8, Select Database (In Single Node HDInsight installation you don’t need to provider User Name and Password in current release but in case of HDInsight on Azure you need to provide User Name and Password) and follow the wizard for selecting tables in our case its EMP and Sales

(15)If you Notice Relationship between Emp and Sales is absent so with one click we can create relationship

(16)Added feature like Calculated Measure name Sales and created a KPI

(17)Select Pivot of your Choice from PivotTable

(18) Finally - you are good to go with PowerPivot Report based on Hive

Refer
HDInsight – What is it?
Configuring: PowerPivot for Excel 2013

HDInsight, PowerPovot, Hadoop

Multinode HDInsight cluster

2013-01-07T02:30:00.000-08:00

Source: http://social.msdn.microsoft.com/Forums/en-US/hdinsight/thread/885efc22-fb67-4df8-8648-4ff38098dac6

Espen S

0 Points

I've setup HDInsight to work with multiple nodes in a lab environment. (Meaning I've ignored security etc. And there's no easy way to control the start-up and shutdown of the nodes in the cluster)

Setting up a multi node cluster on Windows with HDInsight is not significantly different from setting up a multi-node Hadoop cluster on any other platform. I do recommend reading the following link: http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/

Breaking it down to a few simple steps:
1. Install HDInsight on your master (hmaster) and your nodes (hnode1..hnodeN) For ease of configuration make sure the master and the nodes have static IPs.

2. Turn off all firewalls and other port blocking software on all nodes. (Make sure the machines have access to each other)

3. On both the master and on ALL nodes edit %WinDir%\system32\drivers\etc\hosts and add the following lines

<ip-to-master> hmaster
<ip-to-node1> hnode1
......
<ip-to-nodeN> hnodeN

(Replace <ip-to-xxx> with the ip-address of the nodes)

4. On the master edit the file masters (Found in C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\ on my setup)

And make sure the only line in the file is the name of the master (must match what you just entered in the hosts file):
hmaster

5. On the master edit the slaves file:
(Found in C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\ on my setup)
Add the names of all the hnodes, names must match what you just entered in the hosts file

hnode1
...
hnodeN

6. On the master and on ALL nodes edit the file core-site.xml (Found in C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\ on my setup)

Find the property fs.default.name and change it's value to hdfs://hmaster:8020.

7. On the master and on ALL nodes edit the file mapred-site.xml (Found in C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\ on my setup)

Modify all values that make references to localhost to reference hmaster instead.

8. On the master and on ALL nodes edit the file hdfs-site.xml (Found in C:\Hadoop\hadoop-1.1.0-SNAPSHOT\conf\ on my setup)

Modify all values that reference localhost to reference hmaster instead.

Also make sure that the value of property dfs.replication is less or equal to the total number of nodes in your system. (Note that the description on this property seems to be wrong in the HDInsight supplied conf files)

9. Now go to the master and all nodes open a command prompt and execute the shutdown script stop-onebox.cmd. (Found in C:\Hadoop) .

10. During the HDInsight install all the nodes were installed as namenodes and the HDFS filesystem was initialized upon the first service startup. So now the file systems on each node has different IDs. If you try to start your system now you'll get some errors in the log saying something about invalid IDs. The easiest way to fix this on a fresh system is to simply delete the hdfs folder and restart the nodes.

A simple way to fix this is to delete the HDFS folder (c:\Hadoop\hdfs on my system) on each node and let the system recreate the folder on start-up.

11. Log on to the master, start a command prompt and invoke the start-up script (c:\Hadoop\start-onebox.cmd)

Repeat Step 11 for all nodes.

Monitor your namenode status by navigating to
http://hmaster:50070/

And your mapreduce status at
http://hmaster:50030/jobtracker.jsp

There're several drawbacks to this approach, and you should never go live with this setup. One thing is security, another is that all the nodes are configured to be namenodes and they run a few services that should not be on the nodes. The installation includes scripts to start the system as a node but I've tried to keep this walkthrough as simple as possible.

I do recommend starting from scratch when setting up a multi-node cluster, and not migrating from a single node cluster to a multi node cluser. I ran into several issues with hive and the metadata (that it by default stores in derby) when doing this. Hive stores the full hdfs path in the metadata tables (DB_LOCATION_URI column in DBS and LOCATION in SDS). So if you change from a single node to a multinode configuration you're likely to run into the issue of hive trying to access the data on the localhost. To fix the issue you need to modify the data in the derby tables.

About HDInsight

2013-01-07T01:57:00.000-08:00

Source: http://www.sqlmag.com/article/sql-server-2012/microsoft-doug-leland-hekaton-hdinsight-sql-server-2012-144898

Otey: Right; this is the new Windows version of Hadoop that you implemented either on-premise[s] or in Azure as a service?

Leland (Microsoft): Yes; so HDInsight for Windows is the on-premise[s] implementation that was announced at Strata. So now our customers broadly have downloadable access to preview bits, for Hadoop on a Windows implementation -- and a preview for the service.
The benefits of these different approaches are kind of like the following. First, we are the only vendor out there who is providing both on-premise[s] offerings and cloud offerings. So the customer has incredible flexibility in their ability both to choose whatever solutions that they need. If I want to run it in my data center, then I'll run it in my data center. If I want to run it in Microsoft's data center, I could now run it in [a] Microsoft data center. I can move it back and forth with the same implementation, 100 percent code compatible. And I can build hybrid scenarios where I may have a piece of the implementation running on data that lives in my data center; or I've got data that's been born in the cloud and I don't really want to bring it back on the prem, so I'll move it into the service. So the flexibility in terms of what we're offering is incredibly valuable to customers in terms of choices that we offer. The flexibility in the scenarios of my data center, the cloud, or some combination of both is also incredibly helpful. And then you kind of bring that back and say, "Now I can use the power of Excel 2013 to get to that data . . . incredibly valuable propositions.
The value of the service, of the HDInsight service, is simpler to the value proposition than you get for products. As a company, I'm not investing in the capital of the infrastructure to build out the software, and I don't have to buy a bunch of machines to build up a cluster. I don't need to hire a Hadoop expert to know how to deploy and build out a Hadoop cluster because I can go to the service, and with basically three clicks and ten minutes, I can deploy a cluster of any size. I pay for that by capacity, but again I don't need that implementation expertise. What I do need is the individual who knows how to build my jobs.

Source: http://www.microsoft.com/web/gallery/install.aspx?appid=HDINSIGHT-PREVIEW

HDInsight is Microsoft's Hadoop-based distribution, built on the Hortonworks Data Platform for Windows (HDP). It combines the simplicity of Windows with the power and reliability of the HDP to deliver big data insights from Apache Hadoop. This is a preview release targeting developer scenarios and as such only support single-node deployments.

-----
Reads:

India - Mera Bharat Mahaan

2013-01-04T04:21:00.001-08:00

Pratibha Patil spent Rs 18 crore on her last trip as President, RTI query reveals

Source: http://timesofindia.indiatimes.com/india/Pratibha-Patil-spent-Rs-18-crore-on-her-last-trip-as-President-RTI-query-reveals/articleshow/18007898.cms

NEW DELHI: Notwithstanding a huge controversy over expenditure on her foreign travels, the then President Pratibha Patil ran up a bill of Rs 18.08 crore on her last trip abroad shortly before demitting office, according to official information accessed through RTI.

The chartering of the Air India Boeing 747-400 jumbo for her two-nation trip to South Africa and Seychelles from April 29 to May 8 last year alone cost Rs 16.38 crore, the airline said in an RTI reply.

In addition, an expenditure of Rs 1.46 crore was incurred in Pretoria -- the South African capital. Of this, Rs 71.82 lakh was spent on local stay, 52.33 lakh on transportation and 22.12 lakh on miscellaneous expenditure.

In Durban, an expenditure of Rs 23.55 lakh was incurred. Of this hotel stay alone cost nearly 18 lakh and transportation was Rs 5.27 lakh.

The details about the lodging and other expenditures in South Africa were provided by the Indian Missions in Pretoria and Durban under RTI.

A huge controversy broke out last year when it was revealed that Patil had incurred an expenditure of Rs 205 crore on her 12 foreign trips covering 22 countries across four continents during her five-year term which ended on July 25 last year.

These figures were provided before she undertook the last visit to South Africa and Seychelles.

The Rashtrapati Bhavan had then defended these visits terming them "necessary" to deepen bilateral cooperation. The visits were undertaken after careful appraisal and recommendation by the Prime Minister's office and the ministry of external affairs.

Asaram Bapu rants again, calls media 'barking dogs'

Source: http://www.hindustantimes.com/India-news/NewDelhi/Asaram-Bapu-rants-again-calls-media-barking-dogs/Article1-986670.aspx

A day after blaming the Delhi gangrape victim for the henious crime, Asaram Bapu justified his comments calling those opposing his views as 'barking dogs'. "The media has created a controversy but what wrong did I say? A man complained to me that his wife fights with him; I told him you can't clap with one hand," he said.

"One dog barked and more dogs (the media) joined in. Dogs will bark but they can't harm an elephant's dignity. I didn't intend to harm anybody; I wish the well being of all.."

The family of 23-year-old Delhi gangrape victim has taken a strong exception to ... (sigh, there is more to read)

Delhi rape victim as guilty as her rapists: Asaram Bapu

Source: http://www.rediff.com/news/report/delhi-rape-victim-as-guilty-as-her-rapists-asaram-bapu/20130107.htm

Spiritual Guru Asaram Bapu has landed himself in a controversy over his remark that the December 16 Delhi [ Images ] gang-rape victim is as guilty as those responsible for the barbaric sexual assault on her.

"Only 5-6 people are not the culprits. The victim daughter is as guilty as her rapists. She should have called the culprits' brothers and begged before them to stop. This could have saved her dignity and life. Can one hand clap? I don't think so," media reports quoted Asaram Bapu, as saying.

According to media reports, the self-proclaimed godman further said that he is against harsh punishments for the accused, as the law could be mis-utilised.

"We have often seen such laws are made to be mis-utilised. Dowry harassment law is the biggest example," he said.

The spiritual guru's remark comes at a time when the entire nation is mourning the death of the 23-year-old girl who died in a Singapore hospital 13 days later after the heinous crime.

Sonia Gandhi travelled in IAF aircraft 49 times in last 7 years

Source: http://news.rediff.com/commentary/2013/jan/04/liveupdates.htm, http://articles.timesofindia.indiatimes.com/2013-01-04/india/36147762_1_iaf-aircraft-air-travel-iaf-rules

15:50 Sonia travelled in IAF aircraft 49 times in last 7 years: UPA Chairperson Sonia Gandhi travelled in Indian Air Force aircraft and helicopters 49 times in the last seven years which included 23 trips with Prime Minister Manmohan Singh.

Congress General Secretary Rahul Gandhi also travelled in IAF aircraft and helicopters eight times in the last three years, according to a reply received under the Right to Information Act.

Both Sonia Gandhi and Rahul Gandhi are not eligible to travel by IAF aircraft and helicopters so they have to travel in the company of eligible persons which includes the Prime Minister, the Deputy Prime Minister, the Home Minister and the Defence Minister for official purposes.

For unofficial purposes, only the Prime Minister is eligible. Other Cabinet ministers can also travel by these aircraft after taking permission of the Prime Minister.

----------
Reads:

http://texolo.blogspot.in/2012/10/open-letter-to-robert-vadra.html

Extending laptop battery life

2013-01-02T22:22:00.000-08:00

Taken from: http://batterycare.net/en/guide.html

Memory Effect

First of all it's necessary to unfold a myth that persists in many peoples head.
The battery memory effect.

In lithium-based batteries this is in fact a myth, it only applies to older Nickle-based batteries. So fully discharging and charging the battery is completely useless and even harmful as we will see below.

The modern lithium battery can be charged regardless of its current percentage, given that it has absolutely no negative effect in its performance.

Should I remove the battery when A/C is plugged in?

Many laptop users have this question and we will answer it right now:
The answer is: YES and NO, it depends on the situation.

Having a battery fully charged and the laptop plugged in is not harmful, because as soon as the charge level reaches 100% the battery stops receiving charging energy and this energy is bypassed directly to the power supply system of the laptop.

However there's a disadvantage in keeping the battery in its socket when the laptop is plugged in, but only if it's currently suffering from excessive heating caused by the laptop hardware.

So:

- In a normal usage, if the laptop doesn't get too hot (CPU and Hard Disk around 40ºC to 50ºC) the battery should remain in the laptop socket;

- In an intensive usage which leads to a large amount of heat produced (i.e. Games, temperatures above 60ºC) the battery should be removed from the socket in order to prevent unwanted heating.

The heat, among the fact that it has 100% of charge, is the great enemy of the lithium battery and not the plug, as many might think so.

Battery discharges

Full battery discharges (until laptop power shutdown, 0%) should be avoided, because this stresses the battery a lot and can even damage it. It's recommended to perform partial discharges to capacity levels of 20~30% and frequent charges, instead of performing a full discharging followed by a full charging.

Laptop batteries contain a capacity gauge that allows us to know the exact amount of energy stored. However, due to the charging/discharging cycles, this sensor tends to be inaccurate overtime.

Some laptops include in their BIOS, tools to recalibrate this battery gauge, which is nothing more than a full discharge followed by a full charge.
So to calibrate the gauge, it should be performed, in every 30 discharge cycles, a full discharge non-stop , followed by a also, non-stop, full charge.

An inaccurate gauge can lead to the fact that the the battery capacity values are are wrong. The battery may report that it still has 10% of capacity when in fact it has a much lower value, and this causes the computer to shutdown unexpectedly.

Discharge (or charge) cycles consist of using all that battery charge (100%) but not necessarily all at once.
For example, you can use the laptop for some minutes in a day, using half its capacity e then fully charge it. If you did the same thing in the next day, it would be counted a discharge cycle and not two, so it may take several days until a full discharge cycle is completed.

How to perform a calibration (full discharge)?

The most adequate method to do a full discharge (100% to a minimum of 5%) consists of the following procedure:

Fully charge the battery to its maximum capacity (100%);
Let the battery "rest" fully charged for 2 hours or more in order to cool down from the charging process. You may use the computer normally within this period;
Unplug the power cord and set the computer to hibernate automatically at the minimum percentage possible as described by the image sequence below (click images to enlarge);
Leave the computer discharging, non-stop, until it hibernates itself. You may use the computer normally within this period;
When the computer shuts down completely, let it stay in the hibernation state for 5 hours or even more;
Plug the computer to the A/C power to perform a full charge non-stop until its maximum capacity (100%). You may use the computer normally within this period.

After the calibration process, the reported wear level is usually higher than before. This is natural, since it now reports the true current capacity that the battery has to hold charge. Lithium Ion batteries have a limit amount of discharge cycles (generally 200 to 300 cycles) and they will retain less capacity over time.

Many people tend to think "If calibrating gives higher wear level, then it's a bad thing". This is wrong, because like said, the calibration is meant to have your battery report the true capacity it can hold, and it's meant to avoid surprises like, for example, being in the middle of a presentation and suddenly the computer shuts down at 30% of charge.

Prolonged storage

To store a battery for long periods of time, its charge capacity should be around 40% and it should be stored in a place as fresh and dry as possible. A fridge can be used (0ºC - 10ºC), but only if the battery stays isolated from any humidity.
One must say again that the battery's worst enemy is the heat, so leaving the laptop in the car in a hot summer day is half way to kill the battery.

Purchasing a replacement battery

If you intend to purchase another battery, it's recommended that you do it only when the current battery is very degraded. If it's not the case, the non usage of a battery leads to its degradation.
If a spare battery is purchased and won't be used for a long time, the above storage method should be used.
Besides that, when purchasing a battery you must pay attention to the manufacturing date.

Advantages in using BatteryCare

BatteryCare allows you to have the control over the discharge cycles number, and when this reaches 30 (or other configured value), it notifies you that it's time to perform a full discharge in order to keep the battery gauge calibrated.
Like this, it's guaranteed to always have the correct capacity values reported by the battery.

Besides, when using the battery, there's the possibility to suspend some Operating System features that help degrading the autonomy (only in Windows Vista or higher):

- Windows Aero, the theme that allows for visual effects like window transparency, requires graphics card acceleration, which obviously will help decreasing the battery lifetime;

- SuperFetch, ReadyBoost and SearchIndexer are three Windows Vista (and higher) services that, even in battery mode, are using the hard disk a lot and increase total power consumption, thus decreasing battery lifetime. Suspending these services has absolutely no negative impact on the performance or security of the system.

These features are resumed once the laptop is plugged in to A/C power.

External links:

Main reference
Apple.com
Table with information about other types of batteries

Big data store - HBase

2012-12-27T22:26:00.001-08:00

Taken from: http://blogs.igalia.com/dpino/2012/10/31/introduction-to-hbase-and-nosql-systems/

Introduction to HBase and NoSQL systems

October 31st, 2012diego pinoGo to comment

s

HBase is an open source, non-relational, distributed database modelled after Google’s BigTable (Source: HBase, Wikipedia). BigTable is a data store that relies on GFS (Google Filesystem). Since Hadoop is an open source implementation of GFS and MapReduce, it perfectly made sense to build HBase on top of Hadoop.

Usually HBase is categorized as a NoSQL database. NoSQL is a term often used to refer tonon-relational databases. For instance, Graph databases, Object-Oriented databases, Key-Value data stores or Columnar databases. All of them are NoSQL databases. In the recent years there have been an emerging interest in this type of systems as the relational model has proved to be no effective to solve certain problems, especially those related to storing and handling large amounts of data.

In the year 2000, Berkeley researcher Eric Brewer published a now foundational paper known as the CAP Theorem. This theorem states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:

Consistency. All nodes see the same data at the same time.
Availability. A guarantee that every request receives a response about whether it was successful or failed.
Partition tolerance. The system continues to operate despite arbitrary message loss or failure of part of the system.

According to the theorem, a distributed system can satisfy any two of these guarantees at the same time, but not all three (Source: CAP theorem, Wikipedia).

Usually NoSQL systems are depicted within a triangle representing the CAP Theorem, being each of the angles one the aforementioned guarantees: consistency, availability andpartition tolerance. Each system is located in one of the sides of the triangle, depending on the pair of features favoured.

Visual Guide to NoSQL Systems (Source: http://blog.nahurst.com/visual-guide-to-nosql-systems)

As it is shown in the figure above, HBase is a columnar database that guarantees consistencyof data and partition tolerance. On the other hand , systems like Cassandra or Tokyo Cabinetfavour availability and partition tolerance.

Why NoSQL sytems have become relevant in the recent years? For the last 20 years the storage capacity of hard drives have multiply by several orders of magnitude, however seek and transfer times have not evolved at the same pace. Websites like Twitter receive more data everyday that it can write to a single hard drive, so data has to be written in clusters. Twitter users generate more than 12 TB per day, about 4 PB per year. Other sites, such as Google, Facebook or Netflix, handle similar figures, which means facing the same type of problems. But, big data storage and analysis is not something that only affects websites, for instance, the Large Scale Hadron Collider of Geneva produces about 12 PB per year.

HBase Data Model

HBase is a Columnar data store, also called Tabular data store. The main difference of acolumn-oriented database compared to a row-oriented database (RBMS) is about how data is stored in disk. Check how the following table would be serialized using a row-oriented and a column-oriented approach (Source: Columnar Database, Wikipedia).

EmpId	Lastname	Firstname	Salary
1	Smith	Joe	40000
2	Jones	Mary	50000
3	Johnson	Cathy	44000

Row-oriented

1,Smith,Joe,40000;
2,Jones,Mary,50000;
3,Johnson,Cathy,44000;

Column-oriented

1,2,3;
Smith,Jones,Johnson;
Joe,Mary,Cathy;
40000,50000,44000;

Physical organization has an impact on features such as partitioning, indexing, caching, views, OLAP cubes, etc. For instance, since common data is stored together, column-oriented excel at operations about aggregating data.

In HBase data is stored in tables, same as in the relational model. Each table consists of rows, each identified by a RowKey. Each row has a fixed number of column families. Each column family can contain a sparse number of columns. Columns also support versioning, that means, that different versions of the same column can exist at the same time. Versioning is usually implemented using a timestamp.

So, to fetch a value from a table we will need to specify three values: <RowKey, ColumnFamily, Timestamp>.

Perhaps the most obscure concept of this model are Column Families. Column Families consist of two parts: a prefix and a qualifier. The prefix is always fixed and it has to be specified when the table is created. However, the qualifier is dynamic and new qualifiers can be added to prefixes at run time. This allows to created an infinite collection of columns inside of column families. Take a look at the table below representing information about Students.

RowKey	Timestamp	ColumnFamily
Student1	t1	courses:history=”H0112″
Student1	t2	courses:math=”M0212″
Student2	t3	courses:history=”H0112″
Student2	t4	courses:geography=”G0112″
Student2	t5	courses:geography=”G0212″

It is possible to add new courses just by storing new column families with the prefix courses. To get the code of the history subject Student1 is enrolled in , we need to provide three values: Student1; courses:history; timestamp. It is possible to retrieve all values in case a column family, with a different timestamp, is repeated. In the table above, we can see howStudent2 is enrolled in the Geography course (timestamp=t4, courses:geography=”G0112″, however he enrolled again because the code of the subject changed (timestamp=t5; courses:geography=”G0212″).

Column Families work as a sort of light schema for the tables. Column families have to be specify when a table is created, and it is actually very hard to modify this schema. However, as the qualifier part of a column family is dynamic, it is very easy to add new columns to existingcolumn families.

For further explanation about HBase Data Model I recommend the following articles: Hadoop Wiki - HBase DataModel, Understanding HBase and BigTable.

Features of HBase

HBase is built on top of Hadoop, that means, it relies on HDFS and it integrates very well with the MapReduce framework. Relying on HDFS provides a series of benefits:

A distributed data storage running on top of commodity hardware
Redundancy of data
Fault-tolerant

In addition, HBase provides other series of benefits (among other features):

Random reads and writes (this is not possible with plain Hadoop).
Autosharding. Sharding, horizontal data distribution, is done automatically.
Automatic failover based on Apache Zookeeper.
Linear scaling of capacity. Just add new nodes as you need them.

Installing HBase

HBase depends on Hadoop, so it is necessary to install Hadoop before installing HBase. Currently there are several Hadoop branches being developed at the same time, so it is strongly recommended to install a HBase version that is compatible with a specific version of Hadoop (v1.0, v.0.22, v.0.23, etc). HBase v.0.90.6 is compatible with Hadoop v.1.x. To install HBase follow these steps:

Install Hadoop if necessary.
Download HBase.
Move the downloaded tar file to /usr/local/share.

Now, uncompress it:

$ sudo tar xfvz hbase-0.90.6.tar.gz

Run HBase:

bin/start-hbase.sh

By default, HBase listens on port 60010. When a HBase server is running it is possible to check its state by connecting to http://locahost:60010.

Interacting with the HBase shell

First, start a HBase shell:

$ hbase shell

Once you are into a HBase session, create a new table:

> create 'students', 'courses'

Now insert some data into the table ‘students’.

put 'students', 'Student1', 'courses:history', 'H0112'

put 'students', 'Student1', 'courses:math', 'M0212'

put 'students', 'Student2', 'courses:history', 'H0112'

put 'students', 'Student2', 'courses:geography', 'G0112'

put 'students', 'Student2', 'courses:geography', 'G0212'

And now try the command ‘scan’ to show the contents of a table.

> scan 'students'

ROW                                    COLUMN+CELL

Student1                              column=courses:history, timestamp=1351332046854, value=H0112

Student1                              column=courses:math, timestamp=1351332046914, value=M0212

Student2                              column=courses:geography, timestamp=1351332047022, value=G0212

Student2                              column=courses:history, timestamp=1351332046950, value=H0112

2 row(s) in 0.0390 seconds

The command ‘scan’ returns two rows (Student1 and Student2). Notice that the column family‘courses:geography’ was inserted twice with different values, but only one value is shown. One of the features of HBase is versioning of data, this means that different versions of the same data can exist in the same table. Generally, this is implemented via a timestamp. So, why those two values don’t show up? To do so, it is necessary to tell scan how many versions of a row we would like to retrieve.

scan 'students', {VERSIONS => 3}

ROW                                    COLUMN+CELL

Student1                              column=courses:history, timestamp=1351332046854, value=H0112

Student1                              column=courses:math, timestamp=1351332046914, value=M0212

Student2                              column=courses:geography, timestamp=1351332047022, value=G0212

Student2                              column=courses:geography, timestamp=1351332046990, value=G0112

Student2                              column=courses:history, timestamp=1351332046950, value=H0112

2 row(s) in 0.0170 seconds

By default, when a table is created, the number of versions per Column Family is 3. However, it is possible to specify more:

> create 'students', {NAME => 'courses', VERSIONS => 10}

To fetch one Student:

> get 'students', 'Student2'

COLUMN                                 CELL

courses:geography                     timestamp=1351332047022, value=G0212

courses:history                       timestamp=1351332046950, value=H0112

2 row(s) in 0.3100 seconds

The result displays all the current subjects Student2 is enrolled in.

If now we want to disenroll Student2 from subject ‘history’:

> delete 'students', 'Student2', 'courses:history'

<pre>get 'students', 'Student2'

COLUMN                                 CELL

courses:geography                     timestamp=1351332047022, value=G0212

1 row(s) in 0.0140 seconds

Lastly, we drop table ‘students‘. Dropping a table is two-step operation:

> disable 'students'

0 row(s) in 2.0420 seconds

And now we can actually drop the table:

> drop 'students'

0 row(s) in 1.0790 seconds

Summary

This post was a brief introduction to HBase. HBase is a Columnar Database, usually categorized as a NoSQL database. HBase is built on top of Hadoop and shares many concepts with Google’s BigData, mainly its data model. In HBase data is stored in tables, being each table composed of rows and column families. A column family is a pair prefix:qualifier, whereprefix is a fixed part and qualifier is variable. HBase also supports versioning by storing timestamp information for every column family. So, to retrieve a single value from a HBase table, the user has to specify three keys: <RowKey, ColumnFamily, Timestamp>. This way of storing/retrieving data makes that HBase is sometimes referred as a large sparse hash map.

As HBase relies on Hadoop it comes with the common features Hadoop provides: a distributed filesystem, data node fault tolerancy and good integration with the MapReduce framework. In addition, HBase provides other very interesting features such as autosharding, automatic failover, linear scaling of capacity and random reads and writes. The last one is a very interesting feature as plain Hadoop only allows to process the whole dataset in a batch.

And this is all for now. On a following post I will explain how to connect HBase with Hadoop, so it is possible to run MapReduce jobs over data stored in a HBase data store.

HDInsight

2012-12-24T04:41:00.002-08:00

Resources

Getting started - http://blogs.msdn.com/b/windowsazure/archive/2012/10/24/getting-started-with-windows-azure-hdinsight-service.aspx
Running map-reduce jobs - http://gettingstarted.hadooponazure.com/hw/mapReduce.html
Dev resources - http://andyelastacloud.azurewebsites.net/?tag=hdinsight
(coming up)

Misc

Setting up an offline shadow copy of HDInsight
(Reference: http://adhocshare.blogspot.in/2012/12/offline-installation-of-products-on-web.html)

Steps:

> webpicmd /Offline /Products:HDINSIGHT-PREVIEW /Path:D:\HDInsight.setup
> WebPICmd.exe /Install /Products:HDINSIGHT-PREVIEW /XML:D:\HDInsight.setup\feeds\shadow-webproductlist.xml

Shame India / Gangrape / terror / I hate India

2012-12-19T07:59:00.002-08:00

Protests rock Delhi over gangrape, protesters face water cannon outside CM Sheila Dikshit's residence

Read more at:http://indiatoday.intoday.in/story/delhi-gangrape-protest-across-capital-sheila-dikshit-sushilkumar-shinde/1/238534.html

I heard a person, a protester protesting that he wanted basic rights, basic security instead of the big dreams that is shown by government. He did not wanted to be an IT person, or a scientist or anybody else contributing to the nation. He wanted his mother, and sisters to be just safe!

With respect to the recent Delhi rape case one of the other things that come to my mind is that, cant the local transport be equipped with surveillance devices? I saw those devices in almost every bus I traveled with in US.

Would these devices be expensive to install? Or would we face another corruption case giving the contract to a money eating vendor? Or the Government will prefer not to spend money over an individual pains and losses (like rape cases! ) ?

If you are reading this, I would like to know of what you think. Pretty sure that the others would as well like to know your opinion.

Offline installation of products on Web platform installer

2012-12-18T04:43:00.000-08:00

Important commands

1. Cache local copy of the product and dependencies

> WebPICMD.exe /Offline /Products:[All|<Title or ID>] /Path:[offline-directory-location]

2. Install from the local copy

> WebPICmd.exe /Install /Products:<products you want> /XML:<Offline main feed>

3. List all the available product on web platform

> WebPICmd.exe /List /ListOption:All

Sample:

> WebPICmd.exe /List /ListOption:All > products.txt

products.txt file is stored in the current working directory. It contains the product IDs of the products.

References:

1. http://social.msdn.microsoft.com/Forums/et-EE/hdinsight/thread/1bd4ee83-d7a6-4491-95f6-a7547c123998

2. Web Platform Installer v4 command line (WebPICMD.exe) Preview Release

http://www.iis.net/learn/install/web-platform-installer/web-platform-installer-v4-command-line-webpicmdexe-preview-release_1072

Hadoop on Windows Azure

2012-12-17T04:18:00.004-08:00

Taken from: http://msdn.microsoft.com/en-us/magazine/jj190805.aspx

Hadoop on Windows Azure

Lynn Langit

There’s been a lot of buzz about Hadoop lately, and interest in using it to process extremely large data sets seems to grow day by day. With that in mind, I’m going to show you how to set up a Hadoop cluster on Windows Azure. This article assumes basic familiarity with Hadoop technologies. If you’re new to Hadoop, see “What Is Hadoop?” As of this writing, Hadoop on Windows Azure is in private beta. To get an invitation, visit hadooponazure.com. The beta is compatible with Apache Hadoop (snapshot 0.20.203+).

What Is Hadoop?

Hadoop is an open source library designed to batch-process massive data sets in parallel. It’s based on the Hadoop distributed file system (HDFS), and consists of utilities and libraries for working with data stored in clusters. These batch processes run using a number of different technologies, such as Map/Reduce jobs, and may be written in Java or other higher-level languages, such as Pig. There are also languages that can be used to query data stored in a Hadoop cluster. The most common language for query is HQL via Hive. For more information, visit hadoop.apache.org.

Setting Up Your Cluster

Once you’re invited to participate in the beta, you can set up your Hadoop cluster. Go to hadooponazure.com and log in with your authorized Windows Live ID. Next, fill out the dialog boxes on the Web portal using the following values:

Cluster (DNS) name: Enter name in the form “<your unique string>.cloudapp.net”.
Cluster size: Choose the number of nodes, from 4 to 32, and their associated storage allocations, from 2TB to 16TB per cluster.
Administrator username and password: Enter a username and password; password complexity restrictions are listed on the page. Once this is set you can connect via remote desktop or via Excel.
Configuration information for a SQL Azure instance: This is an option for storing the Hive Metastore. If it’s selected, you’ll need to supply the URL to your SQL Azure server instance, as well as the name of the target database and login credentials. The login you specify must have the following permissions on the target database: ddl_ddladmin, ddl_datawriter, ddl_datareader.

After you’ve filled in this information, click Request cluster. You’ll see a series of status updates in the Web portal as your cluster (called Isotope in the beta) is being allocated, created and started. For each cluster you allocate, you’ll see many worker nodes and one head node, which is also known as the NameNode.
After some period of time (five to 30 minutes in my experience), the portal will update to show that your cluster is allocated and ready for use. You can then simply explore the Metro-style interface (by clicking the large buttons) to see what types of data processing and management tasks you can perform (see Figure 1). In addition to using the Web portal to interact with your cluster, you might want to open the available ports (closed by default) for FTP or ODBC Server access. I’ll discuss some alternative methods of connecting in a bit.

Figure 1 The Windows Azure Hadoop Portal
In the Your Cluster section of the portal, you can perform basic administrative tasks such as configuring access to your cluster, importing data and managing the cluster via the interactive console. The interactive console supports JavaScript or Hive. As Figure 1 shows, you can also access the Your Tasks section. Here you can run a MapReduce job (via a .jar file) and see the status of any MapReduce jobs that are running as well as those that have recently completed.
The portal buttons display information about three recently completed MapReduce jobs: C# Streaming Example, Word Count Example and 10GB Terasort Example. Each button shows the status of both the Map and the Reduce portion of each job. There are several other options for viewing the status of running (or completed) MapReduce jobs directly from the portal and via other means of connecting to your cluster, such as Remote Desktop Protocol (RDP).

Connecting to Your Data

You can make data available to your Hadoop on Windows Azure cluster in a number of ways, including directly uploading to your cluster and accessing data stored in other locations.
Although FTP allows uploading theoretically any size data files, best practice is to upload files that are in a lower gigabyte-size range. If you want to run batch jobs on data stored outside of Hadoop, you’ll need to perform a couple of configuration steps first. To set up outside connections, click on the Manage Cluster button on the main portal and then configure the storage locations you wish to use, such as a Windows Azure Blob storage location, a Windows Azure Data Market query result or an Amazon Web Services (AWS) S3 storage location:

To configure a connection to an AWS S3 bucket, enter your security keys (both public and private) so you can access data stored on S3 in your Hadoop cluster.
To work with data from the Windows Azure Data Market, fill in values for username (WLID), passkey (for the data source you wish to query and import), source (extract) query and (destination) Hive table name. Be sure to remove the parameter for default query limit (100 rows) from the query generated by the tools in the Data Market before you enter the query into the text box on your cluster.
To access data stored in Windows Azure Blob storage, you’ll need to enter the storage account name (URL) to the Blob storage locations and your passkey (private key) value.

Running a MapReduce Job

After setting up and verifying your Hadoop cluster, and making your data available, you’ll probably want to start crunching this data by running one or more MapReduce jobs. The question is, how best to start? If you’re new to Hadoop, there are some samples you can run to get a feel of what’s possible. You can view and run any of these by clicking the Samples button on the Web portal.
If you’re experienced with Hadoop techniques and want to run your own MapReduce job, there are several methods. The method you select will depend on your familiarity with the Hadoop tools (such as the Hadoop command prompt) and your preferred language. You can use Java, Pig, JavaScript or C# to write an executable MapReduce job for Hadoop on Windows Azure.
I’ll use the Word Count sample to demonstrate how to run a MapReduce job from the portal using a .jar file. As you might expect, this job counts words for some input—in this example a large text file (the contents of an entire published book)—and outputs the result. Click Samples, then WordCount to open the job configuration page on the portal, as shown in Figure 2.

Figure 2 Setting Up the WordCount Sample
You’ll see two configurable parameters for this job, one for the function (word count) and the other for the source data (the text file). The source data (Parameter 1) includes not only the name of the input file, but also the path to its location. This path to the source data file can be text, or it can be “local,” which means that the file is stored on this Hadoop Windows Azure cluster. Alternatively, the source data can be retrieved from AWS S3 (via the S3n:// or the S3:// protocol), from the Windows Azure Blob storage (via the ASV:// protocol) or from the Windows Azure Data Market (by first importing the desired data using a query), or be retrieved directly from the HDFS store. After you enter the path to a remote location, you can click on the verification icon (a triangle) and you should get an OK message if you can connect using the string provided.
After you configure the parameters, click Execute job. You’ll find a number of ways to monitor both job status as the job is executing and job results after the job completes. For example, on the main portal page, the Your Tasks section displays a button with the status of the most recent jobs during execution and after completion. A new button is added for each job, showing the job name, percentage complete for both the Map and the Reduce portions during execution, and the status (OK, failed and so forth) after job completion.
The Job History page, which you can get to from the Manage Your Account section of the main page, provides more detail about the job, including the text (script) used to run the job and the status, with date and time information. You can click the link for each job to get even more information about job execution.
If you decide to run a sample, be sure to read the detailed instructions for that particular sample. Some samples can be run from the Web portal (Your Tasks | Create Job); others require an RDP connection to your cluster.

Using JavaScript to Run Jobs

Click on the Interactive Console button to open the JavaScript console. Here you can run MapReduce jobs by executing .jar files (Java) by running a Pig command from the prompt, or by writing and executing MapReduce jobs directly in JavaScript.
You can also directly upload source data from the js> prompt using the fs.put command. This command opens a dialog box where you can choose a file to upload to your cluster. IIS limits the size of the file you can upload via the JavaScript console to 4GB.
You can also use source data from other remote stores (such as Windows Azure Blobs) or from other cloud vendors. To work with source data from AWS S3, you use a request in the format s3n://<bucket name>/<folder name>.
Using the JavaScript console, you can verify connectivity to your AWS S3 bucket by using the #ls command with the bucket address, like so:

js> #ls s3n://HadoopAzureTest/Books

Found 2 items

-rwxrwxrwx   1          0 2012-03-30 00:20 /Books

-rwxrwxrwx   1    1395667 2012-03-30 00:22 /Books/davinci.txt

When you do, you should get a list of the contents (both folders and files) of your bucket as in this example.
If you’d like to review the contents of the source file before you run your job, you can do so from the console with the #cat command:

js> #cat s3n://HadoopAzureTest/Books/davinci.txt

After you verify that you can connect to your source data, you’ll want to run your MapReduce job. The following is the JavaScript syntax for the Word Count sample MapReduce job (using a .jar file):


var map = function (key, value, context) {
  var words = value.split(/[^a-zA-Z]/);
  for (var i = 0; i < words.length; i++) {
    if (words[i] !== "") {
      context.write(words[i].toLowerCase(), 1);
    }
  }
};
var reduce = function (key, values, context) {
  var sum = 0;
  while (values.hasNext()) {
    sum += parseInt(values.next());
  }
  context.write(key, sum);
};

In the map portion, the script splits the source text into individual words; in the reduce portion, identical words are grouped and then counted. Finally, an output (summary) file with the top words by count (and the count of those words) is produced. To run this WordCount job directly from the interactive JavaScript console, start with the pig keyword to indicate you want to run a Pig job. Next, call the from method, which is where you pass in the location of the source data. In this case, I’ll perform the operation on data stored remotely—in AWS S3.
Now you call the mapReduce method on the Pig job, passing in the name of the file with the JavaScript code for this job, including the required parameters. The parameters for this job are the method of breaking the text—on each word—and the value and datatype of the reduce aggregation. In this case, the latter is a count (sum) of data type long.
You then specify the output order using the orderBy method, again passing in the parameters; here the count of each group of words will be output in descending order. In the next step, the take method specifies how many aggregated values should be returned—in this case the 10 most commonly occurring words. Finally, you call the to method, passing in the name of the output file you want to generate. Here’s the complete syntax to run this job:

pig.from("s3n://HadoopAzureTest/Books").mapReduce("WordCount.js", "word, count:long").orderBy("count DESC").take(10).to("DaVinciTop10Words.txt")

As the job is running, you’ll see status updates in the browser—the percent complete of first the map and then the reduce job. You can also click a link to open another browser window, where you’ll see more detailed logging about the job progress. Within a couple of minutes, you should see a message indicating the job completed successfully. To further validate the job output, you can then run a series of commands in the JavaScript console.
The first command, fs.read, displays the output file, showing the top 10 words and the total count of each in descending order. The next command, parse, shows the same information and will populate the data variable with the list. The last command, graph.bar, displays a bar graph of the results. Here’s what these commands look like:

js> file = fs.read("DaVinciTop10Words.txt")

js> data = parse(file.data, "word, count:long")

js> graph.bar(data)

An interesting aspect of using JavaScript to execute MapReduce jobs is the terseness of the JavaScript code in comparison to the Java. The MapReduce WordCount sample Java job contains more than 50 lines of code, but the JavaScript sample contains only 10 lines. The functionality of both jobs is similar.

Using C# with Hadoop Streaming

Another way you can run MapReduce jobs in Hadoop on Windows Azure is via C# Streaming. You’ll find an example showing how to do this on the portal. As with the previous example, to try out this sample, you need to upload the needed files (davinci.txt, cat.exe and wc.exe) to a storage location such as HDFS, ASV or S3. You also need to get the IP address of your Hadoop HEADNODE. To get the value using the interactive console, run this command:

js>#cat apps/dist/conf/core-site.xml

Fill in the values on the job runner page; your final command will look something like this:

Hadoop jar hadoop-examples-0.20.203.1-SNAPSHOT.jar

-files "hdfs:///example/apps/wc.exe,hdfs:///example/apps/cat.exe"

-input "/example/data/davinci.txt"

-output "/example/data/StreamingOutput/wc.txt"

-mapper "cat.exe"

-reducer "wc.exe"

In the sample, the mapper and the reducer are executable files that read the input from stdin, line by line, and emit the output to stdout. These files produce a Map/Reduce job, which is submitted to the cluster for execution. Both the mapper file, cat.exe, and the reducer file, wc.exe, are shown in Figure 3.

Figure 3 The Mapper and Reducer Files
Here’s how the job works. First the mapper file launches as a process on mapper task initialization. If there are multiple mappers, each will launch as a separate process on initialization. In this case, there’s only a single mapper file—cat.exe. On execution, the mapper task converts the input into lines and feeds those lines to the stdin portion of the MapReduce job. Next, the mapper gathers the line outputs from stdout and converts each line into a key/value pair. The default behavior (which can be changed) is that the key is created from the line prefix up to the first tab character and the value is created from the remainder of the line. If there’s no tab in the line, the whole line becomes the key and the value will be null.
After the mapper tasks are complete, each reducer file launches as a separate process on reducer task initialization. On execution, the reducer converts key/value input pairs into lines and feeds those lines to the stdin process. Next, the reducer collects the line-oriented outputs from the stdout process and converts each line into a key/value pair, which is collected as the output of the reducer.

Using HiveQL to Query a Hive Table

Using the interactive Web console, you can execute a Hive query against Hive tables you’ve defined in your Hadoop cluster. To learn more about Hive, see hive.apache.org.
To use Hive, you first create (and load) a Hive table. Using our WordCount MapReduce sample output file (DavinciTop10Words.txt), you can execute the following command to create and then verify your new Hive table:

hive> LOAD DATA INPATH

 'hdfs://lynnlangit.cloudapp.net:9000/user/lynnlangit/DaVinciTop10Words.txt'

 OVERWRITE INTO TABLE wordcounttable;

 hive> show tables;

 hive> describe wordcounttable:

 hive> select * from wordcounttable;

Hive syntax is similar to SQL syntax, and HiveQL provides similar query functionality. Keep in mind that all data is case-sensitive by default in Hadoop.

Other Ways to Connect to Your Cluster

Using RDP In addition to working with your cluster via the Web portal, you can also establish a remote desktop connection to the cluster’s NameNode server. To connect via RDP, you click the Remote Desktop button on the portal, then click on the downloaded RDP connection file and, when prompted, enter your administrator username and password. If prompted, open firewall ports on your client machine. After the connection is established, you can work directly with your cluster’s NameNode using the Windows Explorer shell or other tools that are included with the Hadoop installation, much as you would with the default Hadoop experience.
My NameNode server is running Windows Server 2008 R2 Enterprise SP1 on a server with two processors and 14GB of RAM, with Apache Hadoop release 0.20.203.1 snapshot installed. Note that the cluster resources consist of the name node and the associated worker nodes, so the total number of processors for my sample cluster is eight.
The installation includes standard Hadoop management tools, such as the Hadoop Command Shell or command-line interface (CLI), the Hadoop MapReduce job tracker (found at http://[namenode]:50030) and the Hadoop NameNode HDFS (found at http://[namenode]:50070). Using the Hadoop Command Shell you can run MapReduce jobs or other administrative tasks (such as managing your DFS cluster state) via your RDP session.
At this time, you can connect via RDP using only a Windows client machine. Currently, the RDP connection uses a cookie to enable port forwarding. The Remote Desktop Connection for Mac client doesn’t have the ability to use that cookie, so it can’t connect to the virtual machine.
Using the Sqoop Connector Microsoft shipped several connectors for Hadoop to SQL Server in late 2011 (for SQL Server 2008 R2 or later or for SQL Server Parallel Data Warehouse). The Sqoop-based SQL Server connector is designed to let you import or export data between Hadoop on Linux and SQL Server. You can download the connector from bit.ly/JgFmm3. This connector requires that the JDBC driver for SQL Server be installed on the same node as Sqoop. Download the driver at bit.ly/LAIU4F.
You’ll find an example showing how to use Sqoop to import or export data between SQL Azure and HDFS in the samples section of the portal.
Using FTP To use FTP, you first have to open a port, which you can do by clicking the Configure Ports button on the portal and then dragging the slider to open the default port for FTPS (port 2226). To communicate with the FTP server, you’ll need an MD5 hash of the password for your account. Connect via RDP, open the users.conf file, copy the MD5 hash of the password for the account that will be used to transfer files over FTPS and then use this value to connect. Note that the MD5 hash of the password uses a self-signed certificate on the Hadoop server that might not be fully trusted.
You can also open a port for ODBC connections (such as to Excel) in this section of the portal. The default port number for the ODBC Server connections is 10000. For more complex port configurations, though, use an RDP connection to your cluster.
Using the ODBC Driver for Hadoop (to Connect to Excel and PowerPivot) You can download an ODBC driver for Hadoop from the portal Downloads page. This driver, which includes an add-in for Excel, can connect from Hadoop to either Excel or PowerPivot. Figure 4 shows the Hive Pane button that’s added to Excel after you install the add-in. The button exposes a Hive Query pane where you can establish a connection to either a locally hosted Hadoop server or a remote instance. After doing so, you can write and execute a Hive query (via HiveQL) against that cluster and then work with the results that are returned to Excel.

Figure 4 The Hive Query Pane in Excel
You can also connect to Hadoop data using PowerPivot for Excel. To connect to PowerPivot from Hadoop, first create an OLE DB for ODBC connection using the Hive provider. On the Hive Query pane, next connect to the Hadoop cluster using the connection you configured previously, then select the Hive tables (or write a HiveQL query) and return the selected data to PowerPivot.
Be sure to download the correct version of the ODBC driver for your machine hardware and Excel. The driver is available in both 32-bit and 64-bit editions.

Easy and Flexible—but with Some Unknowns

The Hadoop on Windows Azure beta shows several interesting strengths, including:

Setup is easy using the intuitive Metro-style Web portal.
You get flexible language choices for running MapReduce jobs and data queries. You can run MapReduce jobs using Java, C#, Pig or JavaScript, and queries can be executed using Hive (HiveQL).
You can use your existing skills if you’re familiar with Hadoop technologies. This implementation is compliant with Apache Hadoop snapshot 0.203+.
There are a variety of connectivity options, including an ODBC driver (SQL Server/Excel), RDP and other clients, as well as connectivity to other cloud data stores from Microsoft (Windows Azure Blobs, the Windows Azure Data Market) and others (Amazon Web Services S3 buckets).

There are, however, many unknowns in the version of Hadoop on Windows Azure that will be publicly released:

The current release is a private beta only; there is little information on a roadmap and planned release features.
Pricing hasn’t been announced.
During the beta, there’s a limit to the size of files you can upload, and Microsoft included a disclaimer that “the beta is for testing features, not for testing production-level data loads.” So it’s unclear what the release-version performance will be like.

To see video demos (screencasts) of the beta functionality of Hadoop on Windows Azure, see my BigData Playlist on YouTube at bit.ly/LyX7Sj.

Lynn Langit (LynnLangit.com) runs her own technical training and consulting company. She designs and builds data solutions that include both RDBMS and NoSQL systems. She recently returned to private practice after working as a developer evangelist for Microsoft for four years. She is the author of three books on SQL Server Business Intelligence, most recently “Smart Business Intelligence Solutions with SQL Server 2008” (Microsoft Press, 2009). She is also the cofounder of the non-profit TKP (TeachingKidsProgramming.org).

Thanks to the following technical expert for reviewing this article: Denny Lee

Hadoop on Azure: An Introduction

2012-12-17T03:45:00.000-08:00

Taken from: http://architects.dzone.com/articles/hadoop-azure-introduction

Hadoop on Azure: An Introduction

11.22.2012

| 4227 views |

I am in complete awe on how this technology is resonating with today’s developers. If I invite developers for an evening event, Big Data is always a sellout. This particular post is about getting everyone up to speed about what Hadoop is at a high level. Big data is a technology that manages voluminous amount of unstructured and semi-structured data. Due to its size and semi-structured nature, it is inappropriate for relational databases for analysis. Big data is generally in the petabytes and exabytes of data.

However, it is not just about the total size of data (volume)
It is also about the velocity (how rapidly is the data arriving)
What is the structure? Does it have variations?

Sources of Big Data

Science	Scientists are regularly challenged by large data sets in many areas, including meteorology, genomics, connectomics, complex physics simulations, and biological and environmental research.
Sensors	Data sets grow in size in part because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, radio-frequency identification readers, and wireless sensor networks.
Social networks	I am thinking of Facebook, LinkedIn, Yahoo, Google
Social influencers	Blog comments, YELP likes, Twitter, Facebook likes, Apple's app store, Amazon, ZDNet, etc
Log files	Computer and mobile device log files, web site tracking information, application logs, and sensor data. But there are also sensors from vehicles, video games, cable boxes or, soon, household appliances
Public Data Stores	Microsoft Azure MarketPlace/DataMarket, The World Bank, SEC/Edgar, Wikipedia, IMDb
Data warehouse appliances	Teradata, IBM Netezza, EMC Greenplum, which includes internal, transactional data that is already prepared for analysis
Network and in-stream monitoring technologies	Packets in TCP/IP, email, etc
Legacy documents	Archives of statements, insurance forms, medical record and customer correspondence

Two problems to solve

Storage Problem	How do I store a petabyte of data reliably? Afterall, a petabyte is over 333 three TB drives.
Money Problem	1 petabyte costs a lot. For just 70 TB you will pay over $100,000. (eBay ad Dell/EMC CLARiiON CX3-40 -70TB- FAST 4G 15K SAN Storage is only 70 TB for $112,000)

Two seminal papers

The Google File System	It is about a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients	http://research.google.com/archive/gfs.html
MapReduce: Simplified Data Processing on Large Clusters	MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key	http://research.google.com/archive/mapreduce.html

Hadoop : What is it?

Hadoop is an open-source software framework that supports data-intensive distributed applications. Hadoop is written in Java.
I met its creator, Doug Cutting, who was working at Yahoo at the time. Hadoop is named after his son's toy elephant. I was hosting a booth at the time, and I remember Doug was curious about finding some cool stuff to bring home from the booth to give to his son. Another great idea, Doug!
One of the goals of Hadoop is to run applications on large clusters of commodity hardware. The cluster is composed of a single master and multiple worker nodes.
Hadoop leverages the the programming model of map/reduce. It is optimized for processing large data sets.
MapReduce is typically used to do distributed computing on clusters of computer. A cluster had many “nodes,” where each node is a computer in a cluster.
The goal of map reduce is to break huge data sets into smaller pieces, distribute those pieces to various slave or worker nodes in the cluster, and process the the data in parallel. Hadoop leverages a distributed file system to store the data on various nodes.

It is about two functions Hadoop comes down to two functions. As long as you can write the map() and reduce() function, your data type is supported, whether we are talking abuot (1) text files (2) xml files (3) json files (4) even graphics, sound or video files.

The core is map() and reduce()

Understanding these methods is the key to mastering Hadoop

Map Step	The map step is all about dividing the problem into smaller sub-problems. A master node has the job of distributing the work to worker nodes. The worker node just does one thing and returns the work back to the master node.
Reduce Step	Once the master gets the work from the worker nodes, the reduce() step takes over and combines all the work. By combining the work you can form some answer and ultimately output.

public class WordCount {

 

public static class Map extends MapReduceBase

implements Mapper<LongWritable, Text, Text, IntWritable>

{

 

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

 

public void map(LongWritable key, Text value,

OutputCollector<Text, IntWritable> output,

Reporter reporter) throws IOException

{

 

String line = value.toString();

StringTokenizer tokenizer = new StringTokenizer(line);

while (tokenizer.hasMoreTokens())

{

word.set(tokenizer.nextToken());

output.collect(word, one);

}

}

}

 

public static class Reduce

extends MapReduceBase

implements Reducer<Text, IntWritable, Text, IntWritable>

{

 

public void reduce(Text key, Iterator<IntWritable> values,

OutputCollector<Text, IntWritable> output,

Reporter reporter) throws IOException

{

int sum = 0;

while (values.hasNext())

{

sum += values.next().get();

}

output.collect(key, new IntWritable(sum));

}

}

 

public static void main(String[] args) throws Exception

{

JobConf conf = new JobConf(WordCount.class);

conf.setJobName("wordcount");

 

conf.setOutputKeyClass(Text.class);

conf.setOutputValueClass(IntWritable.class);

 

conf.setMapperClass(Map.class);

conf.setCombinerClass(Reduce.class);

conf.setReducerClass(Reduce.class);

 

conf.setInputFormat(TextInputFormat.class);

conf.setOutputFormat(TextOutputFormat.class);

 

FileInputFormat.setInputPaths(conf, new Path(args[0]));

FileOutputFormat.setOutputPath(conf, new Path(args[1]));

 

JobClient.runJob(conf);

}

}

The “map” in MapReduce

There is a master node and many slave nodes.
The master node takes the input, divides it into smaller sub-problems, and distributes the input to worker or slave nodes. worker node may do this again in turn, leading to a multi-level tree structure.
The worker/slave nodes processes the data into a smaller problem, and passes the answer back to its master node.
Each mapping operation is independent of the others, all maps can be performed in parallel.

The “reduce” in MapReduce

The master node then collects the answers from the worker or slave nodes. It then aggregates the answers and creates the needed output, which is the answer to the problem it was originally trying to solve.
Reducers can also preform the reduction phase in parallel. That is how the system can process petabytes in a matter of hours.

Their are 3 key methods

The map() function will generate a list of key/value pairs based on the data

The shuffle() phase will bring things together for the reduce() phase

The reduce() phase will take the list of key/value pairs and hand that to you to do something with.

The Hello World sample for Hadoop is a word count example.
Let's assume our quote is this:
- It is time for all good men to come to the aid of their country.

map() function (see the "to" part) finds "to" twice	(It, 1) (is, 1) (time, 1) (for, 1) (all, 1) (good, 1) (to, 1) (men, 1) (to, 1) (come, 1) (the, 1) (aid, 1) (of, 1) (their, 1) (country, 1)
shuffle() function (see the "to" part) creates (to, 1, 1)	(It, 1) (is, 1) (time, 1) (for, 1) (all, 1) (good, 1) (to, 1, 1) (men, 1) (come, 1) (the, 1) (aid, 1) (of, 1) (their, 1) (country, 1)
reduce() function (see the "to" part) creates (to, 2)	(It, 1) (is, 1) (time, 1) (for, 1) (all, 1) (good, 1) (men, 1) (to, 2) (come, 1) (the, 1) (aid, 1) (of, 1) (their, 1) (country, 1)

High-level Architecture

There are two main layers to both the master node and the slave nodes – the MapReduce layer and the Distributed File System Layer. The master node is responsible for mapping the data to slave or worker nodes.

Hadoop is a platform

Hadoop Common	The common utilities that support the other Hadoop modules.
Hadoop Distributed File System (HDFS)	A distributed file system that provides high-throughput access to application data.
Hadoop YARN	A framework for job scheduling and cluster resource management.
Hadoop MapReduce	A YARN-based system for parallel processing of large data sets.

There also related modules that are commonly associated with Hadoop.

Apache Pig

A platform for analyzing large data sets.It includes a high-level language for expressing data analysis programs
A key point of Pig programs is that they support substantial parallelization
Pig consists of a compiler that produces sequences of Map-Reduce programs
Pig's language layer currently consists of a textual language called Pig Latin

Hive

Hive is a data warehouse system for Hadoop.

It provides a SQL-like language.

It helps with data summarization and ad-hoc queries.

I am not sure yet whether this is required with Hadoop on Azure.

If not, just a few command line tasks to do:

Install Hive	tar -xzvf hive-x.y.z.tar.gz
Set the environment variable HIVE_HOME	$ cd hive-x.y.z/ $ export HIVE_HOME={{pwd}}
Add $HIVE_HOME/bin to your PATH:	$ export PATH=$HIVE_HOME/bin:$PATH

But from what I saw here, looks like there is an ODBC Hive Setup Module.
WehnMing Ye has this video:
http://channel9.msdn.com/Events/windowsazure/learn/Hadoop-on-Windows-Azure

I signed up I recently signed up for the Windows Azure HDInsight Service here https://www.hadooponazure.com/. Logging in After logging in, you will be presented with this screen: Next post : Calculate PI with Hadoop

We will create a job name called “Pi Example.” This very simple sample will calculate PI using a cluster of comptuers.
This is not necessarily the best example of big data, it is more of a compute problem.
The final command line will look like this:

Hadoop jar hadoop-examples-0.20.203.1-SNAPSHOT.jar pi 16 10000000

More details on this sample coming soon.

Published at DZone with permission of Bruno Terkaly, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Installing moonlight on Linux (Chrome and Firefox)

2012-12-17T00:27:00.001-08:00

Go to page -> http://www.go-mono.com/moonlight/download.aspx

Need more help? Let me know.

A vendor-independent comparison of NoSQL databases: Cassandra, HBase, MongoDB, Riak

2012-12-05T02:26:00.001-08:00

Sponsored by:

This story appeared on Network World at
http://www.networkworld.com/news/tech/2012/102212-nosql-263595.html

A vendor-independent comparison of NoSQL databases: Cassandra, HBase, MongoDB, Riak

By Sergey Bushik, senior R&D engineer at Altoros Systems Inc., special to Network World
October 22, 2012 04:26 PM ET

What makes this research unique?

Often referred to as NoSQL, non-relational databases feature elasticity and scalability in combination with a capability to store big data and work with cloud computing systems, all of which make them extremely popular. NoSQL data management systems are inherently schema-free (with no obsessive complexity and a flexible data model) and eventually consistent (complying with BASE rather than ACID). They have a simple API, serve huge amounts of data and provide high throughput.
In 2012, the number of NoSQL products reached 120-plus and the figure is still growing. That variety makes it difficult to select the best tool for a particular case. Database vendors usually measure productivity of their products with custom hardware and software settings designed to demonstrate the advantages of their solutions. We wanted to do independent and unbiased research to complement the work done by the folks at Yahoo.
Using Amazon virtual machines to ensure verifiable results and research transparency (which also helped minimize errors due to hardware differences), we have analyzed and evaluated the following NoSQL solutions:
● Cassandra, a column family store
● HBase (column-oriented, too)
● MongoDB, a document-oriented database
● Riak, a key-value store
We also tested MySQL Cluster and sharded MySQL, taking them as benchmarks.
After some of the results had been presented to the public, some observers said MongoDB should not be compared to other NoSQL databases because it is more targeted at working with memory directly. We certainly understand this, but the aim of this investigation is to determine the best use cases for different NoSQL products. Therefore, the databases were tested under the same conditions, regardless of their specifics.

Tools, libraries and methods

For benchmarking, we used Yahoo Cloud Serving Benchmark, which consists of the following components:
● a framework with a workload generator
● a set of workload scenarios
We have measured database performance under certain types of workloads. A workload was defined by different distributions assigned to the two main choices:
• which operation to perform
• which record to read or write
Operations against a data store were randomly selected and could be of the following types:
• Insert: Inserts a new record.
• Update: Updates a record by replacing the value of one field.
• Read: Reads a record, either one randomly selected field, or all fields.
• Scan: Scans records in order, starting at a randomly selected record key. The number of records to scan is also selected randomly from the range between 1 and 100.
Each workload was targeted at a table of 100,000,000 records; each record was 1,000 bytes in size and contained 10 fields. A primary key identified each record, which was a string, such as "user234123." Each field was named field0, field1, and so on. The values in each field were random strings of ASCII characters, 100 bytes each.
Database performance was defined by the speed at which a database computed basic operations. A basic operation is an action performed by the workload executor, which drives multiple client threads. Each thread executes a sequential series of operations by making calls to the database interface layer both to load the database (the load phase) and to execute the workload (the transaction phase). The threads throttle the rate at which they generate requests, so that we may directly control the offered load against the database. In addition, the threads measure the latency and achieved throughput of their operations and report these measurements to the statistics module.

The performance of the system was evaluated under different workloads:
Workload A: Update heavily
Workload B: Read mostly
Workload C: Read only
Workload D: Read latest
Workload E: Scan short ranges
Workload F: Read-modify-write
Workload G: Write heavily
Each workload was defined by:
1) The number of records manipulated (read or written)
2) The number of columns per each record
3) The total size of a record or the size of each column
4) The number of threads used to load the system
This research also specifies configuration settings for each type of the workloads. We used the following default settings:
1) 100,000,000 records manipulated
2) The total size of a record equal to 1Kb
3) 10 fields of 100 bytes each per record
4) Multithreaded communications with the system (100 threads)

Testing environment

To provide verifiable results, benchmarking was performed on Amazon Elastic Compute Cloud instances. Yahoo Cloud Serving Benchmark Client was deployed on one Amazon Large Instance:
• 7.5GB of memory
• four EC2 Compute Units (two virtual cores with two EC2 Compute Units each)
• 850GB of instance storage
• 64-bit platform
• high I/O performance
• EBS-Optimized (500Mbps)
• API name: m1.large
Each of the NoSQL databases was deployed on a four-node cluster in the same geographical region on Amazon Extra Large Instances:
• 15GB of memory
• eight EC2 Compute Units (four virtual cores with two EC2 Compute Units each)
• 1690GB of instance storage
• 64-bit platform
• high I/O performance
• EBS-Optimized (1000Mbps)
• API name: m1.xlarge
Amazon is often blamed for its high I/O wait time and comparatively slow EBS performance. To mitigate these drawbacks, EBS disks had been assembled in a RAID0 array with stripping and after that they were able to provide up to two times higher performance.

The results

When we started our research into NoSQL databases, we wanted to get unbiased results that would show which solution is best suitable for each particular task. That is why we decided to test performance of each database under different types of loads and let the users decide what product better suits their needs.
We started with measuring the load phase, during which 100 million records, each containing 10 fields of 100 randomly generated bytes, were imported to a four-node cluster.

HBase demonstrated by far the best writing speed. With pre-created regions and deferred log flush enabled, it reached 40K ops/sec. Cassandra also showed great performance during the loading phase with around 15K ops/sec. The data is first saved to the commit log, using the append method, which is a fast operation. Then it is written to a per-column family memory store called a Memtable. Once the Memtable becomes full, the data is saved to disk as an SSTable. In the "just in-memory" mode, MySQL Cluster could show much better results, by the way.
* Workload A: Update-heavily mode. Workload A is an update-heavily scenario that simulates the database work, during which typical actions of an e-commerce solution user are recorded. Settings for the workload:
1) Read/update ratio: 50/50
2) Zipfian request distribution

During updates, HBase and Cassandra went far ahead from the main group with the average response latency time not exceeding two milliseconds. HBase was even faster. HBase client was configured with AutoFlush turned off. The updates aggregated in the client buffer and pending writes flushed asynchronously, as soon as the buffer became full. To accelerate updates processing on the server, the deferred log flush was enabled and WAL edits were kept in memory during the flush period.
Cassandra wrote the mutation to the commit log for the transaction purposes and then an in-memory Memtable was updated. This is a slower, but safer scenario if compared to the HBase deferred log flushing.
* Workload A: Read. During reads, per-column family compression provides HBase and Cassandra with faster data access. HBase was configured with native LZO and Cassandra with Google's Snappy compression codecs. Although the computation ran longer, the compression reduces the number of bytes read from the disk.

* Workload B: Read-heavy mode. Workload B consisted of 95% of reads and 5% of writes. Content tagging can serve as an example task matching this workload; adding a tag is an update, but most operations imply reading tags. Settings for the "read-mostly" workload:
1) Read/update ratio: 95/5
2) Zipfian request distribution

Sharded MySQL showed the best performance in reads. MongoDB -- accelerated by the "memory mapped files" type of cache -- was close to that result. Memory-mapped files were used for all disk I/O in MongoDB. Cassandra's key and row caching enabled very fast access to frequently requested data. With the off-heap row caching feature added in Version 0.8, it showed excellent read performance while using less per-row memory. The key cache held locations of keys in memory on a per-column family basis and defined the offset for the SSTable where the rows were stored. With a key cache, there was no need to look for the position of the row in the SSTable index file. Thanks to the row cache, we did not have to read rows from the SSTable data file. In other words, each key cache hit saved us one disk seek and each row cache hit saved two disk seeks. In HBase, random read performance was slower. However, Cassandra and HBase can provide faster data access with per-column-family compression.
* Workload B: Update. Thanks to deferred log flushing, HBase showed very high throughput with extremely small latency under heavy writes. With deferred log flush turned on, the edits were first committed to the memstore. Then the aggregated edits were flushed to HLog asynchronously. On the client side, HBase write buffer cached writes with the autoFlush option set to true, which also improved performance greatly. For security purposes, HBase confirms every write after its write-ahead log reaches a particular number of in-memory HDFS replicas. HBase's write latency with memory commit was roughly equal to the latency of data transmission over the network. Cassandra demonstrated great write throughput, since it first writes to the commit log -- using the append method, which is a pretty fast operation -- and then to a per-column-family memory store called Memtable.

* Workload C: Read-only. Settings for the workload:
1) Read/update ratio: 100/0
2) Zipfian request distribution
This read-only workload simulated a data caching system. The data was stored outside the system, while the application was only reading it. Thanks to B-tree indexes, sharded MySQL became the winner in this competition.

* Workload E: Scanning short ranges. Settings for the workload:
1) Read/update/insert ratio: 95/0/5
2) Latest request distribution
3) Max scan length: 100 records
4) Scan length distribution: uniform
HBase performed better than Cassandra in range scans. HBase scanning is a form of hierarchical fast merge-sort operation performed by HRegionScanner. It merges the results received from HStoreScanners (one per family), which, in their turn, merge the results received from HStoreFileScanners (one for each file in the family). If caching is turned on, the server will simply provide the number of specified records instead of bouncing back to the HRegionServer to process every record.

Cassandra's scan performance with a random partitioner has improved considerably compared to Version 0.6, where this feature was initially introduced.
Cassandra's SSTable is a sorted strings table that can be described as a file of key-value string pairs sorted by keys and key-value pairs written in a particular order. To achieve maximum performance during range scans, we had to use an order-preserving partitioner. Scanning over ranges of order-preserving rows is super-fast. It is similar to moving a cursor through a continuous index. However, the database cannot distribute individual keys and corresponding rows over the cluster evenly, thus a random partitioner is used to ensure even data distribution. This is the default partitioning strategy in Cassandra. Random partitioning ensures good load balancing and provides some additional speed in range scans with an order preserving partitioner.
In MongoDB 2.5, the table scan triggered by the { "_id":{"$gte": startKey}} query showed a maximum throughput of 20 ops/sec with a latency of ≈ 1 sec.
The performance of MySQL Cluster was under 10 ops/sec with a latency of 400 ms. It is partitioned over the nodes in the cluster, so the system uses an optimizer to translate SQL commands into a query plan. The execution of this plan is divided among multiple nodes. For range scans, a B-tree index is used to make column comparisons in such expressions as >, <, or BETWEEN.
Sharded MySQL is based on key hashing on the connector side and does not support true range scans over a cluster. While a single shard did about 10 ops/sec, the whole sharded setup showed near 40 ops/sec with a latency of up to 400 ms. MyISAM caches index blocks but not data blocks. There can be an overhead due to re-reading data blocks from the OS buffer cache.
The Riak bitcask storage engine does not support range scans. This can be done through secondary indexes with eleveldb and special $key index referring to the primary key. Eleveldb showed insufficient performance that started to degrade after 50,000,000 records had been imported and we fell back to bitcask.
* Workload G: Insert-mostly mode. Settings for the workload:
1) Insert/Read: 90/10
2) Latest request distribution
HBase showed the best results under a workload that included large volumes of writes. Cassandra was second. The NDB engine of MySQL Cluster also managed intensive writes perfectly well.

Conclusion

As you can see, there is no perfect NoSQL database. Every database has its advantages and disadvantages that become more or less important depending on your preferences and the type of tasks.
For example, a database can demonstrate excellent performance, but once the amount of records exceeds a certain limit, the speed falls dramatically. It means that this particular solution can be good for moderate data loads and extremely fast computations, but it would not be suitable for jobs that require a lot of reads and writes. In addition, database performance also depends on the capacity of your hardware.
It was hardly possible to include all of the performance diagrams and describe everything in one article. You can download the full version of the research that contains separate chapters dedicated to every database, YCSB and Amazon EC2 configuration details, and appendix with other performance diagrams athttp://altoros.com/nosql-research.
We hope this research will be useful to both developers working with NoSQL solutions and customers trying to choose a database. Altoros's R&D team will regularly revise and update information of this research to cover new databases and releases of the most popular products.
About the author: Sergey Bushik is a senior R&D engineer at Altoros. He has more than seven years of experience in implementation of Java-based projects that include big data processing, data mining and Hadoop computations. Sergey has a number of certificates in Java and is a Sun Certified Enterprise Architect for the Java Platform. He is a regular speaker at international conferences -- most recently, he delivered sessions at Big Data Meetup (Sunnyvale, Calif.), GOTO Copenhagen 2012, Hadoop Evening (Eastern Europe), etc.
Read more about software in Network World's Software section.

bigdata tools comparision

2012-11-23T04:08:00.001-08:00

Resources

http://apandre.wordpress.com/tools/comparison/
http://db-engines.com/en/
http://vschart.com