[∞] Amazedsaint's Tech Journal

Essential Terminologies to start a conversation about the Container Ecosystem

noreply@blogger.com (Anonymous) — Fri, 02 Dec 2016 18:28:00 +0000

You heard the news - Embracing the world of Micro Services and Containers is essential to help you make your organisation's IT operations more agile, providing immediate operational benefits. Let us have a look at few essential concepts to help you start a conversation with your own internal IT, Developers and Vendors.

Micro services

Well, as Martin Fowler put it, "Micro-services - yet another new term on the crowded streets of software architecture”, is what everybody is talking about.

In simple terms, Micro service architecture enables packaging (mentally and physically) each unit of functionality into a service, and you can distribute and scale these services independently. In a traditional monolithic web or enterprise application, if you need to change a simple functionality, you have to rebuild and redeploy the whole application. In a Micro service architecture, you can individually deploy and scale services.

Now, this has got multiple advantages. You can scale only the services you need to distribute the load effectively - i.e, if you see that your customers are using your Order service more than others, you can scale up only the Order service instances from 10 to 20. Though this is nothing new, the evolution of container technologies accelerated Micro service based systems, and enabled organizations to adopt a very agile, continuous delivery based workflow to build and deploy applications faster.

Containers

‘Container’ is probably the most abused term in this year after the term ‘locker room talk’. In it’s original sense as it is used today in Dev Ops, the term emerged from LXC (Linux Containers). LXC is an OS level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

What is the difference between LXC and Virtual Machines? As you are aware, standard virtualization systems (like KVM, VirtualBox etc) lets you boot full operating systems of different kinds, even non-Linux systems. The main difference between this and Linux Containers is that virtual machine require a separate kernel instance to run on - i.e almost a full standalone OS.

How ever, multiple LXCs can be deployed on top of the same Kernel (Ah, Microsoft didn’t see that coming oops) - So, LXCs are much cheaper to create and destroy (from a memory and processor foot print perspective) compared to Virtual Machines. One Linux Container (LXC) can run a single process, and as long as you don’t give root permissions for the process you run, you can impart some level of security to the process your container is running. To be fancy - you can group containers to Pods, and run them in Nodes. (Side note - Microsoft recently announced Windows Containers -have a look)

Platforms like Docker provides an easy workflow for developers to package their application to a container ‘image’, to spin off instances of this container later in a very easy way.

Pods

In simple terms - containers that need to co-exist in the same Kernel/Virtual Machine/Node and related run time information is grouped together as a Pod. So, a Pod is essentially a group of containers that should co-exist. And typically an application running in one container can access another container via the ‘localhost’ as long as both the containers are in the same pod. Containers with in the same pod will also mostly share the same storage context - much like two applications running in a virtual machine.

Volumes

A Volume is an abstraction you can use for storage, and can be used by containers to read/write data. So, containers of the same Pod can use a ‘Volume’. From Kubrnetes perspective, Volumes are attached to Pods - so even if a container crashes, the files etc related to restart the container can be kept in the Volume. But when you remove/delete a Pod, normally you throw away the volume related to the Pod as well (in simple scenarios).

Nodes

You can consider a Node as a worker machine (either a Virtual Machine or a bare metal physical machine). Nodes can run Pods and multiple nodes are managed by one or more master nodes to form a cluster.

Clusters

A cluster is a large group of containers, some of them grouped into pods and some of them not. A cluster normally has one or more master nodes that manages the pods/containers deployed in the nodes - the master is responsible for ensuring the requested number of container instances are up and running all the time, and also providing API access to the containers in the cluster.

Cluster Federations

Typically, a cluster runs in a single on premise data center, or in a single availability zone in case of cloud providers - now what if these clusters can be tied to each other and federate them? This will enable interesting use cases like ability to overflow your work loads from one cluster to another. For example, an application can run in a private/one-premise cloud and burst into a public cloud when the demand of compute overflows a specific limit (typically mentioned as Cloud-Busting). The easiest way is to start with Kubernetes Cluster Federations

Why Micro services Pattern Love Containers?

As containers are easy to spin up and down, this became the favorite model of packaging and shipping your micro services. You can create a service, and package it to a container - and deploy them independently. Docker because so popular because of its ability to build, package and deploy applications/services using a light weight container. You can use a Docker image to spin off multiple container instances.

Kubernetes, Docker Swarm etc went one step further, allowing you to define and deploy containers at scale to form a whole cluster of pods with containers. For example, container orchestration engines like Kubernetes will let you specify the whole cluster configuration - including how many containers you need per service/application and how exactly they should talk to each other.

So, start from here and think how to be more agile - and re-architect your own enterprise to build and deliver business benefits faster, in an agile way, embracing containerization .

PB- This is a very evolving space, and there are lot of players and platforms in the market. Most of the time an apple to apple comparison is not possible between the tools and platforms. But if you are looking for going one step further, have a look at the container platforms like OpenShift(https://www.openshift.com/), Cloud Foundry(https://www.cloudfoundry.org/) etc. Macro level platforms/orchestration tools like Fabric8 https://fabric8.io/ are also becoming mature - allowing you to spin off your entire dev-ops pipe line as a platform - and optimize and manage everything using a unified user experience.

Exploring Some Of The Probable C# 6.0 Features

noreply@blogger.com (Anonymous) — Tue, 22 Apr 2014 03:40:00 +0000

In my last post, we explored how to create a tiny Roslyn app to compile C# code, to test out some of the new C# 6.0 features.

Read it here

Now go ahead and play with some of the preview features. Try things out. Create a file with some C# 6.0 sugar and compile it with our above app. You can explore the language features that are completed in this list, from the Roslyn documentation in CodePlex

Here is some quick code that demos some of the features

1- Support for Primary Constructors and Auto Property Assignments

Primary constructors allow you to specify arguments as part of your class declaration itself. Also, now C# supports assigning to Auto properties. Together, you may use it to initialise classes, as shown below.

//Test.csfile 
//C# 6.0 may support Primary Constructors and Assignment To Auto Properties

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace CSharp6Test
{

   //Feature: Primary Constructors or Class with arguments ;)
    class Point(int x,int y)
    {
        //Feature: Look ma, you can assign to auto properties now
        public int X {get;set;} =x;
        public int Y {get;set;} =y;

    }

    class MainClass
    {
        public static void Main() 
        {
        
            //Using Primary Constructor
            var p=new Point(1,3);

            //Reading the values back from the properties
            Console.WriteLine("X={0}",p.X);
            Console.WriteLine("Y={0}",p.Y);
        }
    }
}

2 – Invocation of static methods directly

Another completed feature seems like the support for direct invocation of static methods directly, with out the full name space

//Just import the namespace
using System.Console;

namespace CSharp6Test
{
    class Class1
    {
   public static void Main() 
    {
  //And use the static methods now directly  
  WriteLine("Look ma, now you can use static methods like this...");
    }
    }

}

And that works too.

3 – Dictionary initializers and indexed member access

Say good bye to the dirty strings when working with dictionary objects and collections. Let us see if the new $ Indexed member syntax is going to work. See it here.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Console;

namespace CSharp6Test
{
    class Class1
    {
  public static void Main() 
    {
      //See the new dictionary initializer syntax
   var d=new Dictionary<string,int>{["item1"]=1,["item2"]=2}; 
   
   //This is even better
   //var d=new Dictionary<string,int>{$item1=1,$item2=2}; 
   
   //And now you can access indexed members using $variablename syntax
   
   //No more dirty strings
   WriteLine("{0}",d.$item1);
   WriteLine("{0}",d.$item2);
    }
    }

}

Try compiling the above apps with our Roslyn test app (Get it here). Compile them :) Leaving all the other features up to you to try out. Happy Coding!!

CSharp6Test // Creating A Tiny Roslyn app To Explore The Features Of C# 6.0

noreply@blogger.com (Anonymous) — Mon, 21 Apr 2014 17:38:00 +0000

Some of the C# 6.0 Features are exciting. And you can try them out now as the new Roslyn preview is out. You can explore the language features that are completed in this list, from the Roslyn documentation in CodePlex. Some of the ‘Done’ features for C#, based on the documentation there include

Primary constructors - class Point(int x, int y) { … }
Auto-property initializers - public int X { get; set; } = x;
Getter-only auto-properties - public int Y { get; } = y;
Using static members - using System.Console; … Write(4);
Dictionary initializer - new JObject { ["x"] = 3, ["y"] = 7 }
Indexed member initializer - new JObject { $x = 3, $y = 7 }
Indexed member access - c.$name = c.$first + " " + c.$last;
Declaration expressions - int.TryParse(s, out var x);
Await in catch/finally - try … catch { await … } finally { await … }
Exception filters - catch(E e) if (e.Count > 5) { … }

In this post, we’ll explore.

How to parse and walk the Roslyn syntax tree and dump it
How to write a small ‘compiler’ using Roslyn’s CSharpCompilation
Use the same to explore if those features are implemented.

We could’ve used REPL/Scripting APIs, but sadly it is not available in the new version pre-release – So let us write a simple app using Roslyn APIs to test the new features in C# .

[Ouch, lazy to write the code? Fork it from Github - https://github.com/amazedsaint?tab=repositories ]

Our CSharp6Test App

Create a new C# Project in Visual Studio 2012/2013, fire up Nuget console, and install Rolsyn pre release bits. I’m doing this in VS 2012.

  Install-Package Microsoft.CodeAnalysis -Pre

Now, let us write some code to parse the Syntax tree and do the compilation

//FILE - MAIN.CS
//amazedsaint.com

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.CSharp.Syntax;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace CSharp6Test
{

    class Program
    {
        static void Main(string[] args)
        {

            try
            {

                Console.WriteLine("C#> Roslyn Code Verifier.");
                if (args.Count() < 0)
                {
                    Console.WriteLine("C#> Usage: CSharp6Test <file>");
                    return;
                }

                //User provided the output filename
                string file = args[0];

                string output = (args.Count() > 2) ? output = args[1] : file + ".exe";

                //Create a syntax tree from the code
                SyntaxTree tree = CSharpSyntaxTree.ParseText(File.ReadAllText(file));


                Console.WriteLine("C#> Dumping Syntax Tree");
                Console.WriteLine();

                //Dumping it using our extension method
                tree.Dump();
                Console.WriteLine();

                //Compile
                Console.WriteLine("C#> Trying to compile Syntax Tree");
                tree.Compile(output);

            }
            catch (Exception ex)
            {
                Console.WriteLine("Oops: {0}", ex.Message);
                Console.WriteLine("Sorry Skywalker, that was an exception - Love, Yoda");
            }

        }
    }
}

Well, as it is evident, we are just parsing the syntax tree from the file, dumping it, and then compiling it. The Dump and Compile extension methods are here for your service.

//FILE - SYNTAXTREEEXTENSIONS.CS
//amazedsaint.com

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace CSharp6Test
{
    public static class SyntaxTreeExtensions
    {
        public static void Dump(this SyntaxTree tree)
        {
            var writer = new ConsoleDumpWalker();
            writer.Visit(tree.GetRoot());
        }

        class ConsoleDumpWalker : SyntaxWalker
        {
            public override void Visit(SyntaxNode node)
            {
                int padding = node.Ancestors().Count();
                //To identify leaf nodes vs nodes with children
                string prepend = node.ChildNodes().Count() > 0 ? "[-]" : "[.]";
                //Get the type of the node
                string line = new String(' ', padding) + prepend +
                                        " " + node.GetType().ToString();
                //Write the line
                System.Console.WriteLine(line);
                base.Visit(node);
            }

        }

        public static void Compile(this SyntaxTree tree, string output)
        {

            //Creating a compilation
            var compilation = CSharpCompilation.Create
         ("CSharp6Test", new List<SyntaxTree> { tree })
                              .AddReferences(
          new MetadataFileReference
    (typeof(object).Assembly.Location),
                               new MetadataFileReference
    (typeof(ASCIIEncoding).Assembly.Location), 
                               new MetadataFileReference
    (typeof(IEnumerable<>).Assembly.Location), 
                               new MetadataFileReference
    (typeof(IEnumerable).Assembly.Location),  
                               new MetadataFileReference
    (typeof(Enumerable).Assembly.Location));   


            //Lets do the diagnostics
            var check = compilation.GetDiagnostics();


            //Report issues if any
            if (check.Count() > 0)
            {

                bool hasError = false;

                Console.WriteLine();
                Console.WriteLine("C#> Few Issues Found");
                foreach (var c in check)
                {
                    Console.WriteLine("{0} : {1} in {2}", c.Severity, c.GetMessage(), c.Location);
                    if (c.Severity == DiagnosticSeverity.Error) hasError = true;
                }
                if (hasError)
                {
                    Console.WriteLine("C#> Errors found. Aborting");
                }

            }

            var emit = compilation.Emit(output);

            if (emit.Success)
                Console.WriteLine("C#> No Errors Found. Created {0}", output);
            else
                Console.WriteLine("C#> Oops, can't create {0}", output);
        }


    }
}

And that is it. Go ahead and compile as you please. You've a tiny tool to test the new C# features. Checkout my next post about experimenting with some of the C# 6.0 features.

Hadoop On Azure / HDInsight– Quick Intro Video On Writing Map Reduce Jobs In C#

noreply@blogger.com (Anonymous) — Sat, 19 Apr 2014 03:07:00 +0000

Here is a quick intro screen cast on Big Data and creating map reduce jobs in C# to distribute the processing of large volumes of data, leveraging Microsoft Azure HDInsight / Hadoop On Azure, based on my virtual tech days presentation.

ScriptCs Templating Support

noreply@blogger.com (Anonymous) — Mon, 20 Jan 2014 18:46:00 +0000

If you havn’t yet heard about ScriptCs, it is not too late. Go here

I just checked in a ScriptCs Templating module to integrate Razor and StringTemplate transformations in ScriptCs workflow. ScrtipCs Templating module can apply a Razor or StringTemplate (ST4) template on top of one or more model files (normally an xml file or json file), for scenarios like code generation or templating. Example below.

The first bits are https://github.com/amazedsaint/scriptcs-engine-templating

Installing template module

Install scriptcs. You need to install the nightly build using Chocoloately - cinst scriptcs -pre -source https://www.myget.org/F/scriptcsnightly/ (Refer https://github.com/scriptcs/scriptcs if you don't understand this)
Clone the repo at https://github.com/amazedsaint/scriptcs-engine-templating or download the source code from there.
Open the solution and build it in Visual Studio.
From the command line create a package: nuget pack -version 0.1.0-alpha
In VS, edit your Nuget package sources (Tools->Library Package Manager->Settings) and add the folder where the package lives locally.
Install the template package globally: scriptcs -install ScriptCs.Engine.Templating -g -pre

Rendering a template

Note that what ever you pass after -- will goto the module as it's arguments.
Run scriptcs with your template, file specifying our template module using -modules switch: scriptcs mytemplate.cst -loglevel debug -modules template - The -modules template argument let scriptcs load our template module.
Check the log output for details.
You can specify the output file using the -out switch -scriptcs mytemplate.cst -modules template -- -out result.txt (The parameters after -- are the template module parameters according to ScriptCs convention)

Rendering a template Using Models

Template module automagically converts xml files/urls and json files/urls to dynamic models that can be used from your template. Technically, it creates a C# fleuent dynamic object that wraps the xml/json.

Quick example: Create a new folder, and create a model.xml file inside that.

<model>
<class name="MyClass1">
  <property name="MyProperty1" type="string"/>
  <property name="MyProperty2" type="string"/>
</class>
  <class name="MyClass2">
    <property name="MyProperty1" type="string"/>
    <property name="MyProperty2" type="string"/>
  </class>
</model>

Now, create your template, and save it as template.cst - Let us use razor syntax.

@model dynamic

@foreach(var item in Model["class"])
{
   <p>@item.name is a class name</p>  
}

Now, you can run the transformation by specifying the model file, like this

scriptcs template.cst -modules template -- -xml model.xml -out result.txt

Let us rewrite the template to generate the class and properties from our XML model file

@model dynamic

@foreach(var c in Model["class"])
{
   @:class @c.name {

   foreach(var p in c["property"])
   {
       @:public @p.type @p.name {get;set;}
   }

   @:}
}

Regenerate the result file and see.

Transforming multiple model files

You may specify multiple model files

scriptcs template.cst -modules template -- -xml model1.xml -out result1.txt -xml model2.xml -out result2.txt

You may also use Json files as the model

scriptcs template.cst -modules template -- -json model.json -out result.cs

Accessing models from templates

For converting XML files/data to a dynamic model object that'll be accessed from templates, ElasticObject is used. Refer https://github.com/amazedsaint/ElasticObject

For converting Json files/data to a dynamic model object that'll be accessed from templates, DynamicJsonConverter is used. Refer http://stackoverflow.com/questions/3142495/deserialize-json-into-c-sharp-dynamic-object

Template module command line

Example usages:

Render using template mytemplate.cst and ElasticObject dynamic model from model1.xml using Razor (Razor is default)
- scriptcs mytemplate.cst -modules template -- -xml model1.xml -out result1.txt
Do the same as above, but using vb as the template language
- scriptcs mytemplate.vbt -modules template -- -vb -xml model1.xml -out result1.txt
Render using template mytemplate.st4 - Using StringTemplate instead of Razor
- scriptcs mytemplate.st4 -modules template -- -st4 -xml model1.xml -out result1.txt

Parameter meaning:

-out - Specify the output file for the given template or model
-xml - Specify an xml model
-json - Specify a json model
-raw - Enforces raw encoding if using Razor engine
-st4 - Uses StringTemplate (http://www.antlr.org/wiki/display/ST4/StringTemplate+4+Documentationinstead of Razor)
-vb - Uses VB in razor files instead of C#

Additional References:

Razor syntax http://haacked.com/archive/2011/01/06/razor-syntax-quick-reference.aspx/
StringTemplate syntax http://www.antlr.org/wiki/display/ST4/StringTemplate+4+Documentation

More tests need to be added, and St4 support is a bit untested. Happy Coding.

CakeRobot - A Gesture Driven Robot That Follows Your Hand Movements Using Arduino, C# and Kinect

noreply@blogger.com (Anonymous) — Mon, 21 Oct 2013 19:29:00 +0000

Over the last few weekends I’ve spend some time building a simple robot that can be controlled using Kinect. You can see it in action below.
Ever since I read this Cisco paper that mentions Internet Of Things will crate a whooping 14.4 Trillion $ at stake, I revamped my interests in hobby electronics and started hacking with DIY boards like Arduino and Raspberry Pi. That turned out to be fun, and ended up with the robot. This post provides you the general steps, and the github code may help you build your own.
Even if you don’t have a Kinect for the controller, you can easily put together a controller using your phone (Windows Phone/Android/iOS) as we are using blue tooth to communicate with the controller and the robot.

Now, here is a quick start guide to build your own. We’ve an app running in the laptop that is communicating to the Robot via blue tooth in this case that pumps the commands based on input from Kinect, you could easily build a phone UI as well.
And if you already got the idea, here is the code – You may read further to build the hardware part.

1 – Build familiarity

You need to build some familiarity with Arduino and/or Netduino – In this example I’ll be using Arduino.

Arduino

Explore Arduino. The best way is to

Download and install the Arduino IDE http://arduino.cc/en/Main/Software
Go through Arduino Tutorial - https://learn.sparkfun.com/tutorials/what-is-an-arduino
Buy an Arduino board to play with –You may try https://www.sparkfun.com/ if you are in US, Rhydolabz if you are in India, Google if you are in any other country.

Mainly you need to understand the Pins in the Arduino board. You can write simple programs with the Arduino IDE (try the Blink sample to blink an LED in IDE File->Samples). PFB the pins description, from SparkFun website.

GND (3): Short for ‘Ground’. There are several GND pins on the Arduino, any of which can be used to ground your circuit.
5V (4) & 3.3V (5): The 5V pin supplies 5 volts of power, and the 3.3V pin supplies 3.3 volts of power. Most of the simple components used with the Arduino run happily off of 5 or 3.3 volts. If you’re not sure, take a look at Spark Fun’s datasheet tutorial then look up the datasheet for your part.
Analog (6): The area of pins under the ‘Analog In’ label (A0 through A5 on the UNO) are Analog In pins. These pins can read the signal from an analog sensor (like a temperature sensor) and convert it into a digital value that we can read.
Digital (7): Across from the analog pins are the digital pins (0 through 13 on the UNO). These pins can be used for both digital input (like telling if a button is pushed) and digital output (like powering an LED).
PWM (8): You may have noticed the tilde (~) next to some of the digital pins (3, 5, 6, 9, 10, and 11 on the UNO). These pins act as normal digital pins, but can also be used for something called Pulse-Width Modulation (PWM). We have a tutorial on PWM, but for now, think of these pins as being able to simulate analog output (like fading an LED in and out).
AREF (9): Stands for Analog Reference. Most of the time you can leave this pin alone. It is sometimes used to set an external reference voltage (between 0 and 5 Volts) as the upper limit for the analog input pins.

Kinect

If you are planning to use Kinect as the controller, download and install Kinect SDK - http://www.microsoft.com/en-us/kinectforwindowsdev/default.aspx – I’m using SDK 1.7
Kinect How To Videos Here

Protocols

Get some familiarity about Bluetooth Protocols - http://en.wikipedia.org/wiki/Bluetooth

2 – Get The Components

Again, you could find an online store to buy these components. Also, you could try the nearby local electronic store and buy some bread boards, jumper wires (get some mail to mail & female to female wires) etc as well. Here is the list of components you need to build CakeRobot.

A Chassis – I used Dagu Magician Chassis – From Spark Fun, Rhydolabz – it comes with two stepper motors that can be controlled by our driver board.
A Arduino board with Motor Driver – I used Dagu Mini Motor Driver – bought from Rhydolabz in India. For other countries you need to search and find. Some description about the board can be found here – It also has a special slot to plug in dagu blue tooth shield. You could also use Micro Magician
- You also need a Micro USB cable to connect your PC to the motor driver to upload code.
A Bluetooth Shield – Get the Dagu blue tooth module if you can find it. I’ve purchased a Class 2 Rn 42 blue tooth shield From Rhydolabz
Few mini modular bread boards
Jumper wires – Get a mixed pack with M/M, F/F, M/F – like this one
A Tiny blue tooth dongle for your PC/Laptop, like this one, to communicate with the blue tooth shield in the robot (if you don’t have built in blue tooth)
Few sensors if you want to have more fun – I had an ultra sonic distance sensor to avoid collisions in my final version. A better alternative is Ping sensor.
A battery pack and battery holder that should supply around 6V to the Mini Motor Driver
Other components for your later creativity/exploration
- LEDs
- Resistors
- More Sensors
Tools
- Few star screw drivers
- Duct tapes/rubber bands (yea, we are prototyping so no soldering as of now)

And I found these reads pretty useful

3 – Programming the components

You need to spend some hours figuring out how to program each of the components.

To start with, play with Arduino a bit, connecting LEDs, switches etc. Then, understand a bit about programming the Digital and Analog pins. Play with the examples
Try programming the ultrasonic sensor if you’ve one using your Arduino, using serial sockets. If you are using Ping sensor, check out this
Try programming the blue tooth module (Code I used for the distance sensor and blue tooth module are in my examples below, but it’ll be cool if you can figure things out yourself).

4 – Put the components together

Assemble the Dagu Magician Chassis, and place/screw/mount the mini motor driver and Bluetooth module on top of the same. Connect the components using jumper wires/plugin as required. A high level schematic below.

Here is a low resolution snap of mine, from top.

5 – Coding the Arduino Mini Driver

You can explore the full code in the Github repo - How ever, here are few pointers. According to the Dagu Arduino Mini driver spec, the following digital pins can be used to control the motors

D9 is left motor speed
D7 is left motor direction
D10 is right motor speed
D8 is right motor direction

To make a motor move, first we need to set the direction by doing a digitalWrite of HIGH or LOW (for Forward/Reverse) to the direction pin. Next set the motor speed by doing an analogWrite of 0~255 to the speed pin. 0 is stopped and 255 is full throttle.
In the Arduino code, we are initiating communication via blue tooth, to accept commands as strings. For example, speedl 100 will set the left motor speed to 100, and speedr 100 will set the right motor speed to 100. Relevant code below.

        //Setting up the communication with Bluetooth shield over serial 

        Serial.begin(115200);  // rn42 bt

       .......more


      //Read the input In getSerialLine (shortened for brevity)

      while(serialIn != '\n')
 {
  if (!(Serial.available() > 0)) 
  {
   return;
  }

  serialIn = Serial.read();
  if (serialIn!='\n') {
   char a = char(serialIn);
   strReceived += a;
  }
 }


        ....more

        //Process the command (shortened for brevity)
        ....
      
 else if (command=="speedl")
 {
  val=getValue(input,' ',1).toInt();
  analogWrite(leftMotorSpeed,val);
 } 
 else if (command=="speedr")
 {
  val=getValue(input,' ',1).toInt();
  analogWrite(rightMotorSpeed,val);
 } 


        ......more

Have a look at the full code of the quick Arduino client here. Then, compile and upload the code to your mini driver board.

6 – Coding the Controller & Kinect

Essentially, what we are doing is just tracking the Skeletal frame, and calculating the distance of your hand from your hip to provide the direction and speed for the motors. Skeletal tracking details here
We are leveraging http://32feet.codeplex.com/ for identifying the Blue tooth shield to send the commands. Please ensure your blue tooth shield is paired with your PC/Laptop/Phone – you can normally do that by clicking the blue tooth icon in system tray in Windows, and clicking Add Device.

       //For each 600 ms, send a new command
       //_btCon is our instance variable for a blue tooth connection, built over the cool 32Feet library
      
       internal void ProcessCommand(Skeleton skeleton)
        {

            var now = DateTime.Now;
            if (now.Subtract(_prevTime).TotalMilliseconds < 600)
                return;

            _prevTime = DateTime.Now;

            Joint handRight = skeleton.Joints[JointType.HandRight];
            Joint handLeft = skeleton.Joints[JointType.HandLeft];
            Joint shoulderRight = skeleton.Joints[JointType.ShoulderRight];
            Joint shoulderLeft = skeleton.Joints[JointType.ShoulderLeft];
            Joint hipLeft = skeleton.Joints[JointType.HipLeft];
            Joint hipRight = skeleton.Joints[JointType.HipRight];
            Joint kneeLeft = skeleton.Joints[JointType.KneeLeft];

            if (handRight.Position.Y < hipRight.Position.Y)
            {
                _btCon.SetSpeed(Motor.Left, 0);
            }

            if (handLeft.Position.Y < hipLeft.Position.Y)
            {
                _btCon.SetSpeed(Motor.Right, 0);
            }

            if (handRight.Position.Y > hipRight.Position.Y)
            {
                var speed = (handRight.Position.Y - hipRight.Position.Y) * 200;
                if (speed > 230) speed = 230;
                _btCon.SetSpeed(Motor.Left, (int)speed);
            }

            if (handLeft.Position.Y > hipLeft.Position.Y)
            {
                var speed = (handLeft.Position.Y - hipLeft.Position.Y) * 200;
                if (speed > 230) speed = 230;
                _btCon.SetSpeed(Motor.Right, (int)speed);
            }

        }

And so, it sets the speed based on your hand movements. Explore the ConnectionHelper and BluetoothConnector classes I wrote.

Conclusion

The code is here in Github. Fork it and play with it, and expand it.

Reactive Extensions Or Rx (More On IEnumerable, IQueryable, IObservable and IQbservable) - Awesome Libraries For C# Developers #2

noreply@blogger.com (Anonymous) — Sat, 31 Aug 2013 08:01:00 +0000

In my last post – we had a look at Interactive Extensions. In this post, we’ll do a recap of Reactive Extensions and LINQ to Event streams.

Reactive Extensions are out there in the wild for some time, and I had a series about Reactive Extensions few years back. How ever, after my last post on Interactive Extensions, I thought we should discuss Reactive extensions in a bit more detail. Also, in this post we’ll touch IQbservables – the most mysteriously named thing/interface in the world, may be after Higgs Boson. Push and Pull sequences are everywhere – and now with the devices on one end and the cloud at the other end, most of the data transactions happen via push/pull sequences. Hence, it is essential to grab the basic concepts regarding the programming models around them.

First Things First

Let us take a step back and discuss IEnumerable and IQueryable first, before discussing further about Reactive IObservable and IQbservable (Qbservables = Queryable Observables – Oh yea, funny name).

IEnumerable<T>

As you may be aware, the IEnumerable model can be viewed as a pull operation. You are getting an enumerator, and then you iterate the collection by moving forward using MoveNext on a set of items till you reach the final item. And Pull models are useful when the environment is requesting data from an external source. To cover some basics - IEnumerable has a GetEnumerator method which returns an enumerator with a MoveNext() method and a Current property. Offline tip - A C# for each statement can iterate on any dumb thing that can return a GetEnumerator. Anyway, here is what the non generic version of IEnumerable looks like.

public interface IEnumerable
{
    IEnumerator GetEnumerator();
}

public interface IEnumerator
{
    Object Current {get;}
    bool MoveNext();
    void Reset();
}

Now, LINQ defines a set of operators as extension methods, on top of the generic version of IEnumerable – i.e, IEnumerable<T> - So by leveraging the type inference support for Generic Methods, you can invoke these methods on any IEnumerable with out specifying the type. I.e, you could say someStringArray.Count() instead of someStringArray.Count<String>(). You can explore Enumerable class to find these static extensions.

The actual query operators in this case (like Where, Count etc) with related expressions are compiled to IL, and they operate in process much like any IL code is executed by CLR. From an implementation point of view, the parameters of LINQ clauses like Where is a lambda expression (As you may be already knowing, the from.. select is just Syntax sugar that gets expanded to extension methods of IEnumerable<T>), and in most cases a delegate like Func<T,..> can represent an expression from an in memory perspective. But what if you want apply query operators on items sitting some where else? For example, how to apply LINQ operators on top of a set of data rows stored in a table in a database that may be in the cloud, instead of an in memory collection that is an IEnumerable<T>? That is exactly what IQueryable<T> is for.

IQueryable<T>

IQueryable<T> is an IEnumerable<T> (It inherits from IEnumerable<T>) and it points to a query expression that can be executed in a remote world. The LINQ operators for querying objects of type IQueryable<T> are defined in Queryable class, and returns Expression<Func<T..>> when you apply them on an IQueryable<T>, which is a System.Linq.Expressions.Expression (you can read about expression trees here). This will be translated to the remote world (say a SQL system) via a query provider. So, essentially, IQueryable concrete implementations points to a query expression and a Query Provider – it is the job of Query Provider to translate the query expression to the query language of the remote world where it gets executed. From an implementation point of view, the parameters you pass for LINQ that is applied on an IQueryable is assigned to an Expression<T,..> instead. Expression trees in .NET provides a way to represent code as data or kind of Abstract Syntax Trees. Later, the query provider will walk through this to construct an equivalent query in the remote world.

    public interface IQueryable : IEnumerable {       
        Type ElementType { get; }
        Expression Expression { get; }
        IQueryProvider Provider { get; }
    }
    public interface IQueryable<T> : IEnumerable<T>, IQueryable, IEnumerable {
       ..
    }

For example, in LINQ to Entity Framework or LINQ to SQL, the query provider will convert the expressions to SQL and hand it over to the database server. You can even view the translation to the target query language (SQL), just by looking at the Or in short, the LINQ query operators you apply on IQueryable will be used to build an expression tree, and this will be translated by the query provider to build and execute a query in a remote world. Read this article if you are not clear about how an expression trees are built using Expression<T> from Lambdas.

Reactive Extensions

So, now let us get into the anatomy and philosophy of observables.

IObservable <T>

As we discussed, objects of type IEnumerable<T> are pull sequences. But then, in real world, at times we push things as well – not just pull. (Health Alert – when you do both together, make sure you do it safe). In a lot of scenarios, push pattern makes a lot of sense – for example, instead of you waiting in a queue infinitely day and night with your neighbors in front of the local post office to collect snail mails, the post office agent will just push you the mails to your home when they arrive.

Now, one of the cool things about push and pull sequences are, they are duals. This also means, IObservable<T> is a dual of IEnumerable<T> – See the code below. So, to keep the story short, the dual interface of IEnumerable, derived using the Categorical Duality is IObservable. The story goes like some members in Erik’s team (he was with Microsoft then) had a well deserved temporal meglomaniac hyperactive spike when they discovered this duality. Here is a beautiful paper from Erik on that if you are more interested – A brief summary of Erik’s paper is below.

//Generic version of IEnumerable, ignoring the non generic IEnumerable base

interface IEnumerable<out T>
{
	IEnumerator<T> GetEnumerator();
}

interface IEnumerator<out T>: IDisposable
{
	bool MoveNext(); // throws Exception
	T Current { get; } 
}


//Its dual IObservable

interface IObservable<out T>
{
	IDisposable Subscribe(IObserver<T> observer);
}

interface IObserver<in T>
{
	void OnCompleted(bool done);
	void OnError(Exception exception);
	T OnNext { set; } 
}

Surprisingly, the IObservable implementation looks like the Observer pattern.

Now, LINQ operators are cool. They are very expressive, and provide an abstraction to query things. So the crazy guys in the Reactive Team thought they should take LINQ to work against event streams. Event streams are in fact push sequences, instead of pull sequences. So, they built IObservable. IObservable fabric lets you write LINQ operators on top of push sequences like event streams, much like the same way you query IEnumerable<T>. The LINQ operators for an object of type IObservable<T> are defined in Observable class. So, how will you implement a LINQ operator, like where, on an observer to do some filtering? Here is a simple example of the filter operator Where for an IEnumerable and an IObservable (simplified for comparison). In the case of IEnumerable, you dispose the enumerator when we are done with traversing.

 //Where for IEnumerable

        static IEnumerable<T> Where<T>(IEnumerable<T> source, Func<T, bool> predicate)
        {
            // foreach(var element in source)
            //   if (predicate(element))
            //        yield return element;
                    
            using (var enumerator = source.GetEnumerator())
            {
                while (enumerator.MoveNext())
                {
                    var value= enumerator.Current;
                    if (predicate(value))
                    {
                        yield return value;
                    }
                }
            }
        }

//Where for IObservable

        static  IObservable<T> Where<T>(this IObserver<T> source, Func<T, bool> predicate)
        {
           return Observable.Create<T>(observer =>
               {
                   return source.Subscribe(Observer.Create<T>(value =>
                       {
                           try
                           {
                               if (predicate(value)) observer.OnNext(value);
                           }
                           catch (Exception e)
                           {
                               observer.OnError(e);
                           }
                       }));
               });
       }

Now, look at the IObservable’s Where implementation. In this case, we return the IDisposable handle to an Observable so that we can dispose it to stop subscription. For filtering, we are simply creating an inner observable that we are subscribing to the source to apply our filtering logic inside that - and then creating another top level observable that subscribes to the inner observable we created. Now, you can have any concrete implementation for IObservable<T> that wraps an event source, and then you can query that using Where!! Cool. Observable class in Reactive extensions has few helper methods to create observables from events, like FromEvent. Let us create an observable, and query the events now. Fortunately, the Rx Team already has the entire implementation of Observables and related Query operators so that we don’t end up in writing customer query operators like this.

You can do a nuget for install-package Rx-Main to install Rx, and try out this example that shows event filtering.

  //Let us print all ticks between 5 seconds and 20 seconds
            
            //Interval in milli seconds
            var timer = new Timer() { Interval = 1000 };
            timer.Start();

            //Create our event stream which is an Observable
            var eventStream = Observable.FromEventPattern<ElapsedEventArgs>(timer, "Elapsed");
            var nowTime = DateTime.Now;

            //Same as eventStream.Where(item => ...);

            var filteredEvents = from e in eventStream
                                 let time = e.EventArgs.SignalTime
                                 where
                                     time > nowTime.AddSeconds(5) &&
                                     time < nowTime.AddSeconds(20)
                                 select e;

            //Subscribe to our observable
            filteredEvents.Subscribe(t => Console.WriteLine(DateTime.Now));

            Console.WriteLine("Let us wait..");
	    
            //Dispose filteredEvents explicitly if you want
            Console.ReadKey();

Obviously, in the above example, we could’ve used Observable.Timer – but I just wanted to show how to wrap an external event source with observables. Similarly, you can wrap your Mouse Events or WPF events. You can explore more about Rx and observables, and few applications here. Let us move on now to IQbservables.

IQbservable<T>

Now, let us focus on IQbservable<T>. IQbservable<T> is the counterpart to IObserver<T> to represent a query on push sequences/event sources as an expression, much like IQueryable<T> is the counterpart of IEnumerable<T>. So, what exactly this means? If you inspect IQbservable, you can see that

public interface IQbservable<out T> : IQbservable, IObservable<T>
    {
    }

    public interface IQbservable
    {
        Type ElementType { get; }
        Expression Expression { get; }
        IQbservableProvider Provider { get; }
    }

You can see that it has an Expression property to represent the LINQ to Observable query much like how IQueryable had an Expression to represent the AST of a LINQ query. The IQbservableProvider is responsible for translating the expression to the language of a remote event source (may be a stream server in the cloud).

Conclusion

This post is a very high level summary of Rx Extensions, and here is an awesome talk from Bart De Smet that you cannot miss.

And let me take the liberty of embedding the drawing created by Charles that is a concrete representation of the abstract drawing Bart did in the white board. This is the summary of this post.

We’ll discuss more practical scenarios where Rx and Ix comes so handy in future – mainly for device to cloud interaction scenarios, complex event processing, task distribution using ISheduler etc - along with some brilliant add on libraries others are creating on top of Rx. But this one was for a quick introduction. Happy Coding!!

Interactive Extensions - Awesome Libraries For C# Developers #1

noreply@blogger.com (Anonymous) — Tue, 20 Aug 2013 18:48:00 +0000

Recently while I was giving a C# talk, I realized that a lot of developers are still not familiar with the advantages of some of the evolving, but very useful .NET libraries. Hence, I thought about writing a high level post introducing some of them as part of my Back To Basics series, generally around .NET and Javascript. In this post we’ll explore Interactive Extensions, which is a set of extensions initially developed for Reactive Extensions by the Microsoft Rx team.

Recap

Interactive Extensions, at its core, has a number of new extensions methods for IEnumerable<T> – i.e it adds a number of utility LINQ to Object query operators. You may have hand coded some of these utility extension methods some where in your helpers or utility classes, but now a lot of them are aggregated together by the Rx team. Also, this post assumes you are familiar with the cold IEnumerable model and iterators in C#. Basically, what C# compiler does is, it takes an yield return statement and generate a class out of that for each iterator. So, in one way, each C# iterator internally holds a state machine. You can examine this using Reflector or something, on a method yield returning an IEnumerator<T>. Or better, there is a cool post from my friend Abhishek Sur here or this post about implementation of Iterators in C#

More About Interactive Extensions

Fire up a C# console application, and install the Interactive Extensions Package using install-package Ix-Main . You can explore the System.Linq.EnumerationsEx namespace in System.Interactive.dll - Now, let us explore some useful extension methods that got added to IEnumerable.

Examining Few Utility Methods In Interactive Extensions

Let us quickly examine few useful Utility methods.

Do

What the simplest version of 'Do' does is pretty interesting. It'll lazily invoke an action on each element in the sequence, when we do the enumeration leveraging the iterator.

            
 //Let us create a set of numbers
 var numbers = new int[] { 30, 40, 20, 40 };
 var result=numbers.Do(n=>Console.WriteLine(n));

 Console.WriteLine("Before Enumeration");

 foreach(var item in result)
            {
                //The action will be invoked when we actually enumerate                
            }
 Console.WriteLine("After Enumeration");

And the result below. Note that the action (in this case, our Console.WriteLine to print the values) is applied place when we enumerate.

Now, the implementation of the simplest version of Do method is something like this, if you have a quick peek at the the Interactive Extensions source code here in Codeplex, you could see how our Do method is actually implemented. Here is a shortened version.

public static class StolenLinqExtensions
    {
        public static IEnumerable<TSource> StolenDo<TSource>(this IEnumerable<TSource> source, Action<TSource> onNext)
        {
            //Get the enumerator
            using (var e = source.GetEnumerator())
            {
                while (true)
                {
                    //Move next
                    if (!e.MoveNext())
                        break;
                    var current = e.Current;

                    //Call our action on top of the current item
                    onNext(current);

                    //Yield return
                    yield return current;
                }
            }
        }

    }

Cool, huh.

DoWhile

DoWhile in Ix is pretty interesting. It generates an enumerable sequence, by repeating the source sequence till the given condition is true.

IEnumerable<TResult> DoWhile<TResult>(IEnumerable<TResult> source, Func<bool> condition)

Consider the following code.

  var numbers = new int[] { 30, 40, 20, 40 };

  var then = DateTime.Now.Add(new TimeSpan(0, 0, 10));
  var results = numbers.DoWhile(() => DateTime.Now < then);

  foreach (var r in results)
  Console.WriteLine(r);

As expected, you’ll see the foreach loop enumerating results repeatedly, till we reach meet the DateTime.Now < then condition – i.e, till we reach 10 seconds.

Scan

Scan will take a sequence, to apply an accumulator function to generate a sequence of accumulated values. For an example, let us create a simple sum accumulator, that'll take a set of numbers to accumulate the sum of each number with the previous one

 var numbers = new int[] { 10, 20, 30, 40 };
 
 //0 is just the starting seed value
 var results = numbers.Scan(0,(sum, num) => sum+num);

 //Print Results. Results will contain 10, 30, 60, 100

 //0+10=10
 //10+20 = 30
 //30 + 30 = 60
 //60 + 40 = 100

And you may have a look at the actual Scan implementation, from the Rx repository in Codeplex . Here is an abbreviated version.

IEnumerable<TAccumulate> StolenScan<TSource, TAccumulate>
   (this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, 
                               TSource, TAccumulate> accumulator)
        {
            var acc = seed;

            foreach (var item in source)
            {
                acc = accumulator(acc, item);
                yield return acc;
            }
        }

Conclusion

We just touched the tip of the iceberg, as the objective of this post was to introduce you to Ix. We may discuss this in a bit more depth, after covering few other libraries including Rx. There is a pretty exciting talk from Bart De Smet here that you should not miss. Ix is specifically very interesting because of it’s functional roots. Have a look at the Reactive Extensions repository in Codeplex for more inspiration, that should give you a lot more ideas about few functional patterns. You may also play with Ix Providers and Ix Async packages.

As usual, happy coding!!

Building A Recommendation Engine - Machine Learning Using Windows Azure HDInsight, Hadoop And Mahout

noreply@blogger.com (Anonymous) — Thu, 11 Jul 2013 16:39:00 +0000

Feel like helping some one today?

Let us help the Stack Exchange guys to suggest questions to a user that he can answer, based on his answering history, much like the way Amazon suggests you products based on your previous purchase history. If you don’t know what Stack Exchange does – they run a number of Q&A sites including the massively popular Stack Overflow.

Our objective here is to see how we can analyze the past answers of a user, to predict questions that he may answer in future. May Stack Exchange’s current recommendation logic may work better than ours, but that won’t prevent us from helping them for our own learning purposes .

We’ll be doing the following tasks.

Extracting the required information from Stack Exchange data set
Using the required information to build a Recommender

But let us start with the basics. If you are totally new to Apache Hadoop and Hadoop On Azure, I recommend you to read these introductory articles before you begin, where I explain HDInsight and Map Reduce model a bit in detail.

Behind the Scenes

Here we go, let us get into some “data science” woo do first. Cool!! Distributed Machine learning is mainly used for

Recommendations - Remember the Amazon Recommendations? Normally used to predict preferences based on history.
Clustering - For tasks like finding grouping together related documents from a set of documents, or finding like minded people from a community
Classification - For identifying which set of category a new item belongs to. This normally includes training the system first, and then asking the system to detect an item.

“Big Data” jargon is often used when you need to perform operations on a very large data set. In this article, we’ll be dealing with extracting some data from a large data set, and building a Recommender using our extracted data.

What is a Recommender?

Broadly speaking, we can build a recommender either by

Finding questions that a user may be interested in answering, based on the questions answered by other users like him
Finding other questions that are similar to the questions he answered already.

The first technique is known as user based recommendation, and the second technique is known as item based recommendations.

In the first case, taste can be determined by how may questions you answered in common with that user (the questions both of you answered). For example, think about User1, User2, User3 and User4 – Answering few questions Q1, Q2, Q3 and Q4. This diagram shows the Questions answered by the users

Based on the above diagram, User1 and User2 answered Q1, Q2 and Q3. Now, User3 answered Q3 and Q2, but not Q1. Now, to some extent, we can safely assume that User3 will be interested in answering Q1 – because two users who answered Q2 and Q3 with him already answered Q1. There is some taste matching here, isn’t it? So, if you have a array of {UserId, QuestionId} – it seems that data is enough for us to build a recommender.

The Logic Side

Now, how exactly we are going to do build a question recommender? In fact it is quite simple.

First, we need to find the number of times a pair of questions co-occur across the available users. Note that this matrix is having no relations with the user. For example, if Q1 and Q2 is appearing together 2 times (as in the above diagram), co occurrence value at {Q1,Q2} will be 2. Here is the co-occurrence matrix (hope I got this right).

Q1 and Q2 co-occurs 2 times (User1 and User2 answered Q1 ,Q2)
Q1 and Q3 co-occurs 2 times (User1 and User2 answered both Q1, Q3)
Q2 and Q3 co-occurs 3 times (User1, User2 and User3 answered Q2, Q3)
Like wise..

The above matrix just captures how many times a pair of questions co-occurred (answered) as discussed above. There is no mapping with users yet. Now, how we’ll relate this to find a user’s preference? To find out how close a question ‘matches’ a user, we just need to

Find out how often that question co occurs with other questions answered by a that user
Eliminate questions already answered by the user.

For the first step, we need to multiply the above matrix with the user’s preference matrix.

For example, let us Take User3. For User3, the Preference mapping with questions [Q1,Q2,Q3,Q4] is [0,1,1,0] because he already answered Q2 and Q3, but not Q1 and Q4. So, let us multiply this with the above co-occurrence matrix. Remember that this is a matrix multiplication /dot product. The Result indicates how often a Question co-occurs with other questions answered by a user (weightage).

We can omit Q2 and Q3 from the results, as we know the User 3 already answered them. Now, from the remaining, Q1 and Q4 – Q1 has the higher value (4) and hence the higher taste matching with User3. Intuitively, this indicated Q1 co-occurred with the questions already answered by User 3 (Q2 and Q3) more than Q4 co-occurred with Q2 and Q3 – so User3 will be interested in answering Q1 more than Q4. In an actual implementation, note that the User’s taste matrix will be a sparse matrix (mostly zeros) as the user will be answering only a very limited subset of questions in the past. The advantage of the above logic is, we can use a distributed map reduce model for compute with multiple map-reduce tasks - Constructing the co-occurrence matrix, Finding the dot product for each user etc.

Now, let us start thinking about the implementation.

Implementation

From the implementation point of view,

We need to provision a Hadoop Cluster
We need to download and extract the data to analyze (Stack Overflow data)
Job 1 – Extract the Data - From each line, extract {UserId, QuestionId} for all questions answered by the user.
Job 2 – Build the Recommender - Use the output from above Map Reduce to build the recommendation model where possible items are listed against each user.

Let us roll!!

Step 1 - Provisioning Your Cluster

Now remember, the Stack Exchange data is huge. So, we need to have a distributed environment to process the same. Let us head over to Windows Azure. If you don’t have an account, sign up for the free trial. Now, head over to the preview page, and request the HDInsight (Hadoop on Azure) preview.

Once you have the HD Insight available, you can create a Hadoop cluster easily. I’m creating a cluster named stackanalyzer.

Once you have the cluster ready, you’ll see the Connect and Manage buttons in your dashboard (Not shown here). Connect to the head node of your cluster by clicking the ‘Connect’ button, which should open a Remote Desktop Connection to the head node. You may also click the ‘Manage’ button to open your web based management dashboard. (If you want, you can read more about HD Insight here)

Step 2 - Getting Your Data To Analyze

Once you connected to your cluster’s head node using RDP, you may download the Stack Exchange data. You can download the Stack Exchange sites data from Clear Bits, under Creative Commons. I installed Mu-Torrent client in the head node, and then downloaded and extracted the data for http://cooking.stackexchange.com/ – The extracted files looks like this – a bunch of XML files.

What we are interested is in the Posts XML File. Each line represents either a question, or an answer. If it is a question, PostTypeId =1, and if it is an answer, PostTypeId=2.The ParentId represents the question’s Id for an answer, and OwnerUserId represents the guy who wrote the answer for this question.

<row Id="16" PostTypeId="2" ParentId="2" CreationDate="2010-07-09T19:13:37.540" Score="3"
Body="<p>...shortenedforbrevity... </p>
"
OwnerUserId="34" LastActivityDate="2010-07-09T19:13:37.540" />

So, for us, we need to extract the {OwnerUserId, ParentId} for all posts where PostTypeId=2 (Answers) which is a representation of {User,Question,Votes}. The Mahout Recommender Job we’ll be using later will take this data, and will build a Recommendation result.

Now, extracting this data itself is a huge task when you consider the Posts file is huge. For the Cooking site, it is not so huge – but if you are analyzing the entire Stack Overflow, the Posts file may come in GBs. For extraction of this data itself, let us leverage Hadoop and write a custom Map Reduce Job.

Step 3 - Extracting The Data We Need From the Dump (User, Question)

To extract the data, we’ll leverage Hadoop to distribute. Let us write a simple Mapper. As mentioned earlier, we need to figure out {OwnerUserId, ParentId} for all posts with PostTypeId=2. This is because, the input for the Recommender Job we may run later is {user, item}. For this, first load the Posts.XML to HDFS. You may use the hadoop fs command to copy the local file to the specified input path.

Now, time to write a custom mapper to extract the data for us. We’ll be using Hadoop On Azure .NET SDK to write our Map Reduce job. Not that we are specifying the input folder and output folder in the configuration section. Fire up Visual Studio, and create a C# Console application. If you remember from my previous articles, hadoop fs <yourcommand> is used to access HDFS file system, and it’ll help if you know some basic *nix commands like ls, cat etc.

Note: See my earlier posts regarding the first bits of HDInsight to understand more about Map Reduce Model and Hadoop on Azure

You need to install the Hadoop Map Reduce package from Hadoop SDK for .NET via Nuget package manager.

install-package Microsoft.Hadoop.MapReduce

Now, here is some code where we

Create A Mapper
Create a Job
Submit the Job to the cluster

Here we go.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text;
using System.Xml.Linq;
using Microsoft.Hadoop.MapReduce;

namespace StackExtractor
{

    //Our Mapper that takes a line of XML input and spits out the {OwnerUserId,ParentId,Score} 
    //i.e, {User,Question,Weightage}
    public class UserQuestionsMapper : MapperBase
    {
        public override void Map(string inputLine, MapperContext context)
        {
            try
            {
                var obj = XElement.Parse(inputLine);
                var postType = obj.Attribute("PostTypeId");
                if (postType != null && postType.Value == "2")
                {
                    var owner = obj.Attribute("OwnerUserId");
                    var parent = obj.Attribute("ParentId");
		   
                    // Write output data. Ignore records will null values if any
                    if (owner != null && parent != null )
                    {
                        context.EmitLine(string.Format("{0},{1}", owner.Value, parent.Value));
                    }
                }
            }
            catch
            {
                //Ignore this line if we can't parse
            }
        }
    }


    //Our Extraction Job using our Mapper
    public class UserQuestionsExtractionJob : HadoopJob<UserQuestionsMapper>
    {
        public override HadoopJobConfiguration Configure(ExecutorContext context)
        {
            var config = new HadoopJobConfiguration();
            config.DeleteOutputFolder = true;
            config.InputPath = "/input/Cooking";
            config.OutputFolder = "/output/Cooking";
            return config;
        }

       
    }

    //Driver that submits this to the cluster in the cloud
    //And will wait for the result. This will push your executables to the Azure storage
    //and will execute the command line in the head node (HDFS for Hadoop on Azure uses Azure storage)
    public class Driver
    {
        public static void Main()
        {
            try
            {
                var azureCluster = new Uri("https://{yoururl}.azurehdinsight.net:563");
                const string clusterUserName = "admin";
                const string clusterPassword = "{yourpassword}";

                // This is the name of the account under which Hadoop will execute jobs.
                // Normally this is just "Hadoop".
                const string hadoopUserName = "Hadoop";

                // Azure Storage Information.
                const string azureStorageAccount = "{yourstorage}.blob.core.windows.net";
                const string azureStorageKey =
                    "{yourstoragekey}";
                const string azureStorageContainer = "{yourcontainer}";
                const bool createContinerIfNotExist = true;
                Console.WriteLine("Connecting : {0} ", DateTime.Now);

                var hadoop = Hadoop.Connect(azureCluster,
                                            clusterUserName,
                                            hadoopUserName,
                                            clusterPassword,
                                            azureStorageAccount,
                                            azureStorageKey,
                                            azureStorageContainer,
                                            createContinerIfNotExist);

                Console.WriteLine("Starting: {0} ", DateTime.Now);
                var result = hadoop.MapReduceJob.ExecuteJob<UserQuestionsExtractionJob>();
                var info = result.Info;

                Console.WriteLine("Done: {0} ", DateTime.Now);
                Console.WriteLine("\nInfo From Server\n----------------------");
                Console.WriteLine("StandardError: " + info.StandardError);
                Console.WriteLine("\n----------------------");
                Console.WriteLine("StandardOut: " + info.StandardOut);
                Console.WriteLine("\n----------------------");
                Console.WriteLine("ExitCode: " + info.ExitCode);
            }
            catch(Exception ex)
            {
                Console.WriteLine("Error: {0} ", ex.StackTrace.ToString(CultureInfo.InvariantCulture)); 
            }
            Console.WriteLine("Press Any Key To Exit..");
            Console.ReadLine();
        }
    }


}

Now, Compile and run the above program. The ExecuteJob will upload the required binaries to your cluster, and will initiate a Hadoop Streaming Job that’ll run our Mappers on the cluster, with input from the Posts file we stored earlier in the Input folder. Our console application will submit the Job to the cloud, and will wait for the result. The Hadoop SDK will upload the map reduce binaries to the blob, and will build the required command line to execute the job (See my previous posts to understand how to do this manually). You can inspect the job by clicking Hadoop Map Reduce status tracker from the desktop short cut in the head node.

If everything goes well, you’ll see the results like this.

As you see above, you can find the output in /output/Cooking folder. If you RDP to your cluster’s head node, and check the output folder now, you should see the files created by our Map Reduce Job.

And as expected, the files contain the extracted data, which represents the UserId,QuestionId – For all questions answered by a user. If you want, you can load the data from HDFS to Hive, and then view the same with Microsoft Excel using the ODBC for Hive. See my previous articles.

Step 4 – Build the recommender And generate recommendations

As a next step, we need to build the co-occurrence matrix and run a recommender job, to convert our {UserId,QuestionId} data to recommendations. Fortunately, we don’t need to write a Map Reduce job for this. We could leverage Mahout library along with Hadoop. Read about Mahout Here

RDP to the head node of our cluster, as we need to install Mahout. Download the latest version of Mahout (0.7) as of this writing, and copy the same to the c:\app\dist folder in the head node of your cluster.

Mahout’s Recommender Job has support for multiple algorithms to build recommendations – In this case, we’ll be using SIMILARITY_COOCCURRENCE. The Algorithms Page of Mahout website has lot more information about Recommendation, Clustering and Classification algorithms. We’ll be using the files we’ve in the /output/Cooking folder to build our recommendation.

Time to run the Recommender job. Create a users.txt file and place the IDs of the users for whom you need recommendations in that file, and copy the same to HDFS.

Now, the following command should start the Recommendation Job. Remember, we’ll use the output files from our above Map Reduce job as input to the Recommender. Let us kick start the Recommendation job. This will generate output in the /recommend/ folder, for all users specified in the users.txt file. You can use the –numRecommendations switch to specify the number of recommendations you need against each user. If there is a preference relation with a user and and item, (like the number of times a user played a song), you could keep the input dataset for a recommender as {user,item,preferencevalue} – In this case, we are omitting the preference weightage.

Note: If the below command fails after re run complaining output directory already exists, just try removing the tmp folder and the output folder using hadoop fs –rmr temp and hadoop fs –rmr /recommend/

hadoop jar c:\Apps\dist\mahout-0.7\mahout-core-0.7-job.jar 
	org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -s SIMILARITY_COOCCURRENCE 
	--input=/output/Cooking 
	--output=/recommend/ 
	--usersFile=/data/users.txt

After the job is finished, examine the /recommend/ folder, and try printing the content in the generated file. You may see the top recommendations, against the user Ids you had in the users.txt.

So, the recommendation engine think User 1393 may answer the questions 6419, 16897 etc if we suggest the same to him. You could experiment with other Similarity classes like SIMILARITY_LOGLIKELIHOOD, SIMILARITY_PEARSON_CORRELATION etc to find the best results. Iterate and optimize till you are happy.

For an though experiment here is another exercise - Examine the Stack Exchange data set, and find out how you may build a Recommender to show a ‘You may also like’ questions based on the questions a user favorite?

Conclusion

In this example, we were doing a lot of manual work to upload the required input files to HDFS, and triggering the Recommender Job manually. In fact, you could automate this entire work flow leveraging Hadoop For Azure SDK. But that is for another post, stay tuned. Real life analysis has much more to do, including writing map/reducers for extracting and dumping data to HDFS, automating creation of hive tables, perform operations using HiveQL or PIG, etc. However, we just examined the steps involved in doing something meaningful with Azure, Hadoop and Mahout.

You may also access this data in your Mobile App or ASP.NET Web application, either by using Sqoop to export this to SQL Server, or by loading it to a Hive table as I explained earlier. Happy Coding and Machine Learning!! Also, if you are interested in scenarios where you could tie your existing applications with HD Insight to build end to end workflows, get in touch with me.

I suggest you to read further.

Mahout In Action – An awesome start if you want to get into the real details.
Mahout Tutorials – A set of good tutorials about mahout

May be this is the best time for Microsoft to Open Source the .NET Framework

noreply@blogger.com (Anonymous) — Wed, 15 May 2013 17:36:00 +0000

Just read a thoughtful post from OdeToCode / Scott Allen – Where is .NET headed.

Scott wrote,

If your business or company still relies solely on components delivered to developers through an MSDN subscription, then it is past time to start looking beyond what Microsoft offers for .NET development so you won’t be left behind in 5 years. Embrace and support open source.

At least, that’s how I see things.

As I commented there, the best move Microsoft could make at this point, with respect to .NET is - to Open Source the .NET Framework. And then launch a program like Apache Incubator around to promote ‘real’ OSS initiatives around the .NET ecosystem.

I believe, the real problem with .NET ecosystem is the unavailability of serious frameworks for solving new age problems - .NET developers are cramped with the non-availability of Proven libraries for Machine Learning, Distributed Processing, Text Processing etc. - Talk about Hadoop, Solr, Lucene, Mahout, Storm, OpenNLP etc - All written in Java and is continuously maturing, despite the fact that Sun started screwing the Java community. The .NET ports for some of these libraries bit the dust long time back, or has very little adoption compared to their prominence in Java ecosystem.

Microsoft is now trying to bridge this gap by bringing in platforms like Hadoop to Azure, and building bridges around the same for .NET interoperability.

CLR and C# are awesome - but MS just can't push forward the development of mature libraries around the same to keep up with the pace of OSS initiatives. The true future of .NET lies in the hand of OSS community at large. Hopefully, if Azure turns out to be a big success story, then Microsoft won't mind open sourcing the Entire .NET stack ;). As Scott pointed in his article, Azure and related hosted offerings (like Office365) + Windows 8 could become the focus points for Microsoft’s revenue.

It is heartening to see that a number of frameworks like ASP.NET MVC, MEF, EF etc already got Open Sourced. But Microsoft can do better in the OSS world to ensure .NET ecosystem will grow further. I believe a lack of buy in from OSS community at large is there –and Microsoft could reduce the friction a little bit by open sourcing .NET.

Hack Raspberry Pi – How To Build Apps In C#, WinForms and ASP.NET Using Mono In Pi

noreply@blogger.com (Anonymous) — Tue, 16 Apr 2013 14:19:00 +0000

Recently I was doing a bit of R&D related to finding a viable, low cost platform for client nodes. Obviously, I came across Raspberry Pi, and found the same extremely interesting. Now, the missing piece of the puzzle was how to get going using C# and .NET in the Pi. C# is a great language, and there are a lot of C# developers out there in the wild who are interested in the Pi.

In this article, I’ll just document my findings so far, and will explain how develop using C# leveraging Mono in a Raspberry Pi. Also, we’ll see how to write few minimal Windows Forms & ASP.NET applications in the Pie as well.

Step 1: What is Raspberry Pi?

Raspberry Pi is an ARM/Linux box for just ~ $30. It was introduced with a vision to teach basic computer science in schools. How ever, it got a lot of attention from hackers all around the world, as it is an awesome low cost platform to hack and experiment cool ideas as Pi is almost a full fledged computer. More About R-Pi From Wikipedia.

The Raspberry Pi is a credit-card-sized single-board computer developed in the UK by theRaspberry Pi Foundation with the intention of promoting the teaching of basic computer science in schools. The Raspberry Pi has a Broadcom BCM2835 system on a chip (SoC),^[3] which includes anARM1176JZF-S 700 MHz processor (The firmware includes a number of "Turbo" modes so that the user can attempt overclocking, up to 1 GHz, without affecting the warranty),^[4] VideoCore IV GPU,^[12] and originally shipped with 256 megabytes of RAM, later upgraded to 512MB.^[13] It does not include a built-in hard disk or solid-state drive, but uses an SD card for booting and long-term storage.^[14] The Foundation's goal is to offer two versions, priced at US$25 and US$35.

Another good introduction from Life Hacker is here

Step 2: Setting up your Development Environment

Let us have a quick look at setting up the development environment.

If you have a physical Raspberry Pi

Raspberry Pi boots from an SD Card. Basically, this means you can transfer an operating system image to an SD Card, and boot your Pi from there. The Raspberry Pi download page lists multiple OS images you can use to load your SD Card. You will need to unzip it and write it to a suitable SD card using the UNIX tool dd. Windows users should use Win32DiskImager. There is a simple guide about how to setup your SD card if you are wondering how to do this. Then, you can connect your Pi device to the TV using an HDMI cable, and can hook up an old USB keyboard to start hacking.

Though multiple images are available in the below mentioned download page, please note that at this point only the Soft-float Debian “wheezy” image can be used to install Mono with out issues. This is because Mono doesn’t support hard-float for Raspberry Pi (simplified version) and some runtime methods like DateTime won’t work if you are using a hard float version. So, you need to download the Soft-Float Debian image and transfer the image file to the SD card. If you feel a bit adventurous, you can try this hard-float port of Mono for the Pi, but it is not supported.

If you don’t have a physical Raspberry Pi

You can still download one of the images from the download page and boot from an emulator like Qemu, which supports ARM architecture.

Get If you are on Windows and if you are not familiar with Qemu, you can download this pre-packaged version of Qemu + Raspberry Pi from here – How ever there is a problem. The Image file that comes with this download is the Hard float version. As we need Mono to work properly, download the Soft-Float Debian Wheezy image – and copy that to the Qemu folder, and modify run.bat in qemu folder to the below command line to point it to the correct image. (If you downloaded Qemu and Soft Float Debian Wheezy image separately, you can create a handy run.bat). Note that as of this writing, the image Soft float image is 2012-08-08-wheezy-armel.img – You need to modify this accordingly. The –cpu switch specify the CPU architecture, –hda attaches the image as the boot device and –m specifies the memory. I’m allocating ~~512~~ 256 MB to my virtual machine. (Note: Updated the command line to use 256 memory based on the comment from Peter below)

qemu-system-arm.exe -M versatilepb -cpu arm1176 -hda 2012-08-08-wheezy-armel.img -kernel kernel-qemu -m 256 -append "root=/dev/sda2"

Step 3: Booting the Pie And Installing Mono

Once you boot the Pie, enter the username and password based on the image/distribution you are using (user/password is normally pi/raspberry). Now, to install mono, we’ll use the apt-get package manager. Let us update and upgrade all the packages before we proceed with the install. Run apt-get with admin permissions, using sudo in the console/terminal.

Each of the below command will take a bit of time - so make sure you've a pressure ball around. And let the force be with you for a successful installation #lol.

sudo apt-get update --fix-missing

sudo apt-get upgrade --fix-missing

sudo apt-get install mono-complete --fix-missing

Step 4: Writing C#

Assuming you’ve got the installation right - Time to write a little bit of C#. Mono has a CSharp command line (REPL), you can bring it up by typing CSharp. So, just play around. If you are setting this up to teach CSharp to kids in schools (original Raspberry Pi vision), the force is stronger in you than you think (Sorry, HBO is now re-telecasting all the old Star Wards stuff). What ever. Here is my CSharp console. Note that if your DateTime.Now is behaving madly, you never read my above paragraph and you’ve got the wrong OS image.

Step 5: Write And Compile Your Code – A Windows Forms app

Quit the CSharp console (quit) and start the Window manager by typing ‘startx’ – Never mind if you are already there. This should bring up the Raspberry Pi desktop. Now, let us create a CSharp file using Leaf pad, a light weight editor in the Raspberry Pi. Either in Run or in a terminal window, type

sudo leafpad /opt/windowapp.cs

And type in some code. Let us create a quick Windows forms application to ensure this is possible in Mono in the Pi.

Now, let us compile the code using the mono compiler (gmcs). Ensure that you are providing the reference libraries we are importing using the /r switch. This should produce the windowapp.exe executable.

Cool. Now time to run it via Mono. This should display a tiny window with width and height 100 – You can drag the corners to see the little Window we created. From here onwards, it is your normal .NET style. Add controls, build apps and have a blast.

Step 6: Installing The XSP Web Server For Some ASP.NET Love

Let us now install Mono XSP server so that we can run some ASP.NET code in the Pie, to use it as a tiny web server. Let us get the XSP bits using apt-get. Fire up your terminal, and type

sudo apt-get install mono-xsp2 mono-xsp2-base asp.net-examples

Which should install XSP.

Cool, you got the XSP installed. Now, let us try some ASP.NET stuff in the Pi. Create a new aspx file in /opt folder using Leafpad, as we did earlier. Here is my aspx file, a simple aspx file I copied from Startdeveloper as I was lazy.

Save the file to your /opt folder, and And start XSP2 in the opt folder. Run XSP2, and open the Internet->Browser and navigate to http://localhost:8080 – You see, it works.

Conclusion

In this post, we explored how to setup Mono to build and run .NET Win forms and Web applications in the Raspberry Pi. Build a computer to learn .NET and C# for under 40$, huh. Follow me in twitter @ https://twitter.com/amazedsaint – Happy Coding.

BIG DATA for .NET Devs: HDInsight, Writing Hadoop Map Reduce Jobs In C# And Querying Results Back Using LINQ

noreply@blogger.com (Anonymous) — Thu, 28 Mar 2013 18:26:00 +0000

Azure HD Insight Services is a 100% Apache Hadoop implementation on top of Microsoft Azure cloud ecosystem.

In this post, we’ll explore

HDInsight/Hadoop on Azure in general and steps for starting with the same
Writing Map Reduce Jobs for Hadoop using C# in particular to store results in HDFS.
Transferring the result data from HDFS to Hive
Reading the data back from the hive using C# and LINQ

Preface

If you are new to Hadoop and Big Data concepts, I suggest you to quickly check out

There are a couple of ways you can start with HDInsight.

You may Go to Azure Preview features and opt in for HDInsight and/or install the same locally

Step 1: Setting up your instance locally in your Windows

For Development, I highly recommend you to install HDInsight developer version locally – You can find it straight inside the Web Platform installer.

Once you install the HDInsight locally, ensure you are running all the Hadoop services.

Also, you may use the following links once your cluster is up and running.

Goto http://localhost:50030 to see the HDInsight dashboard locally
Goto http://localhost:50070 to explore the HDFS file system

Here is the HDInsight dashboard running locally.

And now you are set.

Step 2: Install the Map Reduce package via Nuget

Let us explore how to write few Map Reduce jobs in C#. We’ll write a quick job to count namespaces from C# source files Earlier, in a couple of posts related to Hadoop on Azure - Analyzing some ‘Big Data’ using C# and Extracting Top 500 MSDN Links from Stack Overflow – I showed how to use C# Map Reduce Jobs with Hadoop Streaming to do some meaningful analytics. In this post, we’ll re-write the mapper and reducer leveraging the the new .NET SDK available, and will apply the same on few code files (you can apply that on any dataset).

The new .NET SDK for Hadoop makes it very easy to work with Hadoop from .NET – with more types for supporting Map Reduce Jobs, For creating LINQ to Hive queries etc. Also, the SDK provides an easier way to create and submit your own Map Reduce jobs directly in C# either to the local developer instance or to Azure Hadoop cluster.

To start with, create a console project and install the Microsoft.Hadoop.Mapreduce package via Nuget.

Install-Package Microsoft.Hadoop.Mapreduce

This will add the required dependencies.

Step 3: Writing your Mapper and Reducer

The mapper will read the input from the HDFS file system, and the writer will emit outputs to HDFS. HDFS is Hadoop’s distributed file system, which guarantees high availability. Checkout the Apache HDFS architecture guide for details.

With Hadoop SDK, now you can inherit your Mapper from the MapperBase class, and Reducer from the ReducerCombinerBase class. This is equivalent to the independent Mapper and Reducer exes I demonstrated earlier using Hadoop streaming, just that we’ve got a better way of doing the same. In the Map method, we are just extracting the namespace declarations using reg ex to emit the same (See hadoop streaming details in my previous article)

    //Mapper
    public class NamespaceMapper : MapperBase
    {
        //Override the map method.
        public override void Map(string inputLine, MapperContext context)
        {
            //Extract the namespace declarations in the Csharp files
            var reg = new Regex(@"(using)\s[A-za-z0-9_\.]*\;");
            var matches = reg.Matches(inputLine);

            foreach (Match match in matches)
            {
                //Just emit the namespaces.
                context.EmitKeyValue(match.Value,"1");
            }
        }
    }

    //Reducer
    public class NamespaceReducer : ReducerCombinerBase
    {
        //Accepts each key and count the occurrances
        public override void Reduce(string key, IEnumerable<string> values, 
            ReducerCombinerContext context)
        {
            //Write back  
            context.EmitKeyValue(key,values.Count().ToString());
        }
    }

Next, let us write a Map Reduce Job and configure the same.

Step 4: Writing your Namespace Counter Job

You can simply specify your Mapper and Reducer types and inherit from HadoopJob to create a job class. Here we go.

   //Our Namespace counter job
    public class NamespaceCounterJob : HadoopJob<NamespaceMapper, NamespaceReducer>
    {
        public override HadoopJobConfiguration Configure(ExecutorContext context)
        {
            var config = new HadoopJobConfiguration();
            config.InputPath = "input/CodeFiles";
            config.OutputFolder = "output/CodeFiles";
            return config;
        }
    }

Note that we are overriding the Configure method to specify the configuration parameters. In this case, we are specifying the input and output folders for our mapper/reducer – The lines in the files in input folder will be passed to our mapper instances, and the combined output from the reducer instances will be placed in the output folder.

Step 5: Submitting the job

Finally, we need to connect to the cluster and submit the job, using the ExecuteJob method. Here we go with the main driver.

class Program
    {
        static void Main(string[] args)
        {
            var hadoop = Hadoop.Connect();
            var result=hadoop.MapReduceJob.ExecuteJob<NamespaceCounterJob>();
        }
    }

We are invoking the ExecuteJob method using the NamespaceCounterJob type we just created. In this case, we are submitting the job locally – if you want to submit the job to an Azure HDInsight cluster for the actual execution scenario, you should pass the Azure connection parameters. Details here

Step 6: Executing the job

Before executing the job, you should prepare your input – in this case, you should copy the source code files in the input folder we provided as part of the configuration while creating our Job (see the NamespaceCounterJob). To do this, fire up the Hadoop command line console from the desktop. If your cluster is on Azure, you can remote login to the cluster head node by choosing Remote Login from the HDInsight Dashboard.

Create a folder using the hadoop fs –mkdir input/CodeFiles command
Copy few CSharp files to your folder using hadoop fs –copyFromLocal your\path\*.cs input/CodeFiles

See I’m copying all my CS files under BasicsRevisited folder to input/CodeFiles.

Now, build your project in Visual Studio, open the bin folder and execute your exe file. This will internally kick start MRRunner.exe and your map reduce job will get executed (The name of my executable is simply MapReduce.exe). You can see the detected file dependencies are automatically submitted.

Once the Map Reduce job is completed, you’ll find that the combined output will be placed in output/CodeFiles folder. You can issue the –ls and –cat commands to list the files and view the content of the part-00000 file where the output will be placed (Yes, a little Linux knowledge will help at times ). The part-00000 file contains the combined output of our task – see the name spaces along with their count from the files I submitted.

Step 7: Loading data from HDFS to Hive

As a next step, let us load the data from HDFS to Hadoop Hive so that we can query the same. We'll create a table using the CREATE TABLE hive syntax, and will load the data. You can run ‘hive’ command from the Hadoop command line to run the following statements.

CREATE TABLE nstable (
  namespace STRING,
  count INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;

LOAD DATA INPATH 'output/CodeFiles/part-00000' into table nstable;

And here is what you might see.

Now, you can read the data from the hive.

See my article Querying data from the Hive using LINQ and C#
I wrote a VS Add In so that you can connect to Hive and query the data

And there you go. Now you know everything about writing your own Hadoop Map Reduce Jobs in C#, load the data to the Hive, and query the same back in C# to visualize your data. Happy Coding.

Reflection API Changes in .NET 4.5 Applications and Fun with Custom Reflection Context

noreply@blogger.com (Anonymous) — Mon, 25 Mar 2013 13:06:00 +0000

Before .NET 4.5, as you are aware, the Type class in System namespace was used for most of the reflection scenarios. Though you can use Type class to inspect meta data, this is really heavy weight. As most reflection scenarios are just read only (for example, using attributes for providing meta data etc), Microsoft decided to introduce a light weight, alternate Type API as part of .NET Core profile. If you are wondering about different .NET profiles, here is a good list compiled together.

Type and TypeInfo

So to keep the long story short - in .NET 4.5 onwards, Type class got ‘shrunk’ to provide a shallow, high speed read only view of an object’s structure (definition). And TypeInfo class got introduced that contains the actual type definitions with more detailed information. See Type and TypeInfo – and check out the platform specific support for it’s members. You can read more about the Reflection API changes here

The TypeInfo class represents type definitions and the Type class represents type references. Given a Type object, you can get the name of the type as a string, without any requirement to load anything more. Alternatively, if you need rich information about a type, you can get a TypeInfo object from a Type object.

In .NET full profiles used for desktop and web applications, the old Type API (Accessing everything through Type) is still available to maintain backward compatibility, along with the new Type API (TypeInfo etc). And For .NET Framework profiles for new platforms like WinRT that don’t need backward compatibility, Type class got shrunk by removing a number of members as discussed, and TypeInfo is the only way to dive deep into reflection. If you are writing code that should target multiple framework versions (as in Portable Class Libraries), you should use the new API. Otherwise, you can continue to use the old API as .NET framework versions on full profile actually returns the self contained Type object as previously.

Using TypeInfo

You may call GetTypeInfo on any type object as shown below to get the TypeInfo for a Type, and may use the new API (like DeclaredProperties) to traverse the members.

	//Get the basic type
        Type myType = "somestring".GetType();

        //Get more information about that type
        TypeInfo myTypeDetails = myType.GetTypeInfo();

        foreach(var prop in myTypeDetails.DeclaredProperties)
            {
                Console.WriteLine(prop.Name);
            }

Also, you may use TypeInfo to check the relations between types. For example, here is how to use IsAssignableFrom method of TypeInfo to determine variance.

  //Can I assign a string to an object - True
   var canAssignStringToObject = typeof (object).GetTypeInfo()
                                  .IsAssignableFrom(typeof (string));

  //Can I assign an object to a string - False
  var canAssignObjectToString = typeof(string).GetTypeInfo()
                                 .IsAssignableFrom(typeof(object));

Custom Reflection Context

Custom Reflection Context is a new addition in .NET 4.5, to customize how a consumer using the TypeInfo based new Reflection API views reflected information about your types. This will be extremely helpful when you write Libraries and design components that take advantage of meta data. This is much like customizing Type information using techniques like type descriptors. The new API provides run time virtualization over reflection, and looks quite intuitive - I wish this was there when we spend years writing custom tools for extending Visual Studio few years back, to virtualize type information for existing and new components.

You can override properties in CustomReflectionContext to add virtualized properties, custom attributes etc. For example, here is a quick PropertyAdder context that’ll inject some properties while accessed through the custom reflection context

    //Just a dummy Dog class
    //See that we don't have any properties
    class Dog
    {

    }

//A PropertyAdder context
    //To add custom properties to the reflection context
    class PropertyAdder : CustomReflectionContext
    {
        private readonly Dictionary _properties;

        public PropertyAdder(Dictionary properties)
        {
            _properties = properties;
        }

        protected override IEnumerable AddProperties(Type type)
        {
            if (type == typeof(Dog))
            {
                foreach (var p in _properties)
                {
                    yield return
                        CreateProperty(MapType(p.Value.GetType().GetTypeInfo()),
                                       p.Key,
                                       (o) => _properties[p.Key],
                                       (o, v) => _properties[p.Key] = v);
                }
            }
            else
            {
                base.AddProperties(type);
            }
        }
    }

And here is a quick example for using our PropertyAdder. Essentially,

We are creating a new instance of PropertyAdder
Then decorating our Dog class’s type information by mapping the type information via the custom reflection context.
Finally, we are creating an instance of Dog class, and look ma- we’ve got the properties via reflection

So, here we go.

    //Main Driver
    class Program
    {
        static void Main(string[] args)
        {

            //Properties we want to add
            var properties = new Dictionary
                {
                    {"Name", "Joe"},
                    {"Age", 10}
                };

            //An instance of our custom reflection context
            var adderContext = new PropertyAdder(properties);

            //Get the type in the default reflection context.
            var dogTypeInfo = typeof (Dog).GetTypeInfo();

            //Get the type in the customized reflection context.
            //We'll map the context with our PropertyAdder
            var customDogTypeInfo = adderContext.MapType(dogTypeInfo);


            //Just an instance
            var d = new Dog();

            //Display properties and values using custom reflection context. 
            foreach (var prop in customDogTypeInfo.DeclaredProperties)
            {

                //We'll really see the properties we added above
                Console.WriteLine("{0}={1}",prop.Name, 
                    customDogTypeInfo.GetProperty(prop.Name).GetValue(d));
            }

            Console.ReadLine();
        }
    }

And here is the output. You’ll see that we are accessing the properties we added via the reflection context. Here you go.

So this is fun especially when you work with libraries that leverage meta data and on the tooling side. For example, the RegistrationBuilder class in MEF 2.0 inherits custom reflection context to provide meta data based conventions that describes rules for decorating entities.

Happy Coding . Also, see my other C# posts here

Azure Hadoop HDInsight Tools – Wrote A VS Add In For LINQ To Hive Model Generation and Hive SQL Queries

noreply@blogger.com (Anonymous) — Fri, 08 Mar 2013 13:21:00 +0000

This post is about the HD Insight Tools Visual Studio Add In - for accessing Azure Hadoop Hive from Visual Studio. I hacked together the same on my way back from the MVP summit, and a post is due. If you are new, last week I wrote about issuing LINQ queries against the Hive meta store – read that post for a general introduction about LINQ to Hive.

You may use HDInsight Tools Visual Studio Add In for

Generating wrapper classes for Hive tables for easily executing LINQ to Hive queries.
Running Hive SQL queries from Visual Studio and for viewing the results

Please note that you need to provision a Hadoop cluster in Azure, or need HDInsight installed locally (See these earlier posts related to Hadoop on Azure - Analyzing some ‘Big Data’ using C# and Extracting Top 500 MSDN Links from Stack Overflow ).

Installing The Add In

You can install the Add In from Visual Studio Tools –> Extensions and Updates.

Generating LINQ To Hive Model Classes

Once you install the Add In, Goto View-> Other Windows –> Hive Explorer to bring up the Hive Explorer tool Window. Click Add Connection to add a hive connection, by providing the server details, port, user name and password and click Save.

Once you add a connection, you can expand the connection to view the tables and fields in the Hive. Now, you can right click on the Added connection name in the tool Window, and click ‘Generate Code’ context menu to generate the model classes. Also, make sure you install the Microsoft.Hadoop.Hive nuget package by issuing the install-package Microsoft.Hadoop.Hive –pre nuget command.

You can use the generated model to query Hive using LINQ as mentioned in the previous article.

Issuing Queries

You can use the Queries window of Hive Explorer to issue Hive Queries. Choose the connection in the combo box, and shoot your query to view the results.

Happy Coding. I’m soon uploading the source code of Hive Explore to Github after some clean up once I’ve some time.

About That ‘Feelings’ In Software Industry

noreply@blogger.com (Anonymous) — Wed, 06 Mar 2013 18:47:00 +0000

When I read Built to Last few years back, I got stumbled upon this fact – it is all about building a cult. And a cult is not a bad thing. You can base it on some ethical values, and can sprinkle stories and sutras around the same to make it more exciting. Or, you may brand it as your corporate culture. But in the end of the day, it is all about building a cult which looks original and rallying your employees and customers around the same with out shaking their belief.

In the last ten years, Steve Jobs did that much better than any one else. I think he learned about the art of creating a cult when he was wandering in India, may be from Neem Karoli Baba. Today, people worship Apple.

I believe successful organizations are good in assembling together such a framework that can yield repeatable behavioral patterns from all stakeholders, most importantly, customers. Any successful organization need at least one iconic leader with charisma to grow beyond mediocrity. People buy a brand not just because of its usability – They buy or act when they believe in the cult of the brand, when they buy in to that ideology and find the trust.

As Simon points out in this Ted Talk, great leaders inspire action – they inspire employees and customers and other stake holders to align with the cult or self of the organization. I think the justification for all this lies deep with in, we are wired to follow a cult. For example, our feelings and instincts influence a purchase decision – probably much more than the numbers we have in our head.

So, I believe everything from Brand loyalty to architecture decisions are influenced by aspects like peer recommendations and herd mentality in a big way. That’s why a number of CXOs invest in ‘Big Data’ and ‘Cloud’ with out knowing the intricacies.

Even software has it’s FADs right now. So, it is of no surprise that the industry is also driven by its own fashions and styles at times. Some of these FADs will burst as bubbles. Who cares. The world works on chaos theory, and it is not really predictable all the time – so decisions based on gut feeling and intuitions are acceptable to some extent, isn’t it? Though, generally I don’t recommend subjectivity over objectivity.

AutoCommand For MVVM – More tricks With Lambdas and CallerMemberName

noreply@blogger.com (Anonymous) — Tue, 05 Mar 2013 20:55:00 +0000

Few days back I ranted about Lamdba Patterns in C# and Init Time branching. I also had a generally insane example regarding Init Time branching using CallerMemberName attribute, and got few comments about the usefulness (or the lack of the same).

While I generally agree with the comments, here is a may be useful, still crazy AutoCommand for your next basement MVVM application. First, here is how the AutoCommand will work.

//Our simple View Model
public class SimpleMathViewModel : BaseViewModel
    {
        public SimpleMathViewModel()
        {
            Commands = new Dictionary<string, Action>
                {
                    {"Add", ()=>Result=A+B},
                    {"Substract", ()=>Result=A-B},
                    {"Multiply", ()=>Result=A*B},
                    {"Divide", ()=>Result=A/B}
                };
        }

        //Your commands should get automatically wired up

        public ICommand Add { get { return new AutoCommand(this); } }
        public ICommand Substract { get { return new AutoCommand(this); } }
        public ICommand Multiply { get { return new AutoCommand(this); } }
        public ICommand Divide { get { return new AutoCommand(this); } }

        //Other blah blah properties
        public int A { get; set; }
        public int B { get; set; }

        private int _result;
        public int Result
        {
            get { return _result; }
            set
            {
                _result = value; OnPropertyChanged();
            }
        }
    }

Also, Look ma, there is no dirty property name string in the OnPropertyChanged method call to raise the PropertyChanged notification as it is inferred - but I'm sure you might have read some where else how to use CallerMemberName to do the trick. If you are still wondering, see the method implementation in the BaseViewModel code below.

Leaving that aside – how’z the wiring works for AutoCommand? Alright, guess you guessed it. Here is the AutoCommand implementation.

   public class AutoCommand : ICommand
    {
        private readonly BaseViewModel _instance;
        private readonly string _caller;

        //Just keep the caller name and the base view model instance
        public AutoCommand(BaseViewModel instance, 
                [CallerMemberName]string caller = "")
        {
            _instance = instance;
            _caller = caller;
        }

        public bool CanExecute(object parameter)
        {
            return true; //who cares
        }

        //Fetch the right command to execute from the dictionary
        public void Execute(object parameter)
        {
            _instance.Commands[_caller](); //get the ICommand based on the caller name
        }

        public event EventHandler CanExecuteChanged;
    }

And if you are wondering how we've implemented the BaseViewModel, it is trivial. There only interesting piece there is how we inject the property name to the base class method.

 public abstract class BaseViewModel : INotifyPropertyChanged
    {
        public event PropertyChangedEventHandler PropertyChanged;
        public Dictionary<string, Action> Commands { get; set; }

        protected virtual void OnPropertyChanged([CallerMemberName]string propertyName="")
        {           
            PropertyChangedEventHandler handler = PropertyChanged;
            if (handler != null) handler(this, new PropertyChangedEventArgs(propertyName));
        }
    }

This is not a best practice show case, was just discussing a language feature and few possibilities. Happy and Insane coding

A Quick introduction to Hadoop Hive on Azure and Querying Hive using LINQ in C#

noreply@blogger.com (Anonymous) — Thu, 28 Feb 2013 03:04:00 +0000

Earlier, in a couple of posts related to Hadoop on Azure - Analyzing some ‘Big Data’ using C# and Extracting Top 500 MSDN Links from Stack Overflow – I showed how to use C# Map Reduce Jobs with Hadoop Streaming to do some meaningful analytics.

Now, a preview version of the .NET SDK for Hadoop is available, making it easier to work with Hadoop from .NET – with more types for supporting Map Reduce Jobs, For creating LINQ to Hive queries etc. You can experiment with Hadoop and C# either by creating a cluster in http://hadooponazure.com or you can obtain Hadoop in your machine by installing Microsoft HDInsight using WebPI.

In case you are new to Hadoop on Azure, I suggest you read the introductory concepts here before you start. This post is just a quick example that shows how to use LINQ to Hive.

Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems

Installing the libraries

To start with, you can fire up Visual Studio, Create a console project, and install Microsoft.Hadoop.Hive libraries via Nuget.

install-package Microsoft.Hadoop.Hive -pre

Also, head over to http://hadoopoonazure.com and create a new cluster. And you are now set.

Creating the typed wrappers

To access Hive, you need to create a strongly typed wrapper – as of now, you need to roll this out your own, as there is no automated generation support. When you provision a Hadoop cluster, the Hive will be pre populated with a sample table (hivesampletable), and I’m using the same for the below example for brevity. You can connect to the Hive via ODBC and see the hive tables in Excel.

So, let us go ahead and create a hive connection (much like an EF data context) and a typed representation for a row in the table. HiveConnection and HiveTable types are in the Microsoft.Hadoop.Hive namespace.

    
    //Our concrete hive connection
    
    public class SampleHiveConnection : HiveConnection
    {
        public SampleHiveConnection(string hostName, int port) 
            : base(hostName, port, null, null) { }

        public SampleHiveConnection(string hostName, int port, 
                            string username, string password) 
            : base(hostName, port, username, password) { }

        public HiveTable<DeviceInfo> DeviceInfoTable
        {
            get
            {
                return this.GetTable<DeviceInfo>("hivesampletable");
            }
        }
    }

    //A typed row. Property names based on field names hivesampletable
    
    public class DeviceInfo : HiveRow
    {
        public string DevicePlatform { get; set; }
        public string DeviceMake { get; set; }
        public int ClientId { get; set; }
    }

Querying the Hive using LINQ

Now, you may perform LINQ queries against your Hive context, thanks to the Hadoop SDK we installed via Nuget. Just make sure to substitute the connection string, username and password with your own.

class Program
    {
        static void Main(string[] args)
        {


            //Create a hive connection
            //I've my cluster in https://www.hadooponazure.com
            var hive = new SampleHiveConnection(
                    "saintcluster.cloudapp.net", //your connection string
                    10000,                       //port                    
                    "user",                      //your username
                    "yourpass");                 //your password


            //Get the results
            //Make sure you goto the dashboard and turn on the ODBC port
            var res = from d in hive.DeviceInfoTable
                      where d.ClientId < 100
                      select d;

            //Dump it to the console if you like
            var list = res.ToList();     

        }
    }

That is cool. Your LINQ query will be submitted to the Azure cluster via the ODBC driver, and will be compiled and executed in the Hive.

Book Reviewed - Kinect for Windows SDK Programming Guide By Abhijit Jana

noreply@blogger.com (Anonymous) — Tue, 26 Feb 2013 07:26:00 +0000

Recently, Abhijith Jana published his Kinect For Windows SDK Programming Guide – And I’m one of the three technical reviewers. The Book is very pictorial in the explanations, with crisp code samples and cited annotations. I was particularly impressed with the way Abhijit tied the features of Kinect with Real use cases. The book covers all the hardware and software aspects, with lot of illustrations and examples. I liked the easy to understand, simple narration followed by Abhijit.

The Tone of the book

He starts with unpacking the Kinect, introducing you to the hardware and software components. And in the next couple of chapters, he is slowly familiarizing you to the camera, making sense of the color and depth image streams, Skeleton tracking, Speech API, Gesture detection etc. The book also covers various third party Kinect related APIs (including Coding4Fun Kinect Toolkit, few Gesture Libraries etc).

The last couple of chapters is all about what you can build using Kinect, with some cool examples that demonstrates how to interface Kinect with multiple platforms including Azure, Windows Phone and few Micro Controllers. There are a number of cool graphics/illustrations to enable readers to visualize various aspects like depth detection, gesture detection etc.

About the Content

More importantly, I believe this turned out to be a practical guide for any one to write real world applications using Kinect. One of my observations during the review stage was to include a lot of practical examples. I am glad to see these countless examples in the book and should commend Abhijit for his hard work. There are lot of practical scenarios covered, including

Using Kinect Windows SDK to build gesture capabilities in your Windows applications
Adding night vision using IR sensors
Audio processing
Building an intrusion detector using Kinect
Using Kinect Speech recognizer capabilities to integrate Speech detection
Making Kinect and Netdunio work together to have some serious fun etc.

I recently showed this book to some enthusiasts during my “Natural User Interface using C#” sessions in Rutgers as well as in Cupertino, and the response was quite amazing. As it assumes no prior knowledge in NUIs, this was a good starting point for a number of hackers. Also, one startup I met in CA mentioned that they are building a 3D Scanner in Kinect to use with the 3D printers, by extracting the point cloud and to convert the same to a 3D format that can be passed to the printers.

I may be a bit biased about the book as I was working with Abhi very closely for the review, but I suggest you to have a look at Amazon listing and see the reviews there. Suggest you should buy the same, if you ever tried hacking cool stuff. Or, what about building interactive Advertisement boards using Windows 8? Or a private intrusion detector for your home? Just Go ahead and buy one. – I’m so confident that you’ll love it..

Lambda patterns in C#?

noreply@blogger.com (Anonymous) — Tue, 26 Feb 2013 05:28:00 +0000

I had few interesting discussions during the MVP Summit around using patterns from other programming languages in C#. Earlier, I’ve blogged about some of these scenarios.

I’ve over heard Florian’s session this time during the summit, where he elaborated few interesting patterns from Java script that could be used in C#. He has a pretty good Code Project article regarding the same – Way To Lambda – and it is pretty good to see a formalized approach towards re-using these patterns in C#

One of my favorites is the C# version of Init-time branching (See Javascript Init time branching here)

   enum Mode
    {
        Http,
        WebSocket
    }

    class Messenger
    {

        public Func<string> Read { get; private set; }

        public Messenger(Mode m)
        {
            if (m == Mode.Http)
            {                
                Read = ()=>"Stub read logic using http"; 
            }
            else
            {
                Read = () => "Stub read logic using websocket";
            }
        }
    }

The above is a very simple version of Init time branching – but Instead of setting up the method (in this case the Read method) based on some configuration parameter, you can inject the same via the constructor or to the property – and in this way abstract out the branching as part the DI.

Convention Based Strategy Pattern _

Disclaimer: The remaining portion of the article is just not for regular use cases, and you can easily shoot yourself in the foot.

Now, let us use Init-Time branching to attain some kind of ‘Convention based’ strategy pattern. Branching will be done based on the caller’s semantics. Let us assemble a simple Serializer, that uses some convention based on the caller’s name. Note that we are using the Caller information attributes to extract the caller name.

class ConventionSerializer
    {
        public Func<string> Serialize { get; private set; }

        public ConventionSerializer([CallerMemberName] string memberName = "")
        {
            if (memberName.Contains("Xml"))
            {
                Serialize = () => 
                { /* serialize to xml */
                  return "xml"; 
                };
            }
            else
            {
                Serialize = () =>
                { /* serialize to json */
                    return "json";
                };
            }
        }

    }

And then, you may consume the same from the methods WriteXml and WriteJson.

       static void WriteXml()
        {
            var ser = new ConventionSerializer();
            Console.WriteLine(ser.Serialize()); //output xml
        }

        static void WriteJson()
        {
            var ser = new ConventionSerializer();
            Console.WriteLine(ser.Serialize()); //output json
        }

Was just bringing up an alternate perspective, requesting purists not to get offended. Happy Coding

Portable Library and WinRT support for ElasticObject

noreply@blogger.com (Anonymous) — Fri, 18 Jan 2013 06:19:00 +0000

Read these articles if you are catching up about Elastic Object

Adventures with C# Dynamic – Implementing your own ElasticObject
Creating Test data using ElasticObject

If you loved using my ElasticObject implementation, here is some good news for you

Ariel Ben Horesh recently added Portable Library project type and WinRT support for ElasticObject recently. So, enjoy the wicked evilness of ElasticObject in your Portable Libraries and WinRT projects.

A quick note on Closing lambda loop variables in C# 5.0 - Breaking Changes in C# 5.0 compiler

noreply@blogger.com (Anonymous) — Mon, 17 Dec 2012 05:15:00 +0000

This post is regarding a breaking change in C# 5.0 related to using variables in lambdas with in a loop.

For example, prior to C# 5.0 compiler, the following code

        
     //** Code Example 1 **
 
     static void Main(string[] args)
        {
            var items = new List<int>() { 1, 2, 3, 4, 5 };
            var queue = new List<Func<int>>();
            foreach (var item in items)
                queue.Add(() => item);
            foreach (var q in queue) Console.WriteLine(q());
            Console.Read();
        
    }

will produce the following output

This is because prior the C# 5.0, the compiler closes/captures the reference of the lambda/loop variable than its actual value. And hence the value defaults to the current value at the time of invocation (here, when invoking the q() method) instead of the value at the time of using the variable in the expression (when adding it to the queue). So, earlier, the compiler forced you to declare a local variable in a loop (counter intuitive any day) because if you want to see the expected output, you need to manually close the variable by assigning it to some other variable, like

    //** Code Example 2 **

   static void Main(string[] args)
        {
            var items = new List<int>() {1,2,3,4,5};
            var queue = new List<Func<int>>();
            foreach (var item in items)
            {
                //Prior to C# 5.0 this was required to correctly
                //capture the value
                var val = item;
                queue.Add(() => val);
            }
            foreach (var q in queue) Console.WriteLine(q());
            Console.Read();
        }
    }

And this should give the output as below.

From C# 5.0 onwards, the compiler will capture the value of the variable instead of it’s reference when you use a variable in a lambda expression with in a loop. So, Code Example 1 will provide Output Listing 2 that eliminates the need to capture the value manually, and you can write better code with out being forced to have local variables with in a loop.

‘HTML5 vs. Hybrid vs. Native Dilemma’ When You Build Enterprise Mobile Applications - The Bigger Picture

noreply@blogger.com (Anonymous) — Tue, 27 Nov 2012 18:23:00 +0000

This post is mainly about building Enterprise Mobile applications. Today I was reading Charlie's post about building mobile applications, and was analyzing the hybrid application scenario from an Enterprise Standpoint.

The Bigger Picture

Most of the recent conversations I had about building enterprise mobile applications starts with questions like “Whether I should go Hybrid or Native”. It is true that a lot of organizations are interested to go the ‘hybrid’ way because of cost concerns – but in my opinion, any decision should be customer driven instead of cost driven. So, the answer for this question should be driven by “What will give the best user experience to the end user”, instead or/along with asking questions like “how we can reduce the cost”. This is true for all apps, not just for enterprise mobile apps.

Beyond this, from an Enterprise Standpoint, I think achieving end point independence is another way to reduce your cost and future proof your investments. So, while “HTML5 vs Hybrid vs Native” is just one decision parameter, there are few more parameters to consider when you talk about a mobile strategy. Including,

Concerns during envisioning phase

Target user analysis
Choose a simple framework to begin with, like Forrester’s POST approach
Gap analysis and portfolio rationalization

Deciding a strategy for Mobile Application development & Delivery

Mobile Web/Pure HTML/JS
Hybrid apps (HTML5/JS apps in Native containers - Like Phone Gap and/or native parts/plugs like in Titanium)
Native apps

Concerns regarding a Common Service Layer

Building/Implementing a scalable Common Service Layer
Implementing a scalable broker system between the service layer and legacy systems
Ensuring standards (like using REST)
Identifying and integrating a Content Delivery Network (CDN) for media delivery

Testing Model

Security and Performance
Device based testing

Submission Governance

Payment model
Keep track of submitted applications
Analytics
Feedback tracking
Continuous improvement

So, all these are parts of your mobile strategy. Especially when you talk about Enterprise mobility, achieving Mobile End Point independence is a key concern, and organizations understand the necessity of making their business logic and data available to multiple end points. REST is evolving more and more as a default choice for building a public service layer, around your existing data and intelligence.

Hybrid vs Native

Now, specifically about Hybrid vs Native. Here are the main Pros and Cons as I see it.

For Hybrid applications

Pros:

No need for device specific development
Better ROI as same code base can be re-used (to an extent) across multiple platforms
You can still access some of the device features
Can be packaged for additional channels, (Chrome Packaged Apps, Awesonium)

Cons

UI specialization is difficult
Can’t be as responsive and feature rich as ‘pure’ native applications
May hit limitations when implementing specific, optimized features

For Native Applications

Pros

High degree of customization, can satisfy edge conditions
High degree of usability

Cons

Scattered/multiple platforms
Investment required for each platform
Non unified code base, not much re-usability
Teams with multiple skill sets required

And More

That said, again, think beyond the “Hybrid vs Native” point when you think about Enterprise Mobility.

From a Middleware Perspective

Select a strategy where storage and compute resources of your Mobile apps can reside in Mobile devices, on-premises servers and Cloud computing infrastructure.
Use a brokered service for connecting, communicating and brokering between Mobile apps, on-premise resources and Cloud/Middleware environment.

Custom implementation on top of brokers like Windows Azure Service Bus
Build on a Mobile ready WOA stack like Marlabs Matrix platform
Third party Mobile Middle ware platforms like Syclo etc

Use Cloud based federated authentication services and security systems if you need support for multiple authentication providers – Like Azure Federated Authentication.
Adopt cloud/service based push notification services for sending and receiving notifications from your Mobile devices.
Choose a proper content delivery network that supports bit rate streaming to deliver video content. For example, Azure CDN or Limelight True Reach API.
Enable APIs for provisioning, tracking and feedback reporting.

From a Middleware Perspective

Submission governance includes deciding a proper strategy for payment model, usage tracking, analytics and feedback reporting. The Submission Governance should ensure

Minimum criteria for submitting an application to the app store
Users are heard properly – Feedbacks and ratings should be used for identifying features
Releases and updates are following user expectations

So, happy coding!!

Changing Times For Web Developers – 6 Tips You Should Read To Survive

noreply@blogger.com (Anonymous) — Sat, 24 Nov 2012 07:57:00 +0000

The context of this post is about the changing times for a web developer, and I see a lot of web developers still lagging behind especially in the .NET world. If you haven't yet started mastering your art and adapting to the changing trends, you should start today.

Think about crafting your web applications properly. Use commonsense to mix and match based on scenarios. Here we go with 6 Tips to be a responsible web developer, and to stay on top of what you do.

1 – Learn to write better JavaScript and CSS

If you don’t write proper JavaScript and CSS, you are dead.

Learn the good parts about Javascript
Venture into Advanced Javascript

Familiarize yourself with a higher level abstraction language to write JavaScript once you are comfortable with plain Javascript – Like CoffeeScript or TypeScript. Also, master the common Javascript libraries, including these essentials

JQuery – Alright, you know this, don’t you?
Require.js - A JavaScript file and module loader.
Underscore .js – A pretty good Javascript utility library
modernizr – HTML5/CSS feature detector
history.js – History State/APIs

Oh yes, start learning TypeScript especially if you have a Microsoft technology/C# background.

On the CSS Side, familiarize with SASS and/or Less – Skim through the CSS 3 techniques, keep an eye on the CSS 3 features available in various browsers. Understand responsive concepts and start using media queries.

2 – Familiarize yourself with a Responsive Framework

These days, the usage of web is increasing more and more in mobile devices, so most websites are expected to work in different form factors by default. If you don’t want to hand craft all those web pages using media queries, better familiarize yourself with few responsive frameworks.

Responsive web design (often abbreviated to RWD) is an approach to web design in which a site is crafted to provide an optimal viewing experience—easy reading and navigation with a minimum of resizing, panning, and scrolling—across a wide range of devices (from desktop computer monitors to mobile phones)

My personal choices include Twitter Bootstrap and Zurb’s Foundation. I’ve done quite a lot of work with Bootstrap, and I’m pretty impressed with the capabilities of foundation too.

Essential reads.

3 – Learn the most useful JavaScript MVC frameworks

Be a first class JavaScript developer. There is a jungle of Javascript MVC frameworks out there for client side development, but familiarize yourself with at least five (Yes, at least 5). I’ll pick some of these to start with.

To compare and contrast them, have a look at TodoMVC – I found it pretty useful. It is

A project which offers the same Todo application implemented using MV* concepts in most of the popular JavaScript MV* frameworks of today.

4 – Understand REST and HTTP

Understand Restful services and master a server side technology to create one. For example, Node Js. If you are in the .NET world, start with Web APIs. More than that, understand proper REST. Understand Hypermedia. Go beyond the terms, and carve out some examples. Tie your REST services with one of the above MVC frameworks, and create something useful. Use proper HTTP codes for your services.

Essential reads

The above reads should help, to start with. Also, keep an eye on implementations, like Meteor, SignalR etc that could leverage Web Sockets once they are ready.

5 – Understand HTML5 Beyond the Buzzword

HTML5 standard is not yet finalized and is still in progress, but already there is lot of adoption every where. Along with developing websites, there are lot of containers available (like PhoneGap etc) that’ll allow you to package your HTML5 applications with a browser like container for mobile devices (hybrid applications), to distribute them much like you distribute native applications.

HTML5 is a markup language for structuring and presenting content for the World Wide Web and a core technology of the Internet. It is the fifth revision of the HTML standard (created in 1990 and standardized as HTML4 as of 1997)^[2] and, as of November 2012, is still under development. Its core aims have been to improve the language with support for the latest multimedia while keeping it easily readable by humans and consistently understood by computers and devices (web browsers, parsers, etc.).

There are lot of good articles about HTML5

Step by step article from Nettuts with examples
This article in Smashing Magazine is a pretty good starting point to understand more about the design philosophy and thought processes involved
Top 25 HTML5 Features

Familiarize yourself with the most useful and exciting HTML5 Javascript APIs and features. Like Web Sockets, Application Cache, Canvas API, Vide/Audio features etc. Start using something like HTML5 boiler plate as the baseline. http://www.initializr.com/

Understand the HTML 5 features and value adds you can use.

Here are few quick demos of various HTML5 feautures
Use something like http://html5please.com/ or CanIUse, keep yourself updated about the HTML5 features you can use to improve use cases based on requirements

Also watch for the big players in the space who are leveraging HTML5 as a platform for building cross platform hybrid applications – not for web, but also for desktop and mobile.

6 – Optimize

Optimize what ever you do, optimize when you are ready.

Bundle and minify your CSS and Javascript.
Use some developer tools to inspect your page and check your JavaScript performance
- Google Chrome’s speed tracer
- YSlow
Inspect your network traffic when required and this will come in handy every time– Familiarize with tools like Fiddler
Use tools like Screenfly to check your screen for multiple form factors
Use simulators like Ripple for checking the behavior of your websites in Mobile devices

Alright, that was lot of tips. So, happy coding.

Meet Ms. Roslyn – Article in DNC Magazine

noreply@blogger.com (Anonymous) — Sat, 24 Nov 2012 05:40:00 +0000

I wrote a couple of articles here about Ms. Roslyn earlier. Roslyn provides language services and APIs on top of .NET’s compiler services, and this will enable .NET developers to do a lot more things – including

Using C# and VB.NET as scripting languages
Use compiler as a service in your own applications for code related tasks
Develop better language and IDE extensions etc.
Write code analysis and manipulation tools around Visual Studio IDE

See my previous articles on Roslyn

So, I was more than happy to contribute when the Amazing Dotnet Curry Magazine team reached out to me for an article on Roslyn. Go, download and subscribe the DNC Magazine here. It is a free Magazine, and kudos to the great work from Suprotim, Sumit and team for the awesome work they are doing for the community by bringing this out. You can get the magazine issues for free here.

Happy Coding!!